Enable Private Networking with Confluent Cloud for Apache Flink

Confluent Cloud for Apache Flink®️ supports private networking on AWS. This feature enables Flink to securely read and write data stored in Confluent Cloud clusters that are located in private networking, with no data flowing to the public internet. With private networking you can use Flink and Apache Kafka® together for stream processing in Confluent Cloud, even in the most stringent regulatory environments.

Currently, Confluent Cloud for Apache Flink supports only Private Networking for AWS Dedicated clusters and Enterprise clusters. A small number of Dedicated clusters created with a previous version of the networking stack are not yet available and will be added later.

How does it work?

Flink Private Networking requires a Private Link Attachment (PLATT) to access Kafka clusters with private networking. Private Link Attachments are resources that enable you to connect to Confluent serverless products, like Enterprise Clusters and Flink.

For Flink, the new Private Link Attachment is used only to establish a connection between your clients (like UI, CLI, Terraform, apps using the REST API) and Flink. Flink-to-Kafka is routed internally within Confluent Cloud. As a result, this PLATT is used only for submitting statements and fetching results from the client.

• For Dedicated clusters, regardless of the Kafka cluster connection type (Private Link, Peering, or Transit Gateway), Flink requires that you define a PLATT in the same region and environment of the cluster, even if a Private Link exists for the Dedicated cluster.
• For Enterprise clusters, you can reuse the same PLATT used by your Enterprise clusters.

Important

By creating an environment-wide PrivateLink Attachment, you also enable your data-movement components in Confluent Cloud, including Flink statements and cluster links, to move data between all of the private networks in the environment, including the Confluent Cloud networks associated with any Dedicated Kafka clusters.

Protected resources

Flink PrivateLink Attachment provides comprehensive protection for Kafka clusters, statements, and workspaces, ensuring secure and private connectivity for operations within Confluent Cloud environments.

PrivateLink Attachment protects Kafka clusters and enables Flink to access Kafka Enterprise clusters with any type of networking: public, private link, VPC Peering, and Transit Gateway. This access is governed by strictly enforced rules to ensure proper isolation and prevent data exfiltration to the public internet.

Private networking provides protection for Flink statements and workspaces, which can contain sensitive information. You can secure these resources under a private network, ensuring that they are not accessible over public networking. This includes both the creation and access of private statements and workspaces, which is possible only from an authorized private network attached to the Confluent Cloud environment.

When private networking is enabled, you can read both public and private data. Flink with private networking can read any data that is either accessible from public networking or accessible within the scope of the PrivateLink Attachment defined for the current environment and region.

To prevent data exfiltration when using Flink private networking, statements can only write to clusters using private networking. If you want to write to a public cluster, use Flink from another environment without a PrivateLink Attachment.

Important

After a PrivateLink Attachment is created and private networking is enabled, you can’t disable it. Because a resource may contain sensitive information, this Confluent policy ensures that private resources stay private. If you delete a PrivateLink Attachment, the environment stays private, and you must create a new PrivateLink Attachment to access Flink statements and workspaces. Without a new PrivateLink Attachment, you will not be able to access your private resources.

The following table shows access to public and private resources with and without a PrivateLink Attachment created. “CRUD” stands for “Create, Read, Update, Delete”.

	Connect from public internet	Connect from private connection to the PrivateLink Attachment of the current environment
Public (no PrivateLink Attachment created)	Default connection [1] ✅ CRUD on public statements ✅ CRUD on public workspaces 🚫 CRUD on private statements 🚫 CRUD on private workspaces	✅ CRUD on public statements ✅ CRUD on public workspaces 🚫 CRUD on private statements 🚫 CRUD on private workspaces
Private (PrivateLink Attachment created)	🚫 CRUD on public statements 🚫 CRUD on public workspaces 🚫 CRUD on private statements 🚫 CRUD on private workspaces	Default connection [2] 🟡 RUD on public statements, new statements are private only 🟡 RD on public workspaces, public workspaces are read-only ✅ CRUD on private statements ✅ CRUD on private workspaces

[1]	Default connection for Cloud Console and Confluent CLI with no PrivateLink Attachment.

[2]	Default connection for Cloud Console and Confluent CLI with a PrivateLink Attachment created.

Prerequisites

• Access to Confluent Cloud.
• The OrganizationAdmin, EnvironmentAdmin, or NetworkAdmin role to enable Flink private networking for an environment.
• A VPC in AWS.

Create a PrivateLink Attachment overview

In this walkthrough, you perform the following steps.

1. Create PLATT/PLATTC:
   1. In Confluent Cloud, create a PrivateLink Attachment as shown in Step 1.
   2. In AWS, create a PrivateLink Attachment Connection, and create a VPC Endpoint linked to the PrivateLink Attachment service.
   3. In AWS Route53, set up a DNS resolution for this endpoint.
2. If your client is not in the VPC, enable the Confluent Cloud Console or Confluent CLI to connect to your private network as shown in Step 2.

You can now use Flink from the Confluent Cloud Console or Confluent CLI. When the previous steps are completed, the experience is the same as with public networking.

Step 1: Create a PrivateLink Attachment

Add the network configuration

The following steps show how to create a PrivateLink Attachment by using the Cloud Console.

If a PrivateLink Attachment exists already, you don’t need to create another, because the existing PrivateLink Attachment provides connectivity for the current environment and region.

1. Log in to the Confluent Cloud Console and navigate to an environment that hosts Flink SQL.

2. In the environment details page, click Network management and ensure that For serverless products is selected.

3. Click Add network configuration.

4. In the Add network configuration page, select your cloud service provider and region.

   Note

   Currently, Confluent Cloud for Apache Flink supports only the AWS PrivateLink Attachment.

5. Click Continue.

6. In the Network name textbox, type the name of your PrivateLink Attachment network.

7. Click Add network configuration.

   The environment details page opens and shows your network. Provisioning the network may take a few seconds.

Create a PrivateLink Attachment Connection

1. Click Add connection and follow the instructions in Create a VPC Endpoint.
2. Set the routes by following the instructions in Set up DNS resolution.

Step 2: Connect to the network with Confluent Cloud Console or Confluent CLI

If you don’t connect from a machine in the VPC, you see the following error.

To connect to Confluent Cloud with PrivateLink Attachment, see Use Confluent Cloud with Private Networking. One way to connect is to set up a reverse proxy.

1. Create an EC2 instance

2. Connect to the instance with SSH

3. Install NGINX

4. Configure Routing Table

5. Set up DNS resolution: point to the Flink regional endpoints you use, as described in Step 6 of Configure a proxy.

   <Public IP Address of VM instance> <Flink-private-endpoint>
   

   <Flink-private-endpoint> will resemble flink.<region>.<cloud>.private.confluent.cloud, for example: flink.us-east-2.aws.private.confluent.cloud.

   Find the DNS part of the PrivateLink Attachment by navigating to your environment’s Network management page and finding the DNS domain setting.

   

   You can find the full list of supported Flink regions by using the Regions endpoint API.

Once networking is set up in Cloud Console, the interface uses the correct endpoint automatically, either public or private, based on the presence of a PrivateLink Attachment. If the connection is private, access to the Flink private network works transparently.

Additional Confluent CLI options

Like Cloud Console, with Confluent CLI the interface uses the correct endpoint automatically, either public or private, based on the presence of a PrivateLink Attachment, but you also have the option of overriding the endpoint by using the following command:

# Override to private endpoint
confluent flink connectivity-type use private

# Override to public endpoint:
confluent flink connectivity-type use public

For more information, see Create a connection to the network that hosts the private cluster endpoints.

Related content

• Use Confluent Cloud with Private Networking
• Flink Compute Pools
• Billing on Confluent Cloud for Apache Flink

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.