Use Azure Private Link for Serverless Products on Confluent Cloud

Confluent Cloud supports private connectivity for serverless Confluent Cloud products, such as Enterprise Kafka clusters and Confluent Cloud for Apache Flink®. When you use Private Link Attachment, your Enterprise cluster or Flink resources are only accessible from tenant-specific private endpoints. Public access is blocked with Private Link Attachment.

Note

The Private Link Attachment enables you to access the Flink API with private networking, allowing you to issue Flink SQL queries and retrieve results through the associated Private Link connection. All data movement between Flink queries and Kafka clusters configured with private networking occurs over a secure private path within Confluent Cloud.

Confluent Cloud uses the following private networking resources for Enterprise clusters. These resources are regional and do not have a mapping to specific availability zones.

Private Link Attachment

The Private Link Attachment (PrivateLinkAttachment) resource represents a reservation to establish a Private Link connection from your Virtual Network (VNet) regional services in a Confluent Cloud environment.

A Private Link Attachment belongs to an Environment in the Confluent resource hierarchy.

This resource is referred to as gateways in the Confluent Cloud Console.

Private Link Attachment Connection

A Private Link Attachment Connection (PrivateLinkAttachmentConnection) is a registration of VNet private endpoints that are allowed to connect to Confluent Cloud. A Private Link Attachment Connection belongs to a specific Private Link Attachment.

This resource is referred as access points in the Confluent Cloud Console.

You can use the Confluent Cloud Console, Confluent REST API, Confluent CLI, or Terraform to establish a Private Link connectivity for serverless products, such as Enterprise Kafka clusters or Flink.

The high-level workflow is:

1. In Confluent Cloud, create a Private Link Attachment.

2. In Azure, create a private endpoint to be associated with the Private Link Attachment service.

   If you are using the Confluent Cloud Console for configuration, this step is merged into the next step and shows up as the first step in connection creation.

3. In Confluent Cloud, create a Private Link Attachment Connection.

4. Set up a DNS resolution.

5. Create a Kafka client in your VNet using the bootstrap endpoint of your Enterprise Kafka cluster. This Kafka client can live in Virtual Machine or similar compute infrastructure.

6. Validate produce/consume traffic is successful.

   Once you create a Private Link Attachment resource and establish a Private Link, you can securely send and receive traffic through the Private Link between your VNet and Confluent Cloud.

Requirements and considerations

• You can connect to only one Confluent Cloud environment from a single VNet or from an on-premises network unless multiple DNS servers are used. Multiple connections utilizing a centralized DNS resolver does not work due to the overlap in domain names between Confluent Cloud environments.

  For the workaround to configure cross-environment queries in Flink, see Configure cross-environment queries.

• You can connect to only one region in a specific environment from a single VNet or from an on-premises network.

• For the regions supported for Private Link Attachment on Azure, see Cloud Providers and Regions for Confluent Cloud.

• Confluent Cloud Console components, like topic management and Flink workspaces, require additional configuration to function as they use cluster endpoints.

  For information about using Flink with Azure Private Link, see Private Networking with Confluent Cloud for Apache Flink.

  To use all features of the Confluent Cloud Console with Azure Private Link, see Use the Confluent Cloud Console with Private Networking.

Create a Private Link Attachment

The Private Link Attachment resource, referred as the gateway in the Confluent Cloud Console, represents a reservation to establish a Private Link connection from Virtual Network (VNet) regional services in a Confluent Cloud environment.

When you create a Private Link Attachment in an environment and in a region, the Private Link Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.

Confluent Cloud ConsoleConfluent REST APIConfluent CLITerraform

In the Confluent Cloud Console, the Private Link Attachment resources are labeled and referred as gateways.

1. In the Confluent Cloud Console, select an environment for the Private Link Attachment.

2. In the Network management tab in the environment, click For serverless products.

3. Click + Add gateway configuration.

4. Select the PrivateLink card to select the type of gateway configuration.

5. On the From your VPC or VNet to Confluent Cloud pane, click + Create configuration.

6. On the Configure gateway sliding panel, enter the following information.

   • Gateway name
   • Cloud provider: Click Microsoft Azure.
   • Region

7. Click Submit.

8. You can continue to create an access point for the Ingress Private Link connection.

   Alternatively, you can create an access point at a later time by navigating to this gateway in the Network management tab.

The Private Link Attachment will be provisioned and move to the Waiting for connection state.

A Private Link Attachment can be in one of the following states:

• WAITING FOR CONNECTION: The Private Link Attachment is waiting for a connection to be created.
• READY: Azure Private Link connectivity is ready to be used.
• EXPIRED: A valid connection has not been provisioned within the allotted time. A new Private Link Attachment must be provisioned.

Create an Azure private endpoint

In Azure, create an endpoint that is associated with the Private Link Service ID of the Private Link Attachment you created in Create a Private Link Attachment.

Azure PortalAzure CLI

For details on creating a private endpoint in Azure, see Create a Private Endpoint.

1. On the Private Endpoint page in Azure portal, click + Create.

2. In the Basics pane, specify the following:

   • Subscription: The subscription name that you selected when you created the VNet.
   • Resource group: The same resource group that you selected when you created the VNet.
   • Name: The name for the private endpoint.
   • Network interface name: A network interface name.
   • Region: The region for the private endpoint.

3. Click Next: Resource.

4. In the Resource pane, specify the following:

   • Connection method: Select Connect to an Azure resource by resource ID or alias.

   • Resource ID or alias: Paste in the Confluent Cloud Resource ID or Service Alias.

     This is the alias or ID created in the previous section, Create a Private Link Attachment.

     

     You can also use the value of the Private Link Service ID from your Network overview of the PrivateLink Attachment gateway in the Confluent Cloud Console.

5. Click Next: Virtual Network.

6. In the Virtual Network pane, specify the following:

   • Virtual network: Select the VNet where the private endpoint is to be created.
   • Subnet: Select the subnet where the private endpoint is to be created.
   • Network policy for private endpoints: Select the organization-approved or mandated policy. The default is Disabled.
   • Private IP configuration: Select Dynamically allocate IP address.

7. Click Next: DNS and accept the default values.

8. Click Next: Tags and, optionally, add tags.

9. Click Next: Review + create. Review the details and click Create to create the private endpoint.

10. Wait for the Azure deployment to complete.

Create a Private Link Attachment Connection

Create a Private Link Attachment Connection resource in Confluent Cloud. A Private Link Attachment Connection represents a private endpoint in your VNet.

In the Confluent Cloud Console, the Private Link Attachment Connection resources are labeled and referred to as access points.

Confluent Cloud ConsoleConfluent REST APIConfluent CLITerraform

1. In the Network Management tab of the desired Confluent Cloud environment, click the For serverless products tab.

   Make sure the Private Link Attachment is in the correct region of the private endpoint.

2. Click the gateway to which you want to add the Private Link Endpoint.

   Make sure the gateway is in the correct region of the VPC Private Endpoint.

3. In the Access points tab, click Create access point.

   Using the service ID in Private Link Service ID or the service alias in Private Link Service Alias shown in Step 3 on the sliding panel, create a VPC Endpoint in Azure.

4. Specify the Private Endpoint ID.

   The private endpoint ID is the Azure resource ID of the private endpoint that was created in Create an Azure private endpoint.

5. Specify the access point name.

6. Click Create access point to create the Private Link Endpoint.

   The Private Link Attachment and Private Link Attachment Connection should now move to the READY state once the private endpoint connection is accepted.

Go to the private endpoint resource in Azure Portal and verify that the private endpoint connection status is Approved.

Set up DNS records in Azure

Set up a DNS resolution and a DNS record using the Azure private DNS zone in the Azure console. This section focuses on the settings related to Confluent Cloud. For details, see Create an Azure private DNS zone.

1. Create a private DNS zone.

   1. In the Confluent Cloud Console, copy the DNS Domain name in Private Link Attachment in the Network management tab, and use it as the name for the private DNS zone. It is in the form, <region>.azure.private.confluent.cloud. For example, centralus.azure.private.confluent.cloud.
   2. In Private Zones in the Azure portal, click + Create.
   3. In the Basics pane, enter or select the following values:
      • Subscription: Pre-filled with the subscription name that you selected when you created the VNet.
      • Resource group: Select the resource group that you selected when you created the VNet.
      • Name: Specify the domain name retrieved from the Confluent Cloud Console in the first step. It is in the format of <region>.azure.private.confluent.cloud, for example, centralus.azure.private.confluent.cloud.
   4. Click Next: Tags and, optionally, add tags.
   5. Click Next: Review + create. Review the details and click Create a DNS zone.
   6. Wait for the Azure deployment to complete.

2. To create DNS records, go to the private DNS zone resource you created in the previous step, and click + Record Set for the Confluent Cloud clusters.

   Note

   In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.

   • Name: *

   • Type: A

   • TTL and TTL unit: 1 Minute

   • IP address: The IP address of the private endpoint can be found under its associated network interface under Settings for the private endpoint.

     

3. Attach the private DNS zone to the VNets where clients or applications are present.

4. Go to the private DNS zone resource and click Virtual network links under Settings.

   1. Click + Add.
   2. Specify the required values and click OK to create a virtual network link.

Connectivity scenarios

Below are examples of a few connectivity scenarios that are supported for Enterprise clusters in Confluent Cloud.

Scenario: Access one environment from one VNet

The following resources are configured:

• PLATT-prod as a Private Link Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a Private Link Attachment Connection for the private-endpoint-1 private endpoint in VNet-1
• ProdApp as a Kafka client bootstrapped with lkc-123.centralus.azure.private.confluent.cloud
• Private DNS Zone 1 with the regional wildcard centralus.azure.private.confluent.cloud

The following steps are performed:

1. ProdApp attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.centralus.azure.private.confluent.cloud resolves against returns private-endpoint-1.
2. Application sends traffic to private-endpoint-1.
3. private-endpoint-1 forwards traffic to PLATT-prod, and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.

Scenario: Access one environment from many VNet’s

The following resources are configured:

• PLATT-abc as a Private Link Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a Private Link Attachment Connection for the private-endpoint-1 private endpoint in VNet-1
• PLATTC-456 for the private-endpoint-2 private endpoint in VNet-2
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.eastus.azure.private.confluent.cloud
• ProdApp-2 as a Kafka client bootstrapped with lkc-456.eastus.azure.private.confluent.cloud
• Private DNS Zone 1 with the regional wildcard *.eastus.azure.private.confluent.cloud
• Private DNS Zone 2 with the regional wildcard *.eastus.azure.private.confluent.cloud

The following steps are performed:

1. ProdApp-1 attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.eastus.azure.private.confluent.cloud resolves against Private DNS Zone 1 and returns private-endpoint-1.
2. ProdApp-1 sends traffic to private-endpoint-1.
3. private-endpoint-1 forwards traffic to PLATT-abc, and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.
4. ProdApp-2 attempts to access lkc-456 in the env-prod environment. A DNS query for lkc-456.eastus.azure.private.confluent.cloud resolves against Private DNS Zone 2 and returns private-endpoint-2.
5. ProdApp-2 sends traffic to private-endpoint-2.
6. private-endpoint-2 forwards traffic to PLATT-abc, and lkc-456 can be accessed since PLATTC-456 is associated with private-endpoint-1.

Scenario: Access one environment from an on-premises network

The following resources are configured:

• PLATT-abc as a Private Link Attachment for accessing Kafka clusters in the env-abc environment
• PLATTC-123 as a Private Link Attachment Connection for the private-endpoint-1 endpoint in VNet-1
• On-Prem-1 as a Kafka client bootstrapped with lkc-123.westus.azure.private.confluent.cloud
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.westus.azure.private.confluent.cloud
• Private DNS Zone Forward as a DNS forwarding rule with the regional wildcard *.westus.azure.private.confluent.cloud
• Private DNS Zone 1 with the regional wildcard *.westus.azure.private.confluent.cloud

The following steps are performed:

1. On-Prem-1 attempts to access lkc-123 in the env-abc environment. A DNS query for lkc-123.westus.azure.private.confluent.cloud forwards to Private DNS Zone 1 and returns private-endpoint-1.
2. On-Prem-1 sends traffic to private-endpoint-1 over Azure ExpressRoute.
3. private-endpoint-1 forwards traffic to PLATT-abc and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.