Use Azure Private Link for Serverless Products on Confluent Cloud

Confluent Cloud supports private connectivity for serverless Confluent Cloud products, such as Enterprise Kafka clusters using PrivateLink Attachment. When you use PrivateLink Attachment, your Enterprise cluster is only accessible from tenant-specific private endpoints. Public access is blocked with PrivateLink Attachment.

Confluent Cloud uses the following private networking resources for Enterprise clusters. These resources are regional and do not have a mapping to specific availability zones.

PrivateLink Attachment

The PrivateLink Attachment (PrivateLinkAttachment) resource represents a reservation to establish a Private Link connection from your Virtual Network (VNet) regional services in a Confluent Cloud environment.

A PrivateLink Attachment belongs to an Environment in the Confluent resource hierarchy.

This resource is referred to as gateways in the Confluent Cloud Console.

PrivateLink Attachment Connection

A PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) is a registration of VNet private endpoints that are allowed to connect to Confluent Cloud. A PrivateLink Attachment Connection belongs to a specific PrivateLink Attachment.

This resource is referred as access points in the Confluent Cloud Console.

You can use Confluent Cloud UI, Confluent REST API, Confluent CLI, or Terraform to establish a Private Link connectivity for an Enterprise cluster.

The high-level workflow is:

1. In Confluent Cloud, create a PrivateLinkAttachment.

2. In Azure, create a private endpoint to be associated with the PrivateLinkAttachment service.

3. In Confluent Cloud, create a PrivateLinkAttachmentConnection.

4. Set up a DNS resolution.

5. Create a Kafka client in your VNet using the bootstrap endpoint of your Enterprise Kafka cluster. This Kafka client can live in Virtual Machine or similar compute infrastructure.

6. Validate produce/consume traffic is successful.

   Once you create a PrivateLinkAttachment resource and establish a Private Link, you can securely send and receive traffic through the PrivateLink between your VNet and Confluent Cloud.

Requirements and considerations

• You can connect to only one region in a specific environment from a single VNet or from an on-premises network.

• For the regions supported for PrivateLink Attachment on Azure, see Cloud Providers and Regions for Confluent Cloud.

• Confluent Cloud connectors connecting to sources or sinks with private IP addresses is unsupported at this time.

  An exception to this is the Azure connectors that access the Azure services via service endpoints. For the current list of these connectors, contact Confluent Support.

• Confluent Cloud Console components, like topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Azure Private Link, see Use Confluent Cloud with Private Networking.

Create a PrivateLink Attachment

When you create a PrivateLink Attachment in an environment and in a region, the PrivateLink Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.

Confluent Cloud ConsoleConfluent REST APIConfluent CLI

In the Confluent Cloud Console, the PrivateLink Attachment resources are labeled and referred as gateways.

1. In the Confluent Cloud Console, select an environment for the PrivateLink Attachment.

2. In the Network management tab in the environment, click For serverless products.

3. Click Add network configuration.

4. On the From your VPC or VNet to Confluent Cloud pane, click + Create configuration.

5. On the Create ingress network configure gateway sliding panel, enter the following information.

   • Gateway name
   • Cloud provider
   • Region

6. Click Submit.

7. You can continue to create an access point for an Ingress Private Link Endpoint.

   Alternatively, you can create an access point at a later time by navigating to this gateway in the Network management tab.

The PrivateLink Attachment will be provisioned and move to the Waiting for connection state.

A PrivateLink Attachment can be in one of the following states:

• WAITING FOR CONNECTION: The PrivateLink Attachment is waiting for a connection to be created.
• READY: Azure Private Link connectivity is ready to be used.
• EXPIRED: A valid connection has not been provisioned within the allotted time. A new PrivateLink Attachment must be provisioned.

Create an Azure private endpoint

In Azure, create an endpoint that is associated with the Private Link Service ID of the PrivateLink Attachment you created in Create a PrivateLink Attachment.

Azure PortalAzure CLI

For details on creating a private endpoint in Azure, see Create a Private Endpoint.

1. On the Private Endpoint page in Azure portal, click + Create.

2. In the Basics pane, specify the following:

   • Subscription: The subscription name that you selected when you created the VNet.
   • Resource group: The same resource group that you selected when you created the VNet.
   • Name: The name for the private endpoint.
   • Network interface name: A network interface name.
   • Region: The region for the private endpoint.

3. Click Next: Resource.

4. In the Resource pane, specify the following:

   • Connection method: Select Connect to an Azure resource by resource ID or alias.

   • Resource ID or alias: Paste in the Confluent Cloud Resource ID or Service Alias.

     This is the alias or ID created in the previous section, Create a PrivateLink Attachment.

5. Click Next: Virtual Network.

6. In the Virtual Network pane, specify the following:

   • Virtual network: Select the VNet where the private endpoint is to be created.
   • Subnet: Select the subnet where the private endpoint is to be created.
   • Network policy for private endpoints: Select the organization-approved or mandated policy. The default is Disabled.
   • Private IP configuration: Select Dynamically allocate IP address.

7. Click Next: DNS and accept the default values.

8. Click Next: Tags and, optionally, add tags.

9. Click Next: Review + create. Review the details and click Create to create the private endpoint.

10. Wait for the Azure deployment to complete.

Create a PrivateLink Attachment Connection

Create a PrivateLink Attachment Connection resource in Confluent Cloud. A PrivateLink Attachment Connection represents a private endpoint in your VNet.

In the Confluent Cloud Console, the PrivateLink Attachment Connection resources are labeled and referred to as access points.

The name of the VNet private link service is not required. Confluent will check which private link service is associated with the PrivateLink Attachment that has a pending private endpoint with the given ID.

Confluent Cloud ConsoleConfluent REST APIConfluent CLI

1. In the Network Management tab of the desired Confluent Cloud environment, click the For serverless products tab.

   Make sure the PrivateLink Attachment is in the correct region of the private endpoint.

2. Click Create access point for the gateway to which you want to add the PrivateLink Endpoint.

   Make sure the gateway is in the correct region of the VPC Private Endpoint.

3. Create a VPC Endpoint in Azure.

4. Specify the Private Endpoint ID.

   The private endpoint ID is the Azure resource ID of the private endpoint that was created in Create an Azure private endpoint.

5. Specify the access point name.

6. Click Create access point to create the Private Link Endpoint.

   The PrivateLink Attachment and PrivateLink Attachment Connection should now move to the READY state once the private endpoint connection is accepted.

Go to the private endpoint resource in Azure Portal and verify that the private endpoint connection status is Approved.

Set up DNS records in Azure

Set up a DNS resolution and a DNS record using the Azure private DNS zone in the Azure console. This section focuses on the settings related to Confluent Cloud. For details, see Create an Azure private DNS zone.

1. Create a private DNS zone.

   1. In the Confluent Cloud Console, copy the DNS Domain name in PrivateLink Attachment in the Network management tab, and use it as the name for the private DNS zone. It is in the form, <region>.azure.private.confluent.cloud. For example, centralus.azure.private.confluent.cloud.
   2. In Private Zones in the Azure portal, click + Create.
   3. In the Basics pane, enter or select the following values:
      • Subscription: Pre-filled with the subscription name that you selected when you created the VNet.
      • Resource group: Select the resource group that you selected when you created the VNet.
      • Name: Specify the domain name retrieved from the Confluent Cloud Console in the first step. It is in the format of <region>.azure.private.confluent.cloud, for example, centralus.azure.private.confluent.cloud.
   4. Click Next: Tags and, optionally, add tags.
   5. Click Next: Review + create. Review the details and click Create a DNS zone.
   6. Wait for the Azure deployment to complete.

2. To create DNS records, go to the private DNS zone resource you created in the previous step, and click + Record Set for the Confluent Cloud clusters.

   • Name: *

   • Type: A

   • TTL and TTL unit: 1 Minute

   • IP address: The IP address of the private endpoint can be found under its associated network interface under Settings for the private endpoint.

     

3. Attach the private DNS zone to the VNets where clients or applications are present.

4. Go to the private DNS zone resource and click Virtual network links under Settings.

   1. Click + Add.
   2. Specify the required values and click OK to create a virtual network link.

Connectivity scenarios

Below are examples of a few connectivity scenarios that are supported for Enterprise clusters in Confluent Cloud.

Scenario: Access one environment from one VNet

The following resources are configured:

• PLATT-prod as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a PrivateLink Attachment Connection for the private-endpoint-1 private endpoint in VNet-1
• ProdApp as a Kafka client bootstrapped with lkc-123.centralus.azure.confluent.cloud
• Private DNS Zone 1 with the regional wildcard centralus.azure.confluent.cloud

The following steps are performed:

1. ProdApp attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.centralus.azure.confluent.cloud resolves against returns private-endpoint-1.
2. Application sends traffic to private-endpoint-1.
3. private-endpoint-1 forwards traffic to PLATT-prod, and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.

Scenario: Access one environment from many VNet’s

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a PrivateLink Attachment Connection for the private-endpoint-1 private endpoint in VNet-1
• PLATTC-456 for the private-endpoint-2 private endpoint in VNet-2
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.eastus.azure.confluent.cloud
• ProdApp-2 as a Kafka client bootstrapped with lkc-456.eastus.azure.confluent.cloud
• Private DNS Zone 1 with the regional wildcard *.eastus.azure.confluent.cloud
• Private DNS Zone 2 with the regional wildcard *.eastus.azure.confluent.cloud

The following steps are performed:

1. ProdApp-1 attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.eastus.azure.confluent.cloud resolves against Private DNS Zone 1 and returns private-endpoint-1.
2. ProdApp-1 sends traffic to private-endpoint-1.
3. private-endpoint-1 forwards traffic to PLATT-abc, and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.
4. ProdApp-2 attempts to access lkc-456 in the env-prod environment. A DNS query for lkc-456.eastus.azure.confluent.cloud resolves against Private DNS Zone 2 and returns private-endpoint-2.
5. ProdApp-2 sends traffic to private-endpoint-2.
6. private-endpoint-2 forwards traffic to PLATT-abc, and lkc-456 can be accessed since PLATTC-456 is associated with private-endpoint-1.

Scenario: Access one environment from an on-premises network

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-abc environment
• PLATTC-123 as a PrivateLink Attachment Connection for the private-endpoint-1 endpoint in VNet-1
• On-Prem-1 as a Kafka client bootstrapped with lkc-123.westus.azure.confluent.cloud
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.westus.azure.confluent.cloud
• Private DNS Zone Forward as a DNS forwarding rule with the regional wildcard *.westus.azure.confluent.cloud
• Private DNS Zone 1 with the regional wildcard *.westus.azure.confluent.cloud

The following steps are performed:

1. On-Prem-1 attempts to access lkc-123 in the env-abc environment. A DNS query for lkc-123.westus.azure.confluent.cloud forwards to Private DNS Zone 1 and returns private-endpoint-1.
2. On-Prem-1 sends traffic to private-endpoint-1 over Azure ExpressRoute.
3. private-endpoint-1 forwards traffic to PLATT-abc and lkc-123 can be accessed since PLATTC-123 is associated with private-endpoint-1.