Use Azure Private Link with Confluent Cloud¶

Azure Private Link allows for one-way secure connection access from your VNet to Confluent Cloud with an added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity of setup.

The following diagram summarizes the Azure Private Link architecture between the VNet or subscription and the Confluent Cloud cluster.

Azure Private Link architecture between customer VNet or subscription and Confluent Cloud cluster

Note

For an overview of Azure Private Link and illustrated steps on getting started with Azure Private Link in Confluent Cloud, see Setting Up Secure Networking in Confluent with Azure Private Link.

Prerequisites¶

A Confluent Cloud network (CCN) of type PRIVATELINK in Azure. If a network does not exist, see Confluent Cloud Network on Azure.
To use an Azure Private Link service with Confluent Cloud, your VNet must allow outbound internet connections for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work.
- DNS requests to public authority traversing to private DNS zone is required.
- Confluent Cloud Schema Registry is accessible over the internet.
- Provisioning new ksqlDB instances requires Internet access. After ksqlDB instances are up and running, they are fully accessible over Azure Private Link connections.
- Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane.
Confluent Cloud Console components, like topic management, need additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Azure Private Link, see Access Confluent Cloud Console with Private Networking.

Warning

For limitations of the Azure Private Link, see Limitations below.

Register your Azure subscription with Confluent Cloud¶

Register your Azure subscription with the Confluent Cloud network for automatic approval of private endpoint connections to the Confluent Cloud network. If required, you can register multiple subscriptions.

In the Confluent Cloud Console, go to your network resource in the Network Management tab and click + Private Link Access.
Enter the Azure subscription ID for the account containing the VNets you want to make the Azure Private Link connection from. The Azure subscription number can be found on your Azure subscription page of the Azure Portal.
Click Save.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/private-link-accesses

Authentication

See Authentication.

Request specification

In the request specification, include Confluent Cloud network ID, subscription, environment, and, optionally, add the display name. Update the attributes below with the correct values.

{
   "spec":{
      "display_name":"Azure-PL-CCN-1",
      "cloud":{
         "kind":"AzurePrivateLinkAccess",
         "subscription":"00000000-0000-0000-0000-000000000000"
      },
      "environment":{
         "id":"env-00000"
      },
      "network":{
         "id":"n-000000"
      }
   }
}

Your Azure Private Link connection status will transition from “Pending” to “Active” in the Confluent Cloud Console. You still need to configure the Private Endpoints in your VNet before you can connect to the cluster.

Note the Private Link Service Endpoint to create an Azure Private Link connection from your VNet to the Confluent Cloud cluster. This URL will also be provided later.

Create an Azure Private Link connection to Confluent Cloud¶

Follow this procedure to create an Azure Private Link connection to a Confluent Cloud cluster on Azure using the Confluent Cloud Console or REST API.

Set up the VNet Endpoint for Azure Private Link in your Azure account¶

After the connection status is “Active” in the Confluent Cloud Console, you must configure Private Endpoints in your VNet from the Azure Portal to make the Azure Private Link connection to your Confluent Cloud cluster.

Note

Confluent recommends using a Terraform configuration for setting up Private Link endpoints. This configuration automates the manual steps described below.

Prerequisites¶

In the Confluent Cloud Console, you will find the following information for your Confluent Cloud cluster under the Cluster Settings section. This information is needed to configure Azure Private Link for a Dedicated cluster in Azure.

Kafka Bootstrap (in the General tab)
DNS domain Name (in the Networking card)
Zonal DNS Subdomain Names (in the Networking card)
Service Aliases (in the Networking card)

Create the following Private Endpoints through the Azure Private Link Center:

For Confluent Cloud single availability zone clusters, create a single Private Endpoint to the Confluent Cloud Service Alias. For Confluent Cloud multi-availability zone clusters, create a Private Endpoint to each of the Confluent Cloud zonal Service Aliases.
Create a Private Endpoint for Confluent Cloud by clicking Create Private Endpoint.
Fill in subscription, resource group, name, and region for the virtual endpoint and click next. The selected subscription must be the same as the one registered with Confluent Cloud.
Select the Connect to an Azure resource by resource ID or alias option, paste in the Confluent Cloud Service Alias and click Next. You can find the Confluent Cloud Service Aliases in the Networking tab under Cluster settings in the Confluent Cloud Console.
Fill in virtual network and subnet where the Private Endpoint is to be created.
Click Review + create. Review the details and click Create to create the Private Endpoint.
Wait for the Azure deployment to complete, go to the Private Endpoint resource and verify Private Endpoint connection status is Approved.

Set up DNS records to use Azure Private Endpoints¶

You must update your DNS records to ensure connectivity passes through Azure Private Link in the supported pattern. Any DNS provider that can ensure DNS is routed as follows is acceptable. Azure Private DNS Zone (used in this example) is one option.

DNS resolution options¶

For Azure Private Link Confluent Cloud networks, you can use the default DNS resolution or enable private DNS resolution.

Default DNS resolution¶

The default DNS resolution, which is partially public, is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using Azure Private Link. The default DNS resolution performs the following two-step process:

The Confluent Cloud Global DNS Resolver removes the glb subdomain and returns a CNAME for your bootstrap and broker hostnames.

Example: $lkc-id-$nid.$region.$cloud.glb.confluent.cloud

CNAME returned: $lkc-id.$nid.$region.$cloud.confluent.cloud
The CNAME resolves to your VNet private endpoints based on the Private DNS Zone configuration.

Warning

Some DNS systems, like Windows DNS service, lack the ability to recursively resolve the previously mentioned two-step process within a single DNS node. To solve the issue, use Private DNS resolution.

Private DNS resolution¶

If you enable the Private DNS resolution option, your private DNS zone provides internal DNS resolution for your private networks without requiring external resolution to the Confluent Global DNS Resolver (GLB).

Tip

To identity the CNAME DNS zone records to correctly map to zonal endpoints for Confluent Cloud, you can run the DNS helper script.

Configure DNS zones¶

To update DNS resolution using Azure Private DNS Zone in the Azure console:

Create the Private DNS Zone.
1. Search for the Private DNS Zone resource in the Azure Portal.
2. Click Add.
3. Copy the DNS Domain name from the Networking tab under Cluster Settings in the Confluent Cloud Console and use it as the name for the Private DNS Zone.
  
  For example:
```
4kgzg.centralus.azure.confluent.cloud
```
  Note that there is no glb in the DNS Domain name.
  
  If the Confluent Cloud DNS Domain name includes the logical cluster id which starts with lkc-, omit the logical cluster id when specifying it as the Private DNS Zone name. For example, the DNS Domain name shown as lkc-123abc-4kgzg.centralus.azure.confluent.cloud in Confluent Cloud should be converted to 4kgzg.centralus.azure.confluent.cloud to be used as the Private DNS Zone name.
4. Fill in subscription, resource group and name and click Review + create.
5. Wait for the Azure deployment to complete.
Create DNS records.
1. Go to the Private DNS Zone resource as created above.
2. Click + Record Set.
3. Create the following record set for Confluent Cloud single availability zone clusters. The IP address of the Private Endpoint can be found under its associated network interface.
  1. Select name as “*”, type as “A”, TTL as “1 Minute” and add IP address of the single virtual endpoint as created above.
4. Create the following record sets for Confluent Cloud multi-availability zone clusters. The IP address of the Private Endpoint can be found under its associated network interface.
  1. Select name as “*”, type as “A”, TTL as “1 Minute” and add IP addresses of all three virtual endpoints as created above.
  2. Select name as “*.az1”, type as “A”, TTL as “1 Minute” and add IP address of the az1 virtual endpoint as created above.
  3. Select name as “*.az2”, type as “A”, TTL as “1 Minute” and add IP address of the az2 virtual endpoint as created above.
  4. Select name as “*.az3”, type as “A”, TTL as “1 Minute” and add IP address of the az3 virtual endpoint as created above.
Attach the Private DNS Zone to the VNets where clients or applications are present.
Go to the Private DNS Zone resource and click Virtual network links under settings.
1. Click Add.
2. Fill in link name, subscription and virtual network.

Validate connectivity to Confluent Cloud¶

From an instance within the VNet, or anywhere the DNS is set up, run the following to validate Kafka connectivity through Azure Private Link is working correctly.
1. Set an environment variable with the cluster bootstrap URL.
```
export BOOTSTRAP=$<bootstrap-server-url>
```
  The Bootstrap URL displayed in Confluent Cloud Console includes the port (9092). The BOOTSTRAP value should include the full hostname, but do not include the port. This is so that you can run the openssl s_client -connect <host>:<port> command with the required values.
  
  For example:
```
export BOOTSTRAP=lkc-222v1o-4kgzg.centralus.azure.glb.confluent.cloud
```
2. Test connectivity to your cluster by running the openssl s_client -connect <host>:<port> command, specifying the $BOOTSTRAP environment variable for the <host> value and 9092 for the <port> value.
```
openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
```
  To run the openssl s_client -connect command, the -connect option requires that you specify the host and the port number. For details, see the openssl documentation for the -connect option option in the openssl s_client documentation.
3. If the output returned is -----BEGIN CERTIFICATE----- Verify return code: 0 (ok), then connectivity to the bootstrap is confirmed.
  
  Note
  
  You might need to update the network security tools and firewalls to allow connectivity. If you have issues connecting after following these steps, confirm which network security systems your organization uses and whether their configurations need to be changed. If you still have issues, run the debug connectivity script and provide the output to Confluent Support for assistance with your Azure Private Link setup.
4. Log in to the Confluent Cloud CLI with your Confluent Cloud credentials.
```
confluent login
```
5. List the clusters in your organization.
```
confluent kafka cluster list
```
6. Select the cluster with Azure Private Link you wish to test.
```
confluent kafka cluster use ...
```
  For example:
```
confluent kafka cluster use lkc-222v1o
```
7. Create a cluster API key to authenticate with the cluster.
```
confluent api-key create --resource ... --description ...
```
  For example:
```
confluent api-key create --resource lkc-222v1o --description "connectivity test"
```
8. Select the API key you just created.
```
confluent api-key use ... --resource ...
```
  For example:
```
confluent api-key use R4XPKKUPLYZSHOAT --resource lkc-222v1o
```
9. Create a test topic.
```
confluent kafka topic create test
```
10. Start consuming events from the test topic.
```
confluent kafka topic consume test
```
11. Open another terminal tab or window.
12. Start a producer.
```
confluent kafka topic produce test
```
  Type anything into the produce tab and hit Enter; press Ctrl+D or Ctrl+C to stop the producer.
13. The tab running consume will print what was typed in the tab running produce.

You’re done! The cluster is ready for use.

Limitations¶

Cross-region Azure Private Link connections are not supported.
Azure Private Link is only available for use with Dedicated clusters.
Existing Confluent Cloud clusters cannot be converted to use Azure Private Link.
Fully-managed Confluent Cloud connectors can connect to data sources or sinks using a public IP address. Sources or sinks in the customer network with private IP addresses are not supported.
Availability zone selection for placement of Confluent Cloud cluster and Azure Private Link service is not supported.
See also: Prerequisites.