Azure Private Link for Enterprise Clusters¶
Confluent Cloud supports private connectivity for Enterprise Kafka clusters using PrivateLink Attachment. When you use PrivateLink Attachment, your Enterprise cluster is only accessible from tenant-specific private endpoints. Public access is blocked with PrivateLink Attachment.
Confluent Cloud uses the following private networking resources for Enterprise clusters. These resources are regional and do not have a mapping to specific availability zones.
- PrivateLink Attachment
The PrivateLink Attachment (
PrivateLinkAttachment
) resource represents a reservation to establish a Private Link connection from your Virtual Network (VNet) regional services in a Confluent Cloud environment.A PrivateLink Attachment belongs to an Environment in the Confluent resource hierarchy.
- PrivateLink Attachment Connection
- A PrivateLink Attachment Connection (
PrivateLinkAttachmentConnection
) is a registration of VNet private endpoints that are allowed to connect to Confluent Cloud. A PrivateLink Attachment Connection belongs to a specific PrivateLink Attachment.
You can use Confluent Cloud UI, Confluent REST API, Confluent CLI, or Terraform to establish a Private Link connectivity for an Enterprise cluster.
The high-level workflow is:
In Azure, create a private endpoint to be associated with the PrivateLinkAttachment service.
In Confluent Cloud, create a PrivateLinkAttachmentConnection.
Create a Kafka client in your VNet using the bootstrap endpoint of your Enterprise Kafka cluster. This Kafka client can live in Virtual Machine or similar compute infrastructure.
Validate produce/consume traffic is successful.
Once you create a PrivateLinkAttachment resource and establish a Private Link, you can securely send and receive traffic through the PrivateLink between your VNet and Confluent Cloud.
Requirements and considerations¶
- You can connect to only one environment from a single VNet or from an on-prem network.
- The following regions are supported:
westeurope
eastus
eastus2
centralus
- Fully-managed Confluent Cloud connectors in the customer networks with private IP addresses are not supported. An exception to this is the Azure connectors that access the Azure services from private endpoints. For the current list of these connectors, see Service and gateway endpoints.
- Confluent Cloud Console components, like topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Azure Private Link, see Use Confluent Cloud with Private Networking.
Create a PrivateLink Attachment¶
When you create a PrivateLink Attachment in an environment and in a region, the PrivateLink Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.
- In the Confluent Cloud Console, select an environment for the PrivateLink Attachment.
- In the Network management tab in the environment, click Add network configuration.
- Select For Enterprise Clusters and click Continue.
- Select Azure, select Region, and click Continue.
- Provide the PrivateLink Attachment name in the Network name field and click Add network configuration.
The PrivateLink Attachment will be provisioned and move to the Waiting for
connection
state.
A PrivateLink Attachment can be in one of the following states:
WAITING FOR CONNECTION
: The PrivateLink Attachment is waiting for a connection to be created.READY
: Azure Private Link connectivity is ready to be used.EXPIRED
: A valid connection has not been provisioned within the allotted time. A new PrivateLink Attachment must be provisioned.
Send a request to create a PrivateLink Attachment resource:
REST request
POST https://api.confluent.cloud/networking/v1/private-link-attachments
REST request body
{ "spec": { "display_name": "<name of this resource>", "cloud": "<provider type>", "region": "<region>", "environment": { "id": "<environement id>" } } }
In the REST response,
status.phase
should be set toPROVISIONING
.Check the status of the new PrivateLink Attachment:
REST request
GET https://api.confluent.cloud/networking/v1/private-link-attachments/<platt-id>
REST response example
{ "status": { "phase": "WAITING_FOR_CONNECTIONS", "error_code": "", "error_message": "", "cloud": { "kind": "AzurePrivateLinkAttachmentStatus", "private_link_service":{ "private_link_service_alias": "<pls-plt-abcdef-az1.f5aedb5a-5830-4ca6-9285-e5c81ffca2cb.centralus.azure.privatelinkservice>", "private_link_service_resource_id": "</subscriptions/12345678-9012-3456-7890-123456789012/resourceGroups/rg-abcdef/providers/Microsoft.Network/privateLinkServices/pls-plt-abcdef>" } } } }
status.phase
isWAITING_FOR_CONNECTIONS
because no PrivateLink Attachment Connection has not been associated with this PrivateLink Attachment resource yet.The
status.cloud
object has information about theprivate_link_service_alias
andprivate_link_service_resource_id
that you must connect your PrivateLink Attachment endpoint to.
Use the confluent network private-link attachment create Confluent CLI command to create an Azure private link attachment:
confluent network private-link attachment create <attachment-name> <flags>
The following command-specific flags are supported:
--network
: Required. Confluent Cloud network ID.--region
: Required. Azure region where the resources to be accessed using the private link attachment.
You can specify additional optional CLI flags described in the Confluent
CLI command reference,
such as --context
and --environment
.
The following is an example Confluent CLI command to create a private link attachment:
confluent network private-link attachment create my-private-link-attachment \
--cloud azure \
--region us-west-2
Create an Azure private endpoint¶
In Azure, create an endpoint that is associated with the Private Link Service ID of the PrivateLink Attachment you created in Create a PrivateLink Attachment.
For details on creating a private endpoint in Azure, see Create a Private Endpoint.
On the Private Endpoint page in Azure portal, click + Create.
In the Basics pane, specify the following:
- Subscription: The subscription name that you selected when you created the VNet.
- Resource group: The same resource group that you selected when you created the VNet.
- Name: The name for the private endpoint.
- Network interface name: A network interface name.
- Region: The region for the private endpoint.
Click Next: Resource.
In the Resource pane, specify the following:
Connection method: Select Connect to an Azure resource by resource ID or alias.
Resource ID or alias: Paste in the Confluent Cloud Resource ID or Service Alias.
This is the alias or ID created in the previous section, Create a PrivateLink Attachment.
Click Next: Virtual Network.
In the Virtual Network pane, specify the following:
- Virtual network: Select the VNet where the private endpoint is to be created.
- Subnet: Select the subnet where the private endpoint is to be created.
- Network policy for private endpoints: Accept the default value (Disabled).
- Private IP configuration: Select Dynamically allocate IP address.
Click Next: DNS and accept the default values.
Click Next: Tags and, optionally, add tags.
Click Next: Review + create. Review the details and click Create to create the private endpoint.
Wait for the Azure deployment to complete, go to the private endpoint resource and verify private endpoint connection status is Approved.
Create an endpoint using the following Azure CLI:
az network private-endpoint create \
--connection-name <connection name> \
--name <endpoint name> \
--private-connection-resource-id <resource id> \
--resource-group <resource group name> \
--subnet <subnet for the endpoint>
Create a PrivateLink Attachment Connection¶
Create a PrivateLink Attachment Connection resource in Confluent Cloud. A PrivateLink Attachment Connection represents a private endpoint in your VNet.
The name of the VNet private link service is not required. Confluent will check which private link service is associated with the PrivateLink Attachment that has a pending private endpoint with the given ID.
In Confluent Cloud Console, in the Network management tab in the environment, click the PrivateLink Attachment you want to add a connection to.
Make sure the PrivateLink Attachment is in the correct region of the private endpoint.
Click + Add connection.
Specify the connection name and the private endpoint ID.
The private endpoint ID is the id of the private endpoint that was created in Create an Azure private endpoint.
Click Finish.
The PrivateLink Attachment and PrivateLink Attachment Connection should now move to the READY
state once the
private endpoint connection is accepted.
Send a request to create a PrivateLink Attachment Connection resource:
REST request
POST https://api.confluent.cloud/networking/v1/private-link-attachment-connections
REST request body
{ "spec": { "display_name": "<PrivateLinkAttachmentEndpoint name>", "cloud": { "kind": "AzurePrivateLinkAttachmentConnection", "private_endpoint_id": "<Private Endpoint ID>", }, "environment": { "id": "<Environment ID>", }, "private_link_attachment": { "id": "<PrivateLinkAttachment>", } } }
REST response example
{ "api_version": "networking/v1", "kind": "PrivateLinkAttachmentConnection", "id": "plattc-xyzuvw", "status": { "phase": "PROVISIONING", "error_code": "", "error_message": "", } }
status.phase
isPROVISIONING
because a private endpoint connection has not yet been accepted.Check the status of the new PrivateLink Attachment Connection:
REST request
GET https://api.confluent.cloud/networking/v1/private-link-attachment-connections/<platt-id>
REST response example
{ "api_version": "networking/v1", "kind": "PrivateLinkAttachmentConnection", "id": "plattc-xyzuvw", "status": { "phase": "READY", "error_code": "", "error_message": "", "cloud": { "kind": "AzurePrivateLinkAttachmentConnectionStatus", "phase": "READY", "private_link_service_alias": "pls-plt-abcdef-az1.f5aedb5a-5830-4ca6-9285-e5c81ffca2cb.centralus.azure.privatelinkservice", "private_link_service_resource_id": "/subscriptions/12345678-9012-3456-7890-123456789012/resourceGroups/rg-abcdef/providers/Microsoft.Network/privateLinkServices/pls-plt-abcdef", "private_endpoint_id": "/subscriptions/Microsoft.Network/privateEndpoints/pe-platt-abcdef" } } }
status.phase
isREADY
because the private endpoint connection has been accepted.status.cloud
has an object of kindAzurePrivateLinkConnectionStatus
.
Use the confluent network private-link attachment connection create Confluent CLI command to create an Azure private link attachment connection:
confluent network private-link attachment connection create <connection-name> <flags>
The following command-specific flags are supported:
--cloud
: Required. The cloud provider. Set toazure
.--endpoint
: Required. ID of an Azure private endpoint that is connected to the Azure private link service.--attachment
: Required. Private link attachment ID.
You can specify additional optional CLI flags described in the Confluent
CLI command reference,
such as --context
and --environment
.
The following is an example Confluent CLI command to create a private link attachment connection:
confluent network private-link attachment connection create azure-private-link-attachment-connection \
--cloud azure \
--endpoint vpce-1234567890abcdef0 \
--attachment platt-123456
Set up DNS records in Azure¶
Set up a DNS resolution and a DNS record using the Azure private DNS zone in the Azure console. This section focuses on the settings related to Confluent Cloud. For details, see Create an Azure private DNS zone.
Create a private DNS zone.
- In the Confluent Cloud Console, copy the DNS Domain name in PrivateLink
Attachment in the Network management tab, and use it as the name for
the private DNS zone. It is in the form,
<region>.azure.private.confluent.cloud
. For example,centralus.azure.private.confluent.cloud
. - In Private Zones in the Azure portal, click + Create.
- In the Basics pane, enter or select the following values:
- Subscription: Pre-filled with the subscription name that you selected when you created the VNet.
- Resource group: Select the resource group that you selected when you created the VNet.
- Name: Specify the domain name retrieved from the Confluent Cloud Console in
the first step. It is in the format of
<cluster>.<region>.azure.confluent.cloud
, for example,4kgzg.centralus.azure.confluent.cloud
.
- Click Next: Tags and, optionally, add tags.
- Click Next: Review + create. Review the details and click Create a DNS zone.
- Wait for the Azure deployment to complete.
- In the Confluent Cloud Console, copy the DNS Domain name in PrivateLink
Attachment in the Network management tab, and use it as the name for
the private DNS zone. It is in the form,
To create DNS records, go to the private DNS zone resource you created in the previous step, and click + Record Set for the Confluent Cloud clusters.
Name:
*
Type:
A
TTL and TTL unit:
1
Minute
IP address: The IP address of the private endpoint can be found under its associated network interface under Settings for the private endpoint.
Attach the private DNS zone to the VNets where clients or applications are present.
Go to the private DNS zone resource and click Virtual network links under Settings.
- Click + Add.
- Specify the required values and click OK to create a virtual network link.