Use Azure Egress Private Link Endpoints for Dedicated Clusters on Confluent Cloud¶

Azure Private Link is a networking service that allows one-way connectivity from one VNet to a service provider and is popular for its unique combination of security and simplicity.

Confluent Cloud supports outbound Azure Private Link connections using Egress Private Link Endpoints. Egress Private Link Endpoints are Azure Private Endpoints, and they enable Confluent Cloud clusters to access supported Azure services and other endpoint services powered by Azure Private Link, such as Azure Blob Storage, a SaaS service, or a Private Link service that you create yourself.

The following diagram summarizes the Egress Private Link Endpoint architecture between Confluent Cloud and various potential destinations.

Azure Egress Private Link Endpoint architecture

To set up an Egress Private Link Endpoint from Confluent Cloud to an external system, such as for managed connectors:

Obtain the Azure Private Link service resource ID.

For certain target systems, you can retrieve the service resource ID as part of the guided workflow while creating an Egress PrivateLink Endpoint in the next step.
Create an Egress Private Link Endpoint in Confluent Cloud.
[Optional] Create private DNS records for use with Azure private endpoints.

For service/connector-specific setup, see the target system networking supportability table.

Requirements and considerations¶

Review the following requirements and considerations before you set up an Egress Private Link Endpoint using Azure Private Link:

Egress PrivateLink Endpoints described in this document is only available for use with Dedicated clusters.

For use with Enterprise clusters, see Use Azure Egress Private Link Endpoints for Serverless Products on Confluent Cloud.
Egress Private Link Endpoints can only be used by fully managed connectors.

Note

It is strongly recommended that you do not implement any automated policies for endpoint acceptance. Confluent Cloud uses a shared subscription when provisioning endpoints. You should manually accept each endpoint connection after validating that the private endpoint ID you see in the Azure portal matches what you see in the Confluent Cloud Console.

When using Egress Private Link Endpoints, additional charges may apply, for example, for certain connector configurations. For more information, see the following pricing information:
- Confluent pricing
- Fully-managed Kafka Connector pricing

Obtain Azure Private Link Resource ID¶

To make an Azure Private Link connection from Confluent Cloud to an external system, you must first obtain an Azure Private Link Resource ID for Confluent to establish a connection to.

Depending on the system you wish to connect to, there may be different allowlist requirements to allow Confluent access. It is recommended that you check each system’s allowlist mechanism to verify that Confluent Cloud will be able to create an endpoint targeting that system.

For Azure services¶

Refer to your specific Azure resource to obtain its Resource ID. Along with the specific ID, you will also need to obtain the Sub-resource name depending on the resource you are connecting to.

For 3rd party services¶

Refer to the system provider’s documentation for how to obtain the Azure Resource ID to determine allowlisting requirements.

The following are reference links for some of the popular system providers: :

For Azure Private Link Services you create¶

Refer to the Azure documentation for how to create a private link service available to service consumers.

Create an Egress Private Link Endpoint in Confluent Cloud¶

Confluent Cloud Egress Private Link Endpoints are Azure Private Endpoints used to connect to Azure Private Link Services.

In the Network Management tab of the desired Confluent Cloud environment, click the Confluent Cloud network to which you want to add the PrivateLink Endpoint. The Connection Type of the network you select must be “PrivateLink Access”.
Click Create endpoint in the Egress connections tab.
Click the service you want to connect to.
Follow the guided steps to specify the field values, including:
- Name: The name of the Private Link Endpoint.
- Resource ID: The resource ID of the Private Link service you retrieved as part of this guided workflow or as described in Obtain Azure Private Link Resource ID.
  
  Note that the resource alias is not supported.
- Sub-resource name: The sub-resource name for the specific Azure service you retrieved in Obtain Azure Private Link Resource ID.
Click Create to create the Private Link Endpoint.
If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

Send a request to create an endpoint:

HTTP POST request

POST https://api.confluent.cloud/networking/v1/access-points

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "<The custom name for the endoint>",
    "config": {
      "kind": "AzureEgressPrivateLinkEndpoint",
      "private_link_service_resource_id": "<ID of the Azure Private Link service you wish to connect to>",
      "private_link_subresource_name": "<Names of subresources which the Private Endpoint is able to connect to>"
    },
    "environment": {
      "id": "<The environment ID where the endpoint belongs to>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

private_link_service_resource_id and private_link_subresource_name: See Obtain the Azure Private Link service resource ID.
gateway.id: Issue the following API request:
```
GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}
```
You can find the gateway id in the response under spec.gateway.id.

An example request spec to create an endpoint:

{
  "spec": {
    "display_name": "prod-plap-egress-usw2",
    "config": {
      "kind": "AzureEgressPrivateLinkEndpoint",
      "private_link_service_resource_id": "/subscriptions/0000000/resourceGroups/plsRgName/providers/Microsoft.Network/privateLinkServices/privateLinkServiceName"
    },
    "environment": {
      "id": "env-00000",
    },
    "gateway": {
      "id": "gw-00000",
    }
  }
}

Use the confluent network access-point private-link egress-endpoint create Confluent CLI command to create an Egress Private Link Endpoint:

confluent network access-point private-link egress-endpoint create [name] [flags]

The following are the command-specific flags:

--cloud: Required. The cloud provider. Set to azure.
--service: Required. ID of the Azure Private Link service you wish to connect to.

See Obtain the Azure Private Link service resource ID.
subresource: The sub-resource name for the specific Azure service you retrieved in Obtain Azure Private Link Resource ID.
--gateway: Required. Gateway ID.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create an Azure Egress Private Link Endpoint:

confluent network access-point private-link egress-endpoint create \
  --cloud azure \
  --gateway gw-123456 \
  --service /subscriptions/0000000/resourceGroups/plsRgName/providers/Microsoft.Network/privateLinkServices/privateLinkServiceName \

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Use the confluent_access_point resource to create an Egress Private Link Endpoint.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_access_point" "main" {
  display_name = "access_point"
  environment {
    id = confluent_environment.development.id
  }
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  azure_egress_private_link_endpoint {
    private_link_service_resource_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/s-abcde/providers/Microsoft.Network/privateLinkServices/pls-plt-abcdef-az3"
  }
}

Your Egress Private Link Endpoint status will transition from “Provisioning” to “Ready” in the Confluent Cloud Console when the endpoint has been created and can be used. Some endpoints may need to be manually accepted before transitioning to “Ready”.

Once an endpoint is created, connectors provisioned against Kafka clusters in the same network can leverage the Egress Private Link Endpoint to access the external data systems.

Confluent Cloud exposes various pieces of information for each of the above Egress Private Link Endpoints so that you can use it in various network-related policies.

Create a private DNS record in Confluent Cloud¶

Create private DNS records for use with Azure private endpoints.

Not all service providers set up public DNS records to be used when connecting to them with Azure Private Link. For situations where a system provider requires setting up private DNS records in conjunction with Azure Private Link, you need to create DNS records in Confluent Cloud.

Before you create a DNS Record, first create an Egress Private Link Endpoint and use the Egress Private Link Endpoint ID for the DNS record.

When creating DNS records, Confluent Cloud creates a single * record that maps the domain name you specify to the IP address of the private endpoint.

For example, in setting up DNS records for Snowflake, the DNS zone configuration will look like:

*.eastus2.privatelink.snowflakecomputing.com A 10.2.0.1 TTL 60

In the Network Management tab of your environment, click the Confluent Cloud network you want to add the DNS record to.
In the Egress DNS tab, click Create DNS record.
Specify the following field values.
- Egress Private Link Endpoint: The Egress Private Link Endpoint ID you created in create an Egress Private Link Endpoint.
- Domain: The domain of the private link service you wish to access. For multiple domains, for example, with the Cosmo DB connector, create a separate DNS record for each of the domain values.
  
  Get the domain values from the private link service provider, Azure or a third-party provider.
Click Save.

Send a request to create a DNS Record that is associated with a Private Link Endpoint that is associated with a gateway.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/dns-records

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "The name of this DNS record",
    "domain": "<The fully qualified domain name of the external system>",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "<The ID of the endpoint that you created>"
    },
    "environment": {
      "id": "<The environment ID where this resource belongs to>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

Get the spec.domain value from the private link service provider, Azure or a third-party provider/

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

An example request spec to create a DNS record:

{
  "spec": {
    "display_name": "prod-dns-record1",
    "domain": "example.com",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "plap-12345"
    },
    "environment": {
      "id": "env-00000",
    },
    "gateway": {
      "id": "gw-00000",
    }
  }
}

Use the confluent network dns record create Confluent CLI command to create a DNS record:

confluent network dns record create [name] [flags]

The following are the command-specific flags:

--private-link-access-point: Required. Private Link Private Link Endpoint ID.
--gateway: Required. Gateway ID.
--domain: Required. Fully qualified domain name of the external system. Get the domain value from the private link service provider, Azure or a third-party provider.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create DNS record for an endpoint:

confluent network dns record create my-dns-record \
  --gateway gw-123456 \
  --private-link-access-point ap-123456 \
  --domain xyz.westeurope.privatelink.snowflakecomputing.com

Use the confluent_dns_record Resource resource to create DNS records.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_record" "main" {
  display_name = "dns_record"
  environment {
    id = confluent_environment.development.id
  }
  domain = "example.com"
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  private_link_access_point {
    id = confluent_access_point.main.id
  }
}

Support for Azure Private Link Service configuration¶

Confluent Support can help with issues you may encounter when creating an Access Point to a specific service or resource.

For any service-side problems, such as described below, Confluent is not responsible for proper Azure Private Link Service configuration or setup:

If you need help setting up an Azure Private Link Service for data systems running within your environment or VNet that you want to connect to from Confluent Cloud, contact Azure for configuration help and best practices.
If you need help configuring Azure Private Link Services for those managed by a third-party provider or service, contact that provider for compatibility and proper setup.