Egress PrivateLink Endpoint Setup: Elasticsearch on Azure for Confluent Cloud

This topic presents the steps for configuring the Elasticsearch Sink V2 connector in Confluent Cloud with Microsoft Azure Private Link and Egress Private Link endpoint.

Prerequisites

The following is a list of prerequisites for configuring the Elasticsearch Sink V2 connector with an Egress PrivateLink endpoint:

• In Confluent Cloud, one of the following cluster types is set up with the specified network resource:

  • A Dedicated cluster with a Confluent Cloud network.

    For the steps to create a Confluent Cloud network, see Create a Confluent Cloud network. The Connection type of the network needs to be PrivateLink Access.

  • A Enterprise cluster with a network gateway

    For the steps to create a gateway, see Create a gateway for outbound connectivity in Confluent Cloud.

• An Elasticsearch Cloud deployment is running in Azure within the same region and cloud as the Confluent Cloud network resource.

• Confluent Cloud network (Dedicated or Serverless Egress Gateway) setup within same region and cloud as Elasticsearch.

Step 1. Obtain Azure Private link service alias and Private hosted zone domain name from Elasticsearch

1. From the Elastic documentation, capture the following values for your region:

   • Azure Private Link Service alias: Use this as the Azure Resource ID in Confluent Cloud.

   • Private hosted zone domain name: Use this to construct the Elasticsearch endpoint hostname that you will configure as the Domain when creating the DNS record in Confluent Cloud (see Step 3)

Step 2. Create an Egress PrivateLink endpoint

Dedicated cluster

1. In the Network management page or tab of the desired Confluent Cloud environment, click the Confluent Cloud network you want to add the PrivateLink endpoint to. The Connection Type of the network needs to be PrivateLink Access.

2. Click Create endpoint in the Egress connections tab.

3. Click the service you want to connect to, specifically, Elasticsearch. Select Other if you do not see the specific service.

4. Specify the following field values:

   • SERVICE: Name of service connecting to - Elasticsearch.

   • Endpoint Name: Name of the PrivateLink endpoint.

   • Azure Resource ID: The name of the Azure Private Link service alias you retrieved in Step 1. Obtain Azure Private link service alias and Private hosted zone domain name from Elasticsearch.

5. Click Create to create the PrivateLink endpoint.

6. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

Enterprise cluster

1. In the Network management page or tab of the desired Confluent Cloud environment, click the For serverless products tab.

2. Click the gateway to which you want to add the PrivateLink endpoint.

3. In the Access points tab, click Add access point.

4. Click the service you want to connect to, specifically, Elasticsearch. Select Other if you do not see the specific service.

5. Follow the steps below to specify the following field values:

   • Access point name: Name of the PrivateLink endpoint.

   • Azure Resource ID: The name of the Azure Private Link service alias you retrieved in Step 1. Obtain Azure Private link service alias and Private hosted zone domain name from Elasticsearch.

6. Click Create access point to create the PrivateLink endpoint.

7. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and click Finish.

Step 3. Add Resource name & Resource ID within Elasticsearch Deployment

Important

The Resource ID in the Confluent Cloud UI is a Confluent-specific identifier. It does not match the Azure resource ID required for Elasticsearch Azure Private Link. Azure Private Link requires the underlying Azure resource GUID and resource name, which are not currently available in the Confluent Cloud UI. To obtain the correct resource GUID and resource name, contact Confluent Support. Use these values when you configure Elasticsearch | az| Private Link.

1. Contact Confluent Support and request the Azure Private Link resource GUID and resource name for your Elasticsearch egress PrivateLink endpoint.

2. Log in to the Elastic Cloud Console.

3. From any deployment or project on the home page, select Manage.

4. In the Elastic Cloud Console, navigate to Access and security > Network security.

5. Click the Create dropdown and select Private connection.

6. Specify the following field values:

   • Resource Type: Select hosted deployments.

   • Cloud provider and region: Select the cloud provider and region for the private connection, matching your Confluent Cloud cluster and network region.

   • Connectivity: Select Privatelink.

   • VPC Filter: Enter your private endpoint Resource name and Resource ID. When applied to a deployment, this information is used to filter traffic. This allows requests only from the Confluent Cloud cluster endpoint to Elasticsearch service.

   • Apply to resources: Under Apply to resources, associate the new private connection policy to your deployment. If you specified a VPC filter, then after you associate the filter with a deployment, it starts filtering traffic.

7. Click Create.

To create a new private connection policy in the Elastic Cloud Console, you can also follow Elasticsearch documentation.

Step 4. Create a DNS record

Dedicated cluster

1. When the PrivateLink endpoint status transitions to Ready, in the DNS tab, click Create record on the associated PrivateLink endpoint.

   

2. Specify the following, and click Save.

   • Access point: Select the PrivateLink endpoint you created in Step 2. Create an Egress PrivateLink endpoint.

   • Domain: When creating the DNS record for Azure Private Link, the Domain value must exactly match the Elasticsearch endpoint you plan to use. Elasticsearch supports two valid endpoint formats, and the required DNS record depends on which one you choose.

     • Option 1: Using the alias-based endpoint (recommended)

       If you connect using the alias-based endpoint constructed from your deployment details and the private hosted zone domain name:

       https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
       

       Example:

       https://my-deployment-d53192.es.privatelink.eastus2.azure.elastic-cloud.com
       

       Configure the DNS record with the following Domain:

       my-deployment-d53192.es.privatelink.eastus2.azure.elastic-cloud.com
       

     • Option 2: Using the Elasticsearch deployment (cluster) ID

       If you connect using the Elasticsearch deployment (cluster) ID directly:

       https://{{deployment_id}}.{{private_hosted_zone_domain_name}}
       

       Example:

       https://6b111580caaa4a9e84b18ec7c600155e.privatelink.eastus2.azure.elastic-cloud.com
       

       Configure the DNS record with the following Domain:

       6b111580caaa4a9e84b18ec7c600155e.privatelink.eastus2.azure.elastic-cloud.com
       

   

Enterprise cluster

1. In the Network Management tab of your environment, click the For serverless products tab, and click the Confluent Cloud gateway.

2. In the DNS tab, click Create DNS record.

3. Specify the following field values:

   • Access point: Select the PrivateLink endpoint you created in Step 2. Create an Egress PrivateLink endpoint.

   • Domain: When creating the DNS record for Azure Private Link, the Domain value must exactly match the Elasticsearch endpoint you plan to use. Elasticsearch supports two valid endpoint formats, and the required DNS record depends on which one you choose.

     • Option 1: Using the alias-based endpoint (recommended)

       If you connect using the alias-based endpoint constructed from your deployment details and the private hosted zone domain name:

       https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
       

       Example:

       https://my-deployment-d53192.es.privatelink.eastus2.azure.elastic-cloud.com
       

       Configure the DNS record with the following Domain:

       my-deployment-d53192.es.privatelink.eastus2.azure.elastic-cloud.com
       

     • Option 2: Using the Elasticsearch deployment (cluster) ID

       If you connect using the Elasticsearch deployment (cluster) ID directly:

       https://{{deployment_id}}.{{private_hosted_zone_domain_name}}
       

       Example:

       https://6b111580caaa4a9e84b18ec7c600155e.privatelink.eastus2.azure.elastic-cloud.com
       

       Configure the DNS record with the following Domain:

       6b111580caaa4a9e84b18ec7c600155e.privatelink.eastus2.azure.elastic-cloud.com
       

4. Click Save.

Step 5. Create the Elasticsearch Sink V2 connector

1. While creating the connector, use the following URL structure for the Connection URI on the authentication page. This URL uses the endpoint information from your Elastic deployment and your registered private hosted zone domain name. For more information, see the Elasticsearch Azure Privatelink documentation.

   https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
   

   For example:

   https://my-deployment-d53192.es.privatelink.eastus2.azure.elastic-cloud.com
   

   Note

   • You can use either 443 or 9243 as a port.

   • You can also connect to the cluster using the Elasticsearch cluster ID, for example, https://6b111580caaa4a9e84b18ec7c600155e.privatelink.eastus2.azure.elastic-cloud.com

2. See the Elasticsearch Sink V2 connector documentation for the steps to create the sink connector in Confluent Cloud.