Egress PrivateLink Endpoint Setup: Elasticsearch on AWS for Confluent Cloud

This topic presents the steps for configuring the Elasticsearch Sink V2 connector in Confluent Cloud with AWS PrivateLink and Egress PrivateLink endpoint.

Confluent Cloud is available through AWS Marketplace or directly from Confluent.

Prerequisites

The following are prerequisites for configuring the Elasticsearch Sink V2 connector with an Egress PrivateLink endpoint:

• In Confluent Cloud, one of the following cluster types is set up with the specified network resource:

  • A Dedicated cluster with a Confluent Cloud network.

    For the steps to create a Confluent Cloud network, see Create a Confluent Cloud network. The Connection type of the network needs to be PrivateLink Access.

  • A Enterprise cluster with a network gateway

    For the steps to create a gateway, see Create a gateway for outbound connectivity in Confluent Cloud.

• An Elasticsearch Cloud deployment is running in AWS within the same region and cloud as the Confluent Cloud network resource.

• Confluent Cloud network (Dedicated or Serverless Egress Gateway) setup within same region and cloud as Elasticsearch.

Step 1. Obtain VPC service name and Private hosted zone domain name from Elasticsearch

1. From the Elastic documentation, capture the following values for your region:

   • VPC Service Name: Use this as the PrivateLink service name in Confluent Cloud.

   • Private hosted zone domain name: Use this as the Domain when creating the DNS record in Confluent Cloud.

Step 2. Create an Egress PrivateLink endpoint

Dedicated cluster

1. In the Network management page or tab of the desired Confluent Cloud environment, click the Confluent Cloud network you want to add the PrivateLink endpoint to. The Connection Type of the network needs to be PrivateLink Access.

2. Click Create endpoint in the Egress connections tab.

3. Click the service you want to connect to, specifically, Elasticsearch. Select Other if you do not see the specific service.

4. Specify the following field values:

   • SERVICE: Name of service connecting to - Elasticsearch.

   • Endpoint Name: Name of the PrivateLink endpoint.

   • Create an endpoint with high availability: Select this checkbox to deploy an endpoint with high availability.

   • PrivateLink service name: The VPC service name retrieved in Step 1. Obtain VPC service name and Private hosted zone domain name from Elasticsearch.

5. Click Create to create the PrivateLink endpoint.

6. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

Enterprise cluster

1. In the Network management page or tab of the desired Confluent Cloud environment, click the For serverless products tab.

2. Click the gateway to which you want to add the PrivateLink endpoint.

3. In the Access points tab, click Add access point.

4. Click the service you want to connect to, specifically, Elasticsearch. Select Other if you do not see the specific service.

5. Follow the steps below to specify the following field values:

   • Access point name: Name of the PrivateLink endpoint.

   • Create an endpoint with high availability: Select this checkbox to deploy an endpoint with high availability.

   • PrivateLink service name: The VPC service name retrieved in Step 1. Obtain VPC service name and Private hosted zone domain name from Elasticsearch.

6. Click Create access point to create the PrivateLink endpoint.

7. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and click Finish.

Step 3. Create a DNS record

Dedicated cluster

1. When the PrivateLink endpoint status transitions to Ready, in the DNS tab, click Create record on the associated PrivateLink endpoint.

   

2. Specify the following field values and click Save.

   • Access point: Select the PrivateLink endpoint you created in Step 2. Create an Egress PrivateLink endpoint.

   • Domain: Enter the Private hosted zone domain name retrieved in Step 1. Obtain VPC service name and Private hosted zone domain name from Elasticsearch.

   

Enterprise cluster

1. In the Network Management tab of your environment, click the For serverless products tab, and click the Confluent Cloud gateway.

2. In the DNS tab, click Create DNS record.

3. Specify the following field values:

   • Access point: Select the PrivateLink endpoint you created in Step 2. Create an Egress PrivateLink endpoint.

   • Domain: Enter the Private hosted zone domain name retrieved in Step 1. Obtain VPC service name and Private hosted zone domain name from Elasticsearch.

4. Click Save.

Step 4. Add Private endpoint within Elasticsearch deployment

1. In the Confluent Cloud console, click the Egress connections tab and copy the VPC endpoint ID.

   

2. Log in to the Elastic Cloud Console.

3. From any deployment or project on the home page, select Manage.

4. In the Elastic Cloud Console, navigate to Access and security > Network security.

5. Click the Create dropdown and select Private connection.

6. Specify the following field values:

   • Resource Type: Select hosted deployments.

   • Cloud provider and region: Select the cloud provider and region for the private connection, matching your Confluent Cloud cluster and network region.

   • Connectivity: Select Privatelink.

   • VPC Filter: Enter the VPC endpoint ID retrieved in the previous step.

   • Apply to resources: Under Apply to resources, associate the new private connection policy to your deployment. If you specified a VPC filter, then after you associate the filter with a deployment, it starts filtering traffic.

7. Click Create.

To create a new private connection policy in the Elastic Cloud Console, you can also follow Elasticsearch documentation.

Step 5. Create the Elasticsearch Sink V2 connector

1. While creating the connector, use the following URL structure for the Connection URI on the authentication page. This URL uses the endpoint information from your Elastic deployment and your registered private hosted zone domain name. For more information, see the Elasticsearch AWS Privatelink documentation.

   https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
   

   For example:

   https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com
   

   Note

   • You can use either 443 or 9243 as a port.

   • You can also connect to the cluster using the Elasticsearch cluster ID, for example, https://6b111580caaa4a9e84b18ec7c600155e.vpce.us-east-1.aws.elastic-cloud.com

2. See the Elasticsearch Sink V2 connector documentation for the steps to create the sink connector in Confluent Cloud.