# Use AWS Egress PrivateLink Endpoints for Serverless Products on Confluent Cloud [AWS PrivateLink](https://aws.amazon.com/privatelink/) is a networking service that allows one-way connectivity from one VPC to a service provider. It is popular for its unique combination of security and simplicity. Confluent Cloud, available through the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g5ujul6iovvcy?trk=14575e70-1766-4f20-8083-0c2757a1ec75&sc_channel=el) or [directly from Confluent](https://www.confluent.io/get-started/), supports outbound AWS PrivateLink connections using Egress PrivateLink Endpoints. Egress PrivateLink Endpoints are [AWS interface VPC Endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html), and they enable Confluent Cloud Enterprise clusters to access supported AWS services and other endpoint services powered by AWS PrivateLink. Examples of the services are AWS S3, a SaaS service, or a PrivateLink service that you create yourself. To set up an Egress PrivateLink Endpoint from Confluent Cloud to an external system, create an Egress PrivateLink gateway and an associated access point. #### NOTE All Enterprise clusters in a region within a given environment share the same gateway and its access points. The following diagram summarizes the Egress PrivateLink Endpoint architecture between Confluent Cloud and supported services. ![AWS Egress PrivateLink Endpoint architecture](networking/images/aws-privatelink-esku-egress.png) The high level workflow to set up an Egress PrivateLink Endpoint from Confluent Cloud to an external system, such as for managed or custom connectors is: 1. [Create a gateway for outbound connectivity in Confluent Cloud](#aws-privatelink-egress-create-gateway-esku). 2. [Obtain the AWS PrivateLink Service name](#aws-privatelink-egress-obtain-service-name-esku). For certain target services, you can retrieve the service name as part of the guided workflow while creating an Egress PrivateLink Endpoint in the next step. 3. [Create an Egress PrivateLink Endpoint in Confluent Cloud](#aws-privatelink-egress-create-access-point-esku). 4. [[Optional] Create private DNS records for use with AWS VPC endpoints](#aws-privatelink-egress-dns-records-esku). For service/connector-specific setup, see the [target service networking supportability table](../connectors/networking/internet-resource.md#connect-cloud-target-system-networking-supportability). ## Requirements and considerations Review the following requirements and considerations before you set up an Egress PrivateLink Endpoint using AWS PrivateLink: * Egress PrivateLink Endpoints described in this document is only available for use with Enterprise clusters. * One Egress PrivateLink gateway per region per environment is allowed. * The AWS PrivateLink service must be configured to allow access from Confluent Cloud’s account or IAM role. Due to the differing granularity of the allowlist configuration across SaaS providers, it is recommended that you leverage provider-specific controls (like network rules) for securing access to the PrivateLink services against confused deputy-type issues. * Egress PrivateLink Endpoints can be used by services, specifically: * Connectors You can use Egress PrivateLink for managed and custom-built connectors in Dedicated and Enterprise clusters to access external target systems with private connectivity. * Custom-built connectors You can use Egress PrivateLink with custom-built connectors in any AWS region where [custom connectors](../connectors/bring-your-connector/custom-connector-fands.md#cc-byoc-regions) are supported. * [Cluster Linking](../multi-cloud/cluster-linking/private-networking.md#cloud-cluster-link-private-networking) with Enterprise Kafka clusters * [Flink external table lookups](../flink/operate-and-deploy/private-networking.md#flink-sql-enable-private-networking-egress) * Inferences for external AI models * AWS does not support cross-region connections with PrivateLink. * When using Egress PrivateLink Endpoints, additional charges may apply, for example, for certain connector configurations. For more information, see the following pricing information: * [Confluent pricing](https://www.confluent.io/confluent-cloud/pricing/) * [Fully managed Kafka Connector pricing](https://www.confluent.io/confluent-cloud/connect-pricing/) ## Create a gateway for outbound connectivity in Confluent Cloud In order for Confluent Serverless products to be able to leverage Egress Privatelink Endpoints for outbound connectivity, you first need to enable private egress connectivity by creating a gateway. To create an egress privatelink gateway: ### Confluent Cloud Console 1. In the Confluent Cloud Console, select an environment for the Egress PrivateLink Endpoint. 2. In the **Network management** tab in the environment, click **For serverless products**. 3. Click **Add gateway configuration**. 4. Click **+ Create configuration** on the **From Confluent Cloud to your external data systems** pane. 5. On the **Configure gateway** sliding panel, enter the following information. * Gateway name * Cloud provider * Region 6. Click **Submit**. 7. You can continue to [create an access point for an Egress Private Link Endpoint](#aws-privatelink-egress-create-access-point-esku). Alternatively, you can create an access point at a later time by navigating to this gateway in the **Network management** tab. ### Confluent REST API Send a request to create a gateway: **HTTP POST request** ```text POST https://api.confluent.cloud/networking/v1/gateways ``` **Authentication** See [Authentication](https://docs.confluent.io/cloud/current/api.html/#authentication). **Request specification** ```json { "spec": { "display_name": "", "gateway_type": "AwsEgressPrivateLink", "config": { "kind": "AwsEgressPrivateLinkGatewaySpec", "region": "" }, "environment": { "id": "", "environment": "" } } } ``` ### Confluent CLI Use the [confluent network gateway create](https://docs.confluent.io/confluent-cli/current/command-reference/network/gateway/confluent_network_gateway_create.html) Confluent CLI command to create a gateway: ```bash confluent network gateway create [name] [flags] ``` The following are the command-specific flags: * `--cloud`: Required. The cloud provider. Set to `aws`. * `--region`: Required. AWS region of the gateway. * `--type` : Required. The gateway type. Set to `egress-privatelink`. You can specify additional optional CLI flags described in the [Confluent CLI command reference](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html), such as `--environment`. ### Terraform Use the [confluent_gateway](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_gateway) Confluent Terraform Provider resource to create an outbound gateway. An example snippet of Terraform configuration: ```terraform resource "confluent_environment" "development" { display_name = "Development" } resource "confluent_gateway" "main" { display_name = "my_gateway" environment { id = confluent_environment.development.id } aws_egress_private_link_gateway { region = "us-west-2" } } ``` ## Obtain AWS PrivateLink Service name To make an AWS PrivateLink connection from Confluent Cloud to an external system, you must first obtain an AWS PrivateLink Service name for Confluent to establish a connection to. Depending on the system you wish to connect to, there may be different allowlist requirements to allow Confluent access. It is recommended that you check each system’s allowlist mechanism to verify that Confluent Cloud will be able to create an endpoint targeting that system. ### For AWS services Refer to the AWS documentation for a list of all [AWS services that integrate with AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html) and their associated service names. ### For 3rd party services Refer to the system provider’s documentation for how to obtain the AWS PrivateLink Service name and to determine allowlisting requirements. The following are reference links for some of the popular system providers: * [Snowflake](https://docs.snowflake.com/en/user-guide/admin-security-privatelink#enabling-aws-privatelink) * [MongoDB Atlas](https://www.mongodb.com/docs/atlas/security-private-endpoint/#learn-about-private-endpoints-in-service) * [Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-vpc.html) ### For AWS PrivateLink Services you create Refer to the AWS documentation for how to [make your endpoint service available to service consumers](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html#share-endpoint-service). For step-by-step guide on how to set up an Egress PrivateLink to connect to self-managed services, see the [Confluent Cloud connector document](../connectors/networking/aws-eap-self-managed.md#cc-aws-eap-self-managed). #### Manage access to your service When you stand up your own PrivateLink Service, you may want to [manage its permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions) to restrict who can create endpoints to that service. Confluent Cloud uses a unique IAM Role to create VPC endpoints for each environment you create an Egress PrivateLink Endpoint from. We **highly recommend** only allowlisting this principal for maintaining an optimal security posture. To obtain the IAM Role’s ARN: 1. In the Confluent Cloud Console, in the **Network Management** tab, click the Confluent Cloud gateway you want to use. 2. Copy the IAM Principal in the **Access Points** tab. ## Create an Egress PrivateLink Endpoint in Confluent Cloud Confluent Cloud Egress PrivateLink Endpoints are [AWS interface VPC Endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) used to connect to AWS PrivateLink Services. ### Confluent Cloud Console 1. In the **Network Management** tab of the desired Confluent Cloud environment, click the **For serverless products** tab. 2. Click the gateway to which you want to add the PrivateLink Endpoint. 3. In the **Access points** tab, click **Add access point**. 4. Select the service you want to connect to. Specific services are listed based on the cloud provider for the gateway. 5. Follow the guided steps to specify the field values, including: * **Access point name**: Name of the PrivateLink Endpoint. * **Create an endpoint with high availability**: Check the box if you wish to deploy an endpoint with High Availability. Endpoints deployed with high availability have network interfaces deployed in multiple availability zones. * **PrivateLink service name**: The name of the PrivateLink service you retrieved as part of this guided workflow or as described in [Obtain AWS PrivateLink Service name](#aws-privatelink-egress-obtain-service-name-esku). 6. Click **Create access point** to create the PrivateLink Endpoint. 7. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and click **Finish**. 8. If the service requires a DNS setup, you can create a DNS record when the access point is successfully created. Specify the **Domain** name, and click **Create DNS record**. Alternatively, you can create a DNS record as described in [Create a private DNS record in Confluent Cloud](#aws-privatelink-egress-dns-records-esku) if the access point creation takes too long. ### Confluent REST API Send a request to create an endpoint: **HTTP POST request** ```text POST https://api.confluent.cloud/networking/v1/access-points ``` **Authentication** See [Authentication](https://docs.confluent.io/cloud/current/api.html/#authentication). **Request specification** ```json { "spec": { "display_name": "", "config": { "kind": "AwsEgressPrivateLinkEndpoint", "vpc_endpoint_service_name": "", "enable_high_availability": "", }, "environment": { "id": "", "environment": "" }, "gateway": { "id": "", "environment": "" } } } ``` * `vpc_endpoint_service_name`: See [Obtain AWS PrivateLink Service name](#aws-privatelink-egress-obtain-service-name-esku). * `enable_high_availability`: Set to `true` to deploy an endpoint with high availability. The default is `false`. Endpoints deployed with high availability have network interfaces deployed in multiple availability zones. An example request spec to create an endpoint: ```json { "spec": { "display_name": "prod-plap-egress-usw2", "config": { "kind": "AwsPrivateLinkEndpoint", "vpc_endpoint_service_name": "com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000", "enable_high_availability": false, }, "environment": { "id": "env-00000", }, "gateway": { "id": "gw-00000", } } } ``` ### Confluent CLI Use the [confluent network access-point private-link egress-endpoint create](https://docs.confluent.io/confluent-cli/current/command-reference/network/access-point/private-link/egress-endpoint/confluent_network_access-point_private-link_egress-endpoint_create.html) Confluent CLI command to create an Egress PrivateLink Endpoint: ```bash confluent network access-point private-link egress-endpoint create [name] [flags] ``` The following are the command-specific flags: * `--cloud`: Required. The cloud provider. Set to `aws`. * `--service`: Required. Name of an AWS VPC endpoint service that you retrieved in [Obtain AWS PrivateLink Service name](#aws-privatelink-egress-obtain-service-name-esku). * `--gateway`: Required. Gateway ID. * `--high-availability`: enable high availability for AWS egress endpoint. Endpoints deployed with high availability have network interfaces deployed in multiple availability zones. You can specify additional optional CLI flags described in the [Confluent CLI command reference](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html), such as `--environment`. The following is an example Confluent CLI command to create an AWS Egress PrivateLink Endpoint with high availability: ```bash confluent network access-point private-link egress-endpoint create \ --cloud aws \ --gateway gw-123456 \ --service com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000 \ --high-availability ``` You can specify additional optional CLI flags described in the [Confluent CLI command reference](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html), such as `--environment`. ### Terraform Use the [confluent_access_point](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_access_point) resource to create an Egress PrivateLink Endpoint. An example snippet of Terraform configuration: ```terraform resource "confluent_environment" "development" { display_name = "Development" } resource "confluent_access_point" "main" { display_name = "access_point" environment { id = confluent_environment.development.id } gateway { id = confluent_network.main.gateway[0].id } aws_egress_private_link_endpoint { vpc_endpoint_service_name = "com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000" } } ```
Your Egress PrivateLink Endpoint status will transition from “Provisioning” to “Ready” in the Confluent Cloud Console when the endpoint has been created and can be used. Once the endpoint is created, connectors provisioned against Enterprise Kafka clusters in the same gateway can leverage the Egress PrivateLink Endpoint to access the external data. Confluent Cloud exposes the VPC Endpoint ID for each of the above Egress PrivateLink Endpoints so that you can use it in various network-related policies, such as in an [S3 bucket policy](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html#bucket-policies-s3) or [Snowflake Network rule](https://docs.snowflake.com/en/user-guide/network-rules). ## Create a private DNS record in Confluent Cloud Create private DNS records for use with AWS VPC endpoints. Not all service providers set up public DNS records to be used when connecting to them with AWS PrivateLink. For situations where a system provider requires setting up private DNS records in conjunction with AWS PrivateLink, you need to create DNS records in Confluent Cloud. Before you create a DNS Record, you need to first [create an Egress PrivateLink Endpoint](#aws-privatelink-egress-create-access-point-esku) and use the Access Point ID for the DNS record. [AWS private DNS names](https://docs.aws.amazon.com/vpc/latest/privatelink/interface-endpoints.html#enable-private-dns-names) are not supported. When creating DNS records, Confluent Cloud creates a single `*` record that maps the domain name you specify to the DNS name of the VPC endpoint. For example, in setting up DNS records for Snowflake, the DNS zone configuration will look like: ```text *.xy12345.us-west-2.privatelink.snowflakecomputing.com CNAME vpce-0cb12cd2dc02130cf-8s6uwimu.vpce-svc-03bc1ff023623a033.us-east-1.vpce.amazonaws.com TTL 60 ``` ### Confluent Cloud Console 1. In the **Network Management** tab of your environment, click the **For serverless products** tab, and click the Confluent Cloud gateway you want to add the DNS record to. 2. In the **DNS** tab, click **Create DNS record**. 3. Specify the following field values: * **Egress PrivateLink Endpoint**: The Access Point ID you created in [create an Egress PrivateLink Endpoint](#aws-privatelink-egress-create-access-point-esku) * **Domain**: The domain of the private link service you wish to access. Get the domain value from the private link service provider, AWS or a third-party provider. 4. Click **Save**. ### Confluent REST API Send a request to create a DNS Record that is associated with a PrivateLink Endpoint that is associated with a gateway. **HTTP POST request** ```text POST https://api.confluent.cloud/networking/v1/dns-records ``` **Authentication** See [Authentication](https://docs.confluent.io/cloud/current/api.html/#authentication). **Request specification** ```json { "spec": { "display_name": "The name of this DNS record", "domain": "", "config": { "kind": "PrivateLinkAccessPoint", "resource_id": "" }, "environment": { "id": "", "environment": "" }, "gateway": { "id": "", "environment": "" } } } ``` * `domain`: Get the value from the private link service provider, AWS, or a third-party provider. * `gateway.id`: Issue the following API request to get the gateway ID. ```rest GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID} ``` You can find the gateway ID in the response under `spec.gateway.id`. An example request spec to create a DNS record: ```json { "spec": { "display_name": "prod-dns-record1", "domain": "example.com", "config": { "kind": "PrivateLinkAccessPoint", "resource_id": "plap-12345" }, "environment": { "id": "env-00000", }, "gateway": { "id": "gw-00000", } } } ``` ### Confluent CLI Use the [confluent network dns record create](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html) Confluent CLI command to create a DNS record: ```bash confluent network dns record create [name] [flags] ``` The following are the command-specific flags: * `--private-link-access-point`: Required. Private Link Endpoint ID. * `--gateway`: Required. Gateway ID. * `--domain`: Required. Fully qualified domain name of the external system. Get the domain value from the private link service provider, AWS or a third-party provider. You can specify additional optional CLI flags described in the [Confluent CLI command reference](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html), such as `--environment`. The following is an example Confluent CLI command to create DNS record for an endpoint: ```bash confluent network dns record create my-dns-record \ --gateway gw-123456 \ --private-link-access-point ap-123456 \ --domain xy12345.us-west-2.privatelink.snowflakecomputing.com ``` ### Terraform Use the [confluent_dns_record](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_dns_record) Confluent Terraform Provider resource to create DNS records. An example snippet of Terraform configuration: ```terraform resource "confluent_environment" "development" { display_name = "Development" } resource "confluent_dns_record" "main" { display_name = "dns_record" environment { id = confluent_environment.development.id } domain = "example.com" gateway { id = confluent_gateway.main.id } private_link_access_point { id = confluent_access_point.main.id } } ``` ## Support for AWS PrivateLink Service configuration Confluent Support can help with issues you may encounter when creating an Egress PrivateLink Endpoint for a specific service. For any service-side problems, such as described below, Confluent is not responsible for proper AWS PrivateLink Service configuration or setup: * If you need help setting up an AWS PrivateLink Service for data systems running within your environment or VPC that you want to connect to from Confluent Cloud, contact AWS for configuration help and best practices. * If you need help configuring AWS PrivateLink Services for those managed by a third-party provider or service, contact that provider for compatibility and proper setup. ## Next steps Try [Confluent Cloud on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g5ujul6iovvcy?trk=14575e70-1766-4f20-8083-0c2757a1ec75&sc_channel=el) with $1000 of free usage for 30 days, and pay as you go. No credit card is required.