AWS Egress Access Points for Dedicated Clusters¶

AWS PrivateLink is a networking service that allows one-way connectivity from one VPC to a service provider and is popular for its unique combination of security and simplicity.

Confluent Cloud supports outbound AWS PrivateLink connections using Egress Access Points. Egress Access Points are AWS interface VPC Endpoints, and they enable Confluent Cloud clusters to access supported AWS services and other endpoint services powered by AWS PrivateLink, such as AWS S3, a SaaS service, or a PrivateLink Service that you create yourself.

The following diagram summarizes the Egress Access Point architecture between Confluent Cloud and various potential destinations.

To set up an Egress Access Point from Confluent Cloud to an external system, such as for managed connectors:

Obtain the AWS PrivateLink Service name.
Create an Egress Access Point in Confluent Cloud.
[Optional] Create private DNS records for use with AWS VPC endpoints.

Requirements and considerations¶

Review the following requirements and considerations before you set up an Egress Access Point using AWS PrivateLink:

Egress Access Points are only available for use with Dedicated clusters.

For more information, see Use AWS PrivateLink with Confluent Cloud.
The AWS PrivateLink service must be configured to allow access from Confluent Cloud’s account or IAM role.

Due to differing granularity of the allowlist configuration across SaaS providers, it is recommended that you leverage provider-specific controls (like network rules) for securing access to the PrivateLink services against confused deputy type issues.
Egress Access Points can only be used by fully managed connectors.
AWS does not support cross-region connections with PrivateLink.

Obtain AWS PrivateLink Service name¶

To make an AWS PrivateLink connection from Confluent Cloud to an external system, you must first obtain an AWS PrivateLink Service name for Confluent to establish a connection to.

Depending on the system you wish to connect to, there may be different allowlist requirements to allow Confluent access. It is recommended that you check each system’s allowlist mechanism to verify that Confluent Cloud will be able to create an endpoint targeting that system.

For AWS services¶

Refer to the AWS documentation for a list of all AWS services that integrate with AWS PrivateLink and their associated service names.

For 3rd party services¶

Refer to the system provider’s documentation for how to obtain the AWS PrivateLink Service name as well as to determine allowlisting requirements.

The following are reference links for some of the popular system providers: :

For AWS PrivateLink Services you create¶

Refer to the AWS documentation for how to make your endpoint service available to service consumers.

Manage access to your service¶

When you stand up your own PrivateLink Service, you may want to manage its permissions to restrict who can create endpoints to that service.

Confluent Cloud uses a unique IAM Role to create VPC endpoints for each environment you create an Egress Access Point from. We highly recommend only allowlisting this principal for maintaining an optimal security posture.

To obtain the the IAM Role’s ARN:

In the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network.
Copy the IAM Principal in the Egress Access Points tab.

Create an Egress Access Point in Confluent Cloud¶

Confluent Cloud Egress Access Points are AWS interface VPC Endpoints used to connect to AWS PrivateLink Services.

In the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network you want to add the Access Point.
Click Create access point in the Egress access points tab.
Specify the following field values:
- Name: Name of the Access Point
- PrivateLink service name: The name of the PrivateLink service you retrieved in Obtain AWS PrivateLink Service name.
- Create an access point with high availability: Check the box if you wish to deploy an endpoint with High Availability.
  
  Endpoints deployed with high availability have network interfaces deployed in multiple availability zones.
Click Save.

Send a request to create an access point:

HTTP POST request

POST https://api.confluent.cloud/networking/v1/access-points

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "<The custom name for the access point>",
    "config": {
      "kind": "AwsEgressPrivateLinkEndpoint",
      "vpc_endpoint_service_name": "<The name of the PrivateLine service you wish to connect to>",
      "enable_high_availability": <Provision with high availability>,
    },
    "environment": {
      "id": "<The environment ID where the access point belongs to>",
      "environment": "string"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

enable_high_availability: Set to true to deploy an endpoint with high availability. The default is false.

Endpoints deployed with high availability have network interfaces deployed in multiple availability zones.

An example request spec to create an access point:

{
  "spec": {
    "display_name": "prod-plap-egress-usw2",
    "config": {
      "kind": "AwsPrivateLinkEndpoint",
      "vpc_endpoint_service_name": "com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000",
      "enable_high_availability": false,
    },
    "environment": {
      "id": "env-00000",
      "environment": "string"
    },
    "gateway": {
      "id": "gw-00000",
      "environment": "string"
    }
  }
}

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

Use the confluent network access-point private-link egress-endpoint create Confluent CLI command to create an Egress Access Point:

confluent network access-point private-link egress-endpoint create [name] [flags]

The following are the command-specific flags:

--cloud: Required. The cloud provider. Set to aws.
--service: Required. Name of an AWS VPC endpoint service.
--gateway: Required. Gateway ID.
--high-availability: enable high availability for AWS egress endpoint. Endpoints deployed with high availability have network interfaces deployed in multiple availability zones.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --context and --environment.

The following is an example Confluent CLI command to create an AWS PrivateLink Egress Access Point with high availability:

confluent network access-point private-link egress-endpoint create \
  --cloud aws \
  --gateway gw-123456 \
  --service com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000 \
  --high-availability

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --context and --environment.

Use the confluent_access_point resource to create an Egress Access Point.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_access_point" "main" {
  display_name = "access_point"
  environment {
    id = confluent_environment.development.id
  }
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  aws_egress_private_link_endpoint {
    vpc_endpoint_service_name = "com.amazonaws.vpce.us-west-2.vpce-svc-00000000000000000"
  }
}

Your Egress Access Point status will transition from “Provisioning” to “ready” in the Confluent Cloud Console when the endpoint has been created and can be used.

Once an access point is created, connectors provisioned against Kafka clusters in the same network can leverage the Egress Access Point to access the external data.

Confluent Cloud exposes the VPC Endpoint ID for each of the above Egress Access Points so that you can use it in various network-related policies, such as in an S3 bucket policy or Snowflake Network rule.

Create a private DNS record in Confluent Cloud¶

Create private DNS records for use with AWS VPC endpoints

Not all service providers set up public DNS records to be used when connecting to them with AWS PrivateLink. For situations where a system provider requires setting up private DNS records in conjunction with AWS PrivateLink, you need to create DNS records in Confluent Cloud.

Before you create a DNS Record, you need to first create an Egress Access Point and use the Egress Access Point ID for the DNS record.

AWS private DNS names are not supported.

When creating DNS records, Confluent Cloud creates a single * record that maps the domain name you specify to the DNS name of the VPC endpoint.

For example, in setting up DNS records for Snowflake, the DNS zone configuration will look like:

*.xy12345.us-west-2.privatelink.snowflakecomputing.com CNAME vpce-0cb12cd2dc02130cf-8s6uwimu.vpce-svc-03bc1ff023623a033.us-east-1.vpce.amazonaws.com TTL 60

Open the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network you want to add the DNS record to.
Navigate to the DNS tab.
Click Create DNS record.
Specify the following field values:
- Egress Access Point: The Egress Access Point ID you created in create an Egress Access Point
- Domain: The domain of the private link service you wish to access. Get the domain value from the private link service provider, AWS or a third-party provider.
Click Save.

Send a request to create a DNS Record that is associated with a PrivateLink Access Point that is associated with a gateway.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/dns-records

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "The name of this DNS record",
    "domain": "<The fully qualified domain name of the external system>",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "<The ID of the access point that you created>"
    },
    "environment": {
      "id": "<The environment ID where this resource belongs to>",
      "environment": "string"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

Get the spec.domain value from the private link service provider, AWS or a third-party provider/

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

An example request spec to create a DNS record:

{
  "spec": {
    "display_name": "prod-dns-record1",
    "domain": "example.com",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "plap-12345"
    },
    "environment": {
      "id": "env-00000",
      "environment": "string"
    },
    "gateway": {
      "id": "gw-00000",
      "environment": "string"
    }
  }
}

Use the confluent network dns record create Confluent CLI command to create a DNS record:

confluent network dns record create [name] [flags]

The following are the command-specific flags:

--private-link-access-point: Required. Private Link Access Point ID.
--gateway: Required. Gateway ID.
--domain: Required. Fully qualified domain name of the external system. Get the domain value from the private link service provider, AWS or a third-party provider.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --context and --environment.

The following is an example Confluent CLI command to create DNS record for an access point:

confluent network dns record create my-dns-record \
  --gateway gw-123456 \
  --private-link-access-point ap-123456 \
  --domain xy12345.us-west-2.privatelink.snowflakecomputing.com

Use the confluent_dns_record Resource resource to create DNS records.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_record" "main" {
  display_name = "dns_record"
  environment {
    id = confluent_environment.development.id
  }
  domain = "example.com"
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  private_link_access_point {
    id = confluent_access_point.main.id
  }
}

Support for AWS PrivateLink Service configuration¶

Confluent Support can help with issues you may encounter when creating an Access Point to a specific service.

For any service-side problems, such as described below, Confluent is not responsible for proper AWS PrivateLink Service configuration or setup:

If you need help setting up an AWS PrivateLink Service for data systems running within your environment or VPC that you want to connect to from Confluent Cloud, contact AWS for configuration help and best practices.
If you need help configuring AWS PrivateLink Services for those managed by a third-party provider or service, contact that provider for compatibility and proper setup.