AWS PrivateLink for Enterprise Clusters

Confluent Cloud supports private connectivity for Enterprise Kafka clusters using PrivateLink Attachment. When you use PrivateLink Attachment, your Enterprise cluster is only accessible from tenant-specific private endpoints. Public access is blocked with PrivateLink Attachment.

Confluent Cloud uses the following private networking resources for Enterprise clusters. These resources are regional and do not have a mapping to specific availability zones.

PrivateLink Attachment

The PrivateLink Attachment (PrivateLinkAttachment) resource represents a reservation to establish a PrivateLink connection from your VPC to regional services in a Confluent Cloud environment.

A PrivateLink Attachment belongs to an Environment in the Confluent resource hierarchy.

PrivateLink Attachment Connection

A PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) is a registration of VPC interface endpoints that are allowed to connect to Confluent Cloud. A PrivateLink Attachment Connection belongs to a specific PrivateLink Attachment.

You can use Confluent Cloud UI, Confluent REST API, or Terraform to establish a PrivateLink connectivity for an Enterprise cluster.

The high-level workflow is:

1. In Confluent Cloud, create a PrivateLinkAttachment.

2. In AWS, create a VPC Endpoint to the PrivateLinkAttachment service.

3. In Confluent Cloud, create a PrivateLinkAttachmentConnection.

4. Set up a DNS resolution.

5. Create a Kafka client in your VPC using the bootstrap endpoint of your Enterprise Kafka cluster. This Kafka client can live in Virtual Machine or similar compute infrastructure.

6. Validate produce/consume traffic is successful.

   Once you create a PrivateLinkAttachment resource and establish a PrivateLink, you can securely send and receive traffic through the PrivateLink between your VPC and Confluent Cloud.

Requirements and considerations

• You can connect to only one environment from a single VPC or from an on-prem network.
• The following regions are supported:
  • us-east-1
  • us-east-2
  • us-west-2
  • eu-west-1
  • eu-central-1
  • ap-south-1
  • ap-southeast-2
  • ap-southeast-1
  • af-south-1
• Fully-managed Confluent Cloud connectors can connect to sources or sinks using public IP addresses. Sources or sinks in the customer network with private IP addresses are not supported. An exception to this is the Amazon S3 Sink connector which can connect to an Amazon S3 bucket from a private network.
• Confluent Cloud Console components, like topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with AWS PrivateLink, see Use Confluent Cloud with Private Networking.

Create a PrivateLink Attachment

When you create a PrivateLink Attachment in an environment and in a region, the PrivateLink Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.

Confluent Cloud ConsoleConfluent REST API

1. In the Confluent Cloud Console, select an environment for the PrivateLink Attachment.
2. In the Network management tab in the environment, click Add network configuration.
3. Select For Enterprise Clusters and click Continue.
4. Select AWS, select Region, and click Continue.
5. Provide the PrivateLink Attachment name in the Network name field and click Add network configuration.

The PrivateLink Attachment will be provisioned and move to the Waiting for connection state.

A PrivateLink Attachment can be in one of the following states:

• WAITING FOR CONNECTION: The PrivateLink Attachment is waiting for a connection to be created.
• READY: AWS PrivateLink connectivity is ready to be used.
• EXPIRED: A valid connection has not been provisioned within the allotted time. A new PrivateLink Attachment must be provisioned.

Create a VPC Endpoint

In AWS, create an endpoint that is associated with the PrivateLink Service ID of the PrivateLink Attachment you created in Create a PrivateLink Attachment.

AWS Management ConsoleAWS CLI

1. In the AWS Management Console, go to the the VPC dashboard.
2. Verify that DNS hostnames and DNS resolution are enabled.
   1. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Your VPCs.
   2. Select your VPC and click Edit VPC settings.
   3. Under DNS settings, verify that Enable DNS resolution and Enable DNS hostnames are selected and then click Save.
3. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Endpoints, and click Create endpoint.
4. In Name tag, specify the name of the endpoint.
5. In Service category, select Other endpoint services.
6. In the Service name field, specify the PrivateLink Service ID of the PrivateLink Attachment you created in Create a PrivateLink Attachment.
7. In the VPC field, specify the ID of this VPC.
8. Uncheck the Enable DNS name setting under Additional settings (only appearing after the VPC is selected).
9. In Subnets, select the subnets in which to create an endpoint network interface.
10. Select or create a security group for the VPC Endpoint.
    • Add three inbound rules for each of ports 80, 443, and 9092 from your desired source (your VPC CIDR). The Protocol should be TCP for all three rules.
    • Port 80 is not required, but is available as a redirect only to https/443, if desired.
11. Click Create endpoint.
12. Note that the VPC endpoint ID created.

Create a PrivateLink Attachment Connection

Create a PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) resource in Confluent Cloud. A PrivateLink Attachment Connection represents a VPC Interface Endpoint in your VPC.

The name of the VPC Endpoint Service is not required. Confluent will check which VPC Endpoint Service is associated with the PrivateLink Attachment that has a pending VPC Endpoint with the given ID.

Confluent Cloud ConsoleConfluent REST API

1. In Confluent Cloud Console, in the Network management tab in the environment, click the PrivateLink Attachment you want to add a connection to.

   Make sure the PrivateLink Attachment is in the correct region of the VPC Private Endpoint.

2. Click + Add connection.

3. Specify the connection name and the Private Endpoint ID.

   The Private Endpoint ID is the id of the VPC Endpoint that was created in Create a VPC Endpoint.

4. Click Finish.

The PrivateLink Attachment and PrivateLink Attachment Connection should now move to the READY state once the VPC Endpoint connection is accepted.

Set up DNS resolution

Set up a Route53 Private Hosted Zone in your AWS VPC for DNS resolution.

1. In Confluent Cloud, verify that the status of the PrivateLink Attachment Connection is READY.

2. In Confluent Cloud, browse to Cluster Overview > Networking of the Enterprise cluster, and click the PrivateLink Attachment to get the DNS domain value of Confluent Cloud.

   The value is in the <region>.aws.private.confluent.cloud pattern.

3. In the AWS Route 53 console, create a Route53 Private Hosted Zone:

   1. Specify the following values:
      • Domain name: Confluent Cloud DNS domain value from the previous step
      • Type: Private hosted zone
      • VPC ID: VPC ID where you added the VPC Endpoint
   2. Click Create hosted zone to associate the Private Hosted Zone with your VPC.

4. Create a DNS record for the Hosted Zone you created above.

   1. Click Create Record from within the previously created Hosted Zone.
   2. Specify the following values:
      • Record name: *
      • Record type: CNAME
      • Value: DNS Name of your VPC Interface Endpoint
   3. Click Create Record.

Connectivity scenarios

Below are a few connectivity scenarios that are supported for Enterprise clusters in Confluent Cloud.

Scenario: Access one environment from one VPC

The following resources are configured:

• PLATT-prod as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment.
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 VPC endpoint in VPC-1
• ProdApp as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. ProdApp attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud resolves against phz-1 and returns vpce-1.
2. Application sends traffic to vpce-1.
3. vpce-1 forwards traffic to PLATT-prod, and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.

Scenario: Access one environment from many VPC’s

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 VPC endpoint in VPC-1
• PLATTC-456 for the vpce-2 VPC endpoint in VPC-2
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• ProdApp-2 as a Kafka client bootstrapped with lkc-456.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud
• phz-2 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. ProdApp-1 attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud resolves against phz-1 and returns vpce-1.
2. ProdApp-1 sends traffic to vpce-1.
3. vpce-1 forwards traffic to PLATT-abc, and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.
4. ProdApp-2 attempts to access lkc-456 in the env-prod environment. A DNS query for lkc-456.us-west-2.aws.private.confluent.cloud resolves against phz-2 and returns vpce-2.
5. ProdApp-2 sends traffic to vpce-2.
6. vpce-2 forwards traffic to PLATT-abc, and lkc-456 can be accessed since PLATTC-456 is associated with vpce-2.

Scenario: Access one environment from an on-premise network

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-abc environment
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 endpoint in VPC-1
• On-Prem-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• phz-fwd as a DNS forwarding rule with the regional wildcard *.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. On-Prem-1 attempts to access lkc-123 in the env-abc environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud forwards to phz-1 and returns vpce-1.
2. On-Prem-1 sends traffic to vpce-1 over AWS DirectConnect.
3. vpce-1 forwards traffic to PLATT-abc and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.