Use AWS PrivateLink for Serverless Products on Confluent Cloud

Confluent Cloud, available through AWS Marketplace or directly from Confluent, supports private connectivity for serverless Confluent Cloud products, such as Enterprise Kafka clusters and Confluent Cloud for Apache Flink®, using PrivateLink Attachment. When you use PrivateLink Attachment, your Enterprise cluster or Flink resources are only accessible from customer-specific private endpoints. Public access is blocked with PrivateLink Attachment.

Note

The PrivateLink Attachment enables you to access the Flink API with private networking, allowing you to issue Flink SQL queries and retrieve results through the associated PrivateLink connection. All data movement between Flink queries and Kafka clusters configured with private networking occurs over a secure private path within Confluent Cloud.

Confluent Cloud uses the following private networking resources for serverless Confluent Cloud products. These resources are regional and do not have a mapping to specific availability zones.

PrivateLink Attachment

The PrivateLink Attachment (PrivateLinkAttachment) resource represents a reservation to establish a PrivateLink connection from your virtual private cloud (VPC) to regional services in a Confluent Cloud environment.

A PrivateLink Attachment belongs to an Environment in the Confluent resource hierarchy.

This resource is referred to as gateways in the Confluent Cloud Console.

PrivateLink Attachment Connection

A PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) is a registration of VPC interface endpoints that are allowed to connect to Confluent Cloud. A PrivateLink Attachment Connection belongs to a specific PrivateLink Attachment.

This resource is referred to as access points in the Confluent Cloud Console.

You can use Confluent Cloud UI, Confluent REST API, Confluent CLI, or Terraform to establish a PrivateLink connectivity for serverless products, such as Enterprise Kafka clusters or Confluent Cloud for Apache Flink®.

The high-level workflow is:

1. In Confluent Cloud, create a PrivateLinkAttachment.
2. In AWS, create a VPC Interface Endpoint to the PrivateLinkAttachment service.
3. In Confluent Cloud, create a PrivateLinkAttachmentConnection.
4. Set up a DNS resolution.

Requirements and considerations

• You can connect to only one environment in a region from a VPC or from an on-premises network. A single VPC cannot have private link connections to multiple Confluent Cloud environments.

  For the workaround you can configure in Flink for cross-environment queries, see Cross-environment queries.

• For the regions supported for PrivateLink Attachment on AWS, see Cloud Providers and Regions for Confluent Cloud.

• Cross-region AWS PrivateLink Attachment Connections are not supported.

• Confluent Cloud Console components, such as topic management and Flink workspaces, require additional configuration to function as they use cluster endpoints.

  For information about using Flink with AWS Private Link, see Enable Private Networking with Confluent Cloud for Apache Flink.

  To use all features of the Confluent Cloud Console with AWS PrivateLink, see Use Confluent Cloud with Private Networking.

Create a PrivateLink Attachment

When you create a PrivateLink Attachment in an environment and in a region, the PrivateLink Attachment resource provides connectivity to all Enterprise Kafka clusters within the environment for the specific cloud region.

Confluent Cloud ConsoleConfluent REST APIConfluent CLI

In the Confluent Cloud Console, the PrivateLink Attachment resources are labeled and referred to as gateways.

1. In the Confluent Cloud Console, select an environment for the PrivateLink Attachment.

2. In the Network management tab in the environment, click For serverless products.

3. Click Add network configuration.

4. On the From your VPC or VNet to Confluent Cloud pane, click + Create configuration.

5. On the Create ingress network configure gateway sliding panel, enter the following information.

   • Gateway name
   • Cloud provider
   • Region

6. Click Submit.

7. You can continue to create an access point for an Ingress Private Link Endpoint.

   Alternatively, you can create an access point at a later time by navigating to this gateway in the Network management tab.

The gateway (PrivateLink Attachment) will be provisioned and move to the Waiting for connection state.

A PrivateLink Attachment can be in one of the following states:

• WAITING FOR CONNECTION: The PrivateLink Attachment is waiting for a connection to be created.
• READY: The connectivity is ready to be used.
• EXPIRED: A valid connection has not been provisioned within the allotted time. A new PrivateLink Attachment must be provisioned.

Create a VPC Endpoint

Note

Confluent recommends using a Terraform configuration for setting up VPC endpoints. This configuration automates the manual steps described below.

In AWS, create an endpoint that is associated with the PrivateLink Service ID of the PrivateLink Attachment you created in Create a PrivateLink Attachment.

AWS Management ConsoleAWS CLI

1. In the AWS Management Console, go to the VPC dashboard.

2. Verify that DNS hostnames and DNS resolution are enabled.

   1. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Your VPCs.
   2. Select your VPC and click Edit VPC settings.
   3. Under DNS settings, verify that Enable DNS resolution and Enable DNS hostnames are selected and then click Save.

3. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Endpoints, and click Create endpoint.

4. In Name tag, specify the name of the endpoint.

5. In Service category, select PrivateLink Ready partner services.

6. In the Service name field, specify the PrivateLink Service ID of the gateway (PrivateLink Attachment) you created in Create a PrivateLink Attachment, and click Verify service.

   If you get an error, ensure that your account is allowed to create PrivateLink connections, and try again.

7. In the VPC field, specify the ID of this VPC.

8. Uncheck the Enable DNS name setting under Additional settings (only appearing after the VPC is selected).

9. In Subnets, select the subnets in which to create an endpoint network interface.

10. Select or create a security group for the VPC Endpoint.

    • Add three inbound rules for each of ports 80, 443, and 9092 from your desired source (your VPC CIDR). The Protocol should be TCP for all three rules.
    • Port 80 is not required, but is available as a redirect only to https/443, if desired.

11. Click Create endpoint.

12. Note that the VPC endpoint ID created.

Create a PrivateLink Attachment Connection

Create a PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) resource in Confluent Cloud. A PrivateLink Attachment Connection represents a VPC Interface Endpoint in your VPC.

In the Confluent Cloud Console, the PrivateLink Attachment Connection resources are labeled and referred to as access points.

The name of the VPC Endpoint Service is not required. Confluent will check which VPC Endpoint Service is associated with the PrivateLink Attachment that has a pending VPC Endpoint with the given ID.

Confluent Cloud ConsoleConfluent REST APIConfluent CLI

1. In the Network Management tab of the desired Confluent Cloud environment, click the For serverless products tab.

2. Click Create access point for the gateway to which you want to add the PrivateLink Endpoint.

   Make sure the gateway is in the correct region of the VPC Private Endpoint.

3. Create a VPC Interface Endpoint in AWS.

4. Specify the Private Endpoint ID. The Private Endpoint ID is the ID of the VPC Interface Endpoint you created in the previous step.

5. Specify the access point name.

6. Click Create access point to create the PrivateLink Endpoint.

   The PrivateLink Attachment and PrivateLink Attachment Connection should now move to the READY state once the VPC Endpoint connection is accepted.

Set up DNS resolution

Set up a Route53 Private Hosted Zone in your AWS VPC for DNS resolution.

1. In Confluent Cloud, verify that the status of the PrivateLink Attachment Connection is READY.

2. In Confluent Cloud, open the newly created PrivateLink Attachment to get the DNS domain value of Confluent Cloud.

   The value is in the <region>.aws.private.confluent.cloud pattern.

3. In the AWS Route 53 console, create a Route53 Private Hosted Zone:

   1. Specify the following values:

      • Domain name: Confluent Cloud DNS domain value from the previous step
      • Type: Private hosted zone
      • VPC ID: VPC ID where you added the VPC Endpoint

      For example:

      

   2. Click Create hosted zone to associate the Private Hosted Zone with your VPC.

4. Create a DNS record for the Hosted Zone you created above.

   This record is regional DNS and is used for all the target Confluent Cloud resources in the region.

   1. Click Create Record from within the previously created Hosted Zone.

   2. Specify the following values:

      • Record name: *

        Enter * as the subdomain name.

        The Record name consists of the subdomain and the DNS domain name. The DNS domain name is filled in with the Confluent Cloud DNS domain value you specified when you created the Route53 Private Hosted Zone in the previous step.

        

        If you are creating DNS resolution for Schema Registry for a single VPC connecting to multiple Schema Registry clusters in the same region across different environments, enter the id of the Schema Registry, lsrc-xxxxx in the Record name field to connect to a specific Schema Registry.

      • Record type: CNAME

      • Value: DNS Name of your VPC Interface Endpoint, such as vpce-<>.vpce-svc-<>.us-east-2.vpce.amazonaws.com.

   3. Click Create Record.

      You will see the summary of the new record. For example:

      

Connectivity scenarios

Below are a few connectivity scenarios that are supported for Enterprise clusters in Confluent Cloud.

Scenario: Access one environment from one VPC

The following resources are configured:

• PLATT-prod as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 VPC endpoint in VPC-1
• ProdApp as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. ProdApp attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud resolves against phz-1 and returns vpce-1.
2. Application sends traffic to vpce-1.
3. vpce-1 forwards traffic to PLATT-prod, and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.

Scenario: Access one environment from many VPC’s

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-prod environment
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 VPC endpoint in VPC-1
• PLATTC-456 for the vpce-2 VPC endpoint in VPC-2
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• ProdApp-2 as a Kafka client bootstrapped with lkc-456.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud
• phz-2 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. ProdApp-1 attempts to access lkc-123 in the env-prod environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud resolves against phz-1 and returns vpce-1.
2. ProdApp-1 sends traffic to vpce-1.
3. vpce-1 forwards traffic to PLATT-abc, and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.
4. ProdApp-2 attempts to access lkc-456 in the env-prod environment. A DNS query for lkc-456.us-west-2.aws.private.confluent.cloud resolves against phz-2 and returns vpce-2.
5. ProdApp-2 sends traffic to vpce-2.
6. vpce-2 forwards traffic to PLATT-abc, and lkc-456 can be accessed since PLATTC-456 is associated with vpce-2.

Scenario: Access one environment from an on-premises network

The following resources are configured:

• PLATT-abc as a PrivateLink Attachment for accessing Kafka clusters in the env-abc environment
• PLATTC-123 as a PrivateLink Attachment Connection for the vpce-1 endpoint in VPC-1
• On-Prem-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• ProdApp-1 as a Kafka client bootstrapped with lkc-123.us-west-2.aws.private.confluent.cloud
• phz-fwd as a DNS forwarding rule with the regional wildcard *.us-west-2.aws.private.confluent.cloud
• phz-1 as a Route53 Private Hosted Zone with the regional wildcard *.us-west-2.aws.private.confluent.cloud

The following steps are performed:

1. On-Prem-1 attempts to access lkc-123 in the env-abc environment. A DNS query for lkc-123.us-west-2.aws.private.confluent.cloud forwards to phz-1 and returns vpce-1.
2. On-Prem-1 sends traffic to vpce-1 over AWS DirectConnect.
3. vpce-1 forwards traffic to PLATT-abc and lkc-123 can be accessed since PLATTC-123 is associated with vpce-1.

Next steps

Try Confluent Cloud on AWS Marketplace with $1000 of free usage for 30 days, and pay as you go. No credit card is required.