Configure private networking¶

Unified Stream Manager (USM) requires a secure, private network connection between your Confluent Platform cluster and Confluent Cloud that uses AWS PrivateLink. When you use AWS PrivateLink, you can only access your Confluent Cloud resources from your private endpoints. This ensures that all metadata is transmitted over a private channel without exposure to the public internet.

You can create a new private network or use an existing private network.

Private networking resources¶

Confluent Cloud uses the following private networking resources for registering Confluent Platform clusters. These resources are regional and do not map to specific availability zones.

PrivateLink Attachment: The PrivateLink Attachment (PrivateLinkAttachment) resource represents a reservation to establish a PrivateLink connection from your virtual private cloud (VPC) to regional services in a Confluent Cloud environment. A PrivateLink Attachment belongs to an environment in the Confluent resource hierarchy. In the Confluent Cloud Console, this resource is called a gateway.
PrivateLink Attachment Connection: A PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) is a registration of VPC interface endpoints that are allowed to connect to Confluent Cloud. A PrivateLink Attachment Connection belongs to a specific PrivateLink Attachment. In the Confluent Cloud Console, this resource is called an access point.

You can establish PrivateLink connectivity by using the Confluent Cloud Console, Confluent REST API, Confluent CLI, or Terraform.

The overall process consists of the following steps:

In Confluent Cloud, create a PrivateLinkAttachment (gateway).
In AWS, create a VPC Interface Endpoint that connects to the gateway’s service.
In Confluent Cloud, create a PrivateLinkAttachmentConnection (access point) for your endpoint.
Set up your DNS resolution.

The setup wizard in the Confluent Cloud guides you through this process.

If you have an existing AWS PrivateLink connection for Confluent Cloud account and have previously configured a private network using these Use AWS PrivateLink for Serverless Products on Confluent Cloud, you can select that existing network from the wizard and proceed to the next section.

Requirements and considerations¶

You can connect to only one environment in a region from a single VPC or on-premises network. A VPC cannot have private link connections to multiple Confluent Cloud environments.
Cross-region AWS PrivateLink Attachment Connections are not supported.

AWS supported regions¶

The following list shows the supported AWS regions, grouped geographically.

Asia Pacific:
- ap-south-1 (Mumbai)
- ap-southeast-1 (Singapore)
- ap-southeast-2 (Sydney)
Canada:
- ca-central-1 (Canada Central)
Europe:
- eu-central-1 (Frankfurt)
- eu-west-1 (Ireland)
- eu-west-2 (London)
United States:
- us-east-1 (N. Virginia)
- us-east-2 (Ohio)
- us-west-2 (Oregon)

Create a network gateway in Confluent Cloud¶

To establish a private network connection, first create a network gateway in the Confluent Cloud wizard. This process generates the unique service name that you need to create the VPC endpoint in your AWS account.

On the Select network configuration page of the wizard, click Add network configuration.
In the Configure gateway panel, enter the following information:
- Gateway name: A descriptive name for your network connection.
- Cloud provider: Select aws.
- Region: Select the AWS region that matches your VPC.
Click Continue.
The next panel displays the PrivateLink Service ID. Copy this ID (for example, com.amazonaws.vpce.us-west-2.vpce-svc-...) to use it in the next part in the AWS console.
Keep this Confluent Cloud browser tab open. Confluent Cloud provisions the gateway, and its status changes to Waiting for connection.

A PrivateLink Attachment can have one of the following statuses:
- WAITING FOR CONNECTION: The PrivateLink Attachment is waiting for a connection to be created.
- READY: The connectivity is ready to be used.
- EXPIRED: A valid connection has not been provisioned in the allotted time. You must provision a new PrivateLink Attachment.

Send a request to create a PrivateLink Attachment resource:

REST request

POST https://api.confluent.cloud/networking/v1/private-link-attachments

REST request body

{
  "spec":
  {
    "display_name": "<name of this resource>",
    "cloud": "<provider type>",
    "region": "<region>",
    "environment":
    {
      "id": "<environement id>"
    }
  }
}

In the REST response, status.phase should be PROVISIONING because a VPC Endpoint Service has not yet been allocated.

Check the status of the new PrivateLink Attachment:

REST request

GET https://api.confluent.cloud/networking/v1/private-link-attachments/<platt-id>

REST response example

{
  "status":
  {
    "phase": "WAITING_FOR_CONNECTIONS",
    "error_code": "",
    "error_message": "",
    "cloud":
    {
      "kind": "AwsPrivateLinkAttachmentStatus",
      "vpc_endpoint_service_id": "com.amazonaws.vpce.us-east-1.vpce-svc-123abcc1298abc123",
    }
  }
}

The status.phase is WAITING_FOR_CONNECTIONS because no PrivateLink Attachment Connection has been associated with this PrivateLink Attachment resource.

The status.cloud object has information about the vpc_endpoint_service that you must connect your PrivateLink Attachment endpoint to.

Use the confluent network private-link attachment create Confluent CLI command to create an AWS PrivateLink attachment:

confluent network private-link attachment create <attachment-name> <flags>

The following command-specific flags are supported:

--cloud: Required. The name of the cloud provider. Set to aws.
--region: Required. AWS region where the resources to be accessed using the PrivateLink attachment.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a private link attachment:

confluent network private-link attachment create my-private-link-attachment \
  --cloud aws \
  --region us-west-2

Use the confluent_private_link_attachment resource from the Confluent Terraform Provider to create a PrivateLink Attachment.

An example snippet of Terraform configuration for PrivateLink Attachment:

resource "confluent_private_link_attachment" "main" {
  cloud = "AWS"
  region = "us-west-2"
  display_name = "staging-platt"
  environment {
    id = "env-3732nw"
  }
}

output "private_link_attachment" {
  value = confluent_private_link_attachment.main
}

See Terraform configuration example for creating a PrivateLink Attachment with ACLs using Terraform.

Create a VPC endpoint in AWS¶

In a new browser tab, log in to your AWS Management Console. Use the network gateway ID that you created in Confluent Cloud to create and configure the VPC endpoint.

In the AWS Management Console, go to the VPC dashboard.
Verify that DNS hostnames and DNS resolution are enabled for your VPC.
1. In the navigation menu, under VIRTUAL PRIVATE CLOUD, click Your VPCs.
2. Select your VPC and click Edit VPC settings.
3. Under DNS settings, ensure that Enable DNS resolution and Enable DNS hostnames are selected, and then click Save changes.
In the navigation menu under VIRTUAL PRIVATE CLOUD, click Endpoints, and then click Create endpoint.
1. In the Name tag field, enter a name for the endpoint.
2. For Service category, select PrivateLink Ready partner services.
3. In the Service name field, enter the PrivateLink Service ID of the gateway (PrivateLink Attachment) you created in Create a network gateway in Confluent Cloud, and then click Verify service.
  
  You can also use the value of the PrivateLink Service ID from your Network overview of the PrivateLink Attachment gateway in the Confluent Cloud Console.
  
  If you get an error, verify that your account is allowed to create PrivateLink connections, and try again.
4. In the VPC field, enter the ID of this VPC.
5. Under Additional settings, uncheck Enable DNS name. This option appears after you select a VPC.
6. In Subnets, select the subnets in which to create an endpoint network interface.
7. Select or create a security group for the VPC Endpoint.
  - Add three inbound rules for each of ports 80, 443, and 9092 from your desired source (your VPC CIDR). The Protocol should be TCP for all three rules.
  - Port 80 is not required, but is available as a redirect only to https/443, if desired.
Click Create endpoint.
After the endpoint is created, make a note of the VPC endpoint ID.

aws ec2 create-vpc-endpoint --vpc-id <id of this VPC> \
  --service-name <privatelink attachment service id> \
  --subnet-ids <subnet IDs for the endpoint> \
  --region <region to use> \
  --private-dns-enabled false \
  --vpc-endpoint-type Interface

From the command output, make a note of the VPC endpoint ID.

For example, using the information from status.cloud.vpc_endpoint_service_id in the PrivateLink Attachment status:

aws ec2 create-vpc-endpoint --vpc-id vpc-097799943f9fc059d \
  --service-name com.amazonaws.vpce.us-east-1.vpce-svc-123abcc1298abc123 \
  --subnet-ids subnet-7b16de0c \
  --region us-east-1 \
  --private-dns-enabled false \
  --vpc-endpoint-type Interface

Create a PrivateLink Attachment Connection¶

Return to Confluent Cloud and create a PrivateLink Attachment Connection (PrivateLinkAttachmentConnection) resource. A PrivateLink Attachment Connection represents a VPC Interface Endpoint in your VPC.

In the Confluent Cloud Console, PrivateLink Attachment connection resources are called access points.

The name of the VPC Endpoint Service is not required. Confluent checks which VPC Endpoint Service is associated with the PrivateLink Attachment that has a pending VPC Endpoint with the given ID.

In the Access points tab, click Create access point.
1. Create a VPC Interface Endpoint in AWS using the PrivateLink Service ID shown in step 3 on the sliding panel.
2. In VPC Endpoint ID from AWS, specify the VPC Interface Endpoint ID you created in the previous step.
3. In the Access point name field, enter a name for the access point.
4. Click Create access point.
  
  After the VPC Endpoint connection is accepted, the statuses for the PrivateLink Attachment and the PrivateLink Attachment connection change to READY.

Send a request to create a PrivateLink Attachment Connection resource:

REST request

POST https://api.confluent.cloud/networking/v1/private-link-attachment-connections

REST request body

{
  "spec":
  {
    "display_name": "<PrivateLinkAttachmentEndpoint name>",
    "cloud":
    {
      "kind": "AwsPrivateLinkAttachmentConnection",
      "vpc_endpoint_id": "<VPC Private Endpoint ID>",
    },
    "environment":
    {
      "id": "<Environment ID>",
    },
    "private_link_attachment":
    {
      "id": "<PrivateLinkAttachment>",
    }
  }
}

REST response example

{
  "api_version": "networking/v1",
  "kind": "PrivateLinkAttachmentConnection",
  "id": "plattc-xyzuvw",
  "status": {
    "phase": "PROVISIONING",
    "error_code": "",
    "error_message": "",
  }
}

The status.phase is PROVISIONING because a VPC Endpoint connection has not yet been accepted.

Check the status of the new PrivateLink Attachment Connection:

REST request

GET https://api.confluent.cloud/networking/v1/private-link-attachment-connections/<platt-id>

REST response example

{
  "api_version": "networking/v1",
  "kind": "PrivateLinkAttachmentConnection",
  "id": "plattc-xyzuvw",
  "status": {
    "phase": "READY",
    "error_code": "",
    "error_message": "",
    "cloud": {
      "kind": "AwsPrivateLinkAttachmentConnectionStatus",
      "phase": "READY",
      "vpc_endpoint_service": "com.amazonaws.vpce.us-east-1.vpce-svc-123abcc1298abc123",
      "vpc_endpoint_id": "vpce-bbbbbb2222222333"
    }
  }
}

status.phase is READY because a VPC Endpoint connection has been accepted.
status.cloud contains an object of the kind AwsPrivateLinkConnectionStatus.
vpc_endpoint_id shows the value vpce-bbbbbb2222222333 that you provided.

Use the confluent network private-link attachment connection create Confluent CLI command to create a PrivateLink Attachment Connection:

 confluent network private-link attachment connection create <connection-name> <flags>

This command requires the following flags:

--cloud: Required. The cloud provider. Set this to aws.
--endpoint: Required. The ID of an AWS VPC endpoint that is connected to the AWS VPC endpoint service.
--attachment: Required. The PrivateLink Attachment ID.

For a full list of optional flags, such as --environment, see the Confluent CLI command reference.

The following example shows how to create a private link attachment connection:

confluent network private-link attachment connection create aws-private-link-attachment-connection \
  --cloud aws \
  --endpoint vpce-1234567890abcdef0 \
  --attachment platt-123456

Use the confluent_private_link_attachment_connection Confluent Terraform resource to create a PrivateLink Attachment Connection.

Example Terraform configuration:

resource "confluent_private_link_attachment_connection" "aws" {
  display_name = "my_endpoint"
  environment {
    id = "env-8gv0v5"
  }
  aws {
    vpc_endpoint_id = "vpce-0ed4d51f5d6ef9b6d"
  }
  private_link_attachment {
    id = "platt-plyvyl"
  }
}

output "private_link_attachment_connection" {
  value = confluent_private_link_attachment_connection.aws
}

Set up DNS resolution¶

Set up a Route53 private hosted zone in your AWS VPC for DNS resolution.

In Confluent Cloud, verify that the status of the PrivateLink Attachment connection is READY.
In Confluent Cloud, open the newly created PrivateLink Attachment to get the DNS domain value of Confluent Cloud.

The value follows the pattern <region>.aws.private.confluent.cloud.
In the AWS Route 53 console, create a private hosted zone with the following settings:
- Domain name: Enter the Confluent Cloud DNS domain value that you copied in the previous step.
- Type: select Private hosted zone.
- VPC ID: Enter the ID of the VPC where you added the VPC endpoint.
1. Click Create hosted zone.
Create a DNS record for the hosted zone you created above.

This record is regional DNS and is used for all the target Confluent Cloud resources in the region.
1. Click Create Record from within the previously created hosted zone.
2. Specify the following values:
  - Record name: *
    
    Enter * as the subdomain name.
    
    The record name consists of the subdomain and the DNS domain name. The DNS domain name is filled in with the Confluent Cloud DNS domain value you specified when you created the Route53 Private hosted zone in the previous step.
    
    If you are creating DNS resolution for Schema Registry for a single VPC connecting to multiple Schema Registry clusters in the same region across different environments, enter the id of the Schema Registry, lsrc-xxxxx in the Record name field to connect to a specific Schema Registry.
  - Record type: Select CNAME.
  - Value: Enter the DNS name of the VPC endpoint that you created in Create a VPC endpoint in AWS.
    
    The value must be the fully qualified DNS name of the VPC endpoint. For example: vpce-012c2200321aff207-gz49hgc1.vpce-svc-00da8c4990b89436d.us-west-2.vpce.amazonaws.com. Do not specify the VPC endpoint name.
    
    To find this value, go to the Endpoint details page in the AWS console and look in the DNS names section.
  Note
  
  In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.
3. Click Create Record.
  
  A summary of the new record appears.

What’s next¶

Configure a service account

Configure private networking¶

Private networking resources¶

Requirements and considerations¶

AWS supported regions¶

Create a network gateway in Confluent Cloud¶

Create a VPC endpoint in AWS¶

Create a PrivateLink Attachment Connection¶

Set up DNS resolution¶

What’s next¶

Related content¶