VPC Peering in Confluent Cloud

You can use virtual private clouds (VPC) with Confluent Cloud Enterprise and Confluent Cloud Dedicated Clusters to maximize the security of your cloud infrastructure. All communication in Confluent Cloud is encrypted, but a VPC can decrease the available surface area for potential attackers.

You can create VPCs with private IP CIDR (Classless Inter-Domain Routing) blocks and run your instances inside the VPCs on these private networks. The VPC can include applications and all of your cloud services. You can then peer your VPCs with Confluent VPCs so that you can access Confluent Cloud within the linked private networks.

Important

  • If you use VPC peering, your clusters will not have public endpoints and you can only access them from peered VPCs.
  • After a cluster has been provisioned with VPC peering, you cannot change the cluster to have public endpoints.

Supported Features

Confluent Cloud Enterprise and Confluent Cloud Dedicated Clusters support the following VPC peering features:

  • Connect up to five VPCs to a single Confluent Cloud VPC.
  • Confluent Cloud VPC can contain any number of clusters.

Limitations

  • For Google Cloud Platform, the customer VPC and Confluent Cloud VPC must be in the same region.
  • Transitive VPC and VNet peering are not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud Enterprise, applications running in Network A will not be able to access Confluent Cloud Enterprise. You can use shared GCP and Amazon Web Services VPCs to enable transitive VPC connectivity. For more information, see AWS Working with Shared VPCs and GCP Shared VPC overview. To achieve transitivity, you can also link an AWS Transit Gateway to a Confluent Cloud AWS cluster.
  • You cannot directly connect from an on-premises datacenter to Confluent Cloud. To do this, you must first be routed to a shared services VPC or VNet that you own, and that is peered to Confluent Cloud. If you are interested in this environment for Confluent Cloud, contact your Confluent sales representative.
  • You can co-locate multiple Confluent Cloud dedicated clusters in the same VPC. However, this is limited by the expected number and size of these clusters given the finite number of IPs available. If you require multiple clusters in the same VPC, contact your Confluent account team to further understand these limitations.

Getting Started

To implement VPC peering in your environment, order Confluent Cloud Enterprise or a Confluent Cloud Dedicated Cluster with VPC peering enabled. You will be asked to provide the following information to your Confluent representative.

Confluent Cloud on AWS

Provide the following information to your Confluent representative.

  • The AWS account ID associated with the VPC that you are peering to Confluent Cloud.
  • The VPC ID that you are peering with Confluent Cloud.
  • The AWS region of the VPC that you are peering with Confluent Cloud.
  • The CIDR block of the VPC that you are peering with Confluent Cloud. This is used by Confluent Cloud to route traffic back to your network. Must be a private range.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Amazon Web Services (AWS), because the Confluent and AWS routes are shared.

For more information about VPC peering with AWS, see What Is Amazon VPC?.

Confluent Cloud on AWS Using Transit Gateway

You can enable Confluent Cloud for use on the AWS Transit Gateway. To enable, you must provide the following information to your Confluent representative:

  1. The full AWS Resource Name (ARN) for the AWS Resource Access Manager (RAM) Share-ID of the Transit Gateways that you want Confluent Cloud attached to.
    • Create a transit gateway. The transit gateway must have Default route table association, Default route table propagation, and Auto accept shared attachments enabled.
    • Create a shared transit gateway using AWS Resource Access Manager (RAM) share. The RAM share ID must be shared with the Confluent AWS Account: 050879227952.
  2. The VPC CIDR block for Confluent Cloud to use.
    • It must be a private /16 CIDR.
    • It must not match or overlap with the CIDR used by your VPC.
    • 10.255.0.0/16 and 198.18.0.0/15 are reserved by Confluent.
    • You might need to increase your route quota when you use VPC peering in Amazon Web Services, because the Confluent and AWS routes are shared.

After the Confluent Cloud clusters have been provisioned:

  1. Confluent accepts RAM share and attaches the Confluent Cloud VPC to the AWS Transit Gateway.
  2. You can set up the desired routing in the AWS Transit Gateway to route traffic to Confluent Cloud.

Confluent Cloud on Google Cloud Platform

Provide the following information to your Confluent representative.

  • The project ID associated with the VPC that you are peering to Confluent Cloud.
  • The network name of the VPC that you are peering with Confluent Cloud.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Google Cloud Platform (GCP) because the Confluent Cloud and GCP routes are shared.

For more information about VPC peering with GCP, see VPC Network Peering.

Confluent Cloud on Microsoft Azure

Provide the following information to your Confluent representative.

  • The Tenant ID associated with the VNet that you are peering to Confluent Cloud. Retrieve your Tenant/Directory ID in portal.azure.com under Azure Active Directory -> Properties.
  • The VNet ID of the VNet that you are peering with Confluent Cloud. Retrieve your VNet Resource ID in portal.azure.com under Virtual Networks -> Target VNet -> Properties.
  • The virtual network CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with any of the CIDRs used by your virtual network.

Your Confluent representative will provide a setup script you must run before Confluent can peer to your VNet.

For more information about VNet peering with Azure, see Virtual Network Peering.

Configuring Access to the UI Dashboard

When VPC peering is enabled, the Confluent Cloud UI dashboard components like topic management and KSQL are set up with private endpoints that are not publicly reachable. You must configure internal access to these components.

Important

You might have to configure multiple endpoints for topic management, consumer lag, and KSQL.

Example Topology

In this this example topology, the customer network is running outside of a cloud VPC using HAProxy to connect to Confluent Cloud.

../_images/cloud-vpc-topology.png

Example topology

HAProxy Example Configuration

Here is an example HAProxy configuration that configures access to topic management.

  1. Create an HAProxy configuration file (/etc/haproxy/haproxy.cfg) with these values:

    • Bind port *:443 for front end listening
    • Configure front end to back end mapping
    • Use the IP address 10.10.1.115 for the front end of HAProxy
    global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
    
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
    
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
    
    defaults
        log global
        mode    http
        option  httplog
        option  dontlognull
            timeout connect 5000
            timeout client  50000
            timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
    
    frontend confluent-cloud-topic-mgmt
            mode tcp
            bind 10.10.1.115:443
        log global
            default_backend confluent-cloud-topic-mgmt
    
    backend confluent-cloud-topic-mgmt
            mode tcp
            server topic1 pkac-4nvdd.eu-west-1.aws.confluent.cloud:443 check
    
  2. Configure the DNS entry to point to the HAProxy front end for topic management endpoint. Here is an example that uses Amazon Route 53. Note that pkac-**** is mapped to the front end of HAProxy:

    $ aws route53 list-resource-record-sets --hosted-zone-id Z03406652PN3OVDPNQJP0
    {
        "ResourceRecordSets": [
            {
                "TTL": 172800,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk."
                    },
                    {
                        "Value": "ns-0.awsdns-00.com."
                    },
                    {
                        "Value": "ns-1024.awsdns-00.org."
                    },
                    {
                        "Value": "ns-512.awsdns-00.net."
                    }
                ],
                "Type": "NS",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 900,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"
                    }
                ],
                "Type": "SOA",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 300,
                "ResourceRecords": [
                    {
                        "Value": "10.10.1.115"
                    }
                ],
                "Type": "A",
                "Name": "pkac-4nvdd.eu-west-1.aws.confluent.cloud."
            }
        ]
    }
    

Tip

  • For more information on HAProxy hardware and operating system requirements, see HAProxy operating system and hardware requirements.

  • You can also edit the /etc/hosts file in Linux or macOS, for example:

    10.10.1.115 pkac-4nvdd.eu-west-1.aws.confluent.cloud
    

Using Confluent Cloud Schema Registry in a VPC Peered Environment

If you have VPC peered environment and you want to use Confluent Cloud Schema Registry, you must open outbound calls (egress) to a public Schema Registry endpoint. This is because Confluent Cloud Schema Registry is a multi-tenant Schema Registry.

Prerequisites
Confluent Cloud Schema Registry is enabled and configured. For more information about supported features, see Confluent Cloud Schema Registry.
  1. Click Schemas on the left-side panel. On step 2 you should see the Schema Registry endpoint. For example https//confluent.us-east-2.aws.confluent.cloud.

    ../_images/cloud-sr-view.png
  2. Open outbound calls to the Confluent Cloud Schema Registry endpoint. Follow the instructions based on your cloud provider.

    Azure VPC

    Configure outbound call access for Azure networking components.

    AWS VPC

    Configure outbound call access for these AWS VPC networking components:

    GCP VPC

    Configure outbound call access for GCP networking components.

  3. Optional: Verify that your Schema Registry credentials are properly configured, where Schema Registry API key (<schema-registry-api-key>), API secret (<schema-registry-api-secret>), and endpoint (<schema-registry-url>) are specified.

    Run this command to authenticate with the cluster and list the topics registered in your schema.

    curl -u <schema-registry-api-key>:<schema-registry-api-secret> \
    <schema-registry-url>/subjects
    

    If no subjects are created, your output will be empty ([]). If you have subjects, your output should resemble:

    ["test2-value"]
    

    Here is an example command:

    curl -u GFS3CFEX5ELQTB6O:2Nc69uGg+oKOdajqGXfZzuUoN1mBZarD0ZLiE5I9sVIQQhuNti1NSBnzslpgNG5J \
    https://psrc-lq2dm.us-east-2.aws.confluent.cloud/subjects