VPC Peering in Confluent Cloud

You can use virtual private clouds (VPC) with Confluent Cloud Enterprise and Confluent Cloud Dedicated Clusters to maximize the security of your cloud infrastructure. All communication in Confluent Cloud is encrypted, but a VPC can decrease the available surface area for potential attackers.

You can create VPCs with private IP CIDR (Classless Inter-Domain Routing) blocks and run your instances inside the VPCs on these private networks. The VPC can include applications and all of your cloud services. You can then peer your VPCs with Confluent VPCs so that you can access Confluent Cloud within the linked private networks.

Important

  • If you use VPC peering, your clusters will not have public endpoints and you can only access them from peered VPCs.
  • After a cluster has been provisioned with VPC peering, you cannot change the cluster to have public endpoints.

Supported Features

Confluent Cloud Enterprise and Confluent Cloud Dedicated Clusters support the following VPC peering features:

  • Connect up to five VPCs to a single Confluent Cloud VPC.
  • Confluent Cloud VPC can contain any number of clusters.

Limitations

  • All clusters in a Confluent Cloud VPC must have the same availability configuration (that is, all single AZ or multi-AZ clusters).
  • For Google Cloud Platform, customer and Confluent Cloud VPCs must be in the same region.
  • You cannot directly connect from an on-premises datacenter to Confluent Cloud. To do so, you must first land in a shared services VPC that you own that is peered to Confluent Cloud.
  • Transitive VPC peering is not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud Enterprise, applications running in Network A will not be able to access Confluent Cloud Enterprise. You can link an AWS Transit Gateway to a Confluent Cloud cluster to achieve transitivity, or you can run a proxy to route application traffic to Confluent Cloud.

For more information about how to configure your VPCs to achieve connectivity across VPCs and on-premise networks, see:

Getting Started

To implement VPC peering in your environment, order Confluent Cloud Enterprise or a Confluent Cloud Dedicated Cluster with VPC peering enabled. You will be asked to provide the following information to your Confluent representative.

Confluent Cloud on AWS

Provide the following information to your Confluent representative.

  • The AWS account ID associated with the VPC that you are peering to Confluent Cloud.
  • The VPC ID that you are peering with Confluent Cloud.
  • The AWS region of the VPC that you are peering with Confluent Cloud.
  • The CIDR block of the VPC that you are peering with Confluent Cloud. This is used by Confluent Cloud to route traffic back to your network. Must be a private range.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Amazon Web Services (AWS), because the Confluent and AWS routes are shared.

For more information about VPC peering with AWS, see What Is Amazon VPC?.

Confluent Cloud on Google Cloud Platform

Provide the following information to your Confluent representative.

  • The project ID associated with the VPC that you are peering to Confluent Cloud.
  • The network name of the VPC that you are peering with Confluent Cloud.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Google Cloud Platform (GCP) because the Confluent Cloud and GCP routes are shared.

For more information about VPC peering with GCP, see VPC Network Peering.

Confluent Cloud on Microsoft Azure

Provide the following information to your Confluent representative.

  • The Tenant ID associated with the VNet that you are peering to Confluent Cloud. Retrieve your Tenant/Directory ID in portal.azure.com under Azure Active Directory -> Properties.
  • The VNet ID of the VNet that you are peering with Confluent Cloud. Retrieve your VNet Resource ID in portal.azure.com under Virtual Networks -> Target VNet -> Properties.
  • The virtual network CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with any of the CIDRs used by your virtual network.

Your Confluent representative will provide a setup script you must run before Confluent can peer to your VNet.

For more information about VNet peering with Azure, see Virtual Network Peering.

Configuring Access to the UI Dashboard

When VPC peering is enabled, the Confluent Cloud UI dashboard components like topic management and KSQL are set up with private endpoints that are not publicly reachable. You must configure internal access to these components.

Important

You might have to configure multiple endpoints for topic management, consumer lag, and KSQL.

Example Topology

In this this example topology, the customer network is running outside of a cloud VPC using HAProxy to connect to Confluent Cloud.

../_images/cloud-vpc-topology.png

Example topology

HAProxy Example Configuration

Here is an example HAProxy configuration that configures access to topic management.

  1. Create an HAProxy configuration file (/etc/haproxy/haproxy.cfg) with these values:

    • Bind port *:443 for front end listening
    • Configure SNI routing of topic management endpoint pkac-***
    • Configure front end to back end mapping
    • Use the IP address 10.10.1.115 for the front end of HAProxy
    global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
    
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
    
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
    
    defaults
        log global
        mode    http
        option  httplog
        option  dontlognull
            timeout connect 5000
            timeout client  50000
            timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
    
    frontend kafka
            mode tcp
            bind *:9092
            bind *:443
            log global
    
            tcp-request inspect-delay 5s
            tcp-request content accept if { req.ssl_hello_type 1 }
    
            acl is_kafka0 req.ssl_sni -i b0-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_kafka1 req.ssl_sni -i b1-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_kafka2 req.ssl_sni -i b2-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_kafka3 req.ssl_sni -i b3-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_kafka4 req.ssl_sni -i b4-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_kafka5 req.ssl_sni -i b5-pkc-438qj.eu-west-1.aws.confluent.cloud
            acl is_topic1 req.ssl_sni -i pkac-4nvdd.eu-west-1.aws.confluent.cloud
            use_backend kafka0 if is_kafka0
            use_backend kafka1 if is_kafka1
            use_backend kafka2 if is_kafka2
            use_backend kafka3 if is_kafka3
            use_backend kafka4 if is_kafka4
            use_backend kafka5 if is_kafka5
            use_backend topic1 if is_topic1
    
    backend kafka0
        mode tcp
        server kafka0 b0-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    backend kafka1
        mode tcp
        server kafka1 b1-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    backend kafka2
        mode tcp
        server kafka2 b2-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    backend kafka3
            mode tcp
            server kafka3 b3-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    backend kafka4
            mode tcp
            server kafka4 b4-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    backend kafka5
            mode tcp
            server kafka5 b5-pkc-438qj.eu-west-1.aws.confluent.cloud:9092 check
    
    
    backend topic1
            mode tcp
            server topic1 pkac-4nvdd.eu-west-1.aws.confluent.cloud:443 check
    
  2. Configure the DNS entry to point to the HAProxy front end for topic management endpoint. Here is an example that uses Amazon Route 53. Note that pkac-**** is mapped to the front end of HAProxy:

    $ aws route53 list-resource-record-sets --hosted-zone-id Z03406652PN3OVDPNQJP0
    {
        "ResourceRecordSets": [
            {
                "TTL": 172800,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk."
                    },
                    {
                        "Value": "ns-0.awsdns-00.com."
                    },
                    {
                        "Value": "ns-1024.awsdns-00.org."
                    },
                    {
                        "Value": "ns-512.awsdns-00.net."
                    }
                ],
                "Type": "NS",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 900,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"
                    }
                ],
                "Type": "SOA",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 300,
                "ResourceRecords": [
                    {
                        "Value": "10.10.1.115"
                    }
                ],
                "Type": "A",
                "Name": "pkac-4nvdd.eu-west-1.aws.confluent.cloud."
            }
        ]
    }
    

Tip

For more information on HAProxy hardware and operating system requirements, see HAProxy operating system and hardware requirements.

Using Confluent Cloud Schema Registry in a VPC Peered Environment

If you have VPC peered environment and you want to use Confluent Cloud Schema Registry, you must open outbound calls (egress) to a public Schema Registry endpoint. This is because Confluent Cloud Schema Registry is a multi-tenant Schema Registry.

Prerequisites
Confluent Cloud Schema Registry is enabled and configured. Confluent Cloud Schema Registry is currently available as a preview. For more information, see Confluent Cloud Schema Registry.
  1. Click Schemas on the left-side panel. On step 2 you should see the Schema Registry endpoint. For example https//confluent.us-east-2.aws.confluent.cloud.

    ../_images/cloud-sr-view.png
  2. Open outbound calls to the Confluent Cloud Schema Registry endpoint. Follow the instructions based on your cloud provider.

    AWS VPC

    Configure outbound call access for these AWS VPC networking components:

    GCP VPC

    Configure outbound call access for GCP networking components.

  3. Optional: Verify that your Schema Registry credentials are properly configured, where Schema Registry API key (<schema-registry-api-key>), API secret (<schema-registry-api-secret>), and endpoint (<schema-registry-url>) are specified.

    Run this command to authenticate with the cluster and list the topics registered in your schema.

    curl -u <schema-registry-api-key>:<schema-registry-api-secret> \
    <schema-registry-url>/subjects
    

    If no subjects are created, your output will be empty ([]). If you have subjects, your output should resemble:

    ["test2-value"]
    

    Here is an example command:

    curl -u GFS3CFEX5ELQTB6O:2Nc69uGg+oKOdajqGXfZzuUoN1mBZarD0ZLiE5I9sVIQQhuNti1NSBnzslpgNG5J \
    https://psrc-lq2dm.us-east-2.aws.confluent.cloud/subjects