Networking in Confluent Cloud

Confluent Cloud Dedicated clusters can be accessed via secure internet endpoints, VPC/VNet peering, or AWS Transit Gateway. All Standard clusters are accessible via secure internet endpoints. All connections to Confluent Cloud are encrypted with TLS and require authentication via API keys, regardless of network configuration.

Important

  • If you use VPC peering, your cluster will not have public endpoints and you can only access it from a peered VPC.
  • If you use AWS Transit Gateway, your cluster will not have public endpoints and you can only access it from the linked AWS Transit Gateway network.
  • After a cluster has been provisioned with VPC peering or AWS Transit Gateway, you cannot change it to have public endpoints.
  • After a cluster has been provisioned with secure internet endpoints, you cannot change it to use VPC/VNet peering or AWS Transit Gateway.

Using VPC peering or AWS Transit Gateway is a tradeoff. Your cluster cannot be accessed from the public internet, which eliminates some potential security threats, but it also requires you to manage the peered or linked networks to ensure all your client applications and developers have the access they need to Confluent Cloud.

Supported Features

Confluent Cloud Dedicated Clusters support the following networking features:

  • Peer up to five VPCs to a single Confluent Cloud VPC.
  • Confluent Cloud VPC can contain any number of clusters.
  • AWS only: connect one AWS Transit Gateway

Limitations

  • Transitive VPC and VNet peering are not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud Enterprise, applications running in Network A will not be able to access Confluent Cloud Enterprise. You can use shared GCP and Amazon Web Services VPCs to enable transitive VPC connectivity. For more information, see AWS Working with Shared VPCs and GCP Shared VPC overview. To achieve transitivity, you can also link an AWS Transit Gateway to a Confluent Cloud AWS cluster.
  • You cannot directly connect from an on-premises datacenter to Confluent Cloud. To do this, you must first route to a shared services VPC or VNet that you own, peer that to Confluent Cloud, and proxy traffic. If you are interested in this environment for Confluent Cloud, contact your Confluent sales representative.
  • You can co-locate multiple Confluent Cloud dedicated clusters in the same VPC. However, this is limited by the expected number and size of these clusters given the finite number of IPs available. If you require multiple clusters in the same VPC, contact your Confluent account team to further understand these limitations.

Getting Started

To implement VPC peering in your environment, order Confluent Cloud Enterprise or a Confluent Cloud Dedicated Cluster with VPC peering enabled. You will be asked to provide the following information to your Confluent representative.

Confluent Cloud on AWS

Provide the following information to your Confluent representative.

  • The AWS account ID associated with the VPC that you are peering to Confluent Cloud.
  • The VPC ID that you are peering with Confluent Cloud.
  • The AWS region of the VPC that you are peering with Confluent Cloud.
  • The CIDR block of the VPC that you are peering with Confluent Cloud. This is used by Confluent Cloud to route traffic back to your network. Must be a private range.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Amazon Web Services (AWS), because the Confluent and AWS routes are shared.

For more information about VPC peering with AWS, see What Is Amazon VPC?.

Confluent Cloud on AWS Using Transit Gateway

You can enable Confluent Cloud for use on the AWS Transit Gateway. To enable, you must provide the following information to your Confluent representative:

  1. The full AWS Resource Name (ARN) for the AWS Resource Access Manager (RAM) Share-ID of the Transit Gateways that you want Confluent Cloud attached to.
  2. The VPC CIDR block for Confluent Cloud to use.
    • It must be a private /16 CIDR.
    • It must not match or overlap with the CIDR used by your VPC.
    • 10.255.0.0/16 and 198.18.0.0/15 are reserved by Confluent.
    • You might need to increase your route quota when you use VPC peering in Amazon Web Services, because the Confluent and AWS routes are shared.

After the Confluent Cloud clusters have been provisioned:

  1. Confluent accepts RAM share and attaches the Confluent Cloud VPC to the AWS Transit Gateway. Confluent installs RFC 1918 routes in our VPC to return to the TGW.
  2. You can set up the desired routing in the AWS Transit Gateway to route traffic to Confluent Cloud. Any routes you install on the TGW outside of the RFC 1918 range are not supported and will not work with Confluent Cloud.

Confluent Cloud on Google Cloud Platform

Provide the following information to your Confluent representative.

  • The project ID associated with the VPC that you are peering to Confluent Cloud.
  • The network name of the VPC that you are peering with Confluent Cloud. Your VPC and the Confluent Cloud VPC must be in the same region.
  • The VPC CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with the CIDR used by your VPC.
    • You might need to increase your route quota when you use VPC peering in Google Cloud Platform (GCP) because the Confluent Cloud and GCP routes are shared.

For more information about VPC peering with GCP, see VPC Network Peering.

Confluent Cloud on Microsoft Azure

Provide the following information to your Confluent representative.

  • The Tenant ID associated with the VNet that you are peering to Confluent Cloud. Retrieve your Tenant/Directory ID in portal.azure.com under Azure Active Directory -> Properties.
  • The VNet ID of the VNet that you are peering with Confluent Cloud. Retrieve your VNet Resource ID in portal.azure.com under Virtual Networks -> Target VNet -> Properties.
  • The virtual network CIDR block for Confluent Cloud to use. It must be a private /16 CIDR, per https://tools.ietf.org/html/rfc1918. It must not match or overlap with any of the CIDRs used by your virtual network.

Your Confluent representative will provide a setup script you must run before Confluent can peer to your VNet.

For more information about VNet peering with Azure, see Virtual Network Peering.

Configuring Access to the UI Dashboard

When VPC peering is enabled, the Confluent Cloud UI dashboard components like topic management and KSQL are set up with private endpoints that are not publicly reachable. You must configure internal access to these components.

Important

You might have to configure multiple endpoints for topic management, consumer lag, and KSQL.

Example Topology

In this this example topology, the customer network is running outside of a cloud VPC using HAProxy to connect to Confluent Cloud.

../_images/cloud-vpc-topology.png

Example topology

HAProxy Example Configuration

Here is an example HAProxy configuration that configures access to topic management.

  1. Create an HAProxy configuration file (/etc/haproxy/haproxy.cfg) with these values:

    • Bind port *:443 for front end listening
    • Configure front end to back end mapping
    • Use the IP address 10.10.1.115 for the front end of HAProxy
    global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
    
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
    
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
    
    defaults
        log global
        mode    http
        option  httplog
        option  dontlognull
            timeout connect 5000
            timeout client  50000
            timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
    
    frontend confluent-cloud-topic-mgmt
            mode tcp
            bind 10.10.1.115:443
        log global
            default_backend confluent-cloud-topic-mgmt
    
    backend confluent-cloud-topic-mgmt
            mode tcp
            server topic1 pkac-4nvdd.eu-west-1.aws.confluent.cloud:443 check
    
  2. Configure the DNS entry to point to the HAProxy front end for topic management endpoint. Here is an example that uses Amazon Route 53. Note that pkac-**** is mapped to the front end of HAProxy:

    $ aws route53 list-resource-record-sets --hosted-zone-id Z03406652PN3OVDPNQJP0
    {
        "ResourceRecordSets": [
            {
                "TTL": 172800,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk."
                    },
                    {
                        "Value": "ns-0.awsdns-00.com."
                    },
                    {
                        "Value": "ns-1024.awsdns-00.org."
                    },
                    {
                        "Value": "ns-512.awsdns-00.net."
                    }
                ],
                "Type": "NS",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 900,
                "ResourceRecords": [
                    {
                        "Value": "ns-1536.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"
                    }
                ],
                "Type": "SOA",
                "Name": "eu-west-1.aws.confluent.cloud."
            },
            {
                "TTL": 300,
                "ResourceRecords": [
                    {
                        "Value": "10.10.1.115"
                    }
                ],
                "Type": "A",
                "Name": "pkac-4nvdd.eu-west-1.aws.confluent.cloud."
            }
        ]
    }
    

Tip

  • For more information on HAProxy hardware and operating system requirements, see HAProxy operating system and hardware requirements.

  • You can also edit the /etc/hosts file in Linux or macOS, for example:

    10.10.1.115 pkac-4nvdd.eu-west-1.aws.confluent.cloud
    

Using Confluent Cloud Schema Registry in a VPC Peered Environment

If you have VPC peered environment and you want to use Confluent Cloud Schema Registry, you must open outbound calls (egress) to a public Schema Registry endpoint. This is because Confluent Cloud Schema Registry is a multi-tenant Schema Registry.

Prerequisites
Confluent Cloud Schema Registry is enabled and configured. For more information about supported features, see Confluent Cloud Schema Registry.
  1. Click Schemas on the left-side panel. On step 2 you should see the Schema Registry endpoint. For example https//confluent.us-east-2.aws.confluent.cloud.

    ../_images/cloud-sr-view.png
  2. Open outbound calls to the Confluent Cloud Schema Registry endpoint. Follow the instructions based on your cloud provider.

    Azure VPC

    Configure outbound call access for Azure networking components.

    AWS VPC

    Configure outbound call access for these AWS VPC networking components:

    GCP VPC

    Configure outbound call access for GCP networking components.

  3. Optional: Verify that your Schema Registry credentials are properly configured, where Schema Registry API key (<schema-registry-api-key>), API secret (<schema-registry-api-secret>), and endpoint (<schema-registry-url>) are specified.

    Run this command to authenticate with the cluster and list the topics registered in your schema.

    curl -u <schema-registry-api-key>:<schema-registry-api-secret> \
    <schema-registry-url>/subjects
    

    If no subjects are created, your output will be empty ([]). If you have subjects, your output should resemble:

    ["test2-value"]
    

    Here is an example command:

    curl -u GFS3CFEX5ELQTB6O:2Nc69uGg+oKOdajqGXfZzuUoN1mBZarD0ZLiE5I9sVIQQhuNti1NSBnzslpgNG5J \
    https://psrc-lq2dm.us-east-2.aws.confluent.cloud/subjects