Connect self-managed Kafka clients to Confluent Cloud

To integrate your existing Kafka clients and applications with Confluent Cloud, you can establish secure connections from your self-managed clients and applications. Confluent provides client libraries for seamless integration in various languages. For the following Kafka clients, click the links to learn how to connect them to Confluent Cloud:

For additional examples and demos using these and other clients, see More examples and demos.

Transport Layer Security (TLS) encryption

To protect data in motion between Confluent Cloud resources (for example, Kafka clusters, Confluent Cloud Schema Registry, and API endpoints) and your connecting clients and applications, Confluent Cloud supports Transport Layer Security (TLS) encryption, the successor of the now-deprecated Secure Socket Layer (SSL).

TLS SNI extension requirement

The client libraries you use to connect to Confluent Cloud must include a Server Name Indication (SNI) extension in the TLS handshake.

Note that the client libraries that Confluent provides do include the SNI extension in the TLS handshake.

For Kafka protocol connections (which use port 9092), the SNI extension must be set to the DNS hostname of the bootstrap endpoint or one of the Kafka brokers.

  1. For initial connections to the cluster, they include the configured bootstrap endpoint as the SNI extension.
  2. For the subsequent connections to individual brokers which are made after bootstrapping, they include that specific broker’s endpoint as the SNI extension.

These broker endpoints are automatically discovered as part of client bootstrapping and do not need to be configured.

For REST connections (which use port 443), the SNI extension must be set to the DNS hostname of the REST endpoint.

Any forward proxies deployed in your environment that are in the path of the network traffic to Confluent Cloud must be configured to forward the SNI extension unmodified.

For testing conformance with TLS SNI requirements, see TLS connectivity testing.

See Transport Layer Security (TLS) Extensions: Server Name Indication and The Transport Layer Security (TLS) Protocol Version 1.3: Server Name Indication.

Manage TLS certificates

Confluent Cloud employs TLS certificates from Let’s Encrypt, a trusted certificate authority. The Let’s Encrypt chain of trust is available for verification: Chain of Trust.

Warning

Avoid intermediate certificate pinning: Confluent Cloud periodically updates TLS certificates. Pinning your clients and applications to specific certificates can result in connection failures after updates.

Root certificate pinning: If your organization’s security policy mandates certificate pinning, pin only to the Let’s Encrypt root certificate to minimize the risk of disruptions due to certificate updates.

Confluent Cloud uses the TLS certificates from Let’s Encrypt to sync them to all data plane components that require TLS termination. Because Let’s Encrypt certificates are subject to change, Confluent Cloud might update these intermediate certificates at any time. Review the following recommendations to ensure that your clients and applications can connect to Confluent Cloud without interruption:

  • Avoid certificate pinning because it can introduce connection failures. Confluent might change intermediate certificates or their properties at any time. If the pinned certificate you use is replaced, then your applications will fail to connect.
  • If your organization requires pinning, then pin to the Let’s Encrypt ISRG Root X1 certificate instead of any subordinate (intermediate) certificate.
  • If Confluent switches CA providers, connections to your applications might be impacted. To avoid connection failures, you can pin to the root certificate to avoid subordinate (intermediate) certificate that are subject to change.

Prerequisites

  • Before connecting a client application to Confluent Cloud, increase the timeout for API version requests from the default value of 10 seconds. You can change this with librdkafka api.version.request.timeout.ms parameter. For more information, see the librdkafka configuration documentation.
  • Specify the location of your client configuration for TLS/SSL (ssl.ca.location) if your operating system and configuration requires it, for example, if your OS does not provide root CA certificates or if the certificates are stored in a non-standard location.
  • For librdkafka-based clients (C/C++, .NET, Python, and Go), to connect to Confluent Cloud, you need Let’s Encrypt TLS certificates installed locally. For RHEL and CentOS, these certificates are not included and must be downloaded.
  • For Java clients, use a Java version of at least 7u151 or 8u141.

Connect a .NET Application to Confluent Cloud

To configure the Confluent .NET Client for Kafka to connect to a Kafka cluster in Confluent Cloud:

  1. Install the Confluent .NET Client for Apache Kafka.
  2. Customize the .NET Confluent Cloud example for your Confluent Cloud cluster, specifically bootstrap.servers, sasl.username, and sasl.password.
  3. Integrate with your environment.

Connect a Go application to Confluent Cloud

To configure the Confluent Golang Client for Kafka to connect to a Kafka cluster in Confluent Cloud:

  1. Install the Confluent Golang Client for Apache Kafka.

    The Confluent Golang Client for Kafka depends on librdkafka, which must be installed separately.

  2. Customize the Golang Confluent Cloud example for your Confluent Cloud cluster, specifically bootstrap.servers, sasl.username, and sasl.password.

  3. Integrate with your environment.

Connect a Python application to Confluent Cloud

To configure the Confluent Python Client for Kafka to connect to a Kafka cluster in Confluent Cloud:

  1. Install the Confluent Python Client for Apache Kafka.
  2. Customize the Python Confluent Cloud example for your Confluent Cloud cluster, specifically bootstrap.servers, sasl.username, and sasl.password.
  3. Integrate with your environment.

More examples and demos

  • To view a working example of hybrid Kafka clusters from self-hosted to Confluent Cloud, see cp-demo.
  • For example configurations for all Confluent Platform components and clients connecting to Confluent Cloud, see template examples for components.
  • See the collection of client examples for supported programming languages in Code Examples for Apache Kafka.
    • The “Hello, World!” examples produce to and consume from any Kafka cluster, including Confluent Cloud, using client libraries for the following programming languages: C (librdkafka), Clojure, C-sharp/.NET, Go, Groovy, Java, Java Spring Boot, Kotlin, Node.js, Python, Ruby, Rust, and Scala.
    • For the subset of language client libraries that support it, the examples demonstrate how to use Confluent Cloud Schema Registry and Avro.