What is Confluent Private Cloud Gateway?

Confluent Private Cloud Gateway (Confluent Gateway) is an Apache Kafka®-protocol-aware, cloud-native proxy solution that provides platform teams centralized control over Kafka infrastructure. It streamlines complex operations including migrations, upgrades, and disaster recovery, without requiring code changes or application restarts from application developers.

Unlike generic proxy solutions that operate at the network layer, Confluent Gateway natively understands the Kafka protocol. It manages traffic by:

• Intercepting client traffic through stable, virtualized endpoints.

• Rewriting broker metadata in real time to dynamically route requests to the appropriate cluster.

• Enforcing policies directly at the proxy layer.

Routing changes and cluster switchovers take effect immediately, with zero application downtime and no client reconfiguration.

Why use Confluent Private Cloud Gateway?

• Application-transparent migrations: Move workloads between clusters or to the cloud by updating a single routing rule, rather than coordinating with multiple application teams.

• Centralized security and governance: Enforce mTLS, authentication, and data streaming security policies at the Confluent Private Cloud Gateway level to standardize Kafka network access across your organization, rather than managing individual client configurations.

• Cloud-native agility for on-premises environments: Bring the stable endpoint experience of Confluent Cloud to your private infrastructure, hiding the complexity of broker lists and networking from your developers.

• Low recovery time objective (RTO) during failures: Reduce recovery time during disasters by switching traffic to a standby cluster, without requiring application restarts or client reconfiguration.

• Simplified client configuration: Eliminate the need for clients to track cluster changes, broker lists, and security configurations by centralizing these concerns at the Confluent Gateway.

Confluent Gateway supports the following use cases, reflecting patterns common in enterprise-scale Kafka deployments:

• Cloud migration: Migrate on-premises clients to Confluent Cloud without client changes.

• On-premises disaster recovery: Switch over from an unhealthy cluster to a healthy cluster without client changes.

• Secure external partner access: Grant trusted external partners direct access to a private Kafka cluster through a dedicated Confluent Gateway endpoint, enforcing authentication and access policies at the boundary without modifying the upstream cluster or exposing direct network access to it.

• Custom domains for Kafka listeners: Define organization-branded hostnames for Kafka endpoints, decoupling client-facing DNS naming from physical cluster addresses.

• Blue-green Kafka version upgrades: Validate a new Kafka version under production traffic before completing the cutover. Shift load incrementally and roll back immediately if issues arise, without involvement from application teams.

For example configuration scenarios, see Confluent Gateway GitHub repository.

How Confluent Gateway routes and secures Kafka traffic

Confluent Private Cloud Gateway intercepts all Kafka traffic from client applications, securely routing Kafka protocol messages to the appropriate upstream Kafka clusters.

By rewriting broker addresses within metadata responses, Confluent Gateway exposes only virtualized endpoints to clients. This abstracts the underlying infrastructure, ensuring that any backend cluster changes remain seamless and transparent to your applications.

Core components

• Routes: The virtualized Confluent Gateway endpoints exposed to Kafka client applications to stream data.

  Confluent Gateway uses Routes to create an abstraction layer that decouples client applications from backend Kafka brokers, allowing traffic to be dynamically redirected without client-side configuration changes.

• Streaming Domains: The logical representations of your upstream Kafka clusters within Confluent Gateway. They act as the backend targets that map your virtualized routes to actual physical infrastructure.

Request lifecycle

When a client application sends a request to a Route, the Confluent Gateway performs the following steps:

1. Intercept: A Kafka client sends a request to a Confluent Gateway Route.

2. Resolve: The Confluent Gateway identifies which Streaming Domain the Route is mapped to and determines the upstream cluster to handle the request.

3. Protocol transformation: The Confluent Gateway performs any necessary authentication swapping, such as converting an OIDC token to SASL credentials required by the backend cluster.

4. Route: The Confluent Gateway forwards the request to the appropriate upstream Kafka broker. When the broker responds, the Confluent Gateway rewrites the broker addresses in the metadata response so that clients see only Confluent Gateway virtual endpoints, keeping cluster topology changes transparent to applications.

Supported protocols, features, and limitations

Protocol support

Confluent Gateway is compatible with Kafka protocol versions 3.x and 4.x. It provides transparent proxying for all standard Kafka client libraries without code changes.

Supported features

• Authentication and identity management: Authenticate clients using mTLS, SASL/PLAIN, SASL/SCRAM, OAuth, or OIDC. Confluent Gateway can pass the client identity directly to the upstream cluster or perform authentication swapping between the Confluent Gateway and the brokers. See Authentication and Authentication swapping.

• Secure credential storage: Retrieve broker and Schema Registry credentials from external secret stores, including HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault. This eliminates the need to store plaintext secrets in the gateway.yaml configuration file. See Secret stores.

• Metrics, tracing, and troubleshooting: Monitor request volume, latency, and validation outcomes by exporting Prometheus-format metrics through the admin endpoint. You can also use the fencing filter to block incoming requests and return standard BROKER_NOT_AVAILABLE error codes to clients. See Administration and metrics and Fencing filter.

Current limitations

• No legacy protocol support: Supports Kafka protocol versions 3.x and 4.x only. Kafka server and client versions 2.x or older are not supported.

• Centralized deployment only: Must be deployed as a centralized or ingress proxy tier. Distributed sidecar deployment models are not supported.

• No Schema Registry proxying: Proxying applies strictly to binary Kafka broker traffic. The Confluent Gateway does not intercept or route direct Schema Registry HTTP/REST traffic.

Get started

To get started with Confluent Gateway, deploy Confluent Gateway on Docker or CFK. Review system requirements, supported Kafka and client versions, and step-by-step instructions for both deployment models in Configure and Deploy Gateway.