Configure and Deploy Confluent Private Cloud Gateway using Confluent for Kubernetes

This guide provides instructions for configuring and deploying Confluent Gateway using Confluent for Kubernetes (CFK).

For the configuration and deployment steps using CFK, see Deploy Confluent Gateway using CFK.

The following tasks are described using CFK:

A separate guide is provided for the following security-related tasks. See Configure Security for Confluent Gateway using CFK:

  • Configure Secret Stores

  • Configure SSL

  • Configure Password

  • Configure Authentication Swapping

Confluent Gateway supports deployment on-premises, in private cloud VPCs, or in hybrid environments.

The following are the high-level steps to deploy Confluent Gateway:

  1. Configure Confluent Gateway to connect to Kafka clusters, specifying authentication, routing policies, and security policies according to your organizational needs.

    Additionally, Configure security for the Confluent Gateway.

  2. Provision the Confluent Gateway service using the configuration settings from the previous step.

  3. Reconfigure clients to communicate with the Confluent Gateway endpoint.

    This simplifies credential management and strengthens overall security posture.

Requirements and considerations

Before you begin, make sure that the following requirements and considerations are satisfied:

  • System requirements

    • CPU and Memory

      • Minimum: 2 vCPUs, 4 GB RAM

      • Recommended: 4 vCPUs, 8 GB RAM

    • Network throughput: 45 MB/sec of sustained workload for a 1 Gigabit Link (1Gbps)

    • Storage: 10 GB of disk space

  • Network Access

    The Confluent Gateway container must have access to necessary Kafka clusters. Configure firewall ports for bidirectional traffic as needed.

  • CFK and Confluent Platform

    Ensure your Kubernetes environment is set up and CFK and Confluent Platform are installed. For details on installing CFK, see Deploy Confluent for Kubernetes.

    Use the appropriate version of CFK that supports your Confluent Platform and Confluent Gateway requirements.

Configure Confluent Gateway using CFK

Provide the following configuration settings in the Confluent Gateway custom resource (CR) file.

The top-level layout for the Confluent Gateway CR is as follows:

kind: Gateway
metadata:
  name:                   --- [1]
  namespace:              --- [2]
spec:
  image:
    application:          --- [3]
    init:                 --- [4]
  streamingDomains:       --- [5]
  secretStores:           --- [6]
  routes:                 --- [7]
  admin:                  --- [8]
  advanced:               --- [9]

  • [1] The Confluent Gateway name.

  • [2] The Confluent Gateway namespace.

  • [3] The Confluent Gateway application image in the following pattern: confluentinc/cpc-gateway:<version-tag>. <version-tag> is the Confluent Gateway version.

  • [4] Set to the Confluent Gateway CFK init container image.

  • [5] A list of Streaming Domains that represent Kafka clusters.

  • [6] A list of credential stores. Required for authentication swapping. See Secret store configuration section for details.

  • [7] A list of client-traffic listeners that enforces governance policies and forwards requests to the appropriate streaming domain.

  • [8] Admin/management configuration.

  • [9] Advanced settings.

Configure Streaming Domains

Streaming Domains are logical representations of your Kafka clusters at the Confluent Gateway.

kind: Gateway
spec:
  streamingDomains:
    - name:               --- [1]
      type:               --- [2]
      kafkaCluster:       --- [3]
        name:             --- [4]
        bootstrapServers: --- [5]
          - id:           --- [6]
            endpoint:     --- [7]
            ssl:          --- [8]
        nodeIdRanges:     --- [9]
          - name:         --- [10]
            start:        --- [11]
            end:          --- [12]

  • [1] A unique name for the Streaming Domain.

  • [2] The type of the Streaming Domain. The default value is kafka.

  • [3] A Kafka cluster, including Confluent Server, Confluent Cloud Kafka, or Apache Kafka®, that the Confluent Gateway can route traffic to.

  • [4] The name of the Kafka cluster.

  • [5] A list of Kafka bootstrap servers. At minimum, one bootstrap server is required per Kafka cluster. For each security protocol listener, one can define one bootstrap server.

  • [6] A unique identifier for the bootstrap-server. Recommended to use the protocol, such as SASL_PLAINTEXT, SSL or PLAINTEXT_SASL_PLAIN for clear distinction between both TLS channel type and SASL mechanism.

  • [7] The bootstrap endpoint of the Kafka broker in the format, protocol://host:port

  • [8] Only required for SSL/SASL_SSL configuration.

    See the TLS/SSL configuration section for details.

  • [9] Optional. The node ID ranges. Only required for port-based broker identification. These should be the broker IDs defined in the Kafka deployment.

  • [10] The name of the node ID range.

  • [11] The start of the node ID range, inclusive.

  • [12] The end of the node ID range, inclusive.

The following is an example configuration for a Confluent Gateway Streaming Domain:

kind: Gateway
spec:
  streamingDomains:
    - name: sales # unique across gateway
      type: kafka # default: kafka
      kafkaCluster:
        name: sales-cluster # default: <streamingDomain.name>
        bootstrapServers:
          - id: PLAINTEXT-1
            endpoint: kafka0.example.com:9092
          - id: SASL_PLAINTEXT-1
            endpoint: kafka0.example.com:9093
        nodeIdRanges:
          - name: default
            start: 0              # inclusive
            end: 3                # inclusive

Configure Routes

Routes are Confluent Private Cloud Gateway endpoints where client applications connect to stream data.

Confluent Gateway uses Routes to define how client applications connect to Kafka clusters. Clients connect to the Gateway as if it were a Kafka cluster, while the Gateway handles routing and governance.

kind: Gateway
spec:
  routes:
    - name:                         --- [1]
      endpoint:                     --- [2]
      brokerIdentificationStrategy: --- [3]
        type:                       --- [4]
        pattern:                    --- [5]
      streamingDomain:              --- [6]
        name:                       --- [7]
        bootstrapServerId:          --- [8]
      security:                     --- [9]
    logNetwork:                     --- [10]
    logFrames:                      --- [11]

  • [1] The unique name for the Route.

  • [2] The host:port combination that Confluent Gateway will listen on. This is the external address clients use to bootstrap to the Kafka cluster.

  • [3] Specifies the strategy for mapping client requests to a specific Kafka broker.

  • [4] The type of broker identification strategy. Set to port (default) or host.

    • port strategy: Each Kafka broker is identified using a unique port number. This is the default strategy.

      Clients connect to different ports to reach specific brokers (for example, port 9092 to connect with broker-0, port 9093 to connect with broker-1).

      The nodeIdRanges for the Streaming Domain you set in Configure Streaming Domains is used.

      nodeIdRanges should be present in all the clusters associated with Route’s Streaming Domain.

    • host strategy: Each Kafka broker is represented using a unique hostname.

      Clients use different host names to reach specific brokers (for example, broker-0.kafka.company.com, broker-1.kafka.company.com), and the Confluent Gateway Routes based on the SNI header.

      The pattern setting ([5]) is used.

  • [5] The pattern for the broker identification strategy. Required if the type ([4]) is host. For example, broker-$(nodeId).eu-gw.sales.example.com:9092.

  • [6] The reference to a Streaming Domain.

  • [7] The name of the Streaming Domain. Must be a valid name from the gateway.streamingDomains[].name.

  • [8] The bootstrap server ID. Must match one of kafkaCluster.bootstrapServers[].id.

  • [9] The security configuration. See Configure Security for Confluent Private Cloud Gateway using Confluent for Kubernetes section for details.

  • [10] Optional. Enable network logging of the TPC-level connection details. The default value is false.

  • [11] Optional. Enable frame logging of Kafka protocol frames. The default value is false.

The following is an example configuration for a Confluent Gateway Routes:

kind: Gateway
spec:
routes:
  - name: eu-sales
    endpoint: eu-gw.sales.example.com:9092
    brokerIdentificationStrategy:
      type: host
      pattern: broker-$(nodeId).eu-gw.sales.example.com:9092
    streamingDomain:
      name: sales
      bootstrapServerId: SASL_SSL-1
    logNetwork: true
    logFrames: true

Configure Admin and Metrics settings

kind: Gateway
spec:
  admin:
    bindAddress:             --- [1]
    port:                    --- [2]
    endpoints:
      metrics:               --- [3]
    jvmMetrics:              --- [4]
      - JvmGcMetrics
      - JvmMemoryMetrics
      - JvmThreadMetrics
      - ProcessorMetrics
      - UptimeMetrics
    commonTags:              --- [5]
      host:
      region:

  • [1] The local address to bind admin service and to listen on for incoming connections. The default value is 0.0.0.0.

    This address determines which network interfaces the gateway will use to expose its administrative endpoints, such as metrics and liveness checks.

  • [2] The TCP port for exposing admin endpoints. The default value is 9190.

    The Confluent Gateway will serve metrics at http://{bindAddress}:{port}/metrics and liveness check at http://{bindAddress}:{port}/livez.

  • [3] The metrics endpoints to be enabled and exposed. The default value is true.

  • [4] The standard JVM Metrics (uses Micrometer classes). By default JvmGcMetrics, JvmMemoryMetrics, JvmThreadMetrics, ProcessorMetrics, UptimeMetrics are enabled.

  • [5] Optional. The common tags to be applied to all the metrics.

The following is an example configuration for a Confluent Gateway Admin and Metrics:

kind: Gateway
spec:
  admin:
    bindAddress: 0.0.0.0
    port: 9190
    endpoints:
      metrics: true
    jvmMetrics:
      - JvmGcMetrics
      - JvmMemoryMetrics
      - JvmThreadMetrics
      - ProcessorMetrics
      - UptimeMetrics
    commonTags:
      host: pod-0
      region: us-west-2

Install Confluent Gateway using CFK

Deploy Confluent Gateway using the Confluent Gateway CR file you set up in the previous section, Configure Confluent Gateway using CFK:

kubectl apply -f <gateway-CR-file>