Configure Load Balancers to Access Confluent Components

External access to Kafka using load balancers

When a client accesses a Kafka cluster, it first connects to the bootstrap server to get the metadata list of all the brokers in the cluster. Then the client finds the address of the broker it is interested in and connects directly to the broker to produce or consume data. When configured to use load balancers, CFK creates a load balancer for each broker in addition to a load balancer for the bootstrap server. For N number of Kafka brokers, CFK creates N+1 number of load balancer services:

  • One as the bootstrap service for the initial connection and for receiving the metadata about the Kafka cluster.
  • Another N services, one for each broker, address the brokers directly.

For the additional configuration steps required to allow external access to Metadata Service (MDS), see Configure Networking for RBAC.

To allow external access to Kafka using load balancers:

  1. Set the following in Kafka CR and apply the configuration:

    spec:
      listeners:
        external:
          externalAccess:
            type: loadBalancer
            loadBalancer:
              domain:           --- [1]
              bootstrapPrefix:  --- [2]
              brokerPrefix:     --- [3]
    
    • [1] Required. Set it to the domain where your Kubernetes cluster is running.

    • [2] Optional. Use bootstrapPrefix to change the default Kafka bootstrap prefix. The default bootstrap prefix is the Kafka component name (kafka).

      The value is used for the DNS entry. The bootstrap DNS name becomes <bootstrapPrefix>.<domain>.

      If not set, the default bootstrap DNS name is kafka.<domain>.

      You may want to change the default prefixes for each component to avoid DNS conflicts when running multiple Kafka clusters.

    • [3] Optional. Use brokerPrefix to change the default Kafka broker prefixes. The default Kafka broker prefix is b.

      These are used for DNS entries. The broker DNS names become <brokerPrefix>0.<domain>, <brokerPrefix>1.<domain>, and so on.

      If not set, the default broker DNS names are b0.<domain>, b1.<domain>, and so on.

      You may want to change the default prefixes to avoid DNS conflicts when running multiple Kafka clusters.

  2. Add DNS entries for Kafka bootstrap server and brokers.

    Once the external load balancers are created, you add DNS entries for Kafka brokers and the Kafka bootstrap service to your DNS table (or the method you use to get DNS entries recognized by your provider environment).

    You need the following to derive Kafka broker DNS entries:

    • Domain name of your Kubernetes cluster (domain in Step #1)

    • The external IP of the Kafka load balancers

      You can retrieve the external IP using the following command:

      kubectl get services -n <namespace> | grep LoadBalancer
      
    • Kafka bootstrap prefix and broker prefix

    The example below shows the DNS table entries, using:

    • Domain: example.com
    • Three broker replicas with the default prefix: b
    • The Kafka bootstrap prefix: kafka
    DNS name                 External IP
    b0.example.com         192.50.14.35
    b1.example.com         192.50.28.28
    b2.example.com         192.50.64.18
    kafka.example.com      192.50.34.20
    

External access to other Confluent Platform components using load balancers

The external clients can connect to other Confluent Platform components using load balancers.

Note

Kafka brokers and ZooKeeper maintain a constant connection that can be broken if a load balancer is used for ZooKeeper. Do not configure a load balancer for ZooKeeper.

The access endpoint of each Confluent Platform component is: <component CR name>.<Kubernetes domain>

For example, in the example.com domain with TLS enabled, you access the Confluent Platform components at the following endpoints:

To allow external access to Kafka using load balancers:

  1. Set the following in the component CR and apply the configuration:

    spec:
      externalAccess:
        type: loadBalancer
        loadBalancer:
          domain:          --- [1]
          prefix:          --- [2]
    
    • [1] Required. Set domain to the domain name of your Kubernetes cluster.

    • [2] Optional. Set prefix to change the default load balancer prefixes. The default is the component name, such as controlcenter, connector, replicator, schemaregistry, ksql.

      The value is used for the DNS entry. The component DNS name becomes <prefix>.<domain>.

      If not set, the default DNS name is <component name>.<domain>, for example, controlcener.example.com.

      You may want to change the default prefixes for each component to avoid DNS conflicts when running multiple Kafka clusters.

  2. Add a DNS entry for each Confluent Platform component that you added a load balancer to.

    Once the external load balancers are created, you add a DNS entry associated with component load balancers to your DNS table (or whatever method you use to get DNS entries recognized by your provider environment).

    You need the following to derive Confluent Platform component DNS entries:

    • Domain name of your Kubernetes cluster as set in Step #1.

    • The external IP of the component load balancers retrieved with the

      You can retrieve the external IP using the following command:

      kubectl get services -n <namespace> -ojson
      
    • The component prefix if set in Step #1 above. Otherwise, the default component name.

    A DNS name is made up of the prefix and the domain name. For example, controlcenter.example.com.

For a tutorial scenario on configuring external access using load balancers, see the quickstart tutorial for using load balancer.`