Configure Networking for Confluent for Kubernetes Blueprints

This document describes the networking options you can configure in Confluent for Kubernetes (CFK) Blueprints to enable external access to Kafka and other Confluent Platform components.

The following are the external access methods CFK Blueprints supports. Once you enable external access using one method and deploy Confluent, you cannot change to another external access method.

  • Load balancers

    Clients connect to Confluent Platform using the Kubernetes provider’s load balancer.

  • Host-based static access

    A Kubernetes Ingress controller manages clients’ connection to Confluent Platform using host-based routing.

  • OpenShift Routes

    Clients connect to Confluent Platform using OpenShift routes.

After you set a global external access method in a Blueprint, (a ConfluentPlatformBlueprint custom resource), you can set provider-specific (for example, AWS or GCP) detail information in Kubernetes clusters (KubernetesCluster custom resources).

For example configurations, see CFK GitHub repo.

Configure DNS for external access

For external access, Confluent Platform requires fully qualified domain names (FQDNs) to be configured to the REST and Kafka listeners endpoints. FQDN is made of <subdomain>.<domain>.

For all the supported external access types, you can use one of the following methods to create and configure FQDNs:

CFK Blueprints generates FQDN

In this mode, the Orchestrator generates an FQDN, using a unique id, prefix, region, cluster name, and domain.

Configure the ConfluentPlatformBlueprint custom resource (CR) as follows:

kind: ConfluentPlatformBlueprint
spec:
  dnsConfig:
    domain:                      --- [1]
    providerType: blueprint      --- [2]
    blueprint:
      generateFQDNWithSchema:
        addRegionName:           --- [3]
        addClusterName:          --- [4]
  • [1] Required. Specify the domain name of your Kubernetes cluster.
  • [2] Required to specify that CFK Blueprints generates FQDNs.
  • [3] Specify whether to add the region name to the FQDNs.
  • [4] Specify whether to add the cluster name to the FQDNs.

With this scheme, CFK Blueprints generates FQDNs in the <prefix>-<uid>.<region>.<clusterName>.<domain> pattern, using the following values:

  • <prefix>:

    • For Kafka listeners: spec.confluentPlatform.kafkaListeners.external.externalAccess.dnsConfig.prefix

      If the prefix field is not configured, kafka is used as a prefix value.

      For example, using kafak as the prefix value, the bootstrap prefix value will be kafka and broker prefix will be kafka0 … kafkaN where N is the replica value.

    • For other Confluent components: spec.components..externalAccess.dnsConfig.prefix

      If not configured, standard names, such as schemaregistry, mds, are used.

  • <uid>: An internally generated, unique 5-digit number.

  • <region>: spec.dnsConfig.blueprint.generateFQDNWithSchema.addRegionName

    If addRegionName field is true ([3]), the Orchestrator will check the KubernetesCluster CR mapped to find the region name. If the region name is missing in that CR, the deployment will fail.

  • <clusterName>: spec.dnsConfig.blueprint.generateFQDNWithSchema.clusterName

    If addClusterName is true ([4]), the Orchestrator will check the KubernetesCluster CR mapped to find the cluster name.

  • <domain> comes from spec.dnsConfig.domain boolean field. This field is required.

If TLS is enabled, Blueprint-managed certificates will generate the Subject Alternative Name (SAN).

Provide subdomain for FQDN

In this mode, you must specify the subdomain in the Deployment CR for each Confluent Platform endpoint.

To standardize FQDNs across Confluent Platform for a given Blueprint, you can specify a regular expression in the ConfluentPlatformBlueprint CR to validate subdomains.

  1. Configure the ConfluentPlatformBlueprint CR:

    kind: ConfluentPlatformBlueprint
spec:
  dnsConfig:
    domain:                      --- [1]
    providerType: deployment     --- [2]
    deployment:
      subDomainRegex:            --- [3]
    • [1] Required. Specify the domain name of your Kubernetes cluster.
    • [2] Required to use the provided subdomain name from deployments.
    • [3] Required. Specify the regular expression for subdomains. For example, to enforce all the subdomain names start with cpc, set this to cpc+.

  2. Specify the subdomain in each Confluent component cluster CR.

    • For the Kafka cluster, pass the subdomain for Kafka REST and listeners in the KafkaCluster CR:

      kind: KafkaCluster
spec:
   externalAccess:
     kafkaRest:
       kafka:
         dnsConfig:
           subdomain:       --- [1]
     listeners:
       external:
         dnsConfig:
           bootstrapPrefix: --- [2]
           brokerPrefix:    --- [3]
      • [1] Required. The Kafka cluster subdomain name.
      • [2] Optional. Kafka bootstrap server prefix. The default value is kafka.
      • [3] Optional. Kafka broker prefix. The default value is b.

    • For other Confluent components’ HTTP configurations, set the subdomain name in the component cluster CR:

      kind: <component>Cluster
spec:
  externalAccess:
    dnsConfig:
      subdomain:              --- [1]
      • [1] Required. The component cluster subdomain name.

Configure external access using the Load Balancer

When configured to use load balancers, CFK Blueprints creates a load balancer for each broker in addition to a load balancer for the bootstrap server. For N number of Kafka brokers, CFK Blueprints creates N+1 number of load balancer services:

  • One as the bootstrap service for the initial connection and for receiving the metadata about the Kafka cluster.
  • Other N services, one for each broker, address the brokers directly.

Note

Kafka brokers and ZooKeeper maintain a constant connection that can be broken if a load balancer is used for ZooKeeper. Do not configure a load balancer for ZooKeeper.

To configure load balancers, set the settings as specified in the following steps and apply the CRs.

  1. Configure DNS.

  2. Specify the load balancer access in the ConfluentPlatformBlueprint CR.

    • For Kafka:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    kafkaListeners:
      global:
        externalAccessType: loadBalancer --- [1]
      externalListener:
        externalAccess:
          enabled: true                  --- [2]
          loadBalancer:
            annotations:                 --- [3]
            labels:                      --- [4]
            advertisedPort:              --- [5]
            externalTrafficPolicy:       --- [6]

    • For other Confluent Platform components:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    http:
      externalAccess:
        type: loadBalancer               --- [1]
        loadBalancer:
          annotations:                   --- [3]
          labels:                        --- [4]
          advertisedPort:                --- [5]
          externalTrafficPolicy:         --- [6]

    • [1] Required.

    • [2] Required to enable external access.

    • [3] Optional. Kubernetes annotations for this service.

    • [4] Optional. Kubernetes labels for this service.

    • [5] Optional. Specify the external port for client access. If not configured, the same internal/external port is configured for the component. Information about the port can be retrieved through the status API.

      The default value is 443.

    • [6] Optional. Valid options are Local and Cluster for the external traffic policy.

  3. In the KubernetesCluster CR, specify any necessary annotations or labels for the load balancer access in the Confluent Platform Kubernetes cluster. The configurations in the KubernetesCluster CR takes precedence over the configurations in the ConfluentPlatformBlueprint CR.

    kind: KubernetesCluster
spec:
  network:
    loadBalancer:
      annotations:
      labels:

  4. Add DNS entries.

    Once the external load balancers are created, you add DNS entries for Kafka brokers, the Kafka bootstrap service, and other Confluent Platform components to your DNS table (or the method you use to get DNS entries recognized by your provider environment).

    1. Get the status of the component cluster deployment CR, such as KafkaCluster, ConnectCluster, etc.

      kubectl get <component cluster CR type> <component CR name> -oyaml -n <namespace>

      An example output for Kafka:

      kind: KafkaCluster
status:
  listeners:
    - externalAccess:
        advertisedExternalEndpoint:
          bootstrap:
            dnsEndpoint: kafka.example.com:9092
            ipAddress: 34.118.13.45
          broker:
          - dnsEndpoint: b0.example.com:9092
            ipAddress: 34.116.201.170
          - dnsEndpoint: b1.example.com:9092
            ipAddress: 34.118.67.170
          - dnsEndpoint: b2.example.com:9092
            ipAddress: 34.116.245.44

      An example output for other components:

      kind: ConnectCluster
status:
  externalAccess:
      advertisedExternalEndpoint:
        dnsEndpoint: connect.example.com:443
        ipAddress: 34.118.17.160
        url: https://connect.example.com:443

    2. Add a DNS entry with each dnsEndpoint and ipAddress in the status output.

      Using the example status output, you would add the following DNS entries:

      DNS name               External IP
b0.example.com         34.116.201.170
b1.example.com         34.118.67.170
b2.example.com         34.116.245.44
kafka.example.com      34.118.13.45
connect.example.com    34.118.17.160

Configure external access using Ingress

When you configure Kafka for host-based static access, the Kafka advertised listeners are set up with the broker prefix and the domain name.

You need to explicitly configure external access to Kafka, for example, using the NGINX ingress controller.

This method requires:

  • The Confluent Platform components are configured with TLS for Ingress access.
  • An Ingress controller that supports SSL passthrough is configured in the Control Plane where the Orchestrator is installed.

To configure Ingress, set the settings as specified in the following steps and apply the CRs.

  1. Configure DNS.

  2. Specify the Ingress access in the ConfluentPlatformBlueprint CR.

    • For Kafka:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    kafkaListeners:
      global:
        externalAccessType: ingress      --- [1]
      externalListener:
        externalAccess:
          enabled: true                  --- [2]
          ingress:
            annotations:                 --- [3]
            labels:                      --- [4]
            advertisedPort:              --- [5]

    • For other Confluent Platform components:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    http:
      externalAccess:
        type: ingress                    --- [1]
        ingress:
          annotations:                   --- [3]
          labels:                        --- [4]
          advertisedPort:                --- [5]

    • [1] Required.

    • [2] Required to enable external access.

    • [3] Optional. Kubernetes annotations for this service.

      Example annotations for NGINX:

      annotations:
  nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/ssl-redirect: "false"
  ingress.kubernetes.io/ssl-passthrough: "true"
  nginx.ingress.kubernetes.io/backend-protocol: HTTPS

    • [4] Optional. Kubernetes labels for this service.

    • [5] Optional. Specify the external port for the clients’ consumption. The default value is 443.

  3. In the KubernetesCluster CR, specify any necessary annotations and labels for the Ingress access in the Confluent Platform Kubernetes cluster:

    kind: KubernetesCluster
spec:
  network:
    ingress:
      annotations:
      labels:

  4. Add DNS entries.

    1. Get the external IP of the Ingress controller load balancer, using the following command:

      kubectl get services -n <namespace>

    2. Get the status of the component cluster deployment CR, such as KafkaCluster, ConnectCluster, etc.

      kubectl get <component cluster CR type> <component CR name> -oyaml -n <namespace>

    3. Use the external IP address of the Ingress controller to add a DNS entry with each dnsEndpoint (in the status output above) for your DNS service provider:

      For example:

      DNS name             ExternalIP
b0.example.com       34.71.198.214
b1.example.com       34.71.198.214
b2.example.com       34.71.198.214
kafka.example.com    34.71.198.214
connect.example.com  34.71.198.214

Configure external access using Routes

CFK Blueprints supports OpenShift routes for exposing Confluent Platform component services to the outside of the OpenShift platform.

When you configure Confluent components with routes, CFK Blueprints creates a route resource for the Confluent component service, and external clients access the service at the HTTPS port, 443.

When using a route, you must configure the Confluent component with TLS.

To configure routes, set the settings as specified in the following steps and apply the CRs.

  1. Configure DNS.

  2. Specify the route access in the ConfluentPlatformBlueprint CR.

    • For Kafka:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    kafkaListeners:
      global:
        externalAccessType: route        --- [1]
      externalListener:
        externalAccess:
          enabled: true                  --- [2]
          route:
            annotations:                 --- [3]
            labels:                      --- [4]
            wildcardPolicy:              --- [5]

    • For other Confluent Platform components:

      kind: ConfluentPlatformBlueprint
spec:
  confluentPlatform:
    http:
      externalAccess:
        type: route                      --- [1]
        route:
          annotations:                   --- [3]
          labels:                        --- [4]
          wildcardPolicy:                --- [5]

    • [1] Required.

    • [2] Required to enable external access.

    • [3] Optional. Kubernetes annotations for this service.

    • [4] Optional. Kubernetes labels for this service.

    • [5] Set this to define a route that covers all hosts within a domain. Valid options are Subdomain and None. The default value is None.

  3. In the KubernetesCluster CR, specify any necessary annotations and labels for the route access in the Confluent Platform Kubernetes cluster:

    kind: KubernetesCluster
spec:
  network:
    route:
      annotations:
      labels:

  4. Add DNS entries.

    Once the routes are created, you add a DNS entry associated with component routes to your DNS table (or whatever method you use to get DNS entries recognized by your provider environment).

    1. Get the IP address of the OpenShift router load balancer.

      The HAProxy load balancer serves as the router for route services, and generally, HAProxy runs in the openshift-ingress namespace.

      oc get svc --namespace openshift-ingress

      An example output:

      NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
router-default            LoadBalancer   172.30.84.52     20.189.181.8   80:31294/TCP,443:32145/TCP   42h
router-internal-default   ClusterIP      172.30.184.233   <none>         80/TCP,443/TCP,1936/TCP      42h

      From the above example, we see the external-ip of the router is 20.189.181.8.

    2. Get the DNS names of the Confluent Platform components:

      oc get routes

    3. Use the external IP to add a DNS entry with each dnsEndpoint (in the status output above) your DNS service provider:

      For example:

      DNS name             ExternalIP
b0.example.com       20.189.181.8
b1.example.com       20.189.181.8
b2.example.com       20.189.181.8
kafka.example.com    20.189.181.8
connect.example.com  20.189.181.8