Configure, Deploy, and Manage Unified Stream Manager Using Confluent for Kubernetes

Confluent Unified Stream Manager connects customer managed clusters with Confluent Cloud to enable Confluent Cloud features for Confluent Platform clusters. The Unified Stream Manager Agent acts as a centralized proxy/gateway for Kafka, and Confluent for Kubernetes (CFK) acts as a tool to deploy Unified Stream Manager Agent in a Kubernetes environment.

This topic presents the steps and guidance for deploying Unified Stream Manager in Confluent Platform using CFK. This process is part of the Register your Confluent Platform Kafka cluster in Confluent Cloud workflow. Review the steps provided in the above topic before proceeding with Unified Stream Manager deployment.

Requirements and considerations

Before deploying Unified Stream Manager with Confluent for Kubernetes, ensure you have the following prerequisites and understand the key considerations:

• Ensure your Kubernetes environment is set up and CFK is installed.

  Use the appropriate version that supports your Confluent Platform and Unified Stream Manager requirements.

• Unified Stream Manager Agent to Confluent Cloud communication only supports basic authentication.

• Unified Stream Manager Agent to Confluent Platform components communication supports basic authentication or mTLS. Both together are not supported. There are 3 authentication combinations possible: basic authentication, basic authentication + TLS, and mTLS.

Configure and deploy Unified Stream Manager

1. Configure Unified Stream Manager Agent using the USMAgent custom resource (CR), and then apply the changes to the CR with the kubectl apply command.

   kind: USMAgent
   spec:
     replicas:
     image:
       application:              --- [1]
       init:                     --- [2]
     authentication:
       type:                     --- [3]
       basic:
         secretRef:              --- [4]
     tls:
       secretRef:                --- [5]
     confluentCloudClient:
       endpoint:                 --- [6]
       environmentId:            --- [7]
       authentication:
         type:                   --- [8]
         basic:
           secretRef:            --- [9]
     externalAccess:             --- [10]
       type:                     --- [11]
       loadBalancer:             --- [12]
       nodePort:                 --- [13]
   

   • [1] Set to the Unified Stream Manager application image.

   • [2] Set to the Unified Stream Manager CFK init container image.

   • [3] Set to basic or mtls.

   • [4] Required for basic authentication. Specify the secret containing the basic authentication credentials.

   • [5] For TLS between Unified Stream Manager Agent and Confluent Platform components, specify the secret containing the TLS certificate, the key, and the root certificate authority (CA) files.

   • [6] Specify the Confluent Cloud endpoint.

   • [7] Specify the Confluent Cloud Environment ID.

   • [8] Set to basic for basic authentication.

   • [9] Required for basic authentication. Specify the secret containing the Cloud Api key and secret.

   • [10] Optional. External access is optional for Unified Stream Manager Agent.

   • [11] Set to loadBalancer or nodePort to specify the external access type.

   • [12] Required when externalAccess type ([11]) is set to loadBalancer. For configuring load balancers, see Configure Load Balancers for Confluent Platform in Confluent for Kubernetes.

   • [13] Required when externalAccess type ([11]) is set to nodePort. For configuring node ports, see Configure Node Ports to Access Confluent Platform Components Using Confluent for Kubernetes.

2. Configure the client-side properties in Kafka, KRaft, and Connect for communication with Unified Stream Manager Agent, and then apply the changes to the CRs with the kubectl apply command.

   spec:
     dependencies:
       usmAgentClient:
         url:                    --- [1]
         authentication:
           type:                 --- [2]
           basic:
             secretRef:          --- [3]
             dpic:               --- [4]
         tls:
           enabled:              --- [5]
           secretRef:            --- [6]
           dpic:                 --- [7]
   

   • [1] Specify the Unified Stream Manager Agent URL.

   • [2] Set to basic or mtls to specify the authentication type.

     See Basic authentication credentials for the required format.

   • [3] Specify the secret containing the basic authentication credentials.

   • [4] Specify the basic authentication credential secret path in the container. For details, see Provide secrets in HashiCorp Vault.

   • [5] Set to true or false to enable or disable TLS.

   • [6] Specify the secret containing the TLS certificate.

   • [7] Specify the TLS certificate secret path in the container. For details, see Provide secrets in HashiCorp Vault.