Quick Start: USM with Confluent for Kubernetes

This quickstart shows you how to deploy USM using Confluent for Kubernetes to register your Confluent Platform cluster with Confluent Cloud.

In this guide, you will:

  • Deploy Confluent for Kubernetes and Confluent Platform

  • Register your Confluent Platform cluster in Confluent Cloud

  • Deploy the USM Agent

  • Update Confluent Platform components to use the USM Agent

  • Verify metrics are flowing to Confluent Cloud

  • Register Connect cluster to monitor managed connectors

  • Deregister Confluent Platform and Connect clusters

Before you begin

Ensure your environment meets the following requirements:

  • Confluent Platform: Version 8.1 or later.

  • Kubernetes cluster: A running Confluent for Kubernetes cluster on AWS EKS in a tier 1 region.

  • Confluent Cloud: An organization existing in Confluent Cloud with the Advanced Stream Governance package.

  • Confluent CLI: Version 4.39 or later. See Install the Confluent CLI for installation and upgrade instructions.

Step 1: Deploy Confluent for Kubernetes and Confluent Platform

The Confluent for Kubernetes operator must be installed on your Kubernetes cluster. If you have not installed it, see the Confluent for Kubernetes documentation.

In this section, you deploy Confluent Platform 8.1 using Confluent for Kubernetes without the USM Agent. You add the agent after registering in Confluent Cloud and obtaining the agent configuration. Use the example YAML below.

  1. Create a file named confluent-platform-without-agent.yaml. Expand the YAML configuration dropdown below and copy the content into the file.

    YAML configuration
    apiVersion: platform.confluent.io/v1beta1
kind: KRaftController
metadata:
  name: kraftcontroller
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  dataVolumeCapacity: 10G
  storageClass:
    name: gp2
  image:
    application: confluentinc/cp-server:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  replicas: 3
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  dataVolumeCapacity: 10G
  storageClass:
    name: gp2
  image:
    application: confluentinc/cp-server:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  replicas: 3
  metricReporter:
    enabled: false
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
---
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  build:
    type: onDemand
    onDemand:
      plugins:
        locationType: confluentHub
        confluentHub:
          - name: kafka-connect-datagen
            owner: confluentinc
            version: 0.5.2
  replicas: 3
  image:
    application: confluentinc/cp-server-connect:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  dependencies:
    kafka:
      bootstrapEndpoint: kafka:9071

    Replace the following placeholders in the file:

    • <your-namespace>: Your Kubernetes namespace

    • <your-id>: Your identifier for managed resources

  2. Apply the configuration to your cluster:

    kubectl apply -f confluent-platform-without-agent.yaml

Step 2: Register the Confluent Platform cluster in Confluent Cloud

After deploying Confluent for Kubernetes and Confluent Platform, the next step is to register your cluster in Confluent Cloud to generate the credentials used by the USM Agent. For detailed instructions, see Register your Confluent Platform cluster.

  1. Log in to Confluent Cloud Console at https://confluent.cloud.

  2. Navigate to Environments, select your environment, and click Clusters.

  3. On the Clusters page, click Start Unified Stream Manager setup.

  4. Read the benefits and click Next to proceed. If you have the Essentials governance package, you’ll be prompted to upgrade to the Advanced Stream Governance package (required for USM); complete the upgrade to continue.

  5. Choose your agent deployment type:

    • New agent: Select this option to deploy a fresh agent in the environment where your Confluent Platform cluster resides.

    • Existing agent: Select this to reuse an agent already deployed in that environment.

  6. Click Get started to proceed to the detailed registration steps.

  7. Follow the guided flow in the UI to:

    1. Configure private networking for your AWS VPC and click Next. For detailed instructions, see Configure private networking.

    2. Configure a service account for the USM Agent (create a new one or select an existing one) and click Next. For detailed instructions, see Configure a service account.

  8. On the Deploy Unified Stream Manager agent page, click Generate configuration file, and then click Next. For detailed instructions, see Deploy the USM Agent.

    The JSON file contains credentials for the USM Agent:

    {
  "ENVIRONMENT_ID": "",
  "SERVICE_ACCOUNT_API_KEY": "",
  "SERVICE_ACCOUNT_API_SECRET": "",
  "SCHEMA_REGISTRY_API_KEY": "",
  "SCHEMA_REGISTRY_API_SECRET": "",
  "FRONTDOOR_URL": "",
  "SCHEMA_REGISTRY_URL": ""
}

  9. On the Register Confluent Platform cluster page, do the following:

    1. Get your Confluent Platform cluster ID. Log in to the Kubernetes cluster and run the following commands:

      kubectl get kafka -n <your-namespace>
kubectl describe kafka <kafka-resource-name> -n <your-namespace>

    2. In Confluent Platform cluster ID, enter the cluster ID. The Metadata region is automatically selected based on your private network configuration. You cannot change this setting.

    3. In Cluster name in Confluent Cloud, enter a display name for the cluster.

    4. Click Register cluster.

After successful registration, you’ll see a cluster card with a Metrics delayed status. This status changes to Running after metrics start flowing. Click Close wizard to exit the wizard.

Step 3: Deploy the USM Agent

If you selected Existing agent in the registration wizard, skip this section and Upgrade Confluent Platform components to use the USM Agent.

  1. Create a credentials file named setu-ccloud-cred.txt with the following content:

    username=<SERVICE_ACCOUNT_API_KEY>
password=<SERVICE_ACCOUNT_API_SECRET>

    Replace <SERVICE_ACCOUNT_API_KEY> and <SERVICE_ACCOUNT_API_SECRET> with the values from the configuration file you downloaded during registration.

  2. Create a Kubernetes secret from the credentials file:

    kubectl -n <your-namespace> create secret generic setu-ccloud-cred \
  --from-file=basic.txt=setu-ccloud-cred.txt

  3. Create a file named usmagent.yaml. Expand the YAML configuration dropdown below and copy the content into the file.

    YAML configuration
    apiVersion: platform.confluent.io/v1beta1
kind: USMAgent
metadata:
  name: usm-agent
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  # Enable debug logs
  envVars:
    - name: USM_AGENT_CUSTOM_ARGS
      value: -l trace
  replicas: 1
  image:
    application: confluentinc/cp-usm-agent:1.0.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  externalAccess:
    type: loadBalancer
    loadBalancer:
      domain: my.domain
      prefix: usm-agent
  # Basic authentication
  #authentication:
  #type: basic/mtls
  # if auth type basic
  # basic:
  #secretRef: setu-basic-auth
  # TLS secret
  #tls:
  #secretRef: tls-usmagent
  confluentCloudClient:
    endpoint: <Frontdoor-URL-from-agent-config-downloaded-during-registration>
    environmentId: <env-id-from-agent-config-file>
    authentication:
      type: basic
      basic:
        secretRef: setu-ccloud-cred

    Replace the following placeholders:

    • <your-namespace>: Your Kubernetes namespace

    • <your-id>: Your identifier for managed resources

    • <Frontdoor-URL-from-agent-config-downloaded-during-registration>: The FRONTDOOR_URL value from the configuration file

    • <env-id-from-agent-config-file>: The ENVIRONMENT_ID value from the configuration file

  4. Apply the USM Agent configuration:

    kubectl apply -f usmagent.yaml

Step 4: Upgrade Confluent Platform components to use the USM Agent

Update your Confluent Platform YAML to include the USM Agent dependency and upgrade your deployment.

  1. Create a file named confluent-platform-with-agent.yaml. Expand the YAML configuration dropdown below and copy the content into the file.

    YAML configuration
    apiVersion: platform.confluent.io/v1beta1
kind: KRaftController
metadata:
  name: kraftcontroller
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  dataVolumeCapacity: 10G
  storageClass:
    name: gp2
  image:
    application: confluentinc/cp-server:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  replicas: 3
  dependencies:
    usmAgentClient:
      url: http://usm-agent.<your-namespace>.svc.cluster.local:10000
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  dataVolumeCapacity: 10G
  storageClass:
    name: gp2
  image:
    application: confluentinc/cp-server:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  replicas: 3
  metricReporter:
    enabled: false
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    usmAgentClient:
      url: http://usm-agent.<your-namespace>.svc.cluster.local:10000
---
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect
  namespace: <your-namespace>
  labels:
    confluent.io/managed-by: "<your-id>"
spec:
  podTemplate:
    labels:
      confluent.io/managed-by: "<your-id>"
  build:
    type: onDemand
    onDemand:
      plugins:
        locationType: confluentHub
        confluentHub:
          - name: kafka-connect-datagen
            owner: confluentinc
            version: 0.5.2
  replicas: 3
  image:
    application: confluentinc/cp-server-connect:8.1.0
    init: confluentinc/confluent-init-container:3.1.0
    pullPolicy: IfNotPresent
  #pullSecretRef:
  # - confluent-registry
  dependencies:
    kafka:
      bootstrapEndpoint: kafka:9071
    usmAgentClient:
      url: http://usm-agent.<your-namespace>.svc.cluster.local:10000

    Replace the following placeholders:

    • <your-namespace>: Your Kubernetes namespace

    • <your-id>: Your identifier for managed resources

  2. Apply the updated configuration:

    kubectl apply -f confluent-platform-with-agent.yaml

Step 5: Verify metrics are flowing to Confluent Cloud

Verify that your cluster is successfully registered and metrics are flowing to Confluent Cloud.

  1. Log in to Confluent Cloud and navigate to Environments > Clusters.

  2. Locate your registered Confluent Platform cluster. After metrics start flowing, the cluster card status changes from Metrics delayed to Running.

  3. Click on the cluster to view detailed metrics and verify they are populated.

  4. Add metadata to your resources:

    • Attach descriptions and tags to your cluster, topics, and connectors

    • For topics, define owner details such as name and email

  5. Navigate to Data Portal to discover your Confluent Platform resources, such as topics and connectors.

  6. Inspect data flow for your topics using Stream lineage.

Step 6: Register Connect cluster to monitor managed connectors

Register your Connect cluster to monitor managed connectors.

  1. In the Confluent Cloud console, navigate to the Connector page of your registered Confluent Platform cluster.

  2. Click Register Connect Cluster.

  3. Retrieve your Connect cluster group ID:

    kubectl describe connect <connect_resource_name> -n <your-namespace>

    Look for the group.id value in the output. This is your Connect cluster group ID.

  4. Enter your group.id value from the previous step in the pop-up and click Register.

  5. After the Connect cluster is successfully registered, connector cards start appearing in the Confluent Cloud console.

Step 7: Clean up

Deregistration is supported with CLI only. Use the Confluent CLI to deregister your Confluent Platform cluster or Connect cluster.

To clean up your environment, deregister your Confluent Platform and Connect clusters using the Confluent CLI.

Deregister Confluent Platform cluster

Deregister your Confluent Platform cluster from Confluent Cloud.

  1. Log in to Confluent Cloud with your Confluent CLI:

    confluent login

  2. Get the Environment ID using the following steps:

    1. In the Confluent Cloud Console, navigate to the Environments page and select your environment.

    2. Click the Details tab and copy the ID from the About section.

  3. List your Confluent Platform clusters to retrieve the unique cluster ID:

    confluent unified-stream-manager kafka list --environment <Environment-ID>

  4. Identify the unique Confluent Platform cluster ID starting with usmkc for the cluster you want to deregister.

  5. Deregister the cluster:

    confluent unified-stream-manager kafka deregister <usmkc-xxx> --environment <Environment-ID>

    Upon successful deregistration, the cluster card disappears from the Confluent Cloud.

Deregister Connect cluster

Deregister your Connect cluster from Confluent Cloud.

  1. List your Connect clusters:

    confluent unified-stream-manager connect list --environment <Environment-ID>

  2. Identify the unique cluster ID starting with usmcc- for the Connect cluster you want to deregister.

  3. Deregister the Connect cluster:

    confluent unified-stream-manager connect deregister <usmcc-xxxx> --environment <Environment-ID>

Related content