USM Agent Command Handler

The Command Handler is a component of the Unified Stream Manager (USM) Agent that runs connector operations from Confluent Cloud on your self-managed Kafka Connect clusters.

When you pause or resume a connector from Confluent Cloud, the Command Handler receives the operation, runs it against the Connect REST API in your network, and reports the result back to Confluent Cloud.

Note

Connector management from Confluent Cloud is an Early Access Program feature. The Command Handler is available in USM Agent 1.3.0 and later. It is disabled by default and does nothing until you enable it.

How the Command Handler works

The Command Handler is a separate process that ships in the USM Agent package. It runs on the same host or pod as the agent, because it reaches Confluent Cloud through the agent’s local Envoy proxy.

The Command Handler repeats the following cycle:

Poll. At a configurable interval, the Command Handler asks Confluent Cloud for pending operations. The default and minimum interval is five seconds. The request is outbound and goes through the same authenticated connection that the agent uses to send metadata and metrics.
Run. For each operation it receives, the Command Handler calls the Connect REST API of the target cluster, using the URL and credentials that you declared for that cluster.
Report. The Command Handler reports each result to Confluent Cloud. If Connect returns an error, the Command Handler passes the error through unchanged.

The Command Handler gets the Kafka cluster ID that backs each Connect cluster by periodically probing that cluster. It asks Confluent Cloud for operations only for clusters it has resolved.

Because Confluent Cloud never opens a connection to your network, you don’t need to open an inbound port or add an outbound firewall rule beyond what the agent already requires. The Command Handler connects directly to each Connect cluster inside your network and doesn’t use the proxy for that traffic. For networking requirements, see USM Agent Networking Configuration.

Requirements

Before you enable the Command Handler, confirm that you meet the following requirements:

You run USM Agent 1.3.0 or later. To deploy with CFK, you also run CFK 3.4.0 or later.
Your Confluent Platform version supports connector management. The USM Agent monitors clusters on every supported Confluent Platform version, but the Command Handler works only with Confluent Platform 8.4. For the full compatibility matrix, see USM Agent Support Policy.
You registered the Connect cluster with USM, and the cluster uses a Kafka cluster running on Confluent Cloud as its metadata cluster. The Command Handler doesn’t manage connectors on Connect clusters that use a Confluent Platform Kafka cluster as the metadata cluster. For registration steps, see Confluent Cloud Kafka as the metadata cluster.
The service account that the USM Agent uses has the USMAgent role.
The agent host or pod can reach the Connect REST API of every cluster that you declare.

Supported authentication and transport

The following table lists how the Command Handler connects to the Connect REST API.

Method	Supported	Notes
No authentication, over HTTP or HTTPS	Yes	Use HTTPS in production.
HTTP Basic authentication	Yes	You provide a credential file for each cluster.
HTTPS with a custom certificate authority (CA)	Yes	You provide a trusted CA file for each cluster.
mutual TLS (mTLS) authentication	No
RBAC through Metadata Service (MDS), using a bearer token	No
OAuth	No

The Command Handler doesn’t discover Connect clusters automatically. You declare each cluster that Confluent Cloud can manage.

Enable the Command Handler

The USM Agent package installs the Command Handler but doesn’t start it. You enable it by declaring the Connect clusters to manage in your deployment tool. Declare one entry for each cluster.

Confluent for Kubernetes

Starting in CFK 3.4.0, you enable the Command Handler by adding spec.commandHandler to the USMAgent custom resource (CR) and applying it with kubectl apply. CFK then runs an extra container, usm-agent-command-handler, in the USM Agent pod. This requires the 3.4.0 init container.

In the simplest configuration, reference the Connect CR by name. CFK reads the REST endpoint and group ID from the Connect CR.

kind: USMAgent
spec:
  commandHandler:
    connectClusters:
      - connectClusterRef:
          name: connect

To override the discovered values or to set authentication and TLS, add the optional groupId and connectRest fields. CFK supports no authentication and basic authentication. For basic authentication, store the credentials in a secret in a basic.txt file with username=<user> and password=<password> lines. For TLS, store the PEM CA certificate that signed the Connect certificate in a secret under ca.crt or cacerts.pem.

Changing spec.commandHandler, or a secret that it references, rolls the USM Agent pod, which restarts the proxy along with the Command Handler. If you remove spec.commandHandler, the pod rolls back to the agent alone.

For all fields and the validation rules, see Confluent for Kubernetes.

Command Handler properties

CFK generates the Command Handler configuration from your custom resource. You can tune the following properties with spec.commandHandler.configOverrides.server.

Property	Default	Description
`confluent.usm-agent.command-handler.poll.interval.ms`	`5000`	How often the Command Handler polls Confluent Cloud for operations, in milliseconds. `5000` is also the minimum.
`confluent.usm-agent.command-handler.log.level`	`info`	`debug`, `info`, `warn`, or `error`.
`confluent.usm-agent.command-handler.log.format`	`json`	`json` or `text`.

Manage connectors from only one interface

Use a single interface to manage each connector. What you need to do depends on how you created the connector:

Created from Confluent Cloud. You don’t need to annotate any custom resource. The connector has no Connector CR, and CFK doesn’t touch connectors that have no Connector CR, so you can manage it freely from Confluent Cloud.
Created from a Connector CR in CFK. CFK treats the CR as the source of truth and keeps reconciling the connector to match it. If you change or delete the connector from Confluent Cloud, CFK reverts the change. For example, a connector that you delete from Confluent Cloud is re-created while its Connector CR exists. To manage the connector from Confluent Cloud, annotate the Connector CR with platform.confluent.io/block-reconcile=true.

For instructions, see Manage Connectors.

Enable connector actions in Confluent Cloud

After the Command Handler is running, enable connector actions for each Connect cluster in Confluent Cloud. Confluent Cloud doesn’t send operations to a cluster until you do. For steps, and for how to update existing automation to use the Confluent Cloud endpoint, see Manage self-managed connectors running against Apache Kafka clusters on Confluent Cloud.

Troubleshoot the Command Handler

If a connector operation in Confluent Cloud fails with Errors while trying to update connector state/configuration, check the following:

Confirm that the Command Handler process is running. In CFK, check that the Command Handler container in the agent pod is running and not restarting.
Confirm that connector actions are enabled for the Connect cluster in Confluent Cloud.
Confirm that the cluster is declared in your custom resource or inventory, and that its group ID matches the value you used to register it.
Confirm that the agent host or pod can reach the url of the Connect REST API and, for HTTPS, that the trusted CA file verifies the server certificate.
Confirm that the credential file for a basic cluster contains valid credentials.
Confirm that the agent’s service account has the USMAgent role.

If Connect rejects an operation, the error message from Connect appears in Confluent Cloud.

To diagnose the Command Handler itself, use the following:

Logs. Run kubectl logs <usm_agent_pod> -c usm-agent-command-handler.
Health. The Command Handler serves /healthz on port 9930 and the status of each cluster on /statusz on port 9920. Port 9920 is available only on the loopback interface, so use kubectl port-forward to reach it.
Debug logging. Set confluent.usm-agent.command-handler.log.level to debug. The Command Handler reads this property only at startup, so the change restarts it.

Limitations

The Command Handler has the following limitations in this release:

It manages only Connect clusters that use a Kafka cluster running on Confluent Cloud as the metadata cluster.
It doesn’t support mTLS, RBAC through Metadata Service (MDS), or OAuth when it connects to the Connect REST API.
Any configuration change restarts the Command Handler. With CFK, it also rolls the agent pod.
In Confluent Cloud Console, you can pause and resume connectors. Restarting, deleting, creating, and updating connectors is available only through the Confluent Cloud API.