<a id="usm-command-handler"></a>

# USM Agent Command Handler

The Command Handler is a component of the Unified Stream Manager (USM) Agent that runs
connector operations from Confluent Cloud on your self-managed Kafka Connect
clusters.

When you pause or resume a connector from Confluent Cloud, the Command Handler
receives the operation, runs it against the Connect REST API in your
network, and reports the result back to Confluent Cloud.

#### NOTE
Connector management from Confluent Cloud is an Early Access Program feature. The
Command Handler is available in USM Agent 1.3.0 and later. It is
disabled by default and does nothing until you enable it.

## How the Command Handler works

The Command Handler is a separate process that ships in the USM Agent
package. It runs on the same host or pod as the agent, because it reaches
Confluent Cloud through the agent’s local Envoy proxy.

The Command Handler repeats the following cycle:

1. **Poll**. At a configurable interval, the Command Handler asks Confluent Cloud for
   pending operations. The default and minimum interval is five seconds. The
   request is outbound and goes through the same authenticated connection that
   the agent uses to send metadata and metrics.
2. **Run**. For each operation it receives, the Command Handler calls the
   Connect REST API of the target cluster, using the URL and credentials
   that you declared for that cluster.
3. **Report**. The Command Handler reports each result to Confluent Cloud. If
   Connect returns an error, the Command Handler passes the error through
   unchanged.

The Command Handler gets the Kafka cluster ID that backs each Connect cluster
by periodically probing that cluster. It asks Confluent Cloud for operations only for
clusters it has resolved.

Because Confluent Cloud never opens a connection to your network, you don’t need to
open an inbound port or add an outbound firewall rule beyond what the agent
already requires. The Command Handler connects directly to each Connect
cluster inside your network and doesn’t use the proxy for that traffic. For
networking requirements, see [USM Agent Networking Configuration](networking.md#usm-networking).

## Requirements

Before you enable the Command Handler, confirm that you meet the following
requirements:

* You run USM Agent 1.3.0 or later. To deploy with CFK, you also run
  CFK 3.4.0 or later.
* Your Confluent Platform version supports connector management. The USM Agent
  monitors clusters on every supported Confluent Platform version, but the Command Handler
  works only with Confluent Platform 8.4. For the full compatibility matrix, see
  [USM Agent Support Policy](../support-policy.md#usm-compatibility).
* You registered the Connect cluster with USM, and the cluster
  uses a Kafka cluster running on Confluent Cloud as its metadata cluster. The Command
  Handler doesn’t manage connectors on Connect clusters that use a
  Confluent Platform Kafka cluster as the metadata cluster. For registration steps, see
  [Confluent Cloud Kafka as the metadata cluster](https://docs.confluent.io/cloud/current/usm/register-connect/cloud-kafka.html).
* The service account that the USM Agent uses has the `USMAgent`
  role.
* The agent host or pod can reach the Connect REST API of every cluster that
  you declare.

### Supported authentication and transport

The following table lists how the Command Handler connects to the Connect
REST API.

| Method                                                    | Supported   | Notes                                           |
|-----------------------------------------------------------|-------------|-------------------------------------------------|
| No authentication, over HTTP or HTTPS                     | Yes         | Use HTTPS in production.                        |
| HTTP Basic authentication                                 | Yes         | You provide a credential file for each cluster. |
| HTTPS with a custom certificate authority (CA)            | Yes         | You provide a trusted CA file for each cluster. |
| mutual TLS (mTLS) authentication                          | No          |                                                 |
| RBAC through Metadata Service (MDS), using a bearer token | No          |                                                 |
| OAuth                                                     | No          |                                                 |

The Command Handler doesn’t discover Connect clusters automatically. You
declare each cluster that Confluent Cloud can manage.

## Enable the Command Handler

The USM Agent package installs the Command Handler but doesn’t start
it. You enable it by declaring the Connect clusters to manage in your
deployment tool. Declare one entry for each cluster.

<a id="usm-command-handler-cfk"></a>

### Confluent for Kubernetes

Starting in CFK 3.4.0, you enable the Command Handler by adding
`spec.commandHandler` to the `USMAgent` custom resource (CR) and applying
it with `kubectl apply`. CFK then runs an extra container,
`usm-agent-command-handler`, in the USM Agent pod. This requires the
3.4.0 init container.

In the simplest configuration, reference the Connect CR by name. CFK reads
the REST endpoint and group ID from the Connect CR.

```yaml
kind: USMAgent
spec:
  commandHandler:
    connectClusters:
      - connectClusterRef:
          name: connect
```

To override the discovered values or to set authentication and TLS, add the
optional `groupId` and `connectRest` fields. CFK supports no
authentication and `basic` authentication. For basic authentication, store
the credentials in a secret in a `basic.txt` file with `username=<user>` and
`password=<password>` lines. For TLS, store the PEM CA certificate that signed
the Connect certificate in a secret under `ca.crt` or `cacerts.pem`.

Changing `spec.commandHandler`, or a secret that it references, rolls the
USM Agent pod, which restarts the proxy along with the Command Handler.
If you remove `spec.commandHandler`, the pod rolls back to the agent alone.

For all fields and the validation rules, see [Confluent for Kubernetes](https://docs.confluent.io/operator/current/co-configure-usm.html#configure-the-command-handler-for-connector-management).

### Command Handler properties

CFK generates the Command Handler configuration from your custom resource. You
can tune the following properties with
`spec.commandHandler.configOverrides.server`.

| Property                                               | Default   | Description                                                                                                          |
|--------------------------------------------------------|-----------|----------------------------------------------------------------------------------------------------------------------|
| `confluent.usm-agent.command-handler.poll.interval.ms` | `5000`    | How often the Command Handler polls Confluent Cloud for operations, in<br/>milliseconds. `5000` is also the minimum. |
| `confluent.usm-agent.command-handler.log.level`        | `info`    | `debug`, `info`, `warn`, or `error`.                                                                                 |
| `confluent.usm-agent.command-handler.log.format`       | `json`    | `json` or `text`.                                                                                                    |

### Manage connectors from only one interface

Use a single interface to manage each connector. What you need to do depends on
how you created the connector:

* Created from Confluent Cloud. You don’t need to annotate any custom resource. The
  connector has no `Connector` CR, and CFK doesn’t touch connectors that
  have no `Connector` CR, so you can manage it freely from Confluent Cloud.
* Created from a `Connector` CR in CFK. CFK treats the CR as the
  source of truth and keeps reconciling the connector to match it. If you change
  or delete the connector from Confluent Cloud, CFK reverts the change. For example, a
  connector that you delete from Confluent Cloud is re-created while its `Connector`
  CR exists. To manage the connector from Confluent Cloud, annotate the `Connector`
  CR with `platform.confluent.io/block-reconcile=true`.

For instructions, see [Manage Connectors](https://docs.confluent.io/operator/current/co-manage-connectors.html).

## Enable connector actions in Confluent Cloud

After the Command Handler is running, enable connector actions for each
Connect cluster in Confluent Cloud. Confluent Cloud doesn’t send operations to a cluster
until you do. For steps, and for how to update existing automation to use the
Confluent Cloud endpoint, see [Manage self-managed connectors running against Apache
Kafka clusters on Confluent Cloud](https://docs.confluent.io/cloud/current/usm/connectors-cloud-kafka/manage-connectors-cloud-kafka.html).

## Troubleshoot the Command Handler

If a connector operation in Confluent Cloud fails with **Errors while trying to update
connector state/configuration**, check the following:

1. Confirm that the Command Handler process is running. In CFK, check that the
   Command Handler container in the agent pod is running and not restarting.
2. Confirm that connector actions are enabled for the Connect cluster in
   Confluent Cloud.
3. Confirm that the cluster is declared in your custom resource or inventory,
   and that its group ID matches the value you used to register it.
4. Confirm that the agent host or pod can reach the `url` of the Connect
   REST API and, for HTTPS, that the trusted CA file verifies the server
   certificate.
5. Confirm that the credential file for a `basic` cluster contains valid
   credentials.
6. Confirm that the agent’s service account has the `USMAgent` role.

If Connect rejects an operation, the error message from Connect appears
in Confluent Cloud.

To diagnose the Command Handler itself, use the following:

* **Logs**. Run `kubectl logs <usm_agent_pod> -c usm-agent-command-handler`.
* **Health**. The Command Handler serves `/healthz` on port 9930 and the
  status of each cluster on `/statusz` on port 9920. Port 9920 is available
  only on the loopback interface, so use `kubectl port-forward` to reach it.
* **Debug logging**. Set `confluent.usm-agent.command-handler.log.level` to
  `debug`. The Command Handler reads this property only at startup, so the
  change restarts it.

## Limitations

The Command Handler has the following limitations in this release:

* It manages only Connect clusters that use a Kafka cluster running on
  Confluent Cloud as the metadata cluster.
* It doesn’t support mTLS, RBAC through Metadata Service (MDS), or OAuth when it
  connects to the Connect REST API.
* Any configuration change restarts the Command Handler. With CFK, it also
  rolls the agent pod.
* In Confluent Cloud Console, you can pause and resume connectors. Restarting,
  deleting, creating, and updating connectors is available only through the
  Confluent Cloud API.

## Related content

* [Unified Stream Manager overview](../overview.md#usm-overview)
* [USM Agent Networking Configuration](networking.md#usm-networking)
* [USM Agent: Sizing, High Availability, and Monitoring](sizing.md#agent-sizing)
* [USM Agent Support Policy](../support-policy.md#usm-compatibility)
* [Manage self-managed connectors on Confluent Cloud](https://docs.confluent.io/cloud/current/usm/connectors-cloud-kafka/manage-connectors-cloud-kafka.html)
* [Control access to self-managed connectors with granular RBAC](https://docs.confluent.io/cloud/current/usm/connectors-cloud-kafka/connectors-rbac-cloud-kafka.html)
* [Monitor self-managed connectors running against Apache Kafka clusters on
  Confluent Cloud](https://docs.confluent.io/cloud/current/usm/connectors-cloud-kafka/monitor-connectors-cloud-kafka.html)