# Configure and Deploy Unified Stream Manager Using Ansible Playbooks for Confluent Platform Confluent Unified Stream Manager connects customer-managed on-premises clusters with Confluent Cloud to enable Confluent Cloud features for Confluent Platform clusters. The Unified Stream Manager Agent acts as a centralized proxy/gateway for Kafka, and Ansible Playbooks for Confluent Platform (Confluent Ansible) acts as a tool to deploy the Unified Stream Manager Agent in a virtual environment. The Ansible roles and playbooks automate Unified Stream Manager Agent deployment, configuration, TLS setup, authentication, credential handling, health checks, and integration with other Confluent Platform components (Kafka, KRaft, and Connect). This topic presents the steps and guidance for deploying Unified Stream Manager with Confluent Ansible. It is part of the [Registering your Confluent Platform Kafka cluster in Confluent Cloud](http://docs.confluent.io/platform/current/usm/get-started.html#registration-process-overview) process. Review the steps described in the registration topic before proceeding with Unified Stream Manager deployment. The high-level workflow to deploy Unified Stream Manager with Confluent Ansible is as follows: 1. [Review the prerequisites and considerations](#ansible-usm-requirements). 2. [Register your Confluent Platform cluster in Confluent Cloud](http://docs.confluent.io/platform/current/usm/get-started.html#registration-process-overview). 3. [Configure and deploy Unified Stream Manager Agent](#ansible-usm-configure). 4. [Complete the registration process for the Unified Stream Manager Agent](#ansible-usm-registration). ## Requirements and considerations Before deploying Unified Stream Manager with Confluent Ansible, ensure you have reviewed and addressed the following prerequisites and the key considerations: - Use the versions that support your Confluent Platform and Unified Stream Manager requirements. * Confluent Platform version 8.1.x and later * Confluent Ansible version 8.1.x and later * Unified Stream Manager version 1.0.0 and later Confluent Ansible supports the same versions of Unified Stream Manager as the corresponding Confluent Platform version supports. #### IMPORTANT Unified Stream Manager Agent cannot be deployed on RHEL 8. You must deploy Unified Stream Manager Agent to a host running RHEL 9 or later. - Unified Stream Manager Agent to Confluent Cloud communication only supports basic authentication. - Unified Stream Manager Agent to Confluent Platform components communication supports basic authentication or mTLS. Both together are not supported. - **IMPORTANT**: Configure appropriate networking and authentication to ensure that the host running the Unified Stream Manager Agent is only reachable by the Confluent Platform components, such as Kafka and Connect, which need to send data as part of Unified Stream Manager. ## Configure and deploy Unified Stream Manager Agent You can use the sample inventory files for the Unified Stream Manager Agent in the following GitHub repository as references. ```bash https://github.com/confluentinc/cp-ansible/blob/8.3.0-post/docs/sample_inventories/usm_agent/ ``` To configure and deploy Unified Stream Manager Agent: 1. Update your existing inventory file to add the Unified Stream Manager Agent host. For example: ```yaml usm_agent: hosts: usm-agent1.confluent.io: ``` 2. Configure the Unified Stream Manager Agent Confluent Cloud settings in the inventory file. ```yaml all: vars: ccloud_endpoint: --- [1] ccloud_environment_id: --- [2] ccloud_credential: --- [3] username: password: ``` The values are available in the output file generated when you perform the first step in the registration process. See [Generate the configuration file](https://docs.confluent.io/cloud/current/usm/register/deploy-agent.html#generate-and-download-the-configuration-file). * [1] Specify the Confluent Cloud endpoint. Use the `FRONTDOOR_URL` value in the generated output file, and append the port, for example, `https://api.us-west-2.aws.confluent.cloud:443`. * [2] Specify the Confluent Cloud Environment ID. * [3] Specify the Confluent Cloud credentials. 3. [Configure the Unified Stream Manager Agent authentication settings](#ansible-usm-authentication). 4. [Configure other settings for the Unified Stream Manager Agent in the inventory file](#ansible-usm-configurations). 5. Validate Unified Stream Manager Agent configuration. ```bash ansible-playbook -i hosts.yml confluent.platform.all --tags=validate_usm_agent_configs ``` 6. Deploy the Unified Stream Manager Agent. Once the inventory file is updated with the Unified Stream Manager Agent host and the required configurations, use the Confluent Ansible playbook to deploy the Unified Stream Manager Agent. For metadata and metrics to flow from Kafka, KRaft, and Connect clusters to the Unified Stream Manager Agent, the following Confluent Platform components also need to be redeployed: - KRaft controller - Kafka brokers - Connect Deploy the Unified Stream Manager Agent and the above-mentioned Confluent Platform components together using one of the two methods below. The playbooks ensure that the Unified Stream Manager Agent is deployed first, followed by the other components in the correct order, to prevent “connection refused” errors. To install all components together: ```bash ansible-playbook -i hosts.yml confluent.platform.all ``` To install only Unified Stream Manager Agent and related Confluent Platform components together: ```bash ansible-playbook -i hosts.yml confluent.platform.all --tags=usm_agent,kafka_controller,kafka_broker,kafka_connect ``` ### Authentication configurations This section describes the authentication configuration and examples for Unified Stream Manager Agent. The following variables are used to specify the authentication mechanism for communication between Confluent Platform components and Unified Stream Manager Agent: * `usm_agent_basic_auth_enabled` * `usm_agent_ssl_enabled` * `usm_agent_ssl_mutual_auth_enabled` When all these variables are set to `false` (default values), this enables unauthenticated communication between Confluent Platform components and Unified Stream Manager Agent. **Basic authentication** To enable Basic authentication, set the following variables: * `usm_agent_basic_auth_enabled: true` * `usm_agent_basic_users:` - Required for basic authentication. * `usm_agent_client_username` and `usm_agent_client_password` - Client credentials for Confluent Platform components to authenticate to Unified Stream Manager Agent when basic authentication is enabled. For example: ```yaml usm_agent_basic_auth_enabled: true usm_agent_ssl_enabled: false # To enable TLS encryption with Basic authentication, set this to true usm_agent_basic_users: admin: principal: admin password: admin-secret user1: principal: user1 password: user1-secret user2: principal: user2 password: user2-secret usm_agent_client_username: "user1" usm_agent_client_password: "user1-secret" ``` **Mutual TLS (mTLS) authentication** To enable mTLS authentication, set the following variables: * `usm_agent_basic_auth_enabled: false` * `usm_agent_ssl_enabled: true` * `usm_agent_ssl_mutual_auth_enabled: true` #### NOTE For TLS/mTLS mode, Confluent Ansible supports generating certificates similar to other Confluent Platform components. For more information, see [Configure Encryption for Confluent Platform using Ansible Playbooks](https://docs.confluent.io/ansible/current/ansible-encrypt.html). ### General configurations The following are example settings for the variables that you can specify in the inventory file for configuring the Unified Stream Manager Agent. **Port configurations:** ```yaml usm_agent_admin_port: 9901 usm_agent_controlplane_port: 9999 usm_agent_dataplane_port: 10000 usm_agent_listener_monitoring_port: 9910 ``` **Client Username & Password configurations** (to define different basic authentication credentials for individual Confluent Platform components): These variables default to `usm_agent_client_*` values but can also be defined individually. ```yaml kafka_broker_usm_agent_username: "{{ usm_agent_client_username }}" kafka_broker_usm_agent_password: "{{ usm_agent_client_password }}" kafka_controller_usm_agent_username: "{{ usm_agent_client_username }}" kafka_controller_usm_agent_password: "{{ usm_agent_client_password }}" kafka_connect_usm_agent_username: "{{ usm_agent_client_username }}" kafka_connect_usm_agent_password: "{{ usm_agent_client_password }}" ``` **Health Check configuration:** ```yaml usm_agent_health_check_enabled: true usm_agent_health_check_retries: 3 usm_agent_health_check_delay: 30 ``` **Logrotate configurations:** `logrotate` keeps the log file size under the specified limit. The example below specifies that a maximum of five log files of size 10 MB each (total size = 50 MB) are allowed, after which the oldest log file is deleted automatically. These are the default values in Confluent Ansible. You can alter these values according to your requirements. For more information, see [logrotate(8) - Linux man page](https://linux.die.net/man/8/logrotate). ```yaml usm_agent_logrotate_enabled: true usm_agent_logrotate_config: # Size limit for rotation (e.g., "10M", "100M", "1G") size: "10M" # Number of rotated files to keep rotate: 5 ``` **Client metrics configuration:** To enable client metrics in Unified Stream Manager, set the following variable. The variable is disabled by default. ```yaml usm_client_metrics_enabled: true ``` **Other variables:** For a full list of Unified Stream Manager Agent variables, such as SSL-related variables, and their descriptions, see [Unified Stream Manager Agent variables](https://github.com/confluentinc/cp-ansible/blob/8.1.x/docs/sample_inventories/usm_agent/README_USM_AGENT.md). ## Register your Confluent Platform clusters in Confluent Cloud This section describes the steps to complete registering the Confluent Platform clusters in Confluent Cloud. 1. [Register your Confluent Platform Kafka cluster in Confluent Cloud](http://docs.confluent.io/cloud/current/usm/register/complete-registration.html#complete-registration). You can use the following options to retrieve the Kafka cluster ID that is required to register the Kafka cluster in Confluent Cloud: * If you have the Confluent CLI installed, use the command `confluent kafka cluster list` as described in the above [registration topic](http://docs.confluent.io/cloud/current/usm/register/complete-registration.html#complete-registration). * Use the following command, and fetch the Cluster ID value: ```bash sudo cat /var/lib/kafka/data/meta.properties | grep cluster.id ``` 2. [Register your Confluent Platform Connect cluster in Confluent Cloud](http://docs.confluent.io/cloud/current/usm/register-connect.html). You can use the following options to retrieve the Connect cluster ID (also known as the group ID) that is required to register the Connect cluster in Confluent Cloud: * If you have the Confluent CLI installed, use the command `confluent connect cluster list` as described in the above [registration topic](http://docs.confluent.io/cloud/current/usm/register-connect.html). * Use the following command, and fetch the Connect Cluster ID value: ```bash sudo cat /etc/kafka/connect-distributed.properties | grep group.id ```