Configure and Deploy Unified Stream Manager Using Ansible Playbooks for Confluent Platform

Confluent Unified Stream Manager connects customer-managed on-premises clusters with Confluent Cloud to enable Confluent Cloud features for Confluent Platform clusters. The Unified Stream Manager Agent acts as a centralized proxy/gateway for Kafka, and Ansible Playbooks for Confluent Platform (Confluent Ansible) acts as a tool to deploy the Unified Stream Manager Agent in a virtual environment. The Ansible roles and playbooks automate Unified Stream Manager Agent deployment, configuration, TLS setup, authentication, credential handling, health checks, and integration with other Confluent Platform components (Kafka, KRaft, and Connect).

This topic presents the steps and guidance for deploying Unified Stream Manager with Confluent Ansible. It is part of the Registering your Confluent Platform Kafka cluster in Confluent Cloud process. Review the steps described in the registration topic before proceeding with Unified Stream Manager deployment.

The high-level workflow to deploy Unified Stream Manager with Confluent Ansible is as follows:

Review the prerequisites and considerations.
Register your Confluent Platform cluster in Confluent Cloud.
Configure and deploy Unified Stream Manager Agent.
Complete the registration process for the Unified Stream Manager Agent.

Requirements and considerations

Before deploying Unified Stream Manager with Confluent Ansible, ensure you have reviewed and addressed the following prerequisites and the key considerations:

Use the versions that support your Confluent Platform and Unified Stream Manager requirements.
- Confluent Platform version 8.1.x and later
- Confluent Ansible version 8.1.x and later
- Unified Stream Manager version 1.0.0 and later
Confluent Ansible supports the same versions of Unified Stream Manager as the corresponding Confluent Platform version supports.
Unified Stream Manager Agent to Confluent Cloud communication only supports basic authentication.
Unified Stream Manager Agent to Confluent Platform components communication supports basic authentication or mTLS. Both together are not supported.
IMPORTANT: Configure appropriate networking and authentication to ensure that the host running the Unified Stream Manager Agent is only reachable by the Confluent Platform components, such as Kafka and Connect, which need to send data as part of Unified Stream Manager.

Configure and deploy Unified Stream Manager Agent

You can use the sample inventory files for the Unified Stream Manager Agent in the following GitHub repository as references.

https://github.com/confluentinc/cp-ansible/blob/8.1.0-post/docs/sample_inventory/usm/

To configure and deploy Unified Stream Manager Agent:

Update your existing inventory file to add the Unified Stream Manager Agent host.

For example:
```
usm_agent:
  hosts:
    usm-agent1.confluent.io:
```
Configure the Unified Stream Manager Agent Confluent Cloud settings in the inventory file.
```
all:
  vars:
    ccloud_endpoint:         --- [1]
    ccloud_environment_id:   --- [2]
    ccloud_credential:       --- [3]
      username:
      password:
```
The values are available in the output file generated when you perform the first step in the registration process. See Generate the configuration file.
- [1] Specify the Confluent Cloud endpoint. Use the FRONTDOOR_URL value in the generated output file, and append the port, for example, https://api.us-west-2.aws.confluent.cloud:443.
- [2] Specify the Confluent Cloud Environment ID.
- [3] Specify the Confluent Cloud credentials.
Configure the Unified Stream Manager Agent authentication settings.
Configure other settings for the Unified Stream Manager Agent in the inventory file.

Validate Unified Stream Manager Agent configuration.

ansible-playbook -i hosts.yml confluent.platform.all --tags=validate_usm_agent_configs

Deploy the Unified Stream Manager Agent.

Once the inventory file is updated with the Unified Stream Manager Agent host and the required configurations, use the Confluent Ansible playbook to deploy the Unified Stream Manager Agent.

For metadata and metrics to flow from Kafka, KRaft, and Connect clusters to the Unified Stream Manager Agent, the following Confluent Platform components also need to be redeployed:
- KRaft controller
- Kafka brokers
- Connect
Deploy the Unified Stream Manager Agent and the above-mentioned Confluent Platform components together using one of the two methods below. The playbooks ensure that the Unified Stream Manager Agent is deployed first, followed by the other components in the correct order, to prevent “connection refused” errors.

To install all components together:
```
ansible-playbook -i hosts.yml confluent.platform.all
```
To install only Unified Stream Manager Agent and related Confluent Platform components together:
```
ansible-playbook -i hosts.yml confluent.platform.all --tags=usm_agent,kafka_controller,kafka_broker,kafka_connect
```

Authentication configurations

This section describes the authentication configuration and examples for Unified Stream Manager Agent.

The following variables are used to specify the authentication mechanism for communication between Confluent Platform components and Unified Stream Manager Agent:

usm_agent_basic_auth_enabled
usm_agent_ssl_enabled
usm_agent_ssl_mutual_auth_enabled

When all these variables are set to false (default values), this enables unauthenticated communication between Confluent Platform components and Unified Stream Manager Agent.

Basic authentication

To enable Basic authentication, set the following variables:

usm_agent_basic_auth_enabled: true
usm_agent_basic_users: - Required for basic authentication.
usm_agent_client_username and usm_agent_client_password - Client credentials for Confluent Platform components to authenticate to Unified Stream Manager Agent when basic authentication is enabled.

For example:

usm_agent_basic_auth_enabled: true
usm_agent_ssl_enabled: false  # To enable TLS encryption with Basic authentication, set this to true

usm_agent_basic_users:
  admin:
    principal: admin
    password: admin-secret
  user1:
    principal: user1
    password: user1-secret
  user2:
    principal: user2
    password: user2-secret

usm_agent_client_username: "user1"
usm_agent_client_password: "user1-secret"

Mutual TLS (mTLS) authentication

To enable mTLS authentication, set the following variables:

usm_agent_basic_auth_enabled: false
usm_agent_ssl_enabled: true
usm_agent_ssl_mutual_auth_enabled: true

Note

For TLS/mTLS mode, Confluent Ansible supports generating certificates similar to other Confluent Platform components. For more information, see Configure Encryption for Confluent Platform using Ansible Playbooks.

General configurations

The following are example settings for the variables that you can specify in the inventory file for configuring the Unified Stream Manager Agent.

Port configurations:

usm_agent_admin_port: 9901
usm_agent_controlplane_port: 9999
usm_agent_dataplane_port: 10000
usm_agent_listener_monitoring_port: 9910

Client Username & Password configurations (to define different basic authentication credentials for individual Confluent Platform components):

These variables default to usm_agent_client_* values but can also be defined individually.

kafka_broker_usm_agent_username: "{{ usm_agent_client_username }}"
kafka_broker_usm_agent_password: "{{ usm_agent_client_password }}"
kafka_controller_usm_agent_username: "{{ usm_agent_client_username }}"
kafka_controller_usm_agent_password: "{{ usm_agent_client_password }}"
kafka_connect_usm_agent_username: "{{ usm_agent_client_username }}"
kafka_connect_usm_agent_password: "{{ usm_agent_client_password }}"

Health Check configuration:

usm_agent_health_check_enabled: true
usm_agent_health_check_retries: 3
usm_agent_health_check_delay: 30

Logrotate configurations:

logrotate keeps the log file size under the specified limit.

The example below specifies that a maximum of five log files of size 10 MB each (total size = 50 MB) are allowed, after which the oldest log file is deleted automatically. These are the default values in Confluent Ansible.

You can alter these values according to your requirements. For more information, see logrotate(8) - Linux man page.

usm_agent_logrotate_enabled: true
usm_agent_logrotate_config:
  # Size limit for rotation (e.g., "10M", "100M", "1G")
  size: "10M"
  # Number of rotated files to keep
  rotate: 5

Other variables:

For a full list of Unified Stream Manager Agent variables, such as SSL-related variables, and their descriptions, see Unified Stream Manager Agent variables.

Register your Confluent Platform clusters in Confluent Cloud

This section describes the steps to complete registering the Confluent Platform clusters in Confluent Cloud.

Register your Confluent Platform Kafka cluster in Confluent Cloud.

You can use the following options to retrieve the Kafka cluster ID that is required to register the Kafka cluster in Confluent Cloud:
- If you have the Confluent CLI installed, use the command confluent kafka cluster list as described in the above registration topic.
- Use the following command, and fetch the Cluster ID value:
```
sudo cat /var/lib/kafka/data/meta.properties  | grep cluster.id
```
Register your Confluent Platform Connect cluster in Confluent Cloud.

You can use the following options to retrieve the Connect cluster ID (also known as the group ID) that is required to register the Connect cluster in Confluent Cloud:
- If you have the Confluent CLI installed, use the command confluent connect cluster list as described in the above registration topic.
- Use the following command, and fetch the Connect Cluster ID value:
```
sudo cat /etc/kafka/connect-distributed.properties | grep group.id
```