Install Confluent Platform with Ansible Playbooks

This topic describes the installation steps for Confluent Platform.

Install all Confluent Platform components

1. Configure required settings in the hosts.yml file as described in Configure Ansible Playbooks for Confluent Platform.

2. (Optional) Validate hosts before installing Confluent Platform:

   ansible-playbook -i hosts.yml confluent.platform.validate_hosts
   

   To validate hosts for an individual component, use the --tag flag as below:

   ansible-playbook -i hosts.yml confluent.platform.validate_hosts --tags=<component-name>
   

   <component-name> can be kafka_broker, kafka_controller, schema_registry, kafka_rest, kafka_connect, ksql, or control_center_next_gen.

3. Install or update Control Center.

4. Install or update all Confluent Platform:

   ansible-playbook -i hosts.yml confluent.platform.all
   

Install individual Confluent Platform components

You can use the Ansible tags to install individual components. You must follow the correct order given below to satisfy the dependencies among the components. For example, Kafka brokers will not install until the certificate authority is generated and the KRaft controller is installed.

Similarly, when you update a component section in the hosts.yml file, you can optionally update the component and the preceding components in the order.

1. Generate the certificate authority (CA) if TLS encryption is enabled and using self-signed certificates. Additionally, a key pair for the MDS token, a private key and a public certificate, will be generated.

   ansible-playbook -i hosts.yml confluent.platform.all --tags=certificate_authority
   

2. Install Control Center.

3. Install KRaft controller.

   ansible-playbook -i hosts.yml confluent.platform.all --tags=kafka_controller
   

4. Install Kafka.

   ansible-playbook -i hosts.yml confluent.platform.all --tags=kafka_broker
   

5. Install Schema Registry, REST Proxy, Connect, and ksqlDB in any order.

   ansible-playbook -i hosts.yml confluent.platform.all --tags=schema_registry
   ansible-playbook -i hosts.yml confluent.platform.all --tags=kafka_rest
   ansible-playbook -i hosts.yml confluent.platform.all --tags=kafka_connect
   ansible-playbook -i hosts.yml confluent.platform.all --tags=ksql
   

Install Confluent Control Center

Confluent Control Center is available as a separate package and follows independent versions.

Control Center requires the bcrypt function. Install bcrypt with the following command on the control node:

pip install bcrypt

After you prepare the host and configure, run the following command to install Control Center:

ansible-playbook -i hosts.yml confluent.platform.all --tags=control_center_next_gen

The Control Center package includes Prometheus and Alert Manager.

Now when you install Kafka and KRaft as described above in Install all Confluent Platform components or Install individual Confluent Platform components, Kafka and KRaft start sending metrics to Control Center, and you will see the metrics in Control Center.

Install Confluent Platform without root access

Ansible Playbooks for Confluent Platform (Confluent Ansible) can install and run Confluent Platform as an unprivileged user. Set rootless_enabled: true in your inventory.

Note

Rootless deployment requires Confluent Ansible 7.6.14, 7.7.12, 7.8.11, 7.9.10, 8.0.8, 8.1.6, 8.2.4, 8.3.2, 8.4.0, or later. Use the latest patch release for your Confluent Ansible minor version. Earlier patch releases do not include rootless_enabled support. This requirement is about the Confluent Ansible patch version and is independent of which Confluent Platform version you install.

A rootless deployment has two phases:

1. A one-time, root-only bootstrap that creates the deploy user, enables systemd linger for that user, and optionally installs a JDK and the other prerequisite packages. Either Confluent Ansible or an administrator can perform this step. You do not grant Confluent Ansible root or sudo access again afterward.

2. The rootless deploy, which runs entirely as the unprivileged deploy user and installs, configures, and starts Confluent Platform.

Note

FIPS setup and the Unified Stream Manager agent are out of scope for rootless deployment.

Complete the following steps to deploy Confluent Platform without root access:

1. Configure your inventory for a rootless deployment.

   Start from the non-root sample inventory. At minimum, set:

   all:
     vars:
       ansible_user: cp-user
       ansible_become: false
       rootless_enabled: true
       deployment_user: cp-user
       deployment_group: cp-user
       rootless_deployment_path: /home/cp-user/cp-data
       installation_method: archive
   

   With deployment_user, deployment_group, and rootless_deployment_path set, Confluent Ansible automatically derives every other user, group, and directory path under rootless_deployment_path. This includes logs, data, SSL, CLI, and the JMX exporter.

2. Run the one-time bootstrap.

   An administrator with root or sudo access runs this step once, using either of the following options.

   • Bootstrap with Confluent Ansible:

     ansible-playbook -i hosts.yml confluent.platform.rootless_bootstrap -e ansible_user=ec2-user
     

     ansible_user is the sudo user Confluent Ansible connects as to run the bootstrap and is unrelated to deployment_user, the unprivileged user the bootstrap creates. Add -e rootless_install_packages=true to also install a JDK and the other prerequisite packages. This step is idempotent.

   • Bootstrap manually, without Confluent Ansible: Use this option if your security policy does not allow granting Confluent Ansible sudo access. An administrator with sudo performs the following steps by hand, out of band, before you run the rootless deploy:

     # Create the deployment group and the deploy user with its home directory.
     sudo groupadd <deployment_group>
     sudo useradd -m -g <deployment_group> <deployment_user>
     
     # Install the authorized_keys file so Confluent Ansible can
     # connect as the deploy user.
     sudo mkdir -p /home/<deployment_user>/.ssh
     sudo cp ~/.ssh/authorized_keys /home/<deployment_user>/.ssh/authorized_keys
     sudo chown -R <deployment_user>:<deployment_group> /home/<deployment_user>/.ssh
     sudo chmod 700 /home/<deployment_user>/.ssh
     sudo chmod 600 /home/<deployment_user>/.ssh/authorized_keys
     
     # Create the deployment directory and give the deploy user ownership.
     sudo mkdir <rootless_deployment_path>
     sudo chown <deployment_user>:<deployment_group> <rootless_deployment_path>
     
     # Enable systemd linger so the deploy user's systemd --user units
     # keep running after logout and survive a reboot.
     sudo loginctl enable-linger <deployment_user>
     
     # Install a supported JDK (full JDK, not JRE).
     sudo dnf install java-17-openjdk       # RHEL
     sudo apt-get install openjdk-17-jdk    # Debian/Ubuntu
     # Or unpack any JDK tarball (OpenJDK, Zulu, Temurin, or Oracle) and
     # set custom_java_path in the inventory instead.
     
     # Install the certificate and Python tools the deploy needs.
     sudo dnf install openssl rsync unzip python3-pip python3-pyyaml     # RHEL
     sudo apt-get install openssl rsync unzip python3-pip python3-yaml  # Debian/Ubuntu
     
     # Debian/Ubuntu only: systemd --user requires a per-user D-Bus session.
     sudo apt-get install -y dbus-user-session
     

     The rootless deploy in the next step does not care which of these two options created the deploy user, directory, and linger state, only that they exist.

3. Run the rootless deploy as the deploy user, with no root access:

   ansible-playbook -i hosts.yml confluent.platform.all
   

   Confluent Ansible generates and starts a systemd --user unit for each component (cp-<component>.service) in dependency order.

4. (Optional) Manage and verify the deployment as the deploy user:

   export XDG_RUNTIME_DIR=/run/user/$(id -u)
   systemctl --user status 'cp-*'
   systemctl --user restart cp-kafka_broker
   journalctl --user -u cp-kafka_broker
   

For the full validated reference, including the preceding manual bootstrap steps and the tested configuration matrix, see Rootless (non-root) deployment in the Confluent Ansible repository.

Next step

• Update a Running Confluent Platform Configuration Using Ansible Playbooks.

• Troubleshoot Ansible Playbooks for Confluent Platform.

• Upgrade Confluent Platform with Ansible Playbooks.