Configure Ansible Playbooks for Confluent Platform

This topic describes commonly used settings to configure Ansible Playbooks for Confluent Platform (Confluent Ansible).

Set Ansible host variables

Once you configure the hosts in your inventory file and verify the connections, you can set variables in the inventory which describe your desired Confluent Platform configuration.

Review the commented out variables with the example inventory file at:

https://github.com/confluentinc/cp-ansible/blob/8.4.0-post/docs/hosts_example.yml

For a full list of supported variables, see the Ansible variable file at:

https://github.com/confluentinc/cp-ansible/blob/8.4.0-post/docs/VARIABLES.md

You can apply variables to all hosts or to specific hosts.

In the below example, all hosts get the ssl_enabled=true variable set:

all:
  vars:
    ssl_enabled: true

We generally recommend applying variables in the all group so that each host is aware of how the Confluent Platform is configured as a whole.

You can also make use of group_vars and host_vars directories that are located next to the inventory file to pass variables. See Ansible Directory Layout.

Additionally, consider saving sensitive variables in their own variables file in the above structure and use Ansible Vault to encrypt the variable files.

The remainder of this document describes how to configure Confluent Platform using Ansible variables.

Set the Confluent Platform version

confluent_package_version is required starting with Confluent Ansible 7.6.14, 7.7.12, 7.8.11, 7.9.10, 8.0.8, 8.1.6, 8.2.4, 8.3.2, and 8.4.0. Confluent Ansible no longer ships a hard-coded default Confluent Platform version.

Add confluent_package_version to your inventory and set it to the Confluent Platform version you want to install. See the Confluent Ansible and Confluent Platform compatibility matrix in Prerequisites for Installing Confluent Platform with Ansible Playbooks. If not set, the playbook fails early.

all:
  vars:
    confluent_package_version: 8.3.2  # required: set to the CP version to install

Set Confluent Platform software installation method

Ansible Playbooks for Confluent Platform supports the following methods for installing Confluent Platform software onto host machines.

Package installation, using the packages hosted on packages.confluent.io

This is the default option. It requires internet connectivity from all hosts to packages.confluent.io. No inventory variables are required to use this method.

Package installation, using the packages hosted on your own RPM or DEB package repository

This option works for hosts that do not have outside internet connectivity. It requires you to pull the Confluent Platform packages and put them on your repository.

Set the following in your inventory file to use this method.

For packages on an RHEL/Centos host:

all:
  vars:
    repository_configuration: custom
    custom_yum_repofile_filepath: /tmp/my-repo.repo

For packages on a Debian host:

all:
  vars:
    repository_configuration: custom
    custom_apt_repo_filepath: /tmp/my-source.list

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Deploy Confluent Platform in Air-Gapped Environment Using Ansible Playbooks.

Tar installation, using the tarball hosted on packages.confluent.io

It requires internet connectivity from all hosts to packages.confluent.io.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive

Tar installation, using the tarball hosted on your own web server

This does not require outside internet connectivity, but does require you to pull the tarball and host it on a web server.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: <web server url>/path/confluent-8.3.0.tar.gz

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Deploy Confluent Platform in Air-Gapped Environment Using Ansible Playbooks.

Tar installation using the tarball placed on Ansible control node

This does not require outside internet connectivity, but requires you to pull and copy the tarball to the control node.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: /path/to/confluent-8.3.0.tar.gz
    confluent_archive_file_remote: false

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Deploy Confluent Platform in Air-Gapped Environment Using Ansible Playbooks.

Set custom component properties

When a configuration setting is not directly supported by Ansible Playbooks for Confluent Platform, you can use the custom property feature to configure Confluent Platform components.

Before you set a custom property variable, first check the Ansible variable file at the following location for an existing variable:

https://github.com/confluentinc/cp-ansible/blob/8.4.0-post/docs/VARIABLES.md

If you find an existing variable that directly supports the setting, use the variable in the inventory file instead of using a config override.

Configure the custom properties in the Ansible inventory file, hosts.yml, using the following dictionaries:

kafka_controller_custom_properties
kafka_broker_custom_properties
schema_registry_custom_properties
kafka_rest_custom_properties
kafka_connect_custom_properties
ksql_custom_properties
control_center_next_gen_custom_properties
kafka_connect_replicator_custom_properties
kafka_connect_replicator_consumer_custom_properties
kafka_connect_replicator_producer_custom_properties
kafka_connect_replicator_monitoring_interceptor_custom_properties

In the example below:

The num.io.threads property gets set in the Kafka properties file.
The confluent.controlcenter.ksql.default.advertised.url property gets set in the Control Center properties file.

Note that the default in the confluent.controlcenter.ksql.default.advertised.url property value is the name Control Center should use to identify the ksqlDB cluster.

all:
  vars:
    kafka_broker_custom_properties:
      num.io.threads: 15
    control_center_next_gen_custom_properties:
      confluent.controlcenter.ksql.url: http://ksql-external-dns:1234,http://ksql-external-dns:2345

Set custom properties on a specific host

You can configure a specific host with unique properties. Put the component properties block directly under the host.

In the example below, the broker.rack property is set to us-west-2a for the host, ip-192-24-10-207.us-west.compute.internal.

kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      kafka_broker_custom_properties:
        broker.rack: us-west-2a

Warning

Define each *_custom_properties variable only once, at a single level: all, group, or host. Ansible Playbooks for Confluent Platform does not merge dictionaries defined more than once for the same variable. Only the last definition takes effect.

For example, if kafka_broker_custom_properties is defined twice under all.vars with different properties in each, only the properties in the second definition apply. Any property set only in the first definition, such as log.dirs, is lost and reverts to its default value.

To set many properties for the same component, combine them into a single dictionary definition instead of defining the variable more than once.

Add Confluent license

To add a Confluent license key for Confluent Platform components, use a custom property for each Confluent Platform component in the hosts.yml file as following:

all:
  vars:
    kafka_broker_custom_properties:
     confluent.license: <license-key>
     kafka.rest.confluent.license.topic: "_confluent-command"
   schema_registry_custom_properties:
     confluent.license: <license-key>
   kafka_connect_custom_properties:
     confluent.license: <license-key>
   control_center_next_gen_custom_properties:
     confluent.license: <license-key>
   kafka_rest_custom_properties:
     confluent.license: <license-key>
   ksql_custom_properties:
     confluent.license: <license-key>

Note that Confluent Server (Kafka broker) contains Kafka REST Server, and this component also requires a valid license configuration. Set the kafka.rest.confluent.license.topic property to the _confluent-command topic that stores the Confluent license.

To add license to a connector, use the following config in the hosts.yaml file:

all:
  vars:
    kafka_connect_connectors:
    - name: sample-connector
      config:
        confluent.license: <license-key>

The following example adds a license key for Kafka and Schema Registry. The example creates a variable for the license key and uses the variable in the custom properties.

vars:
  confluent_license: asdfkjkadslkfjaslkdf
  kafka_broker_custom_properties:
    confluent.license: "{{ confluent_license }}"
    kafka.rest.confluent.license.topic: "_confluent-command"
  schema_registry_custom_properties:
    confluent.license: "{{ confluent_license }}"

For additional license configuration parameters you can set with the above custom properties, see License Configurations for Confluent Platform.

Configure Control Center

To learn about Control Center, see Confluent Control Center Overview.

The Control Center variables are defined in the following file in GitHub, prefixed with confluent_control_center_next_gen:

https://github.com/confluentinc/cp-ansible/blob/8.4.0-post/docs/VARIABLES.md

Override the default values for any of the variables you want to customize in the Control Center role section in the inventory file as below:

control_center_next_gen:
  vars:

The following are a few of the commonly customized variables. For the variables related to authentication, see Control Center authentication.

confluent_control_center_next_gen_package_version: Control Center does not follow the same versions as Confluent Platform. Check and update the confluent_control_center_next_gen_package_version variable in hosts.yml for the Control Center version you want to install.
control_center_next_gen_dependency_config_path: Prometheus and Alert Manager configuration file path. The default path is /opt/confluent-control-center/dependencies.

Control Center ships with a Prometheus version that has specific configurations to work with Control Center. By default, you cannot use any other Prometheus version besides that particular version, unless you configure control_center_next_gen_external_prometheus_enabled to use your own external Prometheus instead of the bundled one. For the full configuration, see Bring your own Prometheus (BYOP).

The Prometheus bundled with Control Center can not be used to ingest data to Grafana or any other tool in your environment.
control_center_next_gen_external_prometheus_enabled: Set to true to use your own external Prometheus instance instead of the bundled Prometheus and Alertmanager (BYOP). For the full configuration, see Bring your own Prometheus (BYOP).
control_center_next_gen_user: Set this variable to customize the Linux user that the Control Center service runs with. The default user is cp-control-center.
control_center_next_gen_custom_properties: Set custom Control Center properties.

control_center_next_gen_dependency_prometheus_service_environment_overrides: Overrides default environment variables to the Prometheus service.

For example:

control_center_next_gen_dependency_prometheus_service_environment_overrides:
  CONFIG_PATH: "{{ control_center_next_gen_dep_prometheus.config_path }}"
  # Data path environment variable is different between prometheus and alertmanager
  TSDB_PATH: "{{ control_center_next_gen_dep_prometheus.data_path }}"
  LOG_PATH: "{{ control_center_next_gen_dep_prometheus.log_path }}"
  METRICS_RETENTION_DAYS: "30d"

control_center_next_gen_dependency_alertmanager_service_environment_overrides: Overrides default environment variables to the Alertmanager service.
control_center_next_gen_dependency_alertmanager_web_external_url: The URL that Alertmanager uses in the View Alert links of its emails. Without this variable, the links point to the short hostname of the Alertmanager host. This variable overrides the ALERTMANAGER_WEB_EXTERNAL_URL environment variable that the alertmanager-start script reads. It requires Control Center 2.7.0 or later. If the URL includes a path, Alertmanager adds the path to the View Alert links but does not serve content under it.

For example:
```
control_center_next_gen:
  vars:
    control_center_next_gen_dependency_alertmanager_web_external_url: "https://alerts.example.com"
```

After configuring Control Center, install Control Center as described in Install Confluent Control Center.

Configure Kafka and KRaft to start sending data to Control Center as described in Configure Kafka and KRaft for Control Center.

For a sample inventory file for Control Center, see Confluent Ansible GitHub repo.

Control Center authentication

Confluent Ansible supports the following authentication methods for Control Center:

Plain
Basic
Basic with TLS
mTLS

Control Center can authenticate to the Prometheus and Alertmanager servers with the Basic, Basic with TLS, or mTLS authentication. Use the following variables to enable Control Center authentication to Prometheus and Alertmanager. Set the variable to true to enable a specific setting.

control_center_next_gen_dependency_prometheus_ssl_enabled
control_center_next_gen_dependency_prometheus_basic_auth_enabled
control_center_next_gen_dependency_prometheus_mtls_enabled
control_center_next_gen_dependency_alertmanager_ssl_enabled
control_center_next_gen_dependency_alertmanager_basic_auth_enabled
control_center_next_gen_dependency_alertmanager_mtls_enabled

For authentication configuration details, see Configure Authentication for Confluent Platform with Ansible Playbooks.

Bring your own Prometheus (BYOP)

Instead of the Prometheus and Alertmanager services bundled with Control Center, you can configure Control Center to use your own externally managed Prometheus instance.

Supported versions and compatibility

BYOP requires Control Center 2.7 and Confluent Ansible 8.4, or later. It also requires Prometheus 3.0 or later. Confluent Ansible rejects Control Center 2.6 and earlier when you enable BYOP.

BYOP supports only the following:

Metrics that Kafka brokers and KRaft controllers push to Prometheus over the OpenTelemetry Protocol (OTLP). Pull-based (scrape) ingestion of these metrics is not supported.
Native, vanilla Prometheus. Customized or vendor-specific Prometheus distributions are not tested and are not supported.
A single Prometheus URL. For high availability, see Configure high availability.

Note

Alerting is not supported when using BYOP. Only Prometheus metrics ingestion is supported. The bundled Alertmanager, and the alerting features it powers, are unavailable in this mode.

External Prometheus prerequisites

Before you configure Control Center to use an external Prometheus instance, prepare that Prometheus instance as follows:

Enable the Prometheus OTLP receiver by starting Prometheus with the --web.enable-otlp-receiver flag. Kafka brokers and KRaft controllers push their metrics to the external Prometheus instance over OTLP.

Configure resource-attribute promotion so the labels Control Center relies on are available on the ingested metrics. Prometheus 3.0 or later is required because Control Center depends on OTLP resource-attribute promotion. Add the following to your Prometheus configuration:

otlp:
  promote_resource_attributes:
    - host.hostname
    - java.version
    - kafka.broker.id
    - kafka.cluster.id
    - kafka.version
    - type
    - kafka.process.roles
    - kafka.process.roles.broker
    - kafka.process.roles.controller
    - kafka.kafka.node.id
    - kafka.broker.rack

Configure an out-of-order time window to accommodate metrics that arrive out of order. Add the following to your Prometheus configuration:
```
storage:
  tsdb:
    out_of_order_time_window: 10m
```
Load the Control Center Prometheus recording rules into your external Prometheus instance. The rules are in the recording_rules-generated.yml file. Copy the file from /etc/confluent-control-center/ on a Control Center host. For an archive installation, copy it from etc/confluent-control-center/ in the installation directory. Alternatively, copy the contents from Recording Rules for an External Prometheus.

If the recording rules are missing or the Prometheus version is earlier than 3.0, Control Center shows an error in its UI. Apply the recording rules file as provided, without modification. The number of rules can change in later Control Center versions, so reload the file when you upgrade Control Center.

The promoted labels apply to the whole Prometheus instance. If you share the instance with other tools, the labels also affect their metrics.

Connectivity and security

Control Center, the Kafka brokers, and KRaft controllers that report metrics to Prometheus must reach your external Prometheus endpoint over the network. The endpoint must be reachable from the Control Center host and from every broker and controller.

Three kinds of components connect to the external Prometheus. Control Center reads metrics with PromQL. Every Kafka broker and every KRaft controller pushes metrics over OTLP. Each one is a separate TLS client. In Confluent Ansible, one set of control_center_next_gen_dependency_prometheus_* variables configures both the Control Center read connection and the broker and controller push connections.

Confluent Ansible requires TLS for the connection to an external Prometheus. You can secure these connections in one of the following ways. Use at most one authentication mode. Do not combine Basic authentication with mTLS.

TLS only. Prometheus presents a server certificate, and every client verifies it with the certificate authority (CA) that signed it. Without Basic authentication, any client that can reach Prometheus can read and push metrics.
Basic authentication over TLS. In addition to TLS, every client sends a user name and password.
mTLS. In addition to TLS, every client presents its own certificate, which Prometheus verifies.

Place all certificate and key files on the Ansible control node. Confluent Ansible builds the truststores and keystores that are required.

Configure the following, as they apply to your setup:

The external Prometheus server certificate must have a Subject Alternative Name (SAN) that matches the host name in the Prometheus URL. If you use an IP address, the SAN must include that IP address. Otherwise, the TLS handshake fails.
Set control_center_next_gen_dependency_prometheus_provided_ca_cert_path to the CA certificate that signed the external Prometheus server certificate. Place this file on the Ansible control node. Confluent Ansible imports it into the Control Center, broker, and controller truststores so they trust the external Prometheus when reading and pushing metrics.
For Basic authentication, set control_center_next_gen_dependency_prometheus_basic_auth_enabled to true and define the credentials in control_center_next_gen_dependency_prometheus_basic_users.
For mTLS, set control_center_next_gen_dependency_prometheus_mtls_enabled to true and set control_center_next_gen_dependency_prometheus_provided_cert_path and control_center_next_gen_dependency_prometheus_provided_key_path to the client certificate and key. Place both files on the Ansible control node. The client certificate must be signed by a CA that the external Prometheus trusts. Confluent Ansible processes the key with openssl rsa using ssl_key_password when it is set, so the key must be either unencrypted or encrypted with that same password.

With mTLS, two different client certificates reach the external Prometheus. Control Center presents the client certificate you provide to read metrics. Brokers and controllers present their own node certificates, signed by the Confluent cluster CA, to push metrics. Configure the external Prometheus to trust both CAs for client authentication. Otherwise, the node push is rejected during the TLS handshake while Control Center can still read metrics, so metrics appear partially populated.

Confluent Ansible builds a keystore from the client certificate, gives it its own alias, and configures Control Center to present it, so you do not set an alias. If a client certificate has an extended key usage, it must include clientAuth. Prometheus rejects a certificate that allows only serverAuth, with the error x509: certificate specifies an incompatible key usage. A certificate with no extended key usage is accepted.

Configure TLS and authentication on Prometheus

Prometheus reads its TLS and authentication settings from the file that you pass with the --web.config.file flag.

For TLS, configure the server certificate:

tls_server_config:
  cert_file: /etc/prometheus/certs/server.crt
  key_file: /etc/prometheus/certs/server.key

For Basic authentication, also add a user with a bcrypt-hashed password:

basic_auth_users:
  c3: <bcrypt_hash_of_password>

For mTLS, set client_ca_file and require a verified client certificate:

tls_server_config:
  cert_file: /etc/prometheus/certs/server.crt
  key_file: /etc/prometheus/certs/server.key
  client_ca_file: /etc/prometheus/certs/client-ca-bundle.crt
  client_auth_type: RequireAndVerifyClientCert

The client_ca_file must contain every CA that signed a client certificate: the CA of the Control Center client certificate and the Confluent cluster CA that signed the broker and controller node certificates. If the file has only the Control Center CA, Prometheus rejects the broker and controller pushes, so dashboards miss broker and controller data.

Configure Control Center for BYOP

control_center_next_gen_external_prometheus_enabled: Set to true to skip installing the bundled Prometheus and Alertmanager on new installations. The default is false.
Configure the connection to your external Prometheus instance using the control_center_next_gen_dependency_prometheus_* variables such as host, port, TLS, CA certificate path, and Basic or mTLS credentials. These are the same variables that are used to connect to the bundled Prometheus.

For example, with Basic authentication over TLS:

all:
  vars:
    ssl_enabled: true

    control_center_next_gen_external_prometheus_enabled: true

    control_center_next_gen_dependency_prometheus_host: prometheus.customer.internal
    control_center_next_gen_dependency_prometheus_port: 9090
    control_center_next_gen_dependency_prometheus_ssl_enabled: true
    control_center_next_gen_dependency_prometheus_provided_ca_cert_path: /path/on/control-node/prometheus-ca.crt

    control_center_next_gen_dependency_prometheus_basic_auth_enabled: true
    control_center_next_gen_dependency_prometheus_basic_users:
      admin:
        principal: c3
        password: <external_prometheus_password>

For mTLS, enable mTLS and provide the Control Center client certificate and key, instead of the Basic authentication variables:

all:
  vars:
    ssl_enabled: true

    control_center_next_gen_external_prometheus_enabled: true

    control_center_next_gen_dependency_prometheus_host: prometheus.customer.internal
    control_center_next_gen_dependency_prometheus_port: 9090
    control_center_next_gen_dependency_prometheus_ssl_enabled: true
    control_center_next_gen_dependency_prometheus_provided_ca_cert_path: /path/on/control-node/prometheus-ca.crt

    control_center_next_gen_dependency_prometheus_mtls_enabled: true
    control_center_next_gen_dependency_prometheus_provided_cert_path: /path/on/control-node/c3-client.crt
    control_center_next_gen_dependency_prometheus_provided_key_path: /path/on/control-node/c3-client.key

For a sample BYOP inventory file, including the mTLS option, see external_prometheus.yml in the Confluent Ansible GitHub repo.

Migrate to BYOP

For new deployments, configure BYOP as described earlier from the start. No bundled Prometheus exists, so no migration is needed.

For existing deployments, move from the bundled Prometheus to BYOP as follows:

Prepare your external Prometheus instance as described in External Prometheus prerequisites.
Update your inventory file to set control_center_next_gen_external_prometheus_enabled: true and the external Prometheus connection variables described previously.
Re-run the playbook for all components, so that the brokers and controllers are also configured to push metrics to the external Prometheus:
```
ansible-playbook -i <inventory> confluent.platform.all
```
Optional: After Control Center reports the external Prometheus as online, remove the old bundled Prometheus and Alertmanager by following the steps in Manually clean up the bundled Prometheus and Alertmanager. The cleanup deletes the bundled Prometheus metric history, so back up the data first as described there.

If the external Prometheus endpoint is unreachable or misconfigured, Control Center cannot retrieve metrics and its monitoring views are unavailable until you restore connectivity. This does not affect other Control Center management functions, such as managing topics, schemas, and connectors.

Roll back to the bundled Prometheus

To roll back to the bundled Prometheus, set control_center_next_gen_external_prometheus_enabled to false (or remove it) and remove the external Prometheus connection variables. Then re-run the playbook for all components, confluent.platform.all.

If you did not run the cleanup, the bundled Prometheus uses its existing data after the rollback. Metrics from the period when Control Center used the external Prometheus are only in the external Prometheus.

Manually clean up the bundled Prometheus and Alertmanager

Enabling control_center_next_gen_external_prometheus_enabled stops new installations of the bundled Prometheus and Alertmanager. It does not remove an existing installation. After Control Center reports the external Prometheus as online, run the following steps on each Control Center host.

Warning

Step 3 deletes the metric history of the bundled Prometheus. Back up the data in step 1 before you run it.

Back up the old data:

sudo tar czf /root/bundled-monitoring-backup.tgz \
  /var/lib/confluent/control-center/prometheus \
  /var/lib/confluent/control-center/alertmanager

Stop and disable both services:

sudo systemctl stop prometheus alertmanager
sudo systemctl disable prometheus alertmanager

Remove their files:

sudo rm -rf /etc/systemd/system/prometheus.service.d \
            /etc/systemd/system/alertmanager.service.d \
            /var/lib/confluent/control-center/prometheus \
            /var/lib/confluent/control-center/alertmanager
sudo rm -f /opt/confluent-control-center/dependencies/prometheus-generated.yml \
           /opt/confluent-control-center/dependencies/web-config-prom.yml \
           /opt/confluent-control-center/dependencies/recording_rules-generated.yml \
           /opt/confluent-control-center/dependencies/trigger_rules-generated.yml \
           /opt/confluent-control-center/dependencies/alertmanager-generated.yml \
           /opt/confluent-control-center/dependencies/web-config-am.yml

Reload systemd:
```
sudo systemctl daemon-reload
```

Keep the following in mind:

These steps are for a standard, non-rootless installation with the default paths. If you set control_center_next_gen_data_dir or control_center_next_gen_dependencies_config_path, use your own paths instead.
Do not delete the prometheus.service and alertmanager.service unit files that came with the installation. They are harmless once the services are disabled.
Greenfield installations do not need any cleanup.

Configure high availability

BYOP uses a single Prometheus URL. For high availability, run your own OpenTelemetry Collector in front of your Prometheus replicas. The Collector receives the OTLP pushes from the brokers and controllers and forwards them to the OTLP endpoint of each replica, which is https://<replica>:<port>/api/v1/otlp. Configure the Collector to accept pushes on the path /api/v1/otlp/v1/metrics.

Set control_center_next_gen_dependency_prometheus_host and control_center_next_gen_dependency_prometheus_port to one replica, not to the Collector. Control Center reads metrics from this URL, and the Collector cannot answer those queries. To send only the broker and controller pushes to the Collector, set the following:

kafka_broker_custom_properties:
  confluent.telemetry.exporter._c3.client.base.url: https://<collector>:<port>/api/v1/otlp
kafka_controller_custom_properties:
  confluent.telemetry.exporter._c3.client.base.url: https://<collector>:<port>/api/v1/otlp

The brokers and controllers trust the CA in control_center_next_gen_dependency_prometheus_provided_ca_cert_path, so the Collector’s server certificate must be signed by that CA.

Control Center reads from one replica. If that replica is unavailable, Control Center cannot show metrics until it can reach that replica again, or until you point it at another replica.

Configure Kafka and KRaft for Control Center

Configure Kafka and KRaft to send data to the Prometheus service in Control Center.

The following are a few of the notable client-side Kafka and KRaft variables for Prometheus:

kafka_broker_telemetry_control_center_next_gen_user
kafka_broker_telemetry_control_center_next_gen_password

TLS-enabled Control Center

With Confluent Platform 7.9.0, when SSL is enabled on the Prometheus endpoint of Control Center, you might get the following error log:

ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
java.lang.NoSuchMethodError:
io.confluent.shaded.org.asynchttpclient.DefaultAsyncHttpClientConfig$Builder.setSslContext(Lio/confluent/shaded/io/netty/handler/ssl/SslContext;)Lio/confluent/shaded/org/asynchttpclient/DefaultAsyncHttpClientConfig$Builder;

To solve the issue, use the following JVM-level overrides for Kafka and KRaft to send metrics to Control Center:

kafka_broker_custom_java_args
kafka_controller_custom_java_args

For example:

all:
  vars:
    kafka_broker_custom_java_args: "-Djavax.net.ssl.trustStore=/var/ssl/private/kafka_broker.truststore.jks -Djavax.net.ssl.trustStorePassword=confluenttruststorepass -Djavax.net.ssl.keyStore=/var/ssl/private/kafka_broker.keystore.jks -Djavax.net.ssl.keyStorePassword=confluentkeystorestorepass"
    kafka_controller_custom_java_args: "-Djavax.net.ssl.trustStore=/var/ssl/private/kafka_controller.truststore.jks -Djavax.net.ssl.trustStorePassword=confluenttruststorepass -Djavax.net.ssl.keyStore=/var/ssl/private/kafka_controller.keystore.jks -Djavax.net.ssl.keyStorePassword=confluentkeystorestorepass"

Enable JMX Exporter

JMX Exporter is disabled by default. When enabled, the JMX Exporter jar is pulled and enabled on all Confluent Platform components besides Confluent Control Center.

Enable JMX Exporter in hosts.yml as below:

all:
  vars:
    jmxexporter_enabled: true

For more information on how the JMX exporter works and how to monitor Kafka cluster with the JMX data using Prometheus and Grafana, see Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana.

Enable Jolokia

Jolokia monitoring is disabled by default for Confluent Platform components when installed by Confluent Ansible. When enabled, the Jolokia jar is pulled enabled on all Confluent Platform components.

Enable Jolokia in hosts.yml as shown below:

all:
  vars:
    jolokia_enabled: true

For more information, see Jolokia.

Jolokia access control

This section explains how to configure Jolokia access control variables for secure JMX monitoring in Confluent Platform deployments.

Jolokia access control variables

These variables control the security of the Jolokia JMX agent endpoint. Jolokia access control restricts JMX operations via an XML policy file, providing fine-grained control over which MBeans and operations are accessible.

The following are the key variables for Jolokia access control:

jolokia_access_control_enabled

Enables Jolokia’s access control feature. When enabled, restricts JMX operations via an XML policy file. By default, this is enabled if Jolokia itself is enabled.

Type: boolean
Default: false

jolokia_access_control_custom_file_enabled

Must be set to true or false when access control is enabled. Controls whether to use a custom XML policy file or the built-in secure default.

true: You must provide your own secure XML file via jolokia_access_control_file_src_path
false: The playbooks will use a secure default XML file provided by the role
Type: boolean
Default: false

Component-specific variables

Each component has its own set of Jolokia access control variables:

zookeeper_jolokia_access_control_enabled
zookeeper_jolokia_access_control_custom_file_enabled
zookeeper_jolokia_access_control_file_src_path
kafka_controller_jolokia_access_control_enabled
kafka_controller_jolokia_access_control_custom_file_enabled
kafka_controller_jolokia_access_control_file_src_path
kafka_broker_jolokia_access_control_enabled
kafka_broker_jolokia_access_control_custom_file_enabled
kafka_broker_jolokia_access_control_file_src_path
kafka_connect_jolokia_access_control_enabled
kafka_connect_jolokia_access_control_custom_file_enabled
kafka_connect_jolokia_access_control_file_src_path
kafka_rest_jolokia_access_control_enabled
kafka_rest_jolokia_access_control_custom_file_enabled
kafka_rest_jolokia_access_control_file_src_path
ksql_jolokia_access_control_enabled
ksql_jolokia_access_control_custom_file_enabled
ksql_access_control_file_src_path
schema_registry_jolokia_access_control_enabled
schema_registry_jolokia_access_control_custom_file_enabled
schema_registry_jolokia_access_control_file_src_path
kafka_connect_replicator_jolokia_access_control_enabled
kafka_connect_replicator_jolokia_access_control_custom_file_enabled
kafka_connect_replicator_jolokia_access_control_file_src_path

Usage examples

Example 1: Enable access control with default secure configuration

all:
  vars:
    jolokia_enabled: true
    jolokia_access_control_enabled: true
    jolokia_access_control_custom_file_enabled: false

Example 2: Enable access control with custom XML policy file

all:
  vars:
    jolokia_enabled: true
    jolokia_access_control_enabled: true
    jolokia_access_control_custom_file_enabled: true
    kafka_broker_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    kafka_connect_replicator_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    schema_registry_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    ksql_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    kafka_rest_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    kafka_connect_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    kafka_broker_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    kafka_controller_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"
    zookeeper_jolokia_access_control_file_src_path: "/path/to/your/jolokia-access.xml"

Example 3: Component-specific configuration

all:
  vars:
    jolokia_enabled: true
    kafka_controller_jolokia_access_control_custom_file_enabled: false #will use the default policy when custom_enabled is set to false for a component


kafka_broker:
  vars:
    # Override for brokers only - use custom policy
    kafka_broker_jolokia_access_control_custom_file_enabled: true
    kafka_broker_jolokia_access_control_file_src_path: "/path/to/broker-jolokia-access.xml"
kafka_controller:

Example 4: Production-ready configuration with explicit choices

all:
  vars:
    jolokia_enabled: true
    jolokia_access_control_enabled: true
    # Force explicit choice in production
    jolokia_access_control_custom_file_enabled: false

# This will require each component to explicitly set their custom_file_enabled value

Security considerations

Always enable access control in production environments.
Use custom XML policy files for fine-grained control.
Regularly review and update access control policies.
Test access control policies in development before production deployment.
Monitor Jolokia logs for unauthorized access attempts.

Sample custom access control XML

Create a file, for example, /path/to/jolokia-access.xml, with content similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<restrict>
  <!-- Allow access to specific MBeans only -->
  <allow>
    <mbean>kafka.controller:type=KafkaController,name=*</mbean>
    <operation>read</operation>
  </allow>

  <!-- Deny all other access -->
  <deny>
    <mbean>*:*</mbean>
  </deny>
</restrict>

For more information on Jolokia access control XML format, see the Jolokia Security documentation.

You can refer to the policy files in roles/{component_name}/templates/jolokia_access_control_default.xml.

Sample custom access control XML

Create a file, for example, /path/to/jolokia-access.xml, with content similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<restrict>
  <!-- Allow access to specific MBeans only -->
  <allow>
    <mbean>kafka.controller:type=KafkaController,name=*</mbean>
    <operation>read</operation>
  </allow>

  <!-- Deny all other access -->
  <deny>
    <mbean>*:*</mbean>
  </deny>
</restrict>

For more information on Jolokia access control XML format, see the Jolokia Security documentation.

You can refer to the policy files in roles/{component_name}/templates/jolokia_access_control_default.xml.

Access control policy requirement for KRaft controllers

When creating custom access control policies for KRaft, ensure that your policy explicitly allows access to the required MBean. This is necessary because Ansible relies on this MBean during specific operational tasks. Without explicit permission in your policy, validation will fail.

Include the following allow block in your custom policy to grant the necessary access:

<!-- ALLOW ONLY - Specific MBean for ZK to KRaft Migration -->

<allow>
  <mbean>kafka.controller:name=ZkMigrationState,type=KafkaController</mbean>
  <attribute mode="read">*</attribute>
</allow>

Failure to include this allowance will result in unsuccessful Ansible validations and may disrupt automation workflows.

Configure Log4j 2

Starting in Confluent Ansible 8.0 and Confluent Platform 8.0, Log4j 2 is supported and enabled by default.

Setting custom Log4j 2 configurations is enabled by default. To disable custom Log4j 2 configurations, set custom_log4j2: false in the inventory file.

To enable or disable custom Log4j 2 configuration at the component level, use the variable <component>_custom_log4j2.

When set to false, Confluent Ansible uses the default Log4j 2 configurations for that specific component.
When set to true, you can set the component level <component>_log4j2_root_logger_level and <component>_log4j2_root_appenders.

To see the default variable settings for Kafka, see Kafka broker variables.

Log redactor is supported with Log4j 2.

By default, Confluent Ansible sets up Log4j 2 with the default values mentioned in the VARIABLES.md.

Deploy Confluent Server or Kafka

Confluent Server is the default version deployed with Confluent Platform. To install Kafka instead, set the following property in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: false

Configure Schema Validation

You can configure Schema ID Validation in your Kafka brokers when running Confluent Server. Set the following properties in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: true
    kafka_broker_schema_validation_enabled: true

Copy files to hosts

To have Ansible copy files to your hosts, place the files on the Ansible control node and set the following variables:

all:
  vars:
    kafka_controller_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_broker_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_rest_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_connect_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    schema_registry_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    ksql_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    control_center_next_gen_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt

The files in each list will be copied to all hosts within each group, meaning you will distribute one file to all Kafka hosts.

Specify the Java package version

Confluent Ansible provides an option for you to use pre-installed Java or to instruct Confluent Ansible which Java package to install.

To use pre-existing Java, add in the inventory file:

custom_java_path

A full pre-existing Java path on the custom nodes. Confluent Ansible will use the provided path and will skip installing Java as part of the execution.

Default: None

To specify a Java package to install, add one of the following in the inventory file:

redhat_java_package_name

Java Package to install on RHEL/Centos hosts.

Possible values: java-17-openjdk, java-21-openjdk

Default: java-21-openjdk
debian_java_package_name

Java Package to install on Debian hosts.

Possible values: openjdk-11-jdk, openjdk-8-jdk, openjdk-17-jdk

Default: openjdk-17-jdk
ubuntu_java_package_name

Java Package to install on Ubuntu hosts.

Possible values: openjdk-8-jdk, openjdk-11-jdk, openjdk-17-jdk

Default: openjdk-17-jdk

Add custom Java arguments

To have Ansible add custom Java arguments to each component’s Java process, use the following variables in the inventory file:

all:
  vars:
    kafka_controller_custom_java_args:
    kafka_broker_custom_java_args:
    kafka_rest_custom_java_args:
    kafka_connect_custom_java_args:
    schema_registry_custom_java_args:
    ksql_custom_java_args:
    control_center_next_gen_custom_java_args:

Set environment variables

To have Ansible set the required environment variables to Confluent Platform component processes, for example, KAFKA_OPTS, use the following dictionary variables in the inventory file. Refer to the specific component documentation for the required environment variables.

all:
  vars:
    kafka_controller_service_environment_overrides:
    kafka_broker_service_environment_overrides:
    kafka_rest_service_environment_overrides:
    kafka_connect_service_environment_overrides:
    kafka_connect_replicator_service_environment_overrides:
    schema_registry_service_environment_overrides:
    ksql_service_environment_overrides:
    control_center_next_gen_service_environment_overrides:

For example, the following snippet sets the KAFKA_JMX_OPTS environment variable in the Kafka broker service:

all:
  vars:
    kafka_broker_service_environment_overrides:
      KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false"

Configure listeners

Ansible Playbooks for Confluent Platform configures the following listeners on the broker:

An inter-broker listener on port 9091
A listener for the other Confluent Platform components and external clients on 9092

By default, both of these listeners inherit the security settings you configure for ssl_enabled (encryption) and sasl_protocol (authentication).

If you only need a single listener, add the following variable in the hosts.yml file.

all:
  vars:
    kafka_broker_configure_multiple_listeners: false

You can customize the out-of-the-box listeners by adding the variable, kafka_broker_custom_listeners in the hosts.yml file.

In the example below, the broker, internal, and client listeners all have unique security settings. You can configure multiple additional client listeners, but do not change the dictionary key for the broker and internal listeners, broker and internal.

all:
  vars:
    kafka_broker_custom_listeners:
      broker:
        name: BROKER
        port: 9091
        ssl_enabled: false
        sasl_protocol: none
      internal:
        name: INTERNAL
        port: 9092
        ssl_enabled: true
        sasl_protocol: scram
      client_listener:
        name: CLIENT
        port: 9093
        ssl_enabled: true
        sasl_protocol: plain

Add advertised listener hostnames

When you have a complex networking setup with multiple network interfaces, you need to set up advertised listeners to the external address (host/IP) so that clients can correctly connect to.

To configure advertised listener hostnames on a specific listener, create an advertised listener ([1]) and set the variables on specific hosts ([2] and [3]) as shown in the following example:

all:
  vars:
    kafka_broker_custom_listeners:
      client_listener:                        -------------------------- [1]
        name: CLIENT
        port: 9093
kafka_broker:
  hosts:
    ip-172-31-43-14.us-west-2.compute.internal:
      kafka_broker_custom_listeners:
        client_listener:                      -------------------------- [1]
          hostname: ec2-34-209-19-18.us-west-2.compute.amazonaws.com --- [2]
    ip-172-31-43-15.us-west-2.compute.internal:
      kafka_broker_custom_listeners:          -------------------------- [1]
        client_listener:
          hostname: ec2-34-209-19-19.us-west-2.compute.amazonaws.com --- [3]

The above example sets the AWS external DNS hostnames ([2] and [3]) on the advertised listener ([1]) for clients to connect over the interface.

Configure secrets protection

Confluent Platform secrets allow you to securely store and manage sensitive information.

Secrets protection works on Confluent Platform components, namely Confluent Server, Schema Registry, Connect, ksqlDB, REST Proxy, and Control Center.

Secrets protection is not supported for the community version of Kafka.

To use secrets protection on your component property files with Ansible Playbooks for Confluent Platform, set secrets_protection_enabled to true in your inventory file, then choose one of the following approaches to manage your master key. The first approach is recommended for most deployments.

Provide your own master key and base secrets file
Generate and regenerate the master key with Ansible

Provide your own master key and base secrets file

To have Ansible use your own master key and base secrets file that you generated ahead of time, add:

all:
  vars:
    secrets_protection_enabled: true
    regenerate_masterkey: false
    secrets_protection_masterkey: <masterkey>
    secrets_protection_security_file: <base secrets file path>

For example:

all:
  vars:
    secrets_protection_enabled: true
    regenerate_masterkey: false
    secrets_protection_masterkey: "UWQYODNQVqwbQeFgytYYoMr+FjK9Q6I0F6r16u6Y0EI="
    secrets_protection_security_file: "/tmp/security.properties"

When you provide your own master key, you must also make the corresponding security.properties base secrets file available on the Ansible control node, also called the Ansible host, at the path you set in secrets_protection_security_file. Ansible reads this file from the control node and distributes it to your Confluent Platform hosts. If the file is missing from the control node, the deployment fails.

Generate and regenerate the master key with Ansible

With this approach, Ansible generates your master key and encrypts all properties containing password across all Confluent Platform components.

all:
  vars:
    secrets_protection_enabled: true

The regenerate_masterkey variable controls whether Ansible recreates the master key and base secrets file on each playbook run.

The default value is true. On every run, Ansible generates a new master key and a new base secrets file. With regenerate_masterkey set to true, every host gets its own master key regenerated.
Set regenerate_masterkey to false after your first successful Ansible run to preserve the existing master key and base secrets file.

all:
  vars:
    secrets_protection_enabled: true
    regenerate_masterkey: false

Control which properties are masked

To have more granular control over which properties get masked, use the <component>_secrets_protection_encrypt_passwords and <component>_secrets_protection_encrypt_properties variables.

If <component>_secrets_protection_encrypt_passwords is set to false, then properties containing password no longer get masked.

Set <component>_secrets_protection_encrypt_properties to a list of variables to encrypt.

For an example, to mask only the Kafka properties advertised.listeners and broker.id, set:

all:
  vars:
    secrets_protection_enabled: true
    kafka_broker_secrets_protection_encrypt_passwords: false
    kafka_broker_secrets_protection_encrypt_properties: [advertised.listeners, broker.id]

Configure Ansible Playbooks for Confluent Platform

Set Ansible host variables

Set the Confluent Platform version

Set Confluent Platform software installation method

Set custom component properties

Set custom properties on a specific host

Add Confluent license

Configure Control Center

Control Center authentication

Bring your own Prometheus (BYOP)

Supported versions and compatibility

External Prometheus prerequisites

Connectivity and security

Configure TLS and authentication on Prometheus

Configure Control Center for BYOP

Migrate to BYOP

Roll back to the bundled Prometheus

Manually clean up the bundled Prometheus and Alertmanager

Configure high availability

Configure Kafka and KRaft for Control Center

TLS-enabled Control Center

Enable JMX Exporter

Enable Jolokia

Jolokia access control

Jolokia access control variables

Component-specific variables

Usage examples

Security considerations

Sample custom access control XML

Sample custom access control XML

Access control policy requirement for KRaft controllers

Configure Log4j 2

Deploy Confluent Server or Kafka

Configure Schema Validation

Copy files to hosts

Specify the Java package version

Add custom Java arguments

Set environment variables

Configure listeners

Add advertised listener hostnames

Configure secrets protection

Provide your own master key and base secrets file

Generate and regenerate the master key with Ansible

Control which properties are masked

Next steps