# Configure Advanced Confluent Platform Settings with Ansible Playbooks This section provides information about various deployment configurations for Confluent Platform using Ansible. ## Configure Tiered Storage with GCS buckets Configure [Tiered Storage](/platform/current/kafka/tiered-storage.html) to use the [Custom Properties](ansible-configure.md#ansible-configure) and the [Copy Files](ansible-configure.md#ansible-copy-file) features of Ansible Playbooks for Confluent Platform. 1. Get the GCP credentials JSON file on the Ansible control node. 2. Set the following variables in the `hosts.yml` file: ```none all: vars: kafka_broker_copy_files: - source_path: /tmp/gcloud-a5f9c87c81ae.json destination_path: /etc/security/google/creds.json kafka_broker_custom_properties: confluent.tier.feature: "true" confluent.tier.enable: "true" confluent.tier.backend: GCS confluent.tier.gcs.bucket: bucket-name confluent.tier.gcs.region: us-west2 confluent.tier.gcs.cred.file.path: /etc/security/google/creds.json ``` The credential file destination path must match the `confluent.tier.gcs.cred.file.path` custom property. ## Configure Tiered Storage with S3 buckets Configure [Tiered Storage](/platform/current/kafka/tiered-storage.html) to use the [Custom Properties](ansible-configure.md#ansible-configure) and the [Copy Files](ansible-configure.md#ansible-copy-file) features of Ansible Playbooks for Confluent Platform. 1. Get the AWS credentials file on the Ansible control node. 2. Set the following variables in the `hosts.yml` file: ```none all: vars: kafka_broker_copy_files: - source_path: /tmp/credentials destination_path: /etc/security/aws/credentials kafka_broker_custom_properties: confluent.tier.feature: "true" confluent.tier.enable: "true" confluent.tier.backend: S3 confluent.tier.s3.bucket: bucket-name confluent.tier.s3.region: us-west-2 confluent.tier.s3.cred.file.path: /etc/security/aws/credentials ``` The credential file destination path must match the `confluent.tier.s3.cred.file.path` custom property. ## Deploy Confluent Replicator Starting in the 6.1.0 release, Ansible Playbooks for Confluent Platform supports deployment of [Confluent Replicator](/platform/current/multi-dc-deployments/replicator/replicator-run.html#replicator-executable). Using Ansible, you can deploy Replicator with the following security mechanisms: * SASL/PLAIN * SASL/SCRAM * Kerberos * mTLS * Plaintext (which is no auth no encryption) The general deployment model is to deploy Replicator after both the source and destination clusters have been deployed. We recommend creating an inventory file specifically for the Replicator deployment, excluding other cluster deployment-related configuration. In this section, an example file, `replicator-hosts.yml`, is used. There are two clusters in this example, the source cluster and the destination cluster. Replicator has four client connections split across the two clusters: * Replicator configuration connection to the cluster which is used for storing configuration information in topics. See [Configure Replicator configuration connection](#ansible-replicator-client-connection). * Replicator monitoring connection which is used to produce metrics to the metrics cluster. This is often the same cluster as the cluster used to store configuration information. See [Configure monitoring connection](#ansible-replicator-monitoring-connection). * Replicator consumer connection which is used to consume data from the source cluster. See [Configure consumer connection](#ansible-replicator-consumer-connection). * Replicator producer connection which is used to produce data to the destination cluster. See [Configure producer connection](#ansible-replicator-producer-connection). The following sections list the configuration properties required in the Replicator inventory file. The examples use: * SASL/PLAIN with TLS on the source cluster * Kerberos with TLS on the destination cluster After configuring the replicator, you deploy the replicator with the following command. The command uses the example inventory file, `replicator-hosts.yml`. ```bash ansible-playbook -i replicator-hosts.yml playbooks/all.yml ``` ### Configure Replicator configuration connection 1. Define the `kafka_connect_replicator` group and `hosts` to deploy to. For example: ```yaml kafka_connect_replicator: hosts: ip-172-31-34-246.us-east-2.compute.internal: ``` 2. Define the listener for Replicator configuration cluster. The following is an example of a listener with Kerberos authentication and TLS enabled: ```yaml kafka_connect_replicator_listener: ssl_enabled: true sasl_protocol: kerberos ``` 3. Define the basic configuration for Replicator connection: ```yaml kafka_connect_replicator_white_list: kafka_connect_replicator_bootstrap_servers: ``` 4. Define security configuration for the Replicator connection: ```yaml kafka_connect_replicator_kerberos_principal: kafka_connect_replicator_kerberos_keytab_path: kafka_connect_replicator_ssl_ca_cert_path: kafka_connect_replicator_ssl_cert_path: kafka_connect_replicator_ssl_key_path: kafka_connect_replicator_ssl_key_password: ``` 5. For RBAC-enabled deployment, define the additional security configuration. Specify either the Kafka cluster id (`kafka_connect_replicator_kafka_cluster_id`) or the cluster name (`kafka_connect_replicator_kafka_cluster_name`). ```yaml kafka_connect_replicator_rbac_enabled: true kafka_connect_replicator_erp_tls_enabled: kafka_connect_replicator_erp_host: kafka_connect_replicator_erp_admin_user: kafka_connect_replicator_erp_admin_password: kafka_connect_replicator_kafka_cluster_id: kafka_connect_replicator_kafka_cluster_name: kafka_connect_replicator_erp_pem_file: ``` 6. Set the `CLASSPATH` to the replicator installation directory in `kafka_connect_service_environment_overrides`: ```yaml kafka_connect_service_environment_overrides: CLASSPATH: /* ``` For more information about setting required Confluent Platform environment variables using Ansible, see [Set environment variables](ansible-configure.md#ansible-override-env-varabiles). ### Configure consumer connection 1. Define the configuration for the consumer listener on the source cluster. The following is an example with TLS and SASL/PLAIN enabled: ```yaml kafka_connect_replicator_consumer_listener: ssl_enabled: true sasl_protocol: plain ``` 2. Define the basic configuration for the consumer client connection: ```yaml kafka_connect_replicator_consumer_bootstrap_servers: ``` 3. Define the security configuration for the consumer client connection: ```yaml kafka_connect_replicator_consumer_ssl_ca_cert_path: kafka_connect_replicator_consumer_ssl_cert_path: kafka_connect_replicator_consumer_ssl_key_path: kafka_connect_replicator_consumer_ssl_key_password: ``` 4. Define custom properties for each client connection: ```yaml kafka_connect_replicator_consumer_custom_properties: ``` 5. For RBAC-enabled deployment, define the additional client custom properties. Specify either the Kafka cluster id (`kafka_connect_replicator_consumer_kafka_cluster_id`) or the cluster name (`kafka_connect_replicator_consumer_kafka_cluster_name`). ```yaml kafka_connect_replicator_consumer_erp_tls_enabled: kafka_connect_replicator_consumer_erp_host: kafka_connect_replicator_consumer_erp_admin_user: kafka_connect_replicator_consumer_erp_admin_password: kafka_connect_replicator_consumer_kafka_cluster_id: kafka_connect_replicator_consumer_kafka_cluster_name: kafka_connect_replicator_consumer_erp_pem_file: ``` ### Configure producer connection 1. Define the listener configuration for the producer connection to the destination cluster. The following is an example with TLS and Kerberos for authentication enabled: ```yaml kafka_connect_replicator_producer_listener: ssl_enabled: true sasl_protocol: kerberos ``` 2. Define the basic producer configuration: ```yaml kafka_connect_replicator_producer_bootstrap_servers: ``` 3. Define the security configuration for the producer connection: ```yaml kafka_connect_replicator_producer_kerberos_principal: kafka_connect_replicator_producer_kerberos_keytab_path: kafka_connect_replicator_producer_ssl_ca_cert_path: kafka_connect_replicator_producer_ssl_cert_path: kafka_connect_replicator_producer_ssl_key_path: kafka_connect_replicator_producer_ssl_key_password: ``` 4. Define custom properties for each client connection: ```yaml kafka_connect_replicator_producer_custom_properties: ``` 5. For RBAC-enabled deployment, define the additional producer custom properties. `kafka_connect_replicator_producer` configs default to match `kafka_connect_replicator` configs. The following are required only if you are producing to a different cluster than where you are storing your configs. Specify either the Kafka cluster id (`kafka_connect_replicator_producer_kafka_cluster_id`) or the cluster name (`kafka_connect_replicator_producer_kafka_cluster_name`). ```yaml kafka_connect_replicator_producer_rbac_enabled: true kafka_connect_replicator_producer_erp_tls_enabled: kafka_connect_replicator_producer_erp_host: kafka_connect_replicator_producer_erp_admin_user: kafka_connect_replicator_producer_erp_admin_password: kafka_connect_replicator_producer_kafka_cluster_id: kafka_connect_replicator_producer_kafka_cluster_name: kafka_connect_replicator_producer_erp_pem_file: ``` ### Configure monitoring connection 1. Define the listener configuration for the monitoring interceptors: ```yaml kafka_connect_replicator_monitoring_interceptor_listener: ssl_enabled: true sasl_protocol: kerberos ``` 2. Define the basic monitoring configuration: ```yaml kafka_connect_replicator_monitoring_interceptor_bootstrap_servers: ``` 3. Define the security configuration for the monitoring connection. ```yaml kafka_connect_replicator_monitoring_interceptor_kerberos_principal: kafka_connect_replicator_monitoring_interceptor_kerberos_keytab_path: kafka_connect_replicator_monitoring_interceptor_ssl_ca_cert_path: kafka_connect_replicator_monitoring_interceptor_ssl_cert_path: kafka_connect_replicator_monitoring_interceptor_ssl_key_path: kafka_connect_replicator_monitoring_interceptor_ssl_key_password: ``` 4. For RBAC-enabled deployment, define additional custom properties for the monitoring connection. `kafka_connect_replicator_monitoring_interceptor` configs default to match `kafka_connect_replicator` configs. The following are required only if you are producing metrics to a different cluster than where you are storing your configs. Specify either the Kafka cluster id (`kafka_connect_replicator_monitoring_interceptor_kafka_cluster_id`) or the cluster name (`kafka_connect_replicator_monitoring_interceptor_kafka_cluster_name`). ```yaml kafka_connect_replicator_monitoring_interceptor_rbac_enabled: true kafka_connect_replicator_monitoring_interceptor_erp_tls_enabled: kafka_connect_replicator_monitoring_interceptor_erp_host: kafka_connect_replicator_monitoring_interceptor_erp_admin_user: kafka_connect_replicator_monitoring_interceptor_erp_admin_password: password kafka_connect_replicator_monitoring_interceptor_kafka_cluster_id: kafka_connect_replicator_monitoring_interceptor_kafka_cluster_name: kafka_connect_replicator_monitoring_interceptor_erp_pem_file: ``` ## Deploy Confluent Platform across multiple regions To configure [multi region clusters](https://docs.confluent.io/platform/current/multi-dc-deployments/multi-region.html), use the following properties in the `hosts.yml` inventory file: * Set `replica.selector.class` on `kafka_broker` group. * Set `broker.rack` uniquely on each Kafka broker host. For example: ```none kafka_broker: vars: kafka_broker_custom_properties: replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector hosts: ip-192-24-10-207.us-west.compute.internal: kafka_broker_custom_properties: broker.rack: us-west-2a ip-192-24-5-30.us-west.compute.internal: kafka_broker_custom_properties: broker.rack: us-west-2b ip-192-24-10-0.us-west.compute.internal: kafka_broker_custom_properties: broker.rack: us-west-2a ``` You can apply the `kafka_broker_custom_properties` directly within the `kafka_broker` group as well. ## Configure ksqlDB log streaming To configure [ksqlDB log stream](https://docs.ksqldb.io/en/latest/developer-guide/test-and-debug/processing-log/#log-stream), use the following properties in the `hosts.yml` inventory file. * Without RBAC enabled, set `ksql_log_streaming_enabled` to `true`: ```none all: ksql_log_streaming_enabled: true ``` * With RBAC and Kerberos enabled, set the `ksql_log_streaming_enabled` to `true`, and additionally provide the keytab location, `ksql_kerberos_keytab_path`, and keytab principal, `ksql_kerberos_principal`, for connecting to your internal listener: ```none all: ksql_log_streaming_enabled: true ksql: hosts: ip-192-24-34-224.us-west.compute.internal: ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ``` When RBAC is enabled, you also need to add your keytab principal to your LDAP server in order for the client connection to authenticate. To configure ksqlDB log streaming with RBAC and MTLS enabled, add the Certificate’s CN, without any special formatting to your LDAP server for the client connection to authenticate. For example, LDAP Username should be specified as `ksql1` instead of `cn=ksql1`. ## Configure multiple ksqlDB clusters To configure multiple ksqlDB clusters, create new groups for each cluster and set them as children of the ksqlDB group. The Ansible groups cannot be named `ksql`. The name of these groups determine how each cluster is named in Control Center. Each ksqlDB cluster needs a unique value for the `ksql_service_id` property. By convention, the service ID should end with an underscore. For example: ```none ksql: children: ksql1: ksql2: ksql1: vars: ksql_service_id: ksql1_ hosts: ip-172-31-34-15.us-east-2.compute.internal: ip-172-31-37-16.us-east-2.compute.internal: ksql2: vars: ksql_service_id: ksql2_ hosts: ip-172-31-34-17.us-east-2.compute.internal: ip-172-31-37-18.us-east-2.compute.internal: ``` To configure Control Center for multiple ksqlDB clusters, set the `ksql_cluster_ansible_group_names` property to a list of all ksqlDB children groups. For example: ```none control_center_next_gen: vars: ksql_cluster_ansible_group_names: - ksql1 - ksql2 hosts: ip-172-31-37-15.us-east-2.compute.internal: ``` ## Configure Connect cluster To configure the connector configurations from Confluent Ansible, set the following properties in your `hosts.yaml` inventory file. ```yaml kafka_connect: vars: kafka_connect_plugins_path: --- [1] - kafka_connect_confluent_hub_plugins: --- [2] - kafka_connect_plugins: --- [3] - kafka_connect_plugins_remote: --- [4] - ``` * [1] An array of directories where the Connect worker searches for connector plugins. The values are mapped to the `plugin.path` Connect property. For example: ```yaml kafka_connect_plugins_path: - /usr/share/java - /my/connectors/dir ``` * [2] An array of connectors to be installed from Confluent Marketplace. For example: ```yaml kafka_connect_confluent_hub_plugins: - jcustenborder/kafka-connect-spooldir:2.0.43 ``` * [3] An array of connectors to be installed from archive files local to Confluent Ansible host. For example: ```yaml kafka_connect_plugins: - local/path/to/connect_archive.zip ``` * [4] An array of connectors to be installed from archive files in remote servers. For example: ```yaml kafka_connect_plugins_remote: - http://myhost.com/connect_archive.zip ``` ## Configure Kafka connectors To configure the connector configurations from Confluent Ansible, set the following connector object in your `hosts.yaml` inventory file, one object per connector. Each object should contain the `name` and `config` properties. ```yaml kafka_connect_connectors: --- [1] - name: --- [2] config: --- [3] connector.class: --- [4] tasks.max: --- [5] topics: --- [6] ``` * [1] A list of the connector objects to manage. Each connector must have the `name` and `config` properties. * [2] Required. The name of the connector. * [3] Required. Connector-specific configuration settings as key-value maps. Consult the [connector documentation](https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html) for the required settings. Numeric values must be provided as strings, for example, `"1"`. Commonly used config properties are listed below: * [4] `connector.class`: Required. The class name of the connector. * [5] `tasks.max`: The max number of tasks for the connector. * [6] `topics`: A comma-separated list of topics to consume. The following is an example snippet of an inventory file for installing a connector from Confluent Marketplace: ```yaml kafka_connect_connectors: - name: sample-connector config: connector.class: "org.apache.kafka.connect.tools.VerifiableSinkConnector" tasks.max: "1" file: "path/to/file.txt" topics: "test_topic" kafka_connect: vars: kafka_connect_plugins_path: - /usr/share/java ## Installing connectors from Confluent Hub kafka_connect_confluent_hub_plugins: - confluentinc/kafka-connect-jdbc:10.3.3 hosts: ip-10-100-130-178.ec2.internal: ``` For more information about Kafka connectors, see [Self-managed Connectors for Confluent Platform](https://docs.confluent.io/platform/current/connect/kafka_connectors.html). ### Configure Kafka connectors on an RBAC-enabled cluster To configure a connector on an RBAC-enabled Confluent cluster, set the `kafka_connect_connector_white_list` variable with the list of topics for the connector to produce / consume. For example: ```yaml kafka_connect_connector_white_list: "test_topic1,test" ``` ## Configure multiple Connect clusters To configure multiple Connect clusters, create a new group for each cluster and set it as children of the `kafka_connect` group. The Ansible groups cannot be named `kafka_connect`. Each Connect cluster needs a unique value for the `kafka_connect_group_id` property. The value of `kafka_connect_group_id` will be the name of the connect cluster within Control Center. For example: ```none kafka_connect: children: syslog: elastic: syslog: vars: kafka_connect_group_id: connect_syslog hosts: ip-172-31-34-246.us-east-2.compute.internal: elastic: vars: kafka_connect_group_id: connect-elastic hosts: ip-172-31-34-247.us-east-2.compute.internal: ``` For an example inventory file that configures two Connect clusters on the same host, see: ```bash https://github.com/confluentinc/cp-ansible/blob/8.3.0-post/docs/sample_inventories/multi_connect_workers_on_single_node.yml ``` To configure Control Center for multiple Connect clusters, set the `kafka_connect_cluster_ansible_group_names` property to a list of all `kafka_connect` children groups. For example: ```none control_center_next_gen: vars: kafka_connect_cluster_ansible_group_names: - syslog - elastic hosts: ip-172-31-37-15.us-east-2.compute.internal: ``` ## Configure multiple Connect services on one host To configure multiple Connect services on the same host, give each Connect instance a unique name and enable hostname aliasing. For example: ```yaml kafka_connect: vars: hostname_aliasing_enabled: true hosts: connect01: ansible_host: ec2-34-217-174-252.us-west-2.compute.amazonaws.com hostname: ip-172-31-40-189.us-west-2.compute.internal connect02: ansible_host: ec2-34-217-174-252.us-west-2.compute.amazonaws.com hostname: ip-172-31-40-189.us-west-2.compute.internal ``` When hostname aliasing is enabled, the hostname used in configuration files for a given host will be set to these variables in this precedence: 1. `hostname` 2. `ansible_host` 3. `inventory_hostname` In the above example, two Connect services will be placed on the same server, and `ip-172-31-40-189.us-west-2.compute.internal` will be their hostname alias. Additionally, you must configure variables to make sure each Connect service gets its own configuration files and ports. Using the previous example, on the `connect02` instance, set the following variables: ```yaml connect02: kafka_connect_service_name: confluent-kafka-connect02 kafka_connect_config_filename: connect-distributed02.properties kafka_connect_rest_port: 8084 # If JMX Exporter is enabled: kafka_connect_jmxexporter_port: 8078 # If Jolokia is enabled: kafka_connect_jolokia_port: 7774 kafka_connect_jolokia_config: /etc/kafka/kafka_connect_jolokia02.properties ``` During provisioning, Ansible Playbooks for Confluent Platform will set up a `systemd` service called `confluent-kafka-connect02` for the second Connect instance, colocated with `connect01`. ## Connect to Confluent Cloud You can use Ansible Playbooks for Confluent Platform to configure and deploy on-premises Confluent Platform to connect to Kafka and Schema Registry running in Confluent Cloud. ### Connect to Confluent Cloud Kafka To enable Confluent Platform components to connect to Confluent Cloud Kafka, get the bootstrap servers, api key, and secret, and set the following variables in the `hosts.yml` file: * `ccloud_kafka_enabled` * `ccloud_kafka_bootstrap_servers` * `ccloud_kafka_key` * `ccloud_kafka_secret` For example: ```yaml all: vars: ccloud_kafka_enabled: true ccloud_kafka_bootstrap_servers: pkc-xxxxx.europe-west1.gcp.confluent.cloud:9092,pkc-yyyy.europe-west1.gcp.confluent.cloud:9092,pkc-zzzz.europe-west1.gcp.confluent.cloud:9092 ccloud_kafka_key: YYYYYYYYYYYYYY ccloud_kafka_secret: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz ``` #### NOTE There should not be a `kafka_controller` or `kafka_broker` group in your inventory file. ### Connect to Confluent Cloud Schema Registry To enable components to connect to Confluent Cloud Schema Registry, get the Schema Registry URL, the api key, and the secret, and set the following variables in the `hosts.yml` file: * `ccloud_schema_registry_enabled` * `ccloud_schema_registry_url` * `ccloud_schema_registry_key` * `ccloud_schema_registry_secret` For example: ```yaml all: vars: ccloud_schema_registry_enabled: true ccloud_schema_registry_url: https://psrc-zzzzz.europe-west3.gcp.confluent.cloud ccloud_schema_registry_key: AAAAAAAAAAAAAAAA ccloud_schema_registry_secret: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb ``` See a sample inventory file for Confluent Cloud Kafka and Schema Registry configuration at the following location: ```bash https://github.com/confluentinc/cp-ansible/blob/8.3.0-post/docs/sample_inventories/ccloud.yml ``` ## Next step [Install Confluent Platform with Ansible Playbooks](ansible-install.md#ansible-install).