# Multi-Node Installation for Production You can install Control Center and Confluent Platform on multiple nodes for a production deployment. 1. Provision a new node using any of the Confluent Platform supported operating systems. For more information, see [Supported operating systems](/platform/current/installation/versions-interoperability.html#operating-systems). Log in to the new VM. Install Control Center on a new node or VM. To ensure a smooth transition, allow Control Center (Legacy) users to continue using Control Center (Legacy) until Control Center has gathered 7-15 days of historical metrics. For more information, see [Migrate from Control Center (Legacy)](alert-migrate.md#install-c3-migration). 2. Log in to the VM and install Control Center. For more information, see [Compatibility with Confluent Platform](system-requirements.md#install-c3-supported-cp). Use the instructions for installing Confluent Platform but make sure to use the base URL and properties from these instructions to install Control Center. For more information, see [Confluent Platform System Requirements](/platform/current/installation/system-requirements.html#system-requirements), [Install Confluent Platform using Systemd on Ubuntu and Debian](/platform/current/installation/installing_cp/deb-ubuntu.html#systemd-ubuntu-debian-install), and [Install Confluent Platform using Systemd on RHEL, CentOS, and Fedora-based Linux](/platform/current/installation/installing_cp/rhel-centos.html#systemd-rhel-centos-install). Ubuntu and Debian ```bash export BASE_URL=https://packages.confluent.io/confluent-control-center-next-gen/deb/ sudo apt-get update wget ${BASE_URL}archive.key sudo apt-key add archive.key sudo add-apt-repository -y "deb ${BASE_URL} stable main" sudo apt update ``` ```bash sudo apt install -y confluent-control-center-next-gen ``` RHEL, CentOS, and Fedora-based Linux ```bash export base_url=https://packages.confluent.io/confluent-control-center-next-gen/rpm/ cat < /dev/null [Confluent] name=Confluent repository baseurl=${base_url} gpgcheck=1 gpgkey=${base_url}archive.key enabled=1 EOF ``` ```bash sudo yum install -y confluent-control-center-next-gen cyrus-sasl openssl-devel ``` 3. Install Java for your operating system, if it isn’t already installed. For RHEL, CentOS, or Fedora: ```bash sudo yum install java-17-openjdk -y ``` For Ubuntu or Debian: ```bash sudo apt install openjdk-17-jdk -y ``` 4. Copy `/etc/confluent-control-center/control-center-production.properties` from your current Control Center (Legacy) into the Control Center node on the VM and add these properties: ```properties confluent.controlcenter.id=10 confluent.controlcenter.prometheus.enable=true confluent.controlcenter.prometheus.url=http://localhost:9090 confluent.controlcenter.prometheus.rules.file=/etc/confluent-control-center/trigger_rules-generated.yml confluent.controlcenter.alertmanager.config.file=/etc/confluent-control-center/alertmanager-generated.yml ``` 5. If you are using SSL, copy the certs at `/var/ssl/private` from your current Control Center (Legacy) into the Control Center node on the VM. If you are not using SSL, skip this step. 6. Change ownership of the configuration files. Give the Control Center process write permissions to the alert manager, so that the process can properly manage alert triggers. Use the `chown` command to set the Control Center process as the owner of the `trigger_rules-generated.yml` and `alertmanager-generated.yml` files. ```bash chown -c cp-control-center /etc/confluent-control-center/trigger_rules-generated.yml chown -c cp-control-center /etc/confluent-control-center/alertmanager-generated.yml ``` 7. Start the following services on the Control Center node: ```bash systemctl enable prometheus systemctl start prometheus systemctl enable alertmanager systemctl start alertmanager systemctl enable confluent-control-center systemctl start confluent-control-center ``` 8. Log in to each broker you intend to monitor and verify brokers can reach the Control Center node on port 9090. ```bash curl http://:9090/-/healthy ``` All brokers must have access to the Control Center node on port 9090, but port 9090 does not require public access. Restrict access as you prefer. 9. Update the following properties for every Kafka broker and KRaft controller. Pay attention to the notes on the highlighted lines that follow the code example. KRaft controller properties are located here: `/etc/controller/server.properties` ```bash metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter,io.confluent.metrics.reporter.ConfluentMetricsReporter --- [1] confluent.telemetry.exporter._c3.type=http confluent.telemetry.exporter._c3.enabled=true confluent.telemetry.exporter._c3.metrics.include=io.confluent.kafka.server.request.(?!.*delta).*|io.confluent.kafka.server.server.broker.state|io.confluent.kafka.server.replica.manager.leader.count|io.confluent.kafka.server.request.queue.size|io.confluent.kafka.server.broker.topic.failed.produce.requests.rate.1.min|io.confluent.kafka.server.tier.archiver.total.lag|io.confluent.kafka.server.request.total.time.ms.p99|io.confluent.kafka.server.broker.topic.failed.fetch.requests.rate.1.min|io.confluent.kafka.server.broker.topic.total.fetch.requests.rate.1.min|io.confluent.kafka.server.partition.caught.up.replicas.count|io.confluent.kafka.server.partition.observer.replicas.count|io.confluent.kafka.server.tier.tasks.num.partitions.in.error|io.confluent.kafka.server.broker.topic.bytes.out.rate.1.min|io.confluent.kafka.server.request.total.time.ms.p95|io.confluent.kafka.server.controller.active.controller.count|io.confluent.kafka.server.session.expire.listener.zookeeper.disconnects.total|io.confluent.kafka.server.request.total.time.ms.p999|io.confluent.kafka.server.controller.active.broker.count|io.confluent.kafka.server.request.handler.pool.request.handler.avg.idle.percent.rate.1.min|io.confluent.kafka.server.session.expire.listener.zookeeper.disconnects.rate.1.min|io.confluent.kafka.server.controller.unclean.leader.elections.rate.1.min|io.confluent.kafka.server.replica.manager.partition.count|io.confluent.kafka.server.controller.unclean.leader.elections.total|io.confluent.kafka.server.partition.replicas.count|io.confluent.kafka.server.broker.topic.total.produce.requests.rate.1.min|io.confluent.kafka.server.controller.offline.partitions.count|io.confluent.kafka.server.socket.server.network.processor.avg.idle.percent|io.confluent.kafka.server.partition.under.replicated|io.confluent.kafka.server.log.log.start.offset|io.confluent.kafka.server.log.tier.size|io.confluent.kafka.server.log.size|io.confluent.kafka.server.tier.fetcher.bytes.fetched.total|io.confluent.kafka.server.request.total.time.ms.p50|io.confluent.kafka.server.tenant.consumer.lag.offsets|io.confluent.kafka.server.session.expire.listener.zookeeper.expires.rate.1.min|io.confluent.kafka.server.log.log.end.offset|io.confluent.kafka.server.broker.topic.bytes.in.rate.1.min|io.confluent.kafka.server.partition.under.min.isr|io.confluent.kafka.server.partition.in.sync.replicas.count|io.confluent.telemetry.http.exporter.batches.dropped|io.confluent.telemetry.http.exporter.items.total|io.confluent.telemetry.http.exporter.items.succeeded|io.confluent.telemetry.http.exporter.send.time.total.millis|io.confluent.kafka.server.controller.leader.election.rate.(?!.*delta).*|io.confluent.telemetry.http.exporter.batches.failed confluent.telemetry.exporter._c3.client.base.url=http://c3-internal-dns-hostname:9090/api/v1/otlp --- [2] confluent.telemetry.exporter._c3.client.compression=gzip confluent.telemetry.exporter._c3.api.key=dummy confluent.telemetry.exporter._c3.api.secret=dummy confluent.telemetry.exporter._c3.buffer.pending.batches.max=80 --- [3] confluent.telemetry.exporter._c3.buffer.batch.items.max=4000 --- [4] confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=10 --- [5] confluent.telemetry.metrics.collector.interval.ms=60000 --- [6] confluent.telemetry.remoteconfig._confluent.enabled=false confluent.consumer.lag.emitter.enabled=true ``` - [1] To enable metrics for both Control Center (Legacy) and Control Center, update your existing Control Center (Legacy) property `metric.reporters` to use the following values: ```bash metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter,io.confluent.metrics.reporter.ConfluentMetricsReporter ``` If you decommission Control Center (Legacy), enable only TelemetryReporter plugin with the following value: ```bash metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter ``` - [2] Ensure the URL in `confluent.telemetry.exporter._c3.client.base.url` is the actual Control Center URL, reachable from the broker host. ```bash confluent.telemetry.exporter._c3.client.base.url=http://c3-internal-dns-hostname:9090/api/v1/otlp ``` - [3] [4] [5] [6] Use the following configurations for clusters up to 100,000 or fewer replicas. To get an accurate count of replicas, use the sum of all replicas across all clusters monitored in Control Center (Legacy) (including the Control Center (Legacy) bootstrap cluster). ```bash confluent.telemetry.exporter._c3.buffer.pending.batches.max=80 confluent.telemetry.exporter._c3.buffer.batch.items.max=4000 confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=10 confluent.telemetry.metrics.collector.interval.ms=60000 ```

Configurations for clusters with 100,000 to 400,000 replicas

Clusters with a replica count of 100,000 - 200,000: ```bash confluent.telemetry.exporter._c3.buffer.pending.batches.max=80 confluent.telemetry.exporter._c3.buffer.batch.items.max=4000 confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=20 confluent.telemetry.metrics.collector.interval.ms=60000 ``` Clusters with a replica count of 200,000 - 400,000: ```bash confluent.telemetry.exporter._c3.buffer.pending.batches.max=80 confluent.telemetry.exporter._c3.buffer.batch.items.max=4000 confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=20 confluent.telemetry.metrics.collector.interval.ms=120000 ``` For clusters with a replica count of 200,000 - 400,000, also update the following Control Center (Legacy) configuration: ```bash confluent.controlcenter.prometheus.trigger.threshold.time=2m ```

10. Perform a rolling restart for the brokers (zero downtime). For more information, see [Rolling restart](/platform/current/kafka/post-deployment.html#rolling-restart). ```bash systemctl restart confluent-server ``` 11. (Optional) Setup log rotation for Prometheus and Alertmanager. ### Prometheus 1. Create a new configuration file at `/etc/logrotate.d/prometheus` with the following content: ```none /var/log/confluent/control-center/prometheus.log { size 10MB rotate 5 compress delaycompress missingok notifempty copytruncate } ``` 2. Create a script at `/usr/local/bin/logrotate-prometheus.sh`: ```bash #!/bin/bash /usr/sbin/logrotate -s /var/lib/logrotate/status-prometheus /etc/logrotate.d/prometheus ``` 3. Make the script executable ```bash chmod +x /usr/local/bin/logrotate-prometheus.sh ``` 4. To schedule with Cron, add the following line to your crontab (crontab -e): ```bash */10 * * * * /usr/local/bin/logrotate-prometheus.sh >> /tmp/prometheus-rotate.log 2>&1 ``` 5. Restart Prometheus ```bash systemctl restart prometheus ``` 6. Perform similar steps for Alertmanager logs. ### Alertmanager 1. Create a new configuration file at `/etc/logrotate.d/alertmanager` with the following content: ```none /var/log/confluent/control-center/alertmanager.log { size 10MB rotate 5 compress delaycompress missingok notifempty copytruncate } ``` 2. Create a script at `/usr/local/bin/logrotate-alertmanager.sh`: ```bash #!/bin/bash /usr/sbin/logrotate -s /var/lib/logrotate/status-alertmanager /etc/logrotate.d/alertmanager ``` 3. Make the script executable ```bash chmod +x /usr/local/bin/logrotate-alertmanager.sh ``` 4. To schedule with Cron, add the following line to your crontab (crontab -e): ```bash */10 * * * * /usr/local/bin/logrotate-alertmanager.sh >> /tmp/alertmanager-rotate.log 2>&1 ``` 5. Restart Alertmanager ```bash systemctl restart alertmanager ``` ## Secure your deployment To secure Control Center, including TLS, SASL, authentication, RBAC, and securing the Prometheus and Alertmanager components, see [Control Center Security on Confluent Platform](../security/overview.md#security-control-center). ## Verify Control Center is running After the installation is complete, visit `http(s)://:9021` and wait for the metrics to start showing up in Control Center. It may take a couple of minutes. Control Center looks exactly like Control Center (Legacy). To confirm Control Center is running, use the following steps: 1. Open the network tab in Control Center. 2. Reload Control Center. 3. Locate the following API call: `/2.0/feature/flags` 4. Verify the following key is present in the response: `confluent.controlcenter.prometheus.enable: true` ## Prometheus and Alertmanager startup flags reference Control Center bundles Prometheus and Alertmanager and starts them using the `bin/prometheus-start` and `bin/alertmanager-start` wrapper scripts. These scripts inject the startup flags that Control Center requires to reload alert configuration, receive metrics from brokers and KRaft controllers, and bind to the addresses that Control Center expects. This section documents those flags as a reference, so you can understand the setup that Control Center performs and triage alerting and metrics issues. Control Center does not support a customer-managed Prometheus or Alertmanager. The following is informational reference for the setup that Control Center performs, not a guide to self-managing these services. #### IMPORTANT Always start and restart Prometheus and Alertmanager using the Confluent-provided `bin/prometheus-start` and `bin/alertmanager-start` scripts (or, on package installs, the `prometheus` and `alertmanager` services). Do not start or restart these processes manually. Starting them directly bypasses the wrapper scripts and drops the flags that Control Center requires, which causes alerting and metrics failures. ### Prometheus startup flags The `bin/prometheus-start` script applies the following flags. | Flag | Why Control Center needs it | Symptom if missing | |-----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| | `--web.enable-lifecycle` | Lets Control Center reload the scrape and alert configuration through
`POST /-/reload` when a trigger is saved. | Trigger creation fails with an “oops, something went wrong” error. An
HTTP Archive (HAR) capture shows `403 Lifecycle API is not enabled.` | | `--web.enable-otlp-receiver` | Lets brokers and KRaft controllers push metrics to
`/api/v1/otlp/v1/metrics`. | “Metrics not available” errors and empty dashboards. | | `--web.listen-address=` | Sets the address Prometheus listens on. Must match
`confluent.controlcenter.prometheus.url` (default `0.0.0.0:9090`). | Prometheus shows “Not Running” in the Control Center user interface. | | `--web.config.file=` | Configures TLS and basic authentication for the Prometheus web endpoint. | Secured communication fails. | | `--config.file`, `--storage.tsdb.path`,
`--storage.tsdb.retention.time` (default `15d`), `--log.level=info` | Set the configuration file location, the data path, the metrics
retention period, and the log level. | Incorrect retention or no data. | ### Alertmanager startup flags The `bin/alertmanager-start` script applies the following flags. | Flag | Why Control Center needs it | Symptom if missing | |--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--web.listen-address=` | Sets the address Alertmanager listens on. Must match
`confluent.controlcenter.alertmanager.url`. The default Alertmanager
port is `9093`. For details, see the following port section. | Alertmanager is unreachable. | | `--cluster.advertise-address=` | Binds an advertise address for the gossip mesh when no private IP can be
auto-detected. | Alertmanager fails to start with `couldn't deduce an advertise address:
no private IP found, explicit advertise addr not provided`, followed by
`unable to initialize gossip mesh ... No private IP address found, and
explicit IP not provided`. For triage steps, see
[Alertmanager won’t start with “couldn’t deduce an advertise address”](troubleshooting.md#c3-alertmanager-advertise-address). | | `--config.file`, `--storage.path`, `--web.config.file`,
`--log.level=info` | Set the configuration file location, the data path, TLS configuration,
and the log level. | Alertmanager won’t start or can’t route alerts. | ### Alertmanager port: 9093 versus 9098 Alertmanager listens on one of two ports, depending on your installation type: * The Alertmanager default port is `9093`. This is the default in the `bin/alertmanager-start` script (`ALERTMANAGER_PORT=9093`) and in the shipped `control-center-production.properties` file (`alertmanager.url=http://localhost:9093`). * Port `9098` is used only in single-node local installations, where KRaft controllers also default to `9093` and would otherwise collide with Alertmanager on the same host. In that scenario, the install steps move Alertmanager to `9098` with `export ALERTMANAGER_PORT=9098`. Treat `9098` as a co-located single-node override, not as a general default. ### Validate Prometheus and Alertmanager Use the following endpoints to validate that Prometheus and Alertmanager are running with the expected configuration: * `GET /-/healthy` confirms the process is healthy. * `GET /-/ready` confirms the process is ready to serve requests. * `GET /api/v1/status/config` returns the running Prometheus configuration, which is useful for confirming that the expected flags and scrape or alert configuration are applied. ### Check the bundled versions Control Center ships specific versions of Prometheus and Alertmanager, which can change between releases. To check the versions bundled with your installation, run: ```bash bin/prometheus --version ``` ```bash bin/alertmanager --version ``` ## Related content - [Single-Node Installation](single-node.md#install-c3-single-node) - [High Availability](high-availability.md#install-c3-high-availability) - [Migrate from Control Center (Legacy)](alert-migrate.md#install-c3-migration) - [Control Center Configuration Reference for Confluent Platform](configuration.md#controlcenter-configuration) - [Control Center Security on Confluent Platform](../security/overview.md#security-control-center)