Multi-Node Installation for Production

You can install Control Center and Confluent Platform on multiple nodes for a production deployment.

Provision a new node using any of the Confluent Platform supported operating systems. For more information, see Supported operating systems. Log in to the new VM.
Install Control Center on a new node or VM. To ensure a smooth transition, allow Control Center (Legacy) users to continue using Control Center (Legacy) until Control Center has gathered 7-15 days of historical metrics. For more information, see Migrate from Control Center (Legacy).

Log in to the VM and install Control Center. For more information, see Compatibility with Confluent Platform.

Use the instructions for installing Confluent Platform but make sure to use the base URL and properties from these instructions to install Control Center.

For more information, see Confluent Platform System Requirements, Install Confluent Platform using Systemd on Ubuntu and Debian, and Install Confluent Platform using Systemd on RHEL, CentOS, and Fedora-based Linux.

Ubuntu and Debian

export BASE_URL=https://packages.confluent.io/confluent-control-center-next-gen/deb/
sudo apt-get update
wget ${BASE_URL}archive.key
sudo apt-key add archive.key
sudo add-apt-repository -y "deb ${BASE_URL} stable main"
sudo apt update

sudo apt install -y confluent-control-center-next-gen

RHEL, CentOS, and Fedora-based Linux

export base_url=https://packages.confluent.io/confluent-control-center-next-gen/rpm/
cat <<EOF | sudo tee /etc/yum.repos.d/Confluent.repo > /dev/null
[Confluent]
name=Confluent repository
baseurl=${base_url}
gpgcheck=1
gpgkey=${base_url}archive.key
enabled=1
EOF

sudo yum install -y confluent-control-center-next-gen cyrus-sasl openssl-devel

Install Java for your operating system, if it isn’t already installed.
For RHEL, CentOS, or Fedora:
```
sudo yum install java-17-openjdk -y
```
For Ubuntu or Debian:
```
sudo apt install openjdk-17-jdk -y
```

Copy /etc/confluent-control-center/control-center-production.properties from your current Control Center (Legacy) into the Control Center node on the VM and add these properties:

confluent.controlcenter.id=10
confluent.controlcenter.prometheus.enable=true
confluent.controlcenter.prometheus.url=http://localhost:9090
confluent.controlcenter.prometheus.rules.file=/etc/confluent-control-center/trigger_rules-generated.yml
confluent.controlcenter.alertmanager.config.file=/etc/confluent-control-center/alertmanager-generated.yml

If you are using SSL, copy the certs at /var/ssl/private from your current Control Center (Legacy) into the Control Center node on the VM. If you are not using SSL, skip this step.
Change ownership of the configuration files. Give the Control Center process write permissions to the alert manager, so that the process can properly manage alert triggers. Use the chown command to set the Control Center process as the owner of the trigger_rules-generated.yml and alertmanager-generated.yml files.
```
chown -c cp-control-center /etc/confluent-control-center/trigger_rules-generated.yml
chown -c cp-control-center /etc/confluent-control-center/alertmanager-generated.yml
```

Start the following services on the Control Center node:

systemctl enable prometheus
systemctl start prometheus

systemctl enable alertmanager
systemctl start alertmanager

systemctl enable confluent-control-center
systemctl start confluent-control-center

Log in to each broker you intend to monitor and verify brokers can reach the Control Center node on port 9090.
```
curl http://<c3-internal-dns-url>:9090/-/healthy
```
All brokers must have access to the Control Center node on port 9090, but port 9090 does not require public access. Restrict access as you prefer.

Update the following properties for every Kafka broker and KRaft controller. Pay attention to the notes on the highlighted lines that follow the code example.

KRaft controller properties are located here: /etc/controller/server.properties

metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter,io.confluent.metrics.reporter.ConfluentMetricsReporter --- [1]
confluent.telemetry.exporter._c3.type=http
confluent.telemetry.exporter._c3.enabled=true
confluent.telemetry.exporter._c3.metrics.include=io.confluent.kafka.server.request.(?!.*delta).*|io.confluent.kafka.server.server.broker.state|io.confluent.kafka.server.replica.manager.leader.count|io.confluent.kafka.server.request.queue.size|io.confluent.kafka.server.broker.topic.failed.produce.requests.rate.1.min|io.confluent.kafka.server.tier.archiver.total.lag|io.confluent.kafka.server.request.total.time.ms.p99|io.confluent.kafka.server.broker.topic.failed.fetch.requests.rate.1.min|io.confluent.kafka.server.broker.topic.total.fetch.requests.rate.1.min|io.confluent.kafka.server.partition.caught.up.replicas.count|io.confluent.kafka.server.partition.observer.replicas.count|io.confluent.kafka.server.tier.tasks.num.partitions.in.error|io.confluent.kafka.server.broker.topic.bytes.out.rate.1.min|io.confluent.kafka.server.request.total.time.ms.p95|io.confluent.kafka.server.controller.active.controller.count|io.confluent.kafka.server.session.expire.listener.zookeeper.disconnects.total|io.confluent.kafka.server.request.total.time.ms.p999|io.confluent.kafka.server.controller.active.broker.count|io.confluent.kafka.server.request.handler.pool.request.handler.avg.idle.percent.rate.1.min|io.confluent.kafka.server.session.expire.listener.zookeeper.disconnects.rate.1.min|io.confluent.kafka.server.controller.unclean.leader.elections.rate.1.min|io.confluent.kafka.server.replica.manager.partition.count|io.confluent.kafka.server.controller.unclean.leader.elections.total|io.confluent.kafka.server.partition.replicas.count|io.confluent.kafka.server.broker.topic.total.produce.requests.rate.1.min|io.confluent.kafka.server.controller.offline.partitions.count|io.confluent.kafka.server.socket.server.network.processor.avg.idle.percent|io.confluent.kafka.server.partition.under.replicated|io.confluent.kafka.server.log.log.start.offset|io.confluent.kafka.server.log.tier.size|io.confluent.kafka.server.log.size|io.confluent.kafka.server.tier.fetcher.bytes.fetched.total|io.confluent.kafka.server.request.total.time.ms.p50|io.confluent.kafka.server.tenant.consumer.lag.offsets|io.confluent.kafka.server.session.expire.listener.zookeeper.expires.rate.1.min|io.confluent.kafka.server.log.log.end.offset|io.confluent.kafka.server.broker.topic.bytes.in.rate.1.min|io.confluent.kafka.server.partition.under.min.isr|io.confluent.kafka.server.partition.in.sync.replicas.count|io.confluent.telemetry.http.exporter.batches.dropped|io.confluent.telemetry.http.exporter.items.total|io.confluent.telemetry.http.exporter.items.succeeded|io.confluent.telemetry.http.exporter.send.time.total.millis|io.confluent.kafka.server.controller.leader.election.rate.(?!.*delta).*|io.confluent.telemetry.http.exporter.batches.failed
confluent.telemetry.exporter._c3.client.base.url=http://c3-internal-dns-hostname:9090/api/v1/otlp --- [2]
confluent.telemetry.exporter._c3.client.compression=gzip
confluent.telemetry.exporter._c3.api.key=dummy
confluent.telemetry.exporter._c3.api.secret=dummy
confluent.telemetry.exporter._c3.buffer.pending.batches.max=80 --- [3]
confluent.telemetry.exporter._c3.buffer.batch.items.max=4000 --- [4]
confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=10 --- [5]
confluent.telemetry.metrics.collector.interval.ms=60000 --- [6]
confluent.telemetry.remoteconfig._confluent.enabled=false
confluent.consumer.lag.emitter.enabled=true

[1] To enable metrics for both Control Center (Legacy) and Control Center, update your existing Control Center (Legacy) property metric.reporters to use the following values:
```
metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter,io.confluent.metrics.reporter.ConfluentMetricsReporter
```
If you decommission Control Center (Legacy), enable only TelemetryReporter plugin with the following value:
```
metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter
```
[2] Ensure the URL in confluent.telemetry.exporter._c3.client.base.url is the actual Control Center URL, reachable from the broker host.
```
confluent.telemetry.exporter._c3.client.base.url=http://c3-internal-dns-hostname:9090/api/v1/otlp
```
[3] [4] [5] [6] Use the following configurations for clusters up to 100,000 or fewer replicas. To get an accurate count of replicas, use the sum of all replicas across all clusters monitored in Control Center (Legacy) (including the Control Center (Legacy) bootstrap cluster).
```
confluent.telemetry.exporter._c3.buffer.pending.batches.max=80
confluent.telemetry.exporter._c3.buffer.batch.items.max=4000
confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=10
confluent.telemetry.metrics.collector.interval.ms=60000
```

Configurations for clusters with 100,000 to 400,000 replicas

Clusters with a replica count of 100,000 - 200,000:

confluent.telemetry.exporter._c3.buffer.pending.batches.max=80
confluent.telemetry.exporter._c3.buffer.batch.items.max=4000
confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=20
confluent.telemetry.metrics.collector.interval.ms=60000

Clusters with a replica count of 200,000 - 400,000:

confluent.telemetry.exporter._c3.buffer.pending.batches.max=80
confluent.telemetry.exporter._c3.buffer.batch.items.max=4000
confluent.telemetry.exporter._c3.buffer.inflight.submissions.max=20
confluent.telemetry.metrics.collector.interval.ms=120000

For clusters with a replica count of 200,000 - 400,000, also update the following Control Center (Legacy) configuration:

confluent.controlcenter.prometheus.trigger.threshold.time=2m

Perform a rolling restart for the brokers (zero downtime). For more information, see Rolling restart.
```
systemctl restart confluent-server
```

(Optional) Setup log rotation for Prometheus and Alertmanager.

Prometheus

Create a new configuration file at /etc/logrotate.d/prometheus with the following content:

/var/log/confluent/control-center/prometheus.log {
   size 10MB
   rotate 5
   compress
   delaycompress
   missingok
   notifempty
   copytruncate
}

Create a script at /usr/local/bin/logrotate-prometheus.sh:

#!/bin/bash
/usr/sbin/logrotate -s /var/lib/logrotate/status-prometheus /etc/logrotate.d/prometheus

Make the script executable

chmod +x /usr/local/bin/logrotate-prometheus.sh

To schedule with Cron, add the following line to your crontab (crontab -e):

*/10 * * * * /usr/local/bin/logrotate-prometheus.sh >> /tmp/prometheus-rotate.log 2>&1

Restart Prometheus
```
systemctl restart prometheus
```
Perform similar steps for Alertmanager logs.

Alertmanager

Create a new configuration file at /etc/logrotate.d/alertmanager with the following content:

/var/log/confluent/control-center/alertmanager.log {
   size 10MB
   rotate 5
   compress
   delaycompress
   missingok
   notifempty
   copytruncate
}

Create a script at /usr/local/bin/logrotate-alertmanager.sh:

#!/bin/bash
/usr/sbin/logrotate -s /var/lib/logrotate/status-alertmanager /etc/logrotate.d/alertmanager

Make the script executable

chmod +x /usr/local/bin/logrotate-alertmanager.sh

To schedule with Cron, add the following line to your crontab (crontab -e):

*/10 * * * * /usr/local/bin/logrotate-alertmanager.sh >> /tmp/alertmanager-rotate.log 2>&1

Restart Alertmanager
```
systemctl restart alertmanager
```

Secure your deployment

To secure Control Center, including TLS, SASL, authentication, RBAC, and securing the Prometheus and Alertmanager components, see Control Center Security on Confluent Platform.

Verify Control Center is running

After the installation is complete, visit http(s)://<c3-url>:9021 and wait for the metrics to start showing up in Control Center. It may take a couple of minutes. Control Center looks exactly like Control Center (Legacy).

To confirm Control Center is running, use the following steps:

Open the network tab in Control Center.
Reload Control Center.
Locate the following API call: /2.0/feature/flags
Verify the following key is present in the response: confluent.controlcenter.prometheus.enable: true

Prometheus and Alertmanager startup flags reference

Control Center bundles Prometheus and Alertmanager and starts them using the bin/prometheus-start and bin/alertmanager-start wrapper scripts. These scripts inject the startup flags that Control Center requires to reload alert configuration, receive metrics from brokers and KRaft controllers, and bind to the addresses that Control Center expects.

This section documents those flags as a reference, so you can understand the setup that Control Center performs and triage alerting and metrics issues. Control Center does not support a customer-managed Prometheus or Alertmanager. The following is informational reference for the setup that Control Center performs, not a guide to self-managing these services.

Important

Always start and restart Prometheus and Alertmanager using the Confluent-provided bin/prometheus-start and bin/alertmanager-start scripts (or, on package installs, the prometheus and alertmanager services). Do not start or restart these processes manually. Starting them directly bypasses the wrapper scripts and drops the flags that Control Center requires, which causes alerting and metrics failures.

Prometheus startup flags

The bin/prometheus-start script applies the following flags.

Flag	Why Control Center needs it	Symptom if missing
`--web.enable-lifecycle`	Lets Control Center reload the scrape and alert configuration through `POST /-/reload` when a trigger is saved.	Trigger creation fails with an “oops, something went wrong” error. An HTTP Archive (HAR) capture shows `403 Lifecycle API is not enabled.`
`--web.enable-otlp-receiver`	Lets brokers and KRaft controllers push metrics to `/api/v1/otlp/v1/metrics`.	“Metrics not available” errors and empty dashboards.
`--web.listen-address=<host:port>`	Sets the address Prometheus listens on. Must match `confluent.controlcenter.prometheus.url` (default `0.0.0.0:9090`).	Prometheus shows “Not Running” in the Control Center user interface.
`--web.config.file=<web-config-prom.yml>`	Configures TLS and basic authentication for the Prometheus web endpoint.	Secured communication fails.
`--config.file`, `--storage.tsdb.path`, `--storage.tsdb.retention.time` (default `15d`), `--log.level=info`	Set the configuration file location, the data path, the metrics retention period, and the log level.	Incorrect retention or no data.

Alertmanager startup flags

The bin/alertmanager-start script applies the following flags.

Flag	Why Control Center needs it	Symptom if missing
`--web.listen-address=<host:port>`	Sets the address Alertmanager listens on. Must match `confluent.controlcenter.alertmanager.url`. The default Alertmanager port is `9093`. For details, see the following port section.	Alertmanager is unreachable.
`--cluster.advertise-address=<host:port>`	Binds an advertise address for the gossip mesh when no private IP can be auto-detected.	Alertmanager fails to start with `couldn't deduce an advertise address:no private IP found, explicit advertise addr not provided`, followed by `unable to initialize gossip mesh ... No private IP address found, andexplicit IP not provided`. For triage steps, see Alertmanager won’t start with “couldn’t deduce an advertise address”.
`--config.file`, `--storage.path`, `--web.config.file`, `--log.level=info`	Set the configuration file location, the data path, TLS configuration, and the log level.	Alertmanager won’t start or can’t route alerts.

Alertmanager port: 9093 versus 9098

Alertmanager listens on one of two ports, depending on your installation type:

The Alertmanager default port is 9093. This is the default in the bin/alertmanager-start script (ALERTMANAGER_PORT=9093) and in the shipped control-center-production.properties file (alertmanager.url=http://localhost:9093).
Port 9098 is used only in single-node local installations, where KRaft controllers also default to 9093 and would otherwise collide with Alertmanager on the same host. In that scenario, the install steps move Alertmanager to 9098 with export ALERTMANAGER_PORT=9098. Treat 9098 as a co-located single-node override, not as a general default.

Validate Prometheus and Alertmanager

Use the following endpoints to validate that Prometheus and Alertmanager are running with the expected configuration:

GET /-/healthy confirms the process is healthy.
GET /-/ready confirms the process is ready to serve requests.
GET /api/v1/status/config returns the running Prometheus configuration, which is useful for confirming that the expected flags and scrape or alert configuration are applied.

Check the bundled versions

Control Center ships specific versions of Prometheus and Alertmanager, which can change between releases. To check the versions bundled with your installation, run:

bin/prometheus --version

bin/alertmanager --version