# Troubleshoot Control Center for Confluent Platform
## Common issues
### View processing status in Control Center
View a high-level summary of running and processing status within the Control Center
application. Check the status at any time.
1. In the upper-right corner of Control Center, click the menu icon to open the
**Administration** menu.
2. Click **About Control Center**.
**Processing Status** shows the status of Control Center (Running or Not Running). Consumption
data and Broker data message processing speeds are shown in real-time since
30 minutes ago.
### Control Center is not starting up
Check log files for error. If you see streams related errors, try resetting Control Center Id and restarting. For
more information, see [How to reset Confluent Control Center](https://support.confluent.io/hc/en-us/articles/360052171252-How-to-reset-Confluent-Control-Center)
### Prometheus or Alertmanager show “Not Running” in the user interface
1. Check if Prometheus and Alertmanager are running:
```none
ps aux | grep prometheus
```
```none
ps aux | grep alertmanager
```
2. Check if Prometheus and Alertmanager are healthy
```none
Prometheus GET http(s)://c3-hostname:9090/-/healthy
```
```none
Alertmanager GET http(s)://c3-hostname:9093/-/healthy
```
3. Check log files in case of Confluent Ansible and manual deployment. For CFK, logs are present in support bundle.
```none
Prometheus /var/log/confluent/control-center/prometheus.log
```
```none
Alertmanager /var/log/confluent/control-center/alertmanager.log
```
### Prometheus or Alertmanager won’t start
1. Check if Prometheus and Alertmanager are running:
```none
ps aux | grep prometheus
```
```none
ps aux | grep alertmanager
```
2. Check log files in case of Confluent Ansible and manual deployment. For CFK, logs are present in support bundle.
```none
Prometheus /var/log/confluent/control-center/prometheus.log
```
```none
Alertmanager /var/log/confluent/control-center/alertmanager.log
```
### Trigger creation fails or Prometheus returns “Lifecycle API is not enabled”
When you create or save an alert trigger, Control Center reloads the Prometheus
configuration through `POST /-/reload`. This requires Prometheus to run with
the `--web.enable-lifecycle` flag, which the bundled `bin/prometheus-start`
script applies.
Symptoms:
* Saving a trigger fails with an “oops, something went wrong” error.
* An HTTP Archive (HAR) capture of the request shows `403 Lifecycle API is not enabled.`
This usually happens when Prometheus is restarted manually, outside the bundled
script, which drops the `--web.enable-lifecycle` flag.
To resolve, restart Prometheus using the Confluent-provided
`bin/prometheus-start` script (or the `prometheus` service on package
installs) so the required flags are reapplied. For a reference of the flags
Control Center applies, see [Prometheus and Alertmanager startup flags reference](multi-node.md#c3-prometheus-alertmanager-startup-reference).
### Alertmanager won’t start with “couldn’t deduce an advertise address”
Alertmanager fails to start and the log shows errors similar to:
```none
couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided
unable to initialize gossip mesh ... No private IP address found, and explicit IP not provided
```
This happens when Alertmanager cannot auto-detect a private IP for the gossip
mesh and no advertise address is provided.
To resolve, ensure the `alertmanager-start` invocation includes both
`--web.listen-address` (for example, `0.0.0.0:9093`) and
`--cluster.advertise-address` (for example, `:9093`).
Control Center 2.2.0 and later include these flags by default.
#### NOTE
A manual edit to the bundled `bin/alertmanager-start` script is overwritten
on a package upgrade or reinstall. Upgrade to Control Center 2.2.0 or later to
make the fix durable.
For a reference of the flags Control Center applies, see
[Prometheus and Alertmanager startup flags reference](multi-node.md#c3-prometheus-alertmanager-startup-reference).
### Control Center shows metrics not available with running and healthy Prometheus
This error is displayed when brokers are not emitting metrics or Prometheus is not receiving metrics from brokers.
1. Check if brokers are configured to export metrics to Prometheus.
Broker configurations (server.properties) should have all the configurations specified in the multi-node manual installation.
For more information, see [Confluent Control Center Deployment Overview](overview.md#install-c3).
2. Check if brokers are able to resolve Control Center endpoint:
```none
curl -X POST http(s)://c3-host:9021/api/v1/otlp/v1/metrics
```
3. Check for errors in the Prometheus logs.
- For CFK install, logs are present in support bundle.
For Confluent Ansible or manual install, find log path using this:
```none
ps aux | grep prometheus
```
### Stale metrics in Prometheus
1. Check if brokers are configured to export metrics to Prometheus.
Broker configurations (server.properties) should have all the configurations specified in the multi-node manual installation.
For more information, see [Confluent Control Center Deployment Overview](overview.md#install-c3).
2. Check if brokers are able to resolve Control Center endpoint:
```none
curl -X POST http(s)://c3-host:9021/api/v1/otlp/v1/metrics
```
3. Check for errors in the Prometheus logs.
- For CFK install, logs are present in support bundle.
For Confluent Ansible or manual install, find log path using this:
```none
ps aux | grep prometheus
```
### Broker Count and Partition Count are missing in KRaft Mode
1. Check if KRaft controllers are configured to export metrics to Prometheus.
Broker configurations should have all the configurations specified in the multi-node manual installation.
For more information, see [Confluent Control Center Deployment Overview](overview.md#install-c3).
2. Check if controllers are able to resolve Control Center endpoint:
```none
curl -X POST http(s)://c3-host:9021/api/v1/otlp/v1/metrics
```
3. Check for errors in the Prometheus logs.
- For CFK install, logs are present in support bundle.
For Confluent Ansible or manual install, find log path using this:
```none
ps aux | grep prometheus
```
### Alerts not triggered
1. Check if alert condition if triggered in Prometheus user interface:
```none
http://c3-host:9090/alerts
```
2. Check if Alertmanager user interface shows the alert:
```none
http://c3-host:9093/#/alerts
```
3. Check log files in case of Confluent Ansible or manual deployment at location
Prometheus:
```none
/var/log/confluent/control-center/prometheus.log
```
Alertmanager:
```none
/var/log/confluent/control-center/alertmanager.log
```
### Error message in Prometheus log
You see this message in the Prometheus log:
```none
:time=2024-12-19T13:41:57.309-05:00 level=WARN source=write_handler.go:520 msg="Error translating OTLP metrics to Prometheus write request" component=web
err="the following errors occurred:\n - invalid temporality and type combination for metric \"io.confluent.kafka.server/request/queue_size/delta\"\n -
invalid temporality and type combination for metric \"io.confluent.kafka.server/request/response_queue_time_ms/time/delta\"\n - invalid temporality and
type combination for metric \"io.confluent.kafka.server/request/response_queue_time_ms/total/delta\"\n - invalid temporality and type combination for me
tric \"io.confluent.kafka.server/request/health_check_total_time_ms/time/delta\"\n - invalid temporality and type combination for metric \"io.confluent.
kafka.server/request/health_check_total_time_ms/total/delta\"\n
```
Non-supported metric type have been whitelisted with the `confluent.telemetry.exporter._c3.metrics.include` configuration. Update your
broker and KRaft controller configurations using the recommendations in the installation topic. For more information, see [Confluent Control Center Deployment Overview](overview.md#install-c3).
### After broker recovery, you see negative number of out-of-sync replicas
You may see negative numbers of out-of-sync replicas after a broker starts up again. The metrics may take about two minutes to return to the correct
state.
## Installing and Setup
If you encounter issues during installation and setup, you can try these solutions.
### Bad security configuration
* Check the security configuration for all brokers, Telemetry Reporter, and Control Center (see [debugging check configuration](#check-configurations)).
For example, is it SASL_SSL, SASL_PLAINTEXT, SSL?
* Possible errors include:
```bash
ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
```
```bash
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka configuration
```
```bash
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected handshake request with client mechanism GSSAPI, enabled mechanisms are [GSSAPI]
```
* Verify that the correct Java Authentication and Authorization Service (JAAS) configuration was detected.
* If ACLs are enabled, check them.
* To verify that you can communicate with the cluster, try to produce and consume using `console-*` with the same security settings.
### InvalidStateStoreException
* This error usually indicates that data is corrupted in the configured `confluent.controlcenter.data.dir`. For example, this can be caused by an unclean shutdown. To fix, give Control Center a new ID by changing `confluent.controlcenter.id` and restart.
* Allow permission for the configured `confluent.controlcenter.data.dir`.
### Not enough brokers
Check the logs for the related error `not enough brokers`. Verify the [topic replication factors](#check-configurations) are set correctly and verify that there are enough brokers available.
### Local store permissions
Check the local permissions in Control Center state directory. These settings are as defined in the config `confluent.controlcenter.data.dir` in the `control-center.properties`. You can access that directory with the user ID that was used to start Control Center.
### Multiple instances of Control Center have the same ID
You must use unique IDs for each Control Center instance, including instances in Docker.
Duplicate IDs are not supported and will cause problems.
### License expired
If you see a message similar to this:
```bash
[2017-08-21 14:12:33,812] WARN checking license failure. contact `support@confluent.io `_ for a license key: Unable to process JOSE object (cause: org.jose4j.lang.JoseException: Invalid JOSE….
```
You should verify that the user has a valid license, as specified in `confluent.license=`. This can be either the key or a path to a license file.
For more information, see the [Control Center configuration documentation](configuration.md#controlcenter-configuration). To manage a license in
the Control Center web interface, see [Manage Confluent Platform Licenses Using Control Center](license.md#controlcenter-licenses).
### A schema for message values has not been set for this topic
If you encounter this error message, you should verify that the Schema Registry `listeners=http://0.0.0.0:8081` configuration
matches the Control Center `confluent.controlcenter.schema.registry.url=http://localhost:8081` configuration. For more
information, see [Manage Control Center Logs for Confluent Platform](logging.md#control-center-logging-settings).
### Cannot browse topic messages using Safari and authentication
Due to a
[WebKit bug in Safari](https://bugs.webkit.org/show_bug.cgi?id=80362),
the following unknown error displays when using
authentication and attempting to
[browse topic messages](../topics/messages.md#c3-topic-message-browser):
The Safari browser fails to send authenticated requests through the WebSocket
protocol.
The recommended workaround in the interim is to use the Chrome or Firefox
browsers rather than
Safari.
## System health
### Web interface that is blank or stuck loading
If you experience a web interface that is blank or stuck loading, you can select the cluster in the drop-down and use the information below to troubleshoot.
* Are there [errors or warnings in the logs](#check-logs)? For more information on how to find logs, see the [documentation](configuration.md#controlcenter-configuration).
* [What are you monitoring](#size-of-clusters)? Are you [under-provisioned](#system-check)?
* Is there [a lag in Control Center](#consumer-offset-lag)?
* Use browser debugging tools to check REST calls to find out if the requests have been made successfully and with a valid response, specifically these requests:
* The `/3.0/metrics//maxtime` endpoint should return the latest
timestamp that Control Center has for metrics data.
* If no data is returned from the backend, verify that you’re getting data on
the input topic and review the logs for issues.
### The Control Center is getting ready to launch
If this message appears in the web interface after starting or restarting
Control Center, click **Try again**.
If Control Center doesn’t launch, try the suggestions below to troubleshoot:
* Usually this means that Apache Kafka® doesn’t have any metrics data, but this
message could also indicate a 500 Internal Server Error has occurred.
If you get a 500 error, check the Control Center logs for errors.
* Use browser debugging tools to check the response. An empty
response (`{ }`) from the `/3.0/metrics//maxtime` endpoint
means that Kafka hasn’t received any metrics data.
* Verify that the Telemetry Reporter is set up correctly.
* Check Prometheus.
### No clusters found
If a message appears in the web interface saying `No clusters found` after starting
Control Center, check your [configuration](configuration.md#controlcenter-configuration) in the
appropriate `control.center.properties`
file.
### Control Center cannot start due to temporary directory permissions
Issue: Get an error about `/tmp` mounted with `noexec`:
```bash
java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni3375578050467151433.so:
/tmp/librocksdbjni3375578050467151433.so: failed to map segment from shared object:
Operation not permitted when /tmp is mounted with noexec
```
Resolution: If you do not have write access to the `/tmp` directory because it is set to `noexec`, pass in
a directory path for `rocksdbtmp` that you have write access to and start Control Center.
```bash
CONTROL_CENTER_OPTS="-Djava.io.tmpdir=/my/dir/for/rocksdbtmp" control-center-start /path/to/control-center.properties
```
### Parts of the broker or topic table have blank values
This is a known issue that should be transient until Control Center is caught up. It can be caused by Control Center lagging or having trouble keeping up due to lack of resources.
## RBAC
### Max requests queued per destination 1024 exceeded
This is a rare issue that occurs due to the interaction of the custom MDS client
used with Control Center when RBAC is enabled. If you receive this error, try the following:
- Restart Control Center if possible
- Increase the `confluent.controlcenter.mds.client.max.requests.queued.per.destination`
configuration value and lower the `confluent.controlcenter.mds.client.idle.timeout` value.
For more, see [RBAC](configuration.md#c3-rbac-settings).
### Failed to connect to any MDS server
When RBAC is enabled and you have many Connect, ksqlDB, or Schema Registry clusters
configured, the Connect clusters, ksqlDB, or Schema Registry pages can fail to load.
The pages show empty lists or return a 502 error with the message
`failed to connect to any MDS server`. Restarting Control Center makes the
symptom go away temporarily, only for it to return shortly after.
This error is misleading because the Metadata Service (MDS) is healthy. The actual cause is
your session token expiring mid-request. Control Center fetches the RBAC
scope for each configured cluster to determine which clusters you can see,
and one or more slow or unreachable clusters can make the cumulative wait time
exceed the session token lifetime. When that happens, the final MDS visibility
call goes out with an expired token, which Control Center surfaces as
`failed to connect to any MDS server`.
This issue affects deployments with all of the following:
- RBAC enabled
- Many Connect, ksqlDB, or Schema Registry clusters configured
- At least one of those clusters slow or unreachable
Single-cluster deployments and non-RBAC deployments are not affected.
To resolve the issue, try the following remediations, in this order:
1. Upgrade to Control Center 2.6.0 or later. Starting in 2.6.0, Control Center
fetches the per-cluster RBAC scopes in parallel and handles up to 10 slow
clusters in a single batch by default. For most deployments, this resolves
the issue without any further changes.
2. If you are already on Control Center 2.6.0 or later and still see the error,
increase `confluent.controlcenter.rbac.scope.fetch.parallelism` from the
default `10` to at least the number of slow or unreachable clusters.
Start at `20`, then `30`, and so on, until the symptom clears. Setting
this value higher than the total number of clusters that you have configured
provides no benefit. For more information, see
[confluent.controlcenter.rbac.scope.fetch.parallelism](configuration.md#c3-rbac-scope-fetch-parallelism).
3. If raising the parallelism to match the total cluster count does not
resolve the issue, the bottleneck is in the clusters themselves rather than
in Control Center. Investigate the slow Connect, ksqlDB, or Schema Registry clusters
at the source. Common causes include TLS misconfiguration, DNS issues, or an
unreachable host.
## Connect
### The Control Center is getting ready to launch
If this message appears in the web interface, use the information below to troubleshoot.
* Is the Connect cluster that is defined in `confluent.controlcenter.connect.cluster` available?
* Can you reach the Connect endpoints directly by running a cURL command (e.g., `curl www.example.com`)?
* Check the Connect logs for any errors. Control Center is a proxy to Connect.
## Debugging
### Check logs
These are the Control Center log types. For more information about logging, see [Manage Control Center Logs for Confluent Platform](logging.md#control-center-logging-settings).
* `control-center.log` - Control Center, HTTP activity, anything not related to streams, REST API calls
* `control-center-streams.log` - Streams
* `control-center-kafka.log` - Client, ZooKeeper, and Kafka
Here are things to look for in the logs:
* `ERROR`
* `shutdown`
* `Exceptions` - verify that the brokers can be reached
* `WARN`
* `Healthcheck` errors and warnings
If nothing is obvious, turn [DEBUG logging on](#enable-debug-and-trace-logging)
and restart Control Center.
### Enable debug and trace logging
1. Open the `CONFLUENT_HOME/etc/confluent-control-center/log4j2.yaml`
file. This file is referenced by the `CONTROL_CENTER_log4j2_OPTS` environment
variable.
2. Set and export the `CONTROL_CENTER_log4j2_OPTS` environment variable similar
to this example:
```bash
export CONTROL_CENTER_log4j2_OPTS='-Dlog4j2.configuration=file:/apps/kafka/config/confluent-control-center/log4j2.yaml'
```
3. Set your debugging options:
- To enable debug logging, change the log level to `DEBUG` at the root level:
```yaml
Loggers:
Root:
level: debug
AppenderRef:
- ref: Stdout
```
- To enable trace logging, change the root logger to `TRACE` at the root level:
```yaml
Loggers:
Root:
level: trace
AppenderRef:
- ref: Stdout
```
- To enable additional streams logging, particularly at the request of
Confluent Support, follow this example:
```yaml
Configuration:
status: WARN
Appenders:
Console:
# Standard output appender for general application logs
- name: Stdout
Target: SYSTEM_OUT
PatternLayout:
pattern: "[%d{yyyy-MM-dd HH:mm:ss,SSS}] %p [%t] %m (%c)%n"
# Dedicated appender for Kafka Streams and Control Center streams logging
# Uses rate filtering to prevent log flooding while capturing TRACE details
- name: Streams
Target: SYSTEM_OUT
PatternLayout:
pattern: "[%d{yyyy-MM-dd HH:mm:ss,SSS}] %p [%t] %m (%c)%n"
Filters:
- Log4jRateFilter:
# Allows everything that is greater than or equal to specified level
level: TRACE
# Allows rate/second logs at less than specified level
#rate: 25
Loggers:
Root:
level: debug
AppenderRef:
- ref: Stdout
Logger:
- name: org.apache.kafka.streams
level: info
AppenderRef:
- ref: Streams
additivity: false
- name: io.confluent.controlcenter.streams
level: info
AppenderRef:
- ref: Streams
additivity: false
- name: kafka
level: error
AppenderRef:
- ref: Stdout
- name: org.apache.kafka
level: error
AppenderRef:
- ref: Stdout
- name: org.apache.kafka.clients.consumer
level: info
AppenderRef:
- ref: Stdout
- name: org.apache.zookeeper
level: error
AppenderRef:
- ref: Stdout
- name: org.I0Itec.zkclient
level: error
AppenderRef:
- ref: Stdout
```
4. [Restart](properties.md#c3-restart) Control Center. For more information, see
[Control Center Configuration Examples for Confluent Platform](properties.md#c3-properties-files).
```bash
./bin/control-center-stop
./bin/control-center-start ../etc/confluent-control-center/control-center.properties
```
5. When you are done debugging and tracing, reset the log levels back to their
defaults and restart Control Center:
```yaml
Configuration:
status: WARN
Appenders:
Console:
- name: Stdout
Target: SYSTEM_OUT
PatternLayout:
pattern: "[%d{yyyy-MM-dd HH:mm:ss,SSS}] %p [%t] %m (%c)%n"
- name: Streams
PatternLayout:
pattern: "[%d{yyyy-MM-dd HH:mm:ss,SSS}] %p [%t] %m (%c)%n"
Filters:
- Log4jRateFilter:
level: WARN
rate: 25
Loggers:
Root:
level: info
AppenderRef:
- ref: Stdout
Logger:
- name: org.apache.kafka.streams
level: info
AppenderRef:
- ref: Streams
additivity: false
- name: io.confluent.controlcenter.streams
level: info
AppenderRef:
- ref: Streams
additivity: false
- name: kafka
level: info
AppenderRef:
- ref: Stdout
- name: org.apache.kafka
level: info
AppenderRef:
- ref: Stdout
- name: org.apache.kafka.clients.consumer
level: info
AppenderRef:
- ref: Stdout
- name: org.apache.zookeeper
level: warn
AppenderRef:
- ref: Stdout
- name: org.I0Itec.zkclient
level: warn
AppenderRef:
- ref: Stdout
```
### Check configurations
* Is security enabled? Check the [Control Center Security on Confluent Platform](../security/overview.md#controlcenter-security) on the broker, clients, and Control Center.
* Verify that the prefixes are correct.
* Verify the topic configurations for all Control Center topics: replication factor, timestamp type, min isr, and retention.
```bash
./bin/kafka-topics --bootstrap-server --describe
```
### Size of clusters
For examples on how to size your environment, review the [Control Center example deployments](overview.md#install-c3).
### System check
Check the system level metrics where Control Center is running; including CPU,
memory, disk, and JVM settings. Are the settings within the [recommended values](overview.md#install-c3)?
### Frontend request and response
Using your browser’s web developer tools, view **Network** settings to verify that
requests and responses are showing the correct data.
If you are working with Confluent Support to debug browser issues, they may
ask you to capture a HAR file for them to review. HAR files store all requests/responses
between the browser and the server.
For example, to generate the HAR file with Google Chrome:
1. Open Google Chrome and go to the page where the issue is occurring.
2. From the Chrome menu bar, select **View** > **Developer** > **Developer Tools**.
3. In the Developer Tools panel, select the **Network** tab.
4. Look for a round red Record button in the upper left corner of the tab and confirm
that it is red (activated). If it is grey, click it once to start recording.
5. Check the **Preserve log** box to preserve capture across multiple pages.
6. Reproduce the issue in the browser by interacting with the page.
7. Right-click anywhere on the grid of network requests, select **Save as HAR with content**,
and save the file.
You can upload the HAR file to your Confluent Support ticket (or review the contents
of the file if you are doing your own troubleshooting).
### REST API
Backend REST API calls are logged in `c3.log`.
### Enable GC logging
Enable GC logs, restart Control Center with the following, where directory (``) is specified:
```bash
CONTROL_CENTER_JVM_PERFORMANCE_OPTS="-server -verbose:gc -Xloggc:/gc.log -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Djava.awt.headless=true"
```
### Thread dump
Run this command for a thread dump:
```bash
jstack -l $(jcmd | grep -i 'controlcenter\.ControlCenter' | awk '{print $1}') > jstack.out
```
### Data directory
The Control Center local state is stored in `confluent.controlcenter.data.dir`.
You can use this command to determine the size of your data directory (``).
```bash
du -h
```