Connecting Control Center to Confluent Cloud

Confluent Control Center can monitor data streams that are running in Confluent Cloud.

To configure this, you must enable interceptors for producers and consumers, and enable Control Center to write data to Confluent Cloud.

Prerequisites
  • Access to Confluent Cloud
  • Confluent Cloud CLI installed

Here are the basic configuration steps:

  1. Using an API key and secret, generate a configuration to connect to Confluent Cloud using one of these methods:

    • Use ccloud init to generate your configuration
    • Download the Java configuration from the Confluent Cloud web page and provide the API secret.
  2. Validate that Confluent Cloud can be accessed from the machine where you are installing Control Center.

    • Check connection by using ccloud topic list.
    • Try producing or consuming from the machine.
  3. Install Control Center using the documentation.

  4. Configure Control Center with the Confluent Cloud specific settings. A minimum valid configuration is shown below. These settings are different from the standard Confluent Cloud configuration. Customize the bootstrap.servers and confluent.controlcenter.streams.sasl.jaas.config for your Confluent Cloud cluster.

    bootstrap.servers=<cloud-bootstrap-servers>
    confluent.controlcenter.streams.security.protocol=SASL_SSL
    confluent.controlcenter.streams.sasl.mechanism=PLAIN
    confluent.controlcenter.streams.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="<api-key-id>" \
      password="<secret-access-key>";
    confluent.controlcenter.streams.ssl.endpoint.identification.algorithm=https
    
  1. Configure data stream interceptors by following the documentation. Here is an example of the required security configuration that must be added:

    confluent.monitoring.interceptor.security.protocol=SASL_SSL
    confluent.monitoring.interceptor.sasl.mechanism=PLAIN
    confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api-key-id>" password="<secret-access-key>";
    confluent.monitoring.interceptor.ssl.endpoint.identification.algorithm=https
    

Example

This example shows the topics that are created when Control Center starts:

$ ccloud topic list
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTERTHIS-0000000095-store-changelog
_confluent-controlcenter-4-0-0-1-AlertHistoryStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringStream-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-error-topic
_confluent-controlcenter-4-0-0-1-cluster-rekey
_confluent-controlcenter-4-0-0-1-MonitoringTriggerStore-changelog
_confluent-controlcenter-4-0-0-1-actual-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-Group-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-expected-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-monitoring-message-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-changelog
_confluent-metrics
_confluent-controlcenter-4-0-0-1-Group-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-group-stream-extension-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-monitoring-trigger-event-rekey
_confluent-controlcenter-4-0-0-1-TriggerActionsStore-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition-changelog
_confluent-command
_confluent-controlcenter-4-0-0-1-TriggerEventsStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringVerifierStore-changelog
_confluent-controlcenter-4-0-0-1-metrics-trigger-measurement-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-repartition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE-changelog
_confluent-monitoring
_confluent-controlcenter-4-0-0-1-MonitoringStream-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTEROTHER-0000000096-store-chang

Limitations

Control Center does not show system health details. This is because Confluent Cloud does not provide the instrumentation from Confluent Metrics Reporter outside of the Confluent Cloud. Confluent Platform internally monitors the system and broker health and takes actions based on that monitoring.

Control Center requires an Apache Kafka cluster to store data. Control Center creates partitions for data storage, that count toward partition limits, and data throughput within Confluent Cloud.

Control Center will log this warning which can be ignored:

WARN broker=15 is not instrumented with ConfluentMetricsReporter (io.confluent.controlcenter.util.HealthCheck)elog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS