Connect Confluent Platform Control Center to Confluent Cloud

Confluent Control Center can monitor data streams that are running in Confluent Cloud.

To configure this, you must enable interceptors for producers and consumers, and enable Control Center to write data to Confluent Cloud.

Tip

For an example of how to connect Control Center to Confluent Cloud using Docker, see this example.

Prerequisites
  • Access to Confluent Cloud.
  • Confluent CLI.
  • Confluent Cloud users require an additional subscription to use Control Center. Control Center subscription is included by default for Confluent Cloud users with committed usage.

Configuration steps

Following are the basic configuration steps:

  1. Using an account with OrganizationAdmin access, create an API key and secret to connect to Confluent Cloud. For details, refer to Use API Keys to Authenticate to Confluent Cloud.

  2. Validate that Confluent Cloud can be accessed from the machine where you are installing Control Center.

    • Check connection by using confluent kafka topic list.
    • Try producing or consuming from the machine.

  3. Install Control Center using the documentation.

  4. Configure Control Center with the Confluent Cloud specific settings. A minimum valid configuration is shown below. These settings are different from the standard Confluent Cloud configuration. Customize the bootstrap.servers and confluent.controlcenter.streams.sasl.jaas.config for your Confluent Cloud cluster.

    bootstrap.servers=<cloud-bootstrap-servers>
confluent.controlcenter.streams.security.protocol=SASL_SSL
confluent.controlcenter.streams.sasl.mechanism=PLAIN
confluent.controlcenter.streams.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="<api-key>" \
  password="<api-secret>";
confluent.metrics.topic.max.message.bytes=8388608
confluent.controlcenter.streams.ssl.endpoint.identification.algorithm=https

    Important

    The confluent.metrics.topic.max.message.bytes property must be set to 8388608. See Control Center Cannot Connect to Confluent Cloud for details.

  5. Configure data stream interceptors by following the documentation security configuration that must be added:

    confluent.monitoring.interceptor.security.protocol=SASL_SSL
confluent.monitoring.interceptor.sasl.mechanism=PLAIN
confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api-key>" password="<api-secret>";
confluent.monitoring.interceptor.ssl.endpoint.identification.algorithm=https

  6. (Optional) Add configs for Confluent Cloud Schema Registry per the example in control-center-ccloud.delta on GitHub at ccloud/examples/template_delta_configs. The schema.registry.url for Control Center is specified using an HTTPS protocol prefix which requires an explicit 443 port, as shown in the example.

    # Confluent Schema Registry configuration for Confluent Control Center
confluent.controlcenter.schema.registry.basic.auth.credentials.source=USER_INFO
confluent.controlcenter.schema.registry.basic.auth.user.info=<SCHEMA_REGISTRY_API_KEY>:<SCHEMA_REGISTRY_API_SECRET>
confluent.controlcenter.schema.registry.url=https://<SCHEMA_REGISTRY_ENDPOINT>:443

Troubleshooting

Confluent Control Center relies on the _confluent-metrics, _confluent-monitoring and _confluent-command internal topics to operate. If you receive an UNKNOWN_TOPIC_OR_PARTITION error for one of these topics, you should:

  • For Basic, Standard, and Enterprise clusters, manually create the topics in Confluent Cloud.
  • For Dedicated clusters, enable automatic topic creation.

Example

This example shows the topics that are created when Control Center starts:

 confluent kafka topic list
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTERTHIS-0000000095-store-changelog
_confluent-controlcenter-4-0-0-1-AlertHistoryStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringStream-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-error-topic
_confluent-controlcenter-4-0-0-1-cluster-rekey
_confluent-controlcenter-4-0-0-1-MonitoringTriggerStore-changelog
_confluent-controlcenter-4-0-0-1-actual-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-Group-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-expected-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-monitoring-message-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-changelog
_confluent-metrics
_confluent-controlcenter-4-0-0-1-Group-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-group-stream-extension-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-monitoring-trigger-event-rekey
_confluent-controlcenter-4-0-0-1-TriggerActionsStore-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition-changelog
_confluent-command
_confluent-controlcenter-4-0-0-1-TriggerEventsStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringVerifierStore-changelog
_confluent-controlcenter-4-0-0-1-metrics-trigger-measurement-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-repartition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE-changelog
_confluent-monitoring
_confluent-controlcenter-4-0-0-1-MonitoringStream-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTEROTHER-0000000096-store-chang

Docker environment

You can run a mix of fully-managed services in Confluent Cloud and self-managed components running in Docker. For a Docker environment that connects any Confluent Platform component to Confluent Cloud, see cp-all-in-one-cloud.

Limitations

  • Control Center does not show system health details. This is because Confluent Cloud does not provide the instrumentation from Confluent Metrics Reporter outside of the Confluent Cloud. Confluent Platform internally monitors the system and broker health and takes actions based on that monitoring. We recommend you enable Reduced infrastructure mode for Control Center.

  • Topics hosted in Confluent Cloud and viewed in Control Center do not display Availability, Offset and Size values on their Topics detail page. In this scenario, these fields display 0.

  • Control Center requires an Apache Kafka® cluster to store data. Control Center creates partitions for data storage, that count toward partition limits, and data throughput within Confluent Cloud.

  • When a Control Center cluster is bootstrapped to Confluent Cloud, no metrics are available for the cluster. Therefore, metrics alerts do not work. However, consumer lag and cluster up/down alerts work as expected.

  • Control Center will log this warning, which can be ignored:

    WARN broker=15 is not instrumented with ConfluentMetricsReporter (io.confluent.controlcenter.util.HealthCheck)elog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS

Suggested reading