Confluent Telemetry Reporter

Overview

The Confluent Telemetry Reporter is a plugin that runs inside each Confluent Platform service to push metadata about the service to Confluent. Telemetry Reporter enables product features based on the metadata, like Health+. Data is sent over HTTP using an encrypted connection.

Important

  • If Self-Balancing is enabled on the cluster, the Telemetry Reporter is enabled and running by default as a requirement of Self-Balancing Clusters. The Self-Balancing algorithm consumes from the Telemetry Reporter internal topic _confluent-telemetry-metrics.
  • if Self-Balancing is not enabled on the cluster, the Telemetry Reporter can be fully disabled using the property setting confluent.reporters.telemetry.auto.enable=false in $CONFLUENT_HOME/etc/kafka/server.properties.
  • No data is sent to Confluent Cloud if you set confluent.telemetry.enabled to false.

For the list of available metrics, see Telemetry Reporter Metrics.

Installation

Confluent Telemetry Reporter is installed as part of the full Confluent Platform installation. If you choose to install packages individually, ensure that Telemetry Reporter is installed by using yum install confluent-telemetry.

Enable telemetry reporting to Confluent Cloud

Prerequisites

Create a Cloud API key to authenticate with Confluent Cloud. For example:

ccloud login
ccloud api-key create --resource cloud

Note

You must use a Cloud API Key. Using the Cluster API Key that is used to communicate with Kafka causes an authentication error.

Configuration with Ansible Playbooks for Confluent Platform

If you use Ansible Playbooks for Confluent Platform to manage your Confluent Platform installation, add the following configuration overrides to all Confluent Platform roles and deploy the changes.

telemetry_enabled: true
telemetry_api_key: <api-key>
telemetry_api_secret: <api-secret>

Configuration with Confluent for Kubernetes

When you use Confluent for Kubernetes (CFK) to manage Confluent Platform, you can enable telemetry as described in the Confluent for Kubernetes documentation.

Custom Deployments Configuration

For all Confluent Platform components other than Confluent Server, enable telemetry reporting by adding the following lines to the configuration file for the service and restarting the service to deploy the new configuration. Replace <API_KEY> and <API_SECRET> with the key and secret generated above.

metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter
confluent.telemetry.enabled=true
confluent.telemetry.api.key=<API_KEY>
confluent.telemetry.api.secret=<API_SECRET>

For Confluent Server, the metric.reporters configuration is not needed. Add the following lines to the Confluent Server configuration file and restart Confluent Server. If restarting Confluent Server is undesirable, you can add these configurations by using dynamic configuration and the kafka-config CLI.

confluent.telemetry.enabled=true
confluent.telemetry.api.key=<API_KEY>
confluent.telemetry.api.secret=<API_SECRET>

Configuration with a proxy

Confluent Telemetry Reporter supports routing telemetry data through an authenticated HTTP proxy. In these environments, set the following parameters to have telemetry data egress through the proxy.

confluent.telemetry.proxy.url=http://<proxy_server_ip>:<proxy_server_port>
confluent.telemetry.proxy.username=<proxy-username>
confluent.telemetry.proxy.password=<proxy-password>

What data is sent and how it is used

When a customer has Confluent Telemetry Reporter enabled for a specific Confluent Platform component, Confluent collects metrics about the Confluent Platform component that include runtime performance metrics, the Kafka version, the Confluent Platform version, a unique identifier that’s valid for the applicable Confluent Platform component, a unique identifier for the Kafka cluster, and a unique identifier for the Customer organization. The data is sent using an encrypted connection to Confluent servers once per minute by default. The resulting data is used by Confluent to offer Health+ to its customers and to maintain and improve Confluent Products and Services.For this data to be sent, enable the transmission by setting configuration flags in every Confluent Platform component. You can stop sending data at any time by removing these configuration flags.

Dynamic Configuration

Note

Currently, dynamic configuration applies only to Kafka brokers.

To understand how to apply dynamic configs for your cluster, see Kafka Dynamic Configurations.

Enable Telemetry without a restart

If you’re using Confluent Server, you can enable the Telemetry Reporter without a restart by using the following command. Replace <API_KEY> and <API_SECRET> with the key and secret generated above.

./bin/kafka-configs --bootstrap-server localhost:9092 --alter --entity-type brokers --entity-default --add-config confluent.telemetry.enabled=true,confluent.telemetry.api.key=<API_KEY>,confluent.telemetry.api.secret=<API_SECRET>

Supported dynamic configuration properties

  • confluent.telemetry.enabled
  • confluent.telemetry.api.key
  • confluent.telemetry.api.secret
  • confluent.telemetry.proxy.url
  • confluent.telemetry.proxy.username
  • confluent.telemetry.proxy.password