Kafka Monitoring and Metrics Using JMX with Docker¶
You can monitor Confluent Platform deployments by using Java Management Extensions (JMX) and MBeans.
Confluent offers some alternatives to using JMX monitoring.
- Health+: Consider monitoring and managing your environment with Confluent Health+. Ensure the health of your clusters and minimize business disruption with intelligent alerts, monitoring, and proactive support based on best practices created by the inventors of Kafka.
- Confluent Control Center: You can deploy Control Center for out-of-the-box Kafka cluster monitoring so you don’t have to build your own monitoring system. Control Center makes it easy to manage the entire Confluent Platform deployment. Control Center is a web-based application that allows you to manage your cluster and to alert on triggers. Additionally, Control Center measures how long messages take to be delivered, and determines the source of any issues in your cluster.
Configure environment variables¶
JMX is enabled for Kafka by default. You can set the following environment variables to configure JMX monitoring for your Docker image in a Compose file, a Dockerfile, or from the command line when you run Kafka. For an example of how to set the environment variables when you run Kafka, see Run Kafka with JMX enabled.
- The JMX Port.
- The hostname associated with locally created remote objects.
JMX options. Use this variable to override the default JMX options such as whether authentication is enabled. A JMX client must be able to connect to the
java.rmi.server.hostnamespecified in the
The default options are set as follows:
-Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
The default for a bridged network is the bridged IP address, which means that you will only be able to connect to it from another Docker container. For the host network, this is the IP that the hostname on the host resolves to. Use
hostname -ito see the what the hostname is set to in the Docker container. If you have more than one network configured for the container,
hostname -igives you all the IPs, and the default is the first IP address.
Authentication is disabled for JMX by default in Kafka. The default JMX configuration binds an unauthenticated JMX interface to all network interfaces. You must use environment variables to override this default. To set authentication on JMX, you can follow the SSL and authentication sections in Monitoring and Management Using JMX Technology.
Run Kafka with JMX enabled¶
You run Kafka with JMX enabled in the same way that you normally start it, but you specify the
KAFKA_JMX_HOSTNAME environment variables.
If you are running Kafka in ZooKeeper mode, specify
KAFKA_JMX_HOSTNAME environment variables
when you start both Kafka and ZooKeeper.
Following are example Docker
run commands for Kafka running in KRaft or ZooKeeper mode with JMX configured:
For more information about configuring KRaft in production, see Configure KRaft in Production.
docker run -d \ --name=kafka-jmx \ -h kafka-jmx \ -p 9101:9101 \ -e KAFKA_NODE_ID=1 \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP='CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' \ -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://kafka-jmx:29092,PLAINTEXT_HOST://localhost:9092' \ -e KAFKA_JMX_PORT=9101 \ -e KAFKA_JMX_HOSTNAME=localhost \ -e KAFKA_PROCESS_ROLES='broker,controller' \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e KAFKA_CONTROLLER_QUORUM_VOTERS='1@kafka-jmx:29093' \ -e KAFKA_LISTENERS='PLAINTEXT://kafka-jmx:29092,CONTROLLER://kafka-jmx:29093,PLAINTEXT_HOST://0.0.0.0:9092' \ -e KAFKA_INTER_BROKER_LISTENER_NAME='PLAINTEXT' \ -e KAFKA_CONTROLLER_LISTENER_NAMES='CONTROLLER' \ -e CLUSTER_ID='MkU3OEVBNTcwNTJENDM2Qk' \ confluentinc/cp-kafka:7.4.3
docker run -d \ --name=zk-jmx \ --net=host \ -e ZOOKEEPER_TICK_TIME=2000 \ -e ZOOKEEPER_CLIENT_PORT=32181 \ -e KAFKA_JMX_PORT=9101 \ confluentinc/cp-zookeeper:7.4.3 docker run -d \ --name=kafka-jmx \ --net=host \ -e KAFKA_BROKER_ID=1 \ -e KAFKA_ZOOKEEPER_CONNECT=localhost:32181/jmx \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:39092 \ -e KAFKA_JMX_PORT=9101 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ confluentinc/cp-kafka:7.4.3
View MBeans with JConsole¶
To confirm you have successfully configured JMX monitoring with a Docker container, you can start JConsole, a command-line utility provided with Java.
To start JConsole, use the
jconsole command, and connect to the Kafka process.
For this example, the JMX settings for a Docker container running locally might look like the following:
KAFKA_JMX_PORT: 9101 KAFKA_JMX_HOSTNAME: localhost
After JConsole starts, under Remote Process, enter the hostname and port you specified in your JMX configuration, and click Connect.
If you have not configured authentication, you may be prompted to make an Insecure connection.
After JConsole is running, you can select the MBeans tab and expand the folders to see the JMX events and attributes for those events.
Configure other Confluent Platform components¶
Use the following environment variables to override the default JMX options such as authentication settings for Confluent Platform components.
|Confluent Control Center||CONTROL_CENTER_JMX_OPTS|