Monitoring Confluent Platform Docker Deployments using JMX

You can monitor Confluent Platform Docker deployments by using Java Management Extensions (JMX).

Configure Environment

To use JMX with Docker containers, set the following properties:

java.rmi.server.hostname=<JMX_HOSTNAME>
com.sun.management.jmxremote.local.only=false
com.sun.management.jmxremote.rmi.port=<JMX_PORT>
com.sun.management.jmxremote.port=<JMX_PORT>
Note about hostname:

The JMX client needs to be able to connect to java.rmi.server.hostname. The default for bridged network is the bridged IP so you will only be able to connect from another Docker container. For host network, this is the IP that the hostname on the host resolves to.

The hostname is set to hostname -i in the Docker container. If you have more that one network configured for the container, hostname -i gives you all the IPs, the default is to pick the first IP (or network).

Configure Security

To set security on JMX, you can follow the SSL and authentication sections in this guide: https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html.

Configure Kafka and ZooKeeper

Settings

Use the following environment variables to configure JMX monitoring for Kafka and ZooKeeper.

KAFKA_JMX_PORT
JMX Port.
KAFKA_JMX_OPTS

JMX options. Use this variable to override the default JMX options. The default options are set as follows:

-Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false
KAFKA_JMX_HOSTNAME
Hostname associated with locally created remote objects.

Launching Kafka and ZooKeeper with JMX Enabled

The steps for launching Kafka and ZooKeeper with JMX enabled are the same as shown in the Confluent Platform Quick Start (Docker), with the only difference being that you set KAFKA_JMX_PORT and KAFKA_JMX_HOSTNAME for both. Here are examples of the Docker run commands for each service:

docker run -d \
  --name=zk-jmx \
  --net=host \
  -e ZOOKEEPER_TICK_TIME=2000 \
  -e ZOOKEEPER_CLIENT_PORT=32181 \
  -e KAFKA_JMX_PORT=39999 \
  -e KAFKA_JMX_HOSTNAME=`docker-machine ip confluent`
  confluentinc/cp-zookeeper:5.4.1

docker run -d \
  --name=kafka-jmx \
  --net=host \
  -e KAFKA_BROKER_ID=1 \
  -e KAFKA_ZOOKEEPER_CONNECT=localhost:32181/jmx \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:39092 \
  -e KAFKA_JMX_PORT=49999 \
  -e KAFKA_JMX_HOSTNAME=`docker-machine ip confluent` \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  confluentinc/cp-kafka:5.4.1

Tip: The examples above for KAFKA_JMX_HOSTNAME environment variable assume you have a Docker machine named “confluent”. Substitute this with your Docker machine name where appropriate.

Configure other Confluent Platform components

Authentication is disabled for JMX by default in Kafka. The default JMX configuration binds an unauthenticated JMX interface to all network interfaces. You must use environment variables to override this default.

Use the following environment variables to override the default JMX options for Confluent Platform components.

Component Environment Variable
Confluent Control Center CONTROL_CENTER_JMX_OPTS
REST Proxy KAFKAREST_JMX_OPTS
KSQL KSQL_JMX_OPTS
Rebalancer REBALANCER_JMX_OPTS
Schema Registry SCHEMA_REGISTRY_JMX_OPTS