Monitoring and Metrics in ksqlDB for Confluent Platform

Context

ksqlDB publishes metrics via JMX (Java Management Extensions) which help you monitor what is happening inside of ksqlDB’s servers. For a comprehensive list of metrics, see the reference section.

Enable monitoring

You must enable monitoring explicitly on each ksqlDB server. To enable it in a Docker-based deployment, export an environment variable named KSQL_JMX_OPTS with your JMX configuration and expose the port that JMX will communicate over.

The following Docker Compose example shows how you can configure monitoring for ksqlDB server. The surrounding components, like the broker and CLI, are omitted for brevity. You can see an example of a complete setup in the ksqlDB Quick Start.

ksqldb-server:
  image: confluentinc/cp-ksqldb-server:7.7.2
  hostname: ksqldb-server
  container_name: ksqldb-server
  depends_on:
    - broker
    - schema-registry
  ports:
    - "8088:8088"
    - "1099:1099"
  environment:
    KSQL_LISTENERS: "http://0.0.0.0:8088"
    KSQL_BOOTSTRAP_SERVERS: "broker:9092"
    KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
    KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
    KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
    KSQL_KSQL_QUERY_PULL_METRICS_ENABLED: "true"
    KSQL_JMX_OPTS: >
      -Djava.rmi.server.hostname=localhost
      -Dcom.sun.management.jmxremote
      -Dcom.sun.management.jmxremote.port=1099
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false
      -Dcom.sun.management.jmxremote.rmi.port=1099

With respect to monitoring, here it what this does:

• The environment variable KSQL_JMX_OPTS is supplied to the server with various arguments. The > character lets you write a multi-line string in Yaml, which makes this long argument easier to read. The advertised hostname, port, and security settings are configured. JMX has a wide range of configuration options, and you can set these however you like.
• Port 1099 is exposed, which corresponds to the JMX port set in the KSQL_JMX_OPTS configuration. This enables remote monitoring tools to communicate into ksqlDB’s process.

Verifying your monitoring setup

An easy way to check that ksqlDB is properly emitting metrics is by using jconsole. JConsole is a graphical monitoring tool to monitor the JVM, and it is included in Oracle JDK installations.

On your host machine, run the command:

jconsole

You will be prompted for a host and port. If you used the example configuration above, then entering localhost:1099 will allow JConsole to establish the connection. You should see a series of graphs showing resource utilization. If you don’t, make sure the networking between your machine and the Docker container is configured correctly.