Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Monitoring Consumer Lag

You can monitor consumer lag with Confluent Cloud using the methods described in this document.

Monitor Offset Lag via Java Client Metrics

You can monitor the records-lag-max metric from the Java consumer.

Monitor Consumer Latency via Client Interceptors

You can use Confluent Control Center to track consumer latency and more.

Monitor Offset Lag via Broker APIs

You can also get offsets from the brokers. This section assumes that you have Confluent Cloud CLI installed or can copy the same properties generates in ~/ccloud/config.

Prerequisites
  • Access to Confluent Cloud
  • Confluent Cloud CLI installed
  • Java version 1.7.0_111 or greater, 1.8.0_102 or greater, and 1.9
  • Confluent Platform is installed

  1. Comment out the request time out in the Confluent Cloud config (~/ccloud/config). This line prevents the Confluent Cloud CLI commands from waiting too long before giving up on errors but conflicts with other settings for this use case.

    #request.timeout.ms=20000

  2. Set the BOOTSTRAP_SERVERS variable to the broker list value that was provided when you signed up. You can find this value in ~/.ccloud/config or by clicking Client config from the Confluent Cloud web interface.

    BOOTSTRAP_SERVERS="SASL_SSL://<bootstrap-server1>, <bootstrap-server2>, <bootstrap-server3>"

  3. From the Confluent Platform installation home, list the consumer groups:

    ./bin/kafka-consumer-groups --bootstrap-server ${BOOTSTRAP_SERVERS} --command-config \
 ~/.ccloud/config --list  _confluent-healthcheck  example-group

  4. For each consumer group, check its offsets using this command. This command only shows information about consumers that use the Java consumer API (i.e., non-ZooKeeper-based consumers).

    ./bin/kafka-consumer-groups --bootstrap-server ${BOOTSTRAP_SERVERS} \
 --command-config ~/.ccloud/config --describe --group _confluent-healthcheck

    Your output should resemble:

    TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
_confluent-healthcheck         0          13164704        13164773        69         healthcheck-agent-bf8d1655-63a6-4061-b680-0f11cdf182e5/100.96.67.0                   healthcheck-agent
_confluent-healthcheck         1          13161581        13161650        69         healthcheck-agent-bf8d1655-63a6-4061-b680-0f11cdf182e5/100.96.67.0                   healthcheck-agent
_confluent-healthcheck         2          12229509        12229578        69         healthcheck-agent-bf8d1655-63a6-4061-b680-0f11cdf182e5/100.96.67.0                   healthcheck-agent
_confluent-healthcheck         3          86              86              0          healthcheck-agent-bf8d1655-63a6-4061-b680-0f11cdf182e5/100.96.67.0                   healthcheck-agent
...

    The fourth column shows the lag, the difference between the last committed offset and the latest offset in the log.