Manage Kafka Consumers Using Control Center for Confluent Platform

The Consumers menu provides a view of all consumer groups for all topics in a cluster. From the consumer pages, you can:

• View all consumer groups for a cluster in the All consumer groups page
• View consumer lag across all topics in a cluster
• View consumption metric for a consumer group
• Set up consumer group alerts

Tip

Consumer Lag for an individual topic can be accessed from the Topics menu.

Important

The Consumers feature is enabled by default using an Access Control setting in the Control Center properties files. See Enabling and disabling the Consumers feature. If RBAC is enabled for Control Center, the resources you can access are determined by your assigned role or roles and their corresponding privileges.

All consumer groups page

Use the All consumer groups page to:

• View all consumer groups for a cluster
• Search for a consumer group by its ID
• View the number of consumers and number of source topics for each consumer group
• View details such as Consumer Lag and Consumption metrics for a consumer group
• Set up a pre-populated alert for Consumer Lag

To access the All consumer groups page:

1. Select a cluster from the navigation bar and click the Consumers menu.

   

View consumer lag details for a consumer group

1. From the All consumer groups page, click the ID for your desired Consumer Group. Consumer lag details are displayed, including:

   • Total messages behind and the measured interval.
   • All consumers in the group.
   • A visualization of lag. The Max lag / consumer graph displays a pin point of each partition in the topic.
   

2. To view details for a particular point on the graph, click its pin. View the Consumer, Partition, Current offset, and Messages behind for that particular point.

   

View consumption details for a consumer group

View the percentage (%) of messages consumed and the end-to-end latency key metrics by a selected timeline.

Note

The consumption charts are only populated if there are Confluent Monitoring Interceptors that are set up correctly. For instructions, see Monitor Production and Consumption Using Control Center in Confluent Platform.

1. From the All consumer groups page, click the ID for your desired Consumer Group.

2. Click the Consumption tab. The streams monitoring metrics are displayed for the default timeframe (last 4 hours).

   

3. Select a timeframe from the menu and click Apply.

   

   Timeframe selector

   Use the time range selector to select the time range for data displayed in charts:

   • Static: A specific time range with constant start and end times.
   • Rolling: A time range where the end time is always equal to the current time and the extent of the time range is held constant.
   • Growing: A time range where the end time is always equal to the current time and the start time is held constant.

   The default selection is Last 4 hours.

4. Select the End-to-end latency metric you want to view:

   • average (ms)
   • minimum (ms)
   • maximum (ms)
   • total (ms)
   

   Tip

   You can set up alerts for average and maximum latency values.

Set up an alert for a consumer group

You can set alerts from the Consumer lag page. These options pre-populate most of the consumer group trigger fields.

On the Consumer lag page, click Set up an alert.

For more details, see Consumer lag trigger.

Enabling and disabling the Consumers feature

The Consumers feature for viewing details for consumer lag and consumption in consumer groups is enabled by default. The feature can be disabled by setting the confluent.controlcenter.consumers.view.enable option in the appropriate Control Center properties file to false. After disabling the feature, the Consumer submenu in the Cluster navigation is no longer visible in the Control Center UI.

Important

If RBAC is enabled for Control Center, it takes precedence over Access Control settings.

To disable the Consumers feature in Control Center:

1. Set the confluent.controlcenter.consumers.view.enable option in your control-center.properties file to false.

   ...
   confluent.controlcenter.consumers.view.enable=false
   ...
   

   Note

   Make the change in the appropriate Control Center properties files configured for your environments, including control-center-dev.properties or control-center-production.properties. The properties files are located in /path-to-confluent/etc/confluent-control-center/. For more information, see Control Center Configuration Examples for Confluent Platform.

2. Restart Control Center and pass in the properties file for the configuration to take effect.

   • Local (dev) environment:

     Important

     Do not use confluent local for production use.

     confluent local services control-center stop
     confluent local services control-center start --config ../etc/confluent-control-center/control-center.properties
     

   • Production environment:

     ./bin/control-center-stop
     ./bin/control-center-start ../etc/confluent-control-center/control-center.properties
     

To enable the feature again, set the option back to true and restart Control Center with the updated properties file.

Reset offsets for consumer groups

You can reset offsets for consumer groups. Offsets track the position of a consumer in a partition. Each consumer instance in a consumer group periodically commits its progress through the partition with an offset. If a consumer instance fails, another instance in the group takes over and resumes from the last committed offset.

You might need to reset offsets for a variety of reasons, including troubleshooting, testing or disaster recovery.

Considerations:

• To reset offsets with Control Center, you must first empty the consumer group by stopping any clients or consumers in the consumer group.
• To reset offsets with Control Center for a consumer group that uses the Apache Kafka® assign() function to manually assign consumers to partitions, you must first locate and pause the consumer that is using the assign() function. Consumer groups that use the assign() function always appear empty to Control Center but they may hide unstopped consumers.
• You should not use Control Center to reset offsets for stream processing applications, which require special considerations for resets. For more information, see Reset Kafka Streams Applications in Confluent Platform.

Prerequisites

You must be a member of one of the following roles:

• super.user
• SystemAdmin
• ClusterAdmin
• ResourceOwner
• DeveloperRead

To reset an offset

1. Select your cluster name.

2. Select Consumers and then select Consumer Lag. A list of consumer group IDs is displayed.

3. Select a consumer group ID. A list of clients and consumers is displayed.

4. Select the client or consumer you want to reset, or select all.

5. Select Reset offset. The Reset offset dialog appears.

6. In Reset offset, choose one of the following and then select Reset:

   • Earliest to set the offset to the earliest offset available for the topic and partition.
   • Latest to set the offset to the latest offset available for the topic and partition.
   • Offset value and add a value in Enter offset value to set the offset to a specified integer.

   After you reset an offset, Control Center may need a few minutes to catch up.