Monitor Kafka Streams Applications in Confluent Cloud¶

Confluent Cloud provides tools to monitor and manage your Kafka Streams applications.

Prerequisites¶

Access to Confluent Cloud.
A provisioned Kafka cluster in Confluent Cloud.

Step 1: Create a Kafka Streams application¶

To create a Kafka Streams application, follow the steps in Kafka Streams Quick Start.
Or you can run an existing Kafka Streams application that uses Apache Kafka® topics in Confluent Cloud.

Step 2: Monitor the application¶

Log in to Confluent Cloud Console and navigate to your Kafka cluster.
In the navigation menu, click Kafka Streams.

The Kafka Streams page displays a list of all the Kafka Streams applications running in your Kafka cluster. The displayed metrics are aggregated across all the Kafka Streams applications in your Kafka cluster.
In the list, find the application you want to monitor. You can search for the application by name.

The displayed metrics include:
- Application Name: The name of the Kafka Streams application.
- Client version: The version of the client used by the application.
- Running threads: The number of threads running in the application.
- Total production: The total number of messages produced by the application in the last minute.
- Total consumption: The total number of messages consumed by the application in the last minute.
- Total lag: The total lag of the application.
Click View consumer group to open the Consumer Lag page.

The Consumer Lag page displays the consumer group metrics for the application.

The displayed metrics include:
- Consumer group ID: The ID of the consumer group.
- Total messages behind: The total number of messages behind for the consumer group.
- Consumer lag: The lag for each topic partition.
For more information, see Monitor Confluent Cloud Clients.

Performance considerations¶

Network traffic and metadata fetching¶

Kafka Streams applications have different metadata requirements compared to standard Kafka producers and consumers. Understanding these differences is important for monitoring network usage and application performance.

Metadata fetching behavior¶

Kafka Streams applications fetch more comprehensive metadata than plain Kafka clients. Specifically, Kafka Streams applications need to pull metadata for all topics in the cluster, not just the topics they directly produce to or consume from. This behavior is required for the internal operations of the Kafka Streams framework.

Impact on service accounts with DataDiscovery or DataSteward roles¶

When service accounts configured with DataDiscovery or DataSteward roles are used with Kafka Streams applications, you may observe increased network traffic compared to applications using service accounts with more restrictive permissions.

This increase occurs because:

DataDiscovery and DataSteward roles provide read access to topic metadata across all topics within an environment.
Kafka Streams applications fetch comprehensive topic metadata as part of their normal operation.
The combination results in additional network requests for metadata retrieval.

Monitoring recommendations¶

When monitoring Kafka Streams applications using service accounts with DataDiscovery or DataSteward roles, you should:

Monitor network utilization metrics to understand the baseline metadata traffic
Consider this additional metadata fetching when planning network capacity.
Use the Metrics API to track request_count and other network-related metrics for your applications.
Evaluate whether the DataDiscovery or DataSteward roles are necessary for your specific Kafka Streams application use case.

For more information about role-based access control and service account configuration, see Role-based Access Control (RBAC) on Confluent Cloud and Service Accounts on Confluent Cloud.