<a id="client-kafka-streams-monitor-apps"></a>

# Monitor Kafka Streams Applications in Confluent Cloud

Confluent Cloud provides tools to monitor and manage your Kafka Streams applications.
Access the Kafka Streams monitoring page by navigating to your cluster’s overview
page in Confluent Cloud Console and clicking **Kafka Streams**.

#### NOTE
The Kafka Streams monitoring features in Confluent Cloud Console require your
applications to be built with Kafka Streams and Apache Kafka® client libraries
version 4.0.0 or later. For more information, see
[Kafka Streams Upgrade Guide](upgrade-guide.md#streams-upgrade-guide).

For this guide, you create a Kafka Streams application by using Confluent for VS
Code, or you can run an existing Kafka Streams application that uses Kafka
topics in Confluent Cloud.

If you’re using an existing Kafka Streams application, you can skip to
[Step 7: Monitor the application](#client-kafka-streams-monitor-apps-monitor-application). Ensure that the
application is built with the latest version of the Kafka Streams and Kafka client
libraries. To use all of the monitoring features, Kafka version 4.0.0 or later
is required.

For a complete list of metrics available for Kafka Streams applications, see
[Kafka Streams Metrics](kafka-streams-metrics.md#kafka-streams-metrics).

## Prerequisites

- [Confluent for VS Code](../client-apps/vs-code-extension.md#vscode-installation).
- Docker installed and running in your development environment.
- A Kafka cluster running in Confluent Cloud.
  - Kafka bootstrap server `host:port`, for example,
    `pkc-abc123.<cloud-provider-region>.<cloud-provider-region>.confluent.cloud:9092`,
    which you can get from the **Cluster Settings** page in
    Cloud Console. For more information, see
    [How do I view cluster details with Cloud Console?](../clusters/cluster-faq.md#cloud-cluster-settings).
  - Kafka cluster API key and secret, which you can get from the
    **Cluster Overview > API Keys** page in Cloud Console.

## Step 1: Create the Kafka Streams project

<!-- WARNING: THIS IS A SHARED FILE AND THE SOURCE IS LOCATED IN DOCS-COMMON. DO NOT ADD TO ANY OTHER REPO. -->

Create the Kafka Streams project by using the **Kafka Streams Application**
template and filling in a form with the required parameters.

### Open the template in VS Code directly

To go directly to the **Kafka Streams Application** template in VS Code, click
this button:

<a class="cta-button slim" href="vscode://confluentinc.vscode-confluent/projectScaffold?collection=docs&template=kafka-streams-simple-example&isFormNeeded=true">Open template in VS Code</a>
<br/>

The **Kafka Streams Application** form opens.

Skip the manual steps and proceed to
[Step 2: Fill in the template form](#step-2-fill-in-the-template-form).

### Open the template in VS Code manually

Follow these steps to open the **Kafka Streams Application** template manually.

1. Open VS Code.
2. In the Activity Bar, click the Confluent icon. If you have many
   extensions installed, you might need to click **…** to access
   **Additional Views** and select **Confluent** from the context menu.
3. In the extension’s Side Bar, locate the **Support** section and click
   **Generate Project from Template**.

   The palette opens and shows a list of available project templates.
4. Click **Kafka Streams Application**.

   The **Kafka Streams Application** template opens.

## Step 2: Fill in the template form

The project needs a few parameters to connect with your Kafka cluster.

<!-- The Step 2 heading is in the docs-common file included above. -->
1. In the **Kafka Streams Application** form, provide the following values.
   - **Kafka Bootstrap Server:** Enter the `host:port` string from the
     **Cluster Settings** page in Cloud Console. If you’re logged in
     with Confluent for VS Code, you can right-click on the Kafka cluster in
     the **Resources** pane and select **Copy bootstrap server**.
   - **Kafka Cluster API Key:** Enter the Kafka cluster API key.
   - **Kafka Cluster API Secret:** Enter the Kafka cluster API secret.
   - **Input Topic:** The name of a topic that the Kafka Streams application
     consumes messages from. Enter `input_topic`. You create this topic in a
     later step.
   - **Output Topic:** The name of a topic that the Kafka Streams application
     produces messages to. Enter `output_topic`. You create this topic in a
     later step.
2. Click **Generate & Save**, and in the save dialog, navigate to the directory
   in your development environment where you want to save the project files and
   click **Save to directory**.

   Confluent for VS Code generates the following project files.
   - The Kafka Streams code, in the `src/main/java/examples` directory, in a
     file named `KafkaStreamsApplication.java`.
   - A `docker-compose.yml` file that declares how to build the Kafka Streams
     code.
   - A `config.properties` file that holds configuration settings, like
     `bootstrap.servers`.
   - A `.env` file that holds secrets, like the Kafka cluster API key.
   - A `README.md` file with instructions for compiling and running the
     project.
3. Open the generated `build.gradle` file and update the version of the
   Kafka Streams and Kafka client libraries to the latest version. Kafka version
   4.0.0 or later is required to use the monitoring features.
   ```none
   dependencies {
     implementation 'org.apache.kafka:kafka-streams:4.0.0'
     implementation 'org.apache.kafka:kafka-clients:4.0.0'
   }
   ```

## Step 3: Connect to Confluent Cloud

1. In the extension’s side bar, click **Sign in to Confluent Cloud**.
2. In the dialog that appears, click **Allow**.

   A browser window opens to the Confluent Cloud login page.
3. Enter your Confluent Cloud credentials, and click **Log in**.

   After you authenticate, VS Code displays your Confluent Cloud resources in the
   extension’s Side Bar.

## Step 4: Create topics

<!-- WARNING: THIS IS A SHARED FILE AND THE SOURCE IS LOCATED IN DOCS-COMMON. DO NOT ADD TO ANY OTHER REPO. -->

Confluent for VS Code enables creating Kafka topics within VS Code.

1. In the extension’s Side Bar, open **Local** in the **Resources** section and
   click **cluster-local**.

   The **Topics** section refreshes, and the cluster’s topics are listed.
2. In the **Topics** section, click **＋** to create a new topic.

   The palette opens with a text box for entering the topic name.
3. In the palette, enter *input_topic*. Press ENTER to confirm the default
   settings for the **partition count** and **replication factor** properties.

   The new topic appears in the **Topics** section.
4. Repeat the previous steps for another new topic named *output_topic*.

## Step 5: Compile and run the project

<!-- WARNING: THIS IS A SHARED FILE AND THE SOURCE IS LOCATED IN DOCS-COMMON. DO NOT ADD TO ANY OTHER REPO. -->

Your Kafka Streams project is ready to build and run in a Docker container.

1. In your terminal, navigate to the directory where you saved the project.
2. The Confluent for VS Code extension saves the project files in a
   subdirectory named `kafka-streams-simple-example`. Run the following
   command to navigate to this directory.
   ```bash
   cd kafka-streams-simple-example
   ```
3. Run the following command to build and run the Kafka Streams application.
   ```bash
   docker compose up --build
   ```

   Docker downloads the required images and starts a container that compiles
   the project.

## Step 6: Produce messages to the input topic

The Kafka Streams application you created in the previous step consumes messages
from `input_topic` and produces messages to an `output_topic`. For
convenience, this guide uses a Datagen Source connector to produce messages to
`input_topic`.

1. In your browser, log in to Confluent Cloud Console and navigate to your Kafka
   cluster.
2. In the navigation menu, click **Connectors**.
3. Click **Add connector**.
4. On the **Search** box, type `datagen`.
5. Select the **Datagen Source** connector, and in the **Launch Sample Data**
   dialog, click **Additional configuration**.
6. In the **Choose the topics you want to send data to** section, select
   **input_topic**, and click **Continue**.
7. In the **API key** section, click **Generate API key & download** and click
   **Continue**.
8. In the **Configuration** page, select **JSON** for the output record value
   format, and **Orders** for the schema, and click **Continue**.
9. For **Connector sizing**, leave the slider at the default of **1** task and
   click **Continue**.
10. Name the connector `Kafka_Streams_data_source` and click
    **Launch connector**.

    Confluent Cloud provisions the connector. After a short time, the connector starts
    producing messages to `input_topic`.

<a id="client-kafka-streams-monitor-apps-monitor-application"></a>

## Step 7: Monitor the application

After your Kafka Streams application is running, you can monitor it by using the
Confluent Cloud Console.

1. In Cloud Console, navigate to your Kafka cluster’s overview page,
   and in the navigation menu, click **Kafka Streams**.

   The **Kafka Streams** page displays a list of all the Kafka Streams
   applications running in your Kafka cluster, along with metrics aggregated
   across all the Kafka Streams applications in your Kafka cluster.

   The page shows the following metrics:
   - **Application Name**: The name of the Kafka Streams application.
   - **Client version**: The version of the client used by the application.
   - **Status**: The status of the application.
   - **Running threads**: The number of threads running in the application.
   - **Total production**: The total number of messages produced by the
     application in the last minute.
   - **Total consumption**: The total number of messages consumed by the
     application in the last minute.
   - **Total lag**: The total lag of the application.
2. In the list, find the application you want to monitor. You can search for
   the application by name. If you created it with Confluent for VS Code,
   the application name starts with `vscode-kafka-streams-simple-example-`.
3. Click the link in the **Application** column to open the application’s
   overview page.

   The overview page displays the following metrics:
   - **Size of memtables**
   - **Estimated number of keys**
   - **Block cache usage**

   Graphs display the following metrics:
   - **End-to-end latency**
   - **Process ratio**

   Confluent Cloud takes a few minutes to collect and display the metrics.

## Performance considerations

### Network traffic and metadata fetching

Kafka Streams applications have different metadata requirements than standard
Kafka producers and consumers. Understanding these differences is important
for monitoring network usage and application performance.

### Metadata fetching behavior

Kafka Streams applications fetch more comprehensive metadata than plain Kafka
clients. Specifically, Kafka Streams applications pull metadata for all topics
in the cluster, not only the topics they directly produce to or consume
from. The Kafka Streams framework requires this behavior for its internal
operations.

### Impact on service accounts with DataDiscovery or DataSteward roles

When you use Kafka Streams applications with [service accounts](../security/authenticate/workload-identities/service-accounts/overview.md#service-accounts) that have the [DataDiscovery](../security/access-control/rbac/predefined-rbac-roles.md#datadiscovery-role)
or [DataSteward](../security/access-control/rbac/predefined-rbac-roles.md#datasteward-role) role, you might see increased
network traffic compared to applications that use service accounts with
more restrictive permissions.

This increase occurs because:

- The DataDiscovery and DataSteward roles provide read access to topic
  metadata across all topics in an environment.
- Kafka Streams applications fetch comprehensive topic metadata as part of
  their normal operation.
- The combination results in additional network requests for metadata
  retrieval.

### Monitoring recommendations

When you monitor Kafka Streams applications that use service accounts with
the [DataDiscovery](../security/access-control/rbac/predefined-rbac-roles.md#datadiscovery-role) or [DataSteward](../security/access-control/rbac/predefined-rbac-roles.md#datasteward-role) role, follow these recommendations:

- Monitor network utilization metrics to understand the baseline metadata
  traffic.
- Consider this additional metadata fetching when planning network
  capacity.
- Use the [Metrics API](../monitoring/metrics-api.md#metrics-api) to track `request_count` and
  other network-related metrics for your applications.
- Evaluate whether the DataDiscovery or DataSteward role is necessary for
  your specific Kafka Streams application use case.

For more information about role-based access control and service account
configuration, see [Role-based Access Control (RBAC) on Confluent Cloud](../security/access-control/rbac/overview.md#cloud-rbac) and [Service Accounts on Confluent Cloud](../security/authenticate/workload-identities/service-accounts/overview.md#service-accounts).

## Related content

- [Kafka Streams Metrics](kafka-streams-metrics.md#kafka-streams-metrics)
- [Kafka Streams Quick Start](quickstart.md#client-kafka-streams-quick-start)
- [Apache Kafka Overview](/kafka/overview.html)

#### NOTE
This website includes content developed at the [Apache Software Foundation](https://www.apache.org/)
under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).