Data Flow

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is being introduced to gain early feedback from developers. This feature can be used for evaluation and non-production testing purposes or to provide feedback to Confluent.

Click the Provide feedback link in the preview banner notification to share your thoughts and ideas about this preview feature with the Confluent Cloud team.

Note

Your VPC must communicate with the Confluent Cloud Data Flow public internet endpoint to use the preview feature.

The Data flow feature provides a visual representation of the data flow paths (edges) between producers, topics, and consumers within a cluster. Drill into producers, topics, and consumers for more detailed information. Inspect the flow throughput metrics between producers, topics, consumer groups and consumers.

To access Data flow, select a cluster from the navigation bar and click Data flow.

Prerequisites

If there is not an existing topic in the cluster, the interactive data flow diagram is not displayed. The flows display when data is being produced to and consumed from a topic. For best results, select a cluster that has existing topics that are actively producing and consuming message data.

Tip

If you are new to Confluent Cloud, follow the quick start to create topics, and produce to and consume from topics on an Apache Kafka® cluster using the Confluent Cloud CLI.

Check out the producer/consumer examples of Kafka client applications written in different languages connecting to Confluent Cloud. Click on the examples graphic in the GitHub repo to go directly to the example you’re interested in.

Producers

All producers are listed to the left of topics.

  • Click Inspect for a producer to view metrics such as Total production and Bytes produced per topic.
../../_images/ccloud-data-flow-inspect-producers.png

Data flow inspect producer

Tip

Hover on an edge (path) between a producer and topic, or a topic and consumer, to view throughput.

../../_images/ccloud-dataflow-path-tp.png

Topics

Topics are centrally located in the Data flow interactive diagram. When you hover on a topic or one its partitions, the paths to its producers and consumers are emphasized prominently so you can easily see the data flows between them.

  • Click Inspect for a topic to view metrics such as Total throughput, Bytes in and out per partition, and Bytes written per producer. Hover on a chart to view more detail.

    ../../_images/ccloud-data-flow-chart-hover.png
  • Click Show partitions to view the partitions for a topic and the percentage of messages consumed from a producer.

../../_images/ccloud-data-flow-inspect-topics-parts.png

Data flow inspect, show, and hide partitions in topics

  • Click Hide partitions to collapse the view of the partitions within a topic.

Consumer groups and consumers

All consumer groups and consumers are listed to the right of topics. For a given consumer group, the total message lag is visible as well as the message lag for each consumer.

  • Click Inspect for a consumer group to view metrics for Total consumption.
  • Click Show consumers to view the consumers within a consumer group.
../../_images/ccloud-data-flow-inspect-consumers-2.png

Data flow inspect, show, and hide consumers in a consumer group

  • Click Hide consumers to collapse the view of the consumers within a consumer group.