Kafka Streams Code Examples for Confluent Platform¶
If you’re new to Kafka Streams, here is a curated list of resources to get you started.
Confluent for VS Code¶
Use the Confluent for VS Code extension to generate a new Kafka Streams application that reads messages from a Kafka topic, performs a simple transformation, and writes the transformed data to another topic. This option is ideal if you’re learning about Kafka Streams.
For more information, see Confluent for VS Code with Confluent Platform.
Getting started examples¶
Collections¶
The kafka-streams-examples GitHub repo is a curated repo with examples that demonstrate the use of Kafka Streams DSL, the low-level Processor API, Java 8 lambda expressions, reading and writing Avro data, and implementing unit tests with TopologyTestDriver and end-to-end integration tests using embedded Kafka clusters.
There are also numerous Kafka Streams examples in Kafka Tutorials that provide full code examples with step–by-step instructions.
Java¶
With lambda expressions for Java 8+:
- WordCountLambdaExample
- AnomalyDetectionLambdaExample
- GlobalKTablesExample
- MapFunctionLambdaExample
- PageViewRegionLambdaExample
- UserRegionLambdaExample
- WikipediaFeedAvroLambdaExample
- ApplicationResetExample
- SessionWindowsExample
- SumLambdaExample
- TopArticlesLambdaExample
Without lambda expressions:
Scala¶
Security examples¶
Interactive Queries examples¶
Since Confluent Platform 3.1+ and Kafka 0.10.1+, it is possible to query state stores created via the Kafka Streams DSL and the Processor API. For further information, see Kafka Streams Interactive Queries for Confluent Platform.
End-to-end application examples¶
These demo applications use embedded instances of Kafka and Confluent Schema Registry. They are implemented as integration tests.
See also
To see end-to-end examples of Kafka Streams applications deployed in an event streaming platform with all the services in Confluent Platform and interconnecting other end systems, refer to confluentinc/examples.
Java¶
With lambda expressions for Java 8+:
- WordCountLambdaIntegrationTest
- FanoutLambdaIntegrationTest
- GenericAvroIntegrationTest
- GlobalKTablesExampleTest
- HandlingCorruptedInputRecordsIntegrationTest
- MapFunctionLambdaIntegrationTest
- MixAndMatchLambdaIntegrationTest – how to mix the DSL and the Processor API
- SpecificAvroIntegrationTest
- StateStoresInTheDSLIntegrationTest – how to use state stores in the DSL
- StreamToStreamJoinIntegrationTest
- StreamToTableJoinIntegrationTest
- TableToTableJoinIntegrationTest
- UserCountsPerRegionLambdaIntegrationTest
- ApplicationResetIntegrationTest
- EventDeduplicationLambdaIntegrationTest
- SumLambdaIntegrationTest
- TopArticlesLambdaExampleTest
Without lambda expressions:
Scala¶
- StreamToTableJoinScalaIntegrationTest
- ProbabilisticCountingScalaIntegrationTest - demonstrates how to probabilistically count items in an input stream by implementing a custom state store that is backed by a Count-Min Sketch data structure
- GenericAvroScalaIntegrationTest
- SpecificAvroScalaIntegrationTest
- WordCountScalaIntegrationTest
Event-Driven Microservice example¶
Java¶
The Event-Driven Microservice example implements an Order Service that provides a REST interface to POST and GET orders. Posting an order creates an event in Kafka, which is picked up by three different validation engines: a Fraud Service, an Inventory Service, and an Order Details Service. These services validate the order in parallel, emitting a PASS or FAIL based on whether each validation succeeds.
- Microservices example source code
- Microservices example tests
- Self-paced Kafka Streams tutorial based on the microservices example above: deployed in an event streaming platform with all the services in Confluent Platform and interconnecting other end systems
Note
This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.