Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Kafka Streams

Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in an Apache Kafka® cluster. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka’s server-side cluster technology.

Quick Start Guide

The Kafka Streams Quick Start demonstrates how to run your first Java application that uses the Kafka Streams library by showcasing a simple end-to-end data pipeline powered by Kafka.

Streams Podcasts

Streaming Audio is a podcast from Confluent, the team that built Kafka. Host Tim Berglund (Senior Director of Developer Experience, Confluent) and guests unpack a variety of topics surrounding Kafka, event stream processing, and real-time data.

• Apache Kafka Fundamentals: The Concept of Streams and Tables ft. Michael Noll
• Introducing JSON and Protobuf Support ft. David Araujo and Tushar Thole

Recommended Reading

• Streams and Tables in Apache Kafka: A Primer
• Introducing Kafka Streams: Stream Processing Made Simple

Streams API Screencasts

Watch the Intro to Streams API on YouTube.

Contents

• Introduction
  • The Kafka Streams API in a Nutshell
  • Use Case Examples
  • A Closer Look
  • Requirements
    • Kafka
    • Confluent
• Kafka Streams Quick Start
  • Purpose
  • Start the Kafka cluster
  • Prepare the topics and the input data
  • Process the input data with Kafka Streams
  • Inspect the output data
  • Stop the Kafka cluster
  • Next steps
• Tutorial: Introduction to Streaming Application Development
  • Overview
    • Microservices
    • End-to-end Streaming ETL
  • Pre-requisites
    • Reading
    • Environment Setup
  • Tutorial
    • Setup the Tutorial
    • Exercise 0: Run end-to-end demo
    • Exercise 1: Persist events
    • Exercise 2: Event-driven applications
    • Exercise 3: Enriching streams with joins
    • Exercise 4: Filtering and branching
    • Exercise 5: Stateful operations
    • Exercise 6: State stores
    • Exercise 7: Enrichment with ksqlDB
  • Additional Resources
• Kafka Streams Demo Application
  • Prerequisites
  • Start the music application
  • View messages in Kafka topics
  • Validate the Kafka Streams application
  • Create the ksqlDB application
    • Manually
    • Automatically
  • Stop the music application
  • Troubleshooting
• Connecting Kafka Streams to Confluent Cloud
• Streams Concepts
  • Kafka 101
  • Stream
  • Stream Processing Application
  • Processor Topology
  • Stream Processor
  • Stateful Stream Processing
  • Duality of Streams and Tables
  • KStream
  • KTable
  • GlobalKTable
  • Time
    • event-time
    • processing-time
    • ingestion-time
    • stream-time
    • Timestamps
    • Other aspects of time
  • Aggregations
  • Joins
  • Windowing
  • Interactive Queries
  • Processing Guarantees
  • Out-of-Order Handling
  • Out-of-Order Terminology
  • Suggested Reading
• Streams Architecture
  • Processor Topology
  • Parallelism Model
    • Stream Partitions and Tasks
    • Threading Model
    • Example
  • State
  • Memory management
    • Record caches
  • Fault Tolerance
  • Flow Control with Timestamps
  • Backpressure
• Streams Code Examples
  • Getting started examples
    • Collections
    • Java
    • Scala
  • Security examples
    • Java programming language
  • Interactive Queries examples
    • Java
  • End-to-end application examples
    • Java
    • Scala
  • Event-Driven Microservice example
    • Java
• Streams Developer Guide
  • Writing a Streams Application
    • Libraries and Maven artifacts
    • Using Kafka Streams within your application code
  • Testing Streams Code
    • Importing the Test Utilities
    • Testing a Streams Application
    • Unit Testing for Processors
    • Suggested Resources
  • Configuring a Streams Application
    • Configuration parameter reference
  • Streams DSL
    • Overview
    • Creating source streams from Kafka
    • Transform a stream
    • Controlling KTable emit rate
    • Writing streams back to Kafka
    • Kafka Streams DSL for Scala
  • Naming Kafka Streams DSL Topologies
    • Readability Issues
    • Changing Names
    • Conclusion
  • Optimizing Kafka Streams Topologies
    • Optimization FAQ
    • Optimization Details
  • Processor API
    • Overview
    • Defining a Stream Processor
    • Accessing Processor Context
    • State Stores
    • Connecting Processors and State Stores
    • Describing a Topology
  • Data Types and Serialization
    • Configuring SerDes
    • Overriding default SerDes
    • Available SerDes
    • Implementing custom SerDes
  • Interactive Queries
    • Querying local state stores for an app instance
    • Querying remote state stores for the entire app
    • Demo applications
  • Memory Management
    • Record caches in the DSL
    • Record caches in the Processor API
    • RocksDB
    • Other memory usage
  • Running Streams Applications
    • Starting a Kafka Streams application
    • Elastic scaling of your application
  • Managing Streams Application Topics
    • User topics
    • Internal topics
  • Streams Security
    • Required ACL setting for secure Kafka clusters
    • Security example
  • Application Reset Tool
    • Step 1: Run the application reset tool
    • Step 2: Reset the local environments of your application instances
• Pipelining with Kafka Connect and Kafka Streams
  • Overview
  • Description of Data
  • Demo Prerequisites
  • Run the demo
  • Example 1: Kafka console producer -> Key:String and Value:String
  • Example 2: JDBC source connector with Single Message Transformations -> Key:Long and Value:JSON
  • Example 3: JDBC source connector with SpecificAvro -> Key:String(null) and Value:SpecificAvro
  • Example 4: JDBC source connector with GenericAvro -> Key:String(null) and Value:GenericAvro
  • Example 5: Java client producer with SpecificAvro -> Key:Long and Value:SpecificAvro
  • Example 6: JDBC source connector with Avro to ksqlDB -> Key:Long and Value:Avro
  • Technical Notes
• Streams Operations
  • Capacity planning and sizing
    • Background and context
    • Stateless vs. stateful
    • Examples
    • Troubleshooting
  • Monitoring Kafka Streams Applications
    • Metrics
    • Runtime Status Information
    • Integration with Confluent Control Center
• Streams Upgrade Guide
  • Upgrading from Confluent Platform 5.4.x to Confluent Platform 5.5.15
    • Compatibility
    • Upgrading your Kafka Streams applications to Confluent Platform 5.5.15
    • Streams API changes
  • Upgrading older Kafka Streams applications to Confluent Platform 5.5.x
    • Streams API changes
    • Streams API changes in Confluent Platform 5.3.0
    • Streams API changes in Confluent Platform 5.2.0
    • Streams API changes in Confluent Platform 5.1.0
    • Streams API changes in Confluent Platform 5.0
    • Streams API changes in Confluent Platform 4.1
    • Streams API changes in Confluent Platform 4.0
    • Streams API changes in Confluent Platform 3.3
    • Streams API changes in Confluent Platform 3.2
    • Streams API changes in Confluent Platform 3.1
• Streams FAQ
  • General
    • Is Kafka Streams a project separate from Kafka?
    • Is Kafka Streams a proprietary library of Confluent?
    • Do Kafka Streams applications run inside the Kafka brokers?
    • Why Does My Kafka Streams Application Use So Much Memory?
    • What are the system dependencies of Kafka Streams?
    • How do I migrate my older Kafka Streams applications to the latest Confluent Platform version?
    • Which versions of Kafka clusters are supported by Kafka Streams?
    • What programming languages are supported?
    • Why is my application re-processing data from the beginning?
  • Scalability
    • Maximum parallelism of my application? Maximum number of app instances I can run?
  • Processing
    • How should I retain my Streams application’s processing results from being cleaned up?
    • Accessing record metadata such as topic, partition, and offset information?
    • Difference between map, peek, foreach in the DSL?
    • How to avoid data repartitioning if you know it’s not required?
    • Serdes config method
    • How can I replace RocksDB with a different store?
  • Failure and exception handling
    • Handling corrupted records and deserialization errors (“poison pill records”)?
    • Sending corrupt records to a quarantine topic or dead letter queue?
  • Interactive Queries
    • Handling InvalidStateStoreException: “the state store may have migrated to another instance”?
  • Security
    • Application fails when running against a secured Kafka cluster?
  • Troubleshooting and debugging
    • Easier to interpret Java stacktraces?
    • Visualizing topologies?
    • Inspecting streams and tables?
    • Invalid Timestamp Exception
    • Why do I get an IllegalStateException when accessing record metadata?
    • Why is punctuate() not called?
    • Scala: compile error “no type parameter”, “Java-defined trait is invariant in type T”
    • How can I convert a KStream to a KTable without an aggregation step?
    • RocksDB behavior in 1-core environments
• Streams Javadocs