Examples Overview

ksqlDB Basics

This tutorial demonstrates a simple workflow using ksqlDB to write streaming queries against messages in Apache Kafka®.

Write Streaming Queries with the ksqlDB CLI

                  ===========================================
                  =       _              _ ____  ____       =
                  =      | | _____  __ _| |  _ \| __ )      =
                  =      | |/ / __|/ _` | | | | |  _ \      =
                  =      |   <\__ \ (_| | | |_| | |_) |     =
                  =      |_|\_\___/\__, |_|____/|____/      =
                  =                   |_|                   =
                  =        The Database purpose-built       =
                  =        for stream processing apps       =
                  ===========================================

Copyright 2017-2020 Confluent Inc.

CLI v6.1.15, Server v6.1.15 located at http://localhost:8088
Server Status: RUNNING

Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!

ksql>

Get started with the ksqlDB CLI:

• Write streaming queries against Apache Kafka® using ksqlDB (Local)

Write Streaming Queries with ksqlDB and Confluent Control Center

Get started with ksqlDB and Confluent Control Center:

• Control Center deployed with Docker
• Control Center deployed locally

Common use cases

Use ksqlDB to implement solutions for these common use cases.

Materialized view/cache

Create and query a set of materialized views about phone calls made to a call center. This tutorial demonstrates capturing changes from a MySQL database, forwarding them into Kafka, creating materialized views with ksqlDB, and querying them from your applications.

• Materialized view/cache

Streaming ETL pipeline

Create a streaming ETL pipeline that ingests and joins events together to create a cohesive view of orders that have shipped. This tutorial demonstrates capturing changes from Postgres and MongoDB databases, forwarding them into Kafka, joining them together with ksqlDB, and sinking them out to ElasticSearch for analytics.

• Streaming ETL pipeline

Event-driven microservice

Create an event-driven microservice that identifies suspicious activity and notifies customers. This tutorial demonstrates finding anomalies with ksqlDB and sending alert emails using a simple Kafka consumer with SendGrid.

• Event-driven microservice

Kafka Tutorials

Kafka Tutorials contain ksqlDB tutorials with step-by-step instructions.

Clickstream Data Analysis Pipeline

Clickstream analysis is the process of collecting, analyzing, and reporting aggregate data about which pages a website visitor visits and in what order. The path the visitor takes though a website is called the clickstream.

This tutorial focuses on building real-time analytics of users to determine:

• General website analytics, such as hit count and visitors
• Bandwidth use
• Mapping user-IP addresses to actual users and their location
• Detection of high-bandwidth user sessions
• Error-code occurrence and enrichment
• Sessionization to track user-sessions and understand behavior (such as per-user-session-bandwidth, per-user-session-hits etc)

The tutorial uses standard streaming functions (i.e., min, max, etc) and enrichment using child tables, stream-table join, and different types of windowing functionality.

Get started now with these instructions:

• Clickstream Data Analysis Pipeline Using ksqlDB

If you don’t have Docker, you can also run an automated version of the Clickstream tutorial designed for local Confluent Platform installs. Running the Clickstream demo locally without Docker requires that you have Confluent Platform installed locally, along with Elasticsearch and Grafana.

ksqlDB Examples

These examples provide common ksqlDB usage operations:

• Transform
• Join
• Aggregate, window, and sessionize

You can configure Java streams applications to deserialize and ingest data in multiple ways, including Kafka console producers, JDBC source connectors, and Java client producers. For full code examples, see Pipelining with Kafka Connect and Kafka Streams.

ksqlDB in a Kafka Streaming ETL

To learn how to deploy a Kafka streaming ETL using ksqlDB for stream processing, you can run the Confluent Platform demo. All components in the Confluent Platform demo have encryption, authentication, and authorization configured end-to-end.

ksqlDB Videos

See the latest videos on Confluent Platform ksqlDB and Confluent Cloud ksqlDB at the Confluent YouTube channel.

Video	Description
ksqlDB HowTo: A mini video series	Robin Moffatt’s quick videos about ksqlDB, based on demo scripts that you can run for yourself.
Apache Kafka 101: ksqlDB	Tim Berglund provides a gentle introduction to ksqlDB concepts and queries.
Confluent Cloud Quick Start, ksqlDB, and Project Reactor (Redux)	Viktor Gamov provisions Kafka, Connect, and ksqlDB clusters in Confluent Cloud and accesses with the ksqlDB Reactor client.
Demo: The Event Streaming Database in Action	Tim Berglund builds a movie rating system with ksqlDB to write movie records into a Kafka topic.
Demo: Seamless Stream Processing with Kafka Connect & ksqlDB	Set up and build ksqlDB applications using the AWS source, Azure sink, and MongoDB source connectors in Confluent Cloud.
Introduction to ksqlDB and stream processing	Vish Srinivasan talks Kafka stream processing fundamentals and discusses ksqlDB.
Ask Confluent #16: ksqlDB edition	Gwen Shapira hosts Vinoth Chandar in a wide-ranging talk on ksqlDB.
An introduction to ksqlDB	Robin Moffatt describes how ksqlDB helps you build scalable and fault-tolerant stream processing systems.
ksqlDB and the Kafka Connect JDBC Sink	Robin Moffatt demonstrates how to use ksqlDB with the Connect JDBC sink.
How to Transform a Stream of Events Using ksqlDB	Viktor Gamov demonstrates how to transform a stream of movie data.
ksqlDB Java Client and Confluent Cloud	Viktor Gamov takes the ksqlDB Java client for a spin and tests it against Confluent Cloud.