Batch and Stream Processing in Confluent Cloud for Apache Flink

Confluent Cloud for Apache Flink® unifies batch and stream processing in a single engine that treats batch as a special case of streaming. Write one Flink SQL query and run it against bounded data (batch) or unbounded data (streaming) — the same runtime, APIs, and operators handle both modes. In Confluent Cloud for Apache Flink, batch mode is available through snapshot queries, which automatically bound input sources as of the current time and release resources when finished.

Batch processing

Batch processing in Flink operates on bounded datasets, which are finite, static collections of data. This processing mode has the following key characteristics.

It processes complete, finite datasets, such as files or database snapshots.
Batch jobs run to completion and then terminate.
It optimizes for throughput, focusing on processing large volumes of data efficiently.
Batch processing can sort, aggregate, and join across the entire dataset.
The system can drop state as soon as the data no longer needs it.
Use cases: - Historical data analysis - ETL (Extract, Transform, Load) operations - Report generation - Data warehousing

Stream processing

Stream processing in Flink handles unbounded data streams, which have data that arrives continuously and might never end. This processing mode has the following key characteristics.

It processes infinite, continuous data streams, such as Kafka topics or sensor feeds.
Stream processing jobs run indefinitely, processing data as it arrives.
It focuses on processing data with minimal delay for low latency.
It produces incremental results as new data arrives.
The system must retain state to handle late or out-of-order events.
Use cases: - Real-time analytics - Fraud detection - IoT data processing - Live dashboards

Bounded and unbounded tables comparison

In Flink, tables can be either bounded (batch) or unbounded (streaming). The following table compares the key differences between these two modes.

Aspect	Bounded Mode (Batch)	Unbounded Mode (Streaming)
Data Size	Finite (static)	Infinite (dynamic, continuous)
Processing Style	Batch processing	Real-time/continuous processing
Query Semantics	All data available at once	Data arrives over time
State Management	Minimal, can drop state when done	Must retain state for late/out-of-order data
Use Cases	ETL, reporting, historical analytics	Real-time analytics, monitoring, alerting

Differences between batch and stream processing

The following table compares the important differences between batch and stream processing.

Aspect	Batch Processing	Stream Processing
Data Model	Processes complete, finite datasets.	Processes infinite, continuous data streams.
Execution Model	Jobs run to completion.	Jobs run continuously.
Latency vs. Throughput	Optimized for high throughput.	Optimized for low latency.
State Management	Minimal state, which is dropped when no longer needed.	Robust state, which is retained for late or out-of-order data.
Fault Tolerance	Can restart from the beginning.	Requires checkpointing for fault recovery.
Query Semantics	All data is available at once, so global operations are possible.	Data arrives over time, so results are incremental.
SQL/API Differences	ORDER BY: You can use any sort order. Windowing: Supports time-based windows on static data. Deduplication: Deduplication is global. Joins: Temporal joins on regular tables (`FOR SYSTEM_TIME AS OF`) aren’t supported. Use a lookup join instead.	ORDER BY: The primary sort must be on a time attribute. Windowing: Uses windows to scope aggregations over unbounded data. Deduplication: Deduplication is incremental and often uses windows. Session Windows: Supported. Joins: Temporal joins on regular tables and lookup joins are both supported.

Unified processing model

One important advantage of Flink is its unified processing model: the same runtime engine handles both batch and streaming by treating batch processing as a special case of stream processing. A bounded dataset is simply a stream that has a defined end. The same APIs and operators work for both modes, so you can use the same Flink SQL queries for batch and streaming applications.

This unified approach enables you to:

Build applications that process both historical and real-time data.
Seamlessly transition between batch and streaming modes.
Maintain consistent semantics across processing modes.
Use the same tools and libraries for both paradigms.

Time and watermarks

Time and watermarks are important concepts in Flink that help you process data correctly.

Batch mode: Time is fixed. All data is available, so event time and processing time are equivalent.
Streaming mode: Time is dynamic. Streaming mode uses watermarks to track event time progress and handle out-of-order data.
Windowing: In streaming, you use windows (tumbling, hopping, cumulative, session) to group data for aggregation. In batch, windows apply to static data.

For more information, see Time and Watermarks.

Determinism

Determinism is a key concept in Flink that helps you ensure that your queries always produce the same results.

Batch: Re-running a batch job on the same data yields the same result, except for non-deterministic functions such as UUID().
Streaming: Results can vary due to timing, order of arrival, and late data. Determinism is harder to guarantee.

For more information, see Determinism in Continuous Queries.

Snapshot queries and batch mode

In Confluent Cloud for Apache Flink, batch mode is available by using snapshot queries.

Snapshot queries: These are batch queries that automatically bound the input sources as of the current time.
Batch optimizations: Batch mode enables optimizations such as global sorting, blocking operators, and efficient joins. Snapshot queries benefit from these optimizations.
Resource usage: Batch jobs, which are snapshot queries in Confluent Cloud for Apache Flink, release resources when finished. Streaming jobs hold resources as long as they run.

For more information, see Snapshot Queries.

SQL feature support in batch mode

Most Flink SQL queries produce the same results in batch and streaming mode, but a few features have mode-specific behavior.

Temporal joins: A temporal join on a regular table, using FOR SYSTEM_TIME AS OF, is supported only in streaming mode. Flink rejects this join type in batch mode. Use a lookup join instead, which is supported in both modes. For more information, see Temporal joins.
MATCH_RECOGNIZE: Pattern recognition queries are supported in both modes. The ORDER BY clause differs slightly in batch mode. For more information, see Order of events.
Window deduplication: Supported in both modes, with the same results. For more information, see Window Deduplication Queries in Confluent Cloud for Apache Flink.

Schema versioning with Tableflow

When you run a snapshot query against a topic that has Tableflow enabled, the query can read from both Kafka and Tableflow-materialized Parquet data. Tableflow materializes data asynchronously, with a table freshness of approximately 5 minutes. Immediately after a compatible schema change, the Tableflow part of the query might briefly reflect an earlier schema version than the one most recently registered in Confluent Cloud Schema Registry. The remaining data that the query reads directly from Kafka always reflects the current schema. For more information, see Schema Compatibility and Snapshot Queries and Tableflow.

Private networking

Batch and streaming queries support the same private networking options. There’s no batch-specific restriction. For more information, see Private Networking for Flink.

Compute pool recommendations

Snapshot queries run in batch mode and can consume a burst of compute resources before releasing them. This burst can contend with the steadier resource needs of your streaming workloads. Confluent recommends running snapshot queries in a compute pool that’s separate from the pool that runs your streaming production workloads. Run your production queries in streaming mode by default. For more information, see Isolation and Resource Sharing.

Examples

The following code example shows a batch query.

-- Count all orders in a bounded table
SELECT COUNT(*) FROM orders;

The following code example shows a streaming query.

-- Count orders per minute in an unbounded stream.
SELECT window_start, window_end, COUNT(*)
FROM
  TUMBLE(TABLE orders, DESCRIPTOR(order_time), INTERVAL '1' MINUTE)
GROUP BY window_start, window_end;