# Batch and Stream Processing in Confluent Cloud for Apache Flink Confluent Cloud for Apache Flink® unifies batch and stream processing in a single engine that treats batch as a special case of streaming. Write one Flink SQL query and run it against bounded data (batch) or unbounded data (streaming) — the same runtime, APIs, and operators handle both modes. In Confluent Cloud for Apache Flink, batch mode is available through [snapshot queries](#flink-sql-batch-and-stream-processing-snapshot-queries), which automatically bound input sources as of the current time and release resources when finished. ## Batch processing Batch processing in Flink operates on *bounded datasets*, which are finite, static collections of data. This processing mode has the following key characteristics. - It processes complete, finite datasets, such as files or database snapshots. - Batch jobs run to completion and then terminate. - It optimizes for throughput, focusing on processing large volumes of data efficiently. - Batch processing can sort, aggregate, and join across the entire dataset. - The system can drop state as soon as the data no longer needs it. - Use cases: - Historical data analysis - ETL (Extract, Transform, Load) operations - Report generation - Data warehousing ## Stream processing Stream processing in Flink handles *unbounded data streams*, which have data that arrives continuously and might never end. This processing mode has the following key characteristics. - It processes infinite, continuous data streams, such as Kafka topics or sensor feeds. - Stream processing jobs run indefinitely, processing data as it arrives. - It focuses on processing data with minimal delay for low latency. - It produces incremental results as new data arrives. - The system must retain state to handle late or out-of-order events. - Use cases: - Real-time analytics - Fraud detection - IoT data processing - Live dashboards ## Bounded and unbounded tables comparison In Flink, tables can be either *bounded* (batch) or *unbounded* (streaming). The following table compares the key differences between these two modes. | Aspect | Bounded Mode (Batch) | Unbounded Mode (Streaming) | |------------------|--------------------------------------|----------------------------------------------| | Data Size | Finite (static) | Infinite (dynamic, continuous) | | Processing Style | Batch processing | Real-time/continuous processing | | Query Semantics | All data available at once | Data arrives over time | | State Management | Minimal, can drop state when done | Must retain state for late/out-of-order data | | Use Cases | ETL, reporting, historical analytics | Real-time analytics, monitoring, alerting | ## Differences between batch and stream processing The following table compares the important differences between batch and stream processing. | Aspect | Batch Processing | Stream Processing | |------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Data Model | Processes complete, finite datasets. | Processes infinite, continuous data streams. | | Execution Model | Jobs run to completion. | Jobs run continuously. | | Latency vs. Throughput | Optimized for high throughput. | Optimized for low latency. | | State Management | Minimal state, which is dropped when no longer needed. | Robust state, which is retained for late or out-of-order data. | | Fault Tolerance | Can restart from the beginning. | Requires checkpointing for fault recovery. | | Query Semantics | All data is available at once, so global operations are possible. | Data arrives over time, so results are incremental. | | SQL/API Differences | - **ORDER BY**: You can use any sort order.
- **Windowing**: Supports time-based windows on static data.
- **Deduplication**: Deduplication is global.
- **Joins**: Temporal joins on regular tables
(`FOR SYSTEM_TIME AS OF`) aren’t supported. Use a
[lookup join](../reference/queries/joins.md#flink-sql-lookup-joins) instead. | - **ORDER BY**: The primary sort must be on a time attribute.
- **Windowing**: Uses windows to scope aggregations over unbounded data.
- **Deduplication**: Deduplication is incremental and often uses windows.
- **Session Windows**: Supported.
- **Joins**: Temporal joins on regular tables and lookup joins are
both supported. | ## Unified processing model One important advantage of Flink is its *unified processing model*: the same runtime engine handles both batch and streaming by treating batch processing as a special case of stream processing. A bounded dataset is simply a stream that has a defined end. The same APIs and operators work for both modes, so you can use the same Flink SQL queries for batch and streaming applications. This unified approach enables you to: - Build applications that process both historical and real-time data. - Seamlessly transition between batch and streaming modes. - Maintain consistent semantics across processing modes. - Use the same tools and libraries for both paradigms. ## Time and watermarks Time and watermarks are important concepts in Flink that help you process data correctly. - **Batch mode**: Time is fixed. All data is available, so event time and processing time are equivalent. - **Streaming mode**: Time is dynamic. Streaming mode uses watermarks to track event time progress and handle out-of-order data. - **Windowing**: In streaming, you use windows (tumbling, hopping, cumulative, session) to group data for aggregation. In batch, windows apply to static data. For more information, see [Time and Watermarks](timely-stream-processing.md#flink-sql-timely-stream-processing). ## Determinism Determinism is a key concept in Flink that helps you ensure that your queries always produce the same results. - **Batch**: Re-running a batch job on the same data yields the same result, except for non-deterministic functions such as [UUID()](../reference/functions/numeric-functions.md#flink-sql-uuid-function). - **Streaming**: Results can vary due to timing, order of arrival, and late data. Determinism is harder to guarantee. For more information, see [Determinism in Continuous Queries](determinism.md#flink-sql-determinism). ## Snapshot queries and batch mode In Confluent Cloud for Apache Flink, batch mode is available by using snapshot queries. - **Snapshot queries**: These are batch queries that automatically bound the input sources as of the current time. - **Batch optimizations**: Batch mode enables optimizations such as global sorting, blocking operators, and efficient joins. Snapshot queries benefit from these optimizations. - **Resource usage**: Batch jobs, which are snapshot queries in Confluent Cloud for Apache Flink, release resources when finished. Streaming jobs hold resources as long as they run. For more information, see [Snapshot Queries](snapshot-queries.md#flink-sql-snapshot-queries). ## SQL feature support in batch mode Most Flink SQL queries produce the same results in batch and streaming mode, but a few features have mode-specific behavior. - **Temporal joins**: A temporal join on a regular table, using `FOR SYSTEM_TIME AS OF`, is supported only in streaming mode. Flink rejects this join type in batch mode. Use a [lookup join](../reference/queries/joins.md#flink-sql-lookup-joins) instead, which is supported in both modes. For more information, see [Temporal joins](../reference/queries/joins.md#flink-sql-temporal-joins). - `MATCH_RECOGNIZE`: Pattern recognition queries are supported in both modes. The `ORDER BY` clause differs slightly in batch mode. For more information, see [Order of events](../reference/queries/match_recognize.md#flink-sql-pattern-recognition-order-of-events). - **Window deduplication**: Supported in both modes, with the same results. For more information, see [Window Deduplication Queries in Confluent Cloud for Apache Flink](../reference/queries/window-deduplication.md#flink-sql-window-deduplication). ## Schema versioning with Tableflow When you run a snapshot query against a topic that has Tableflow enabled, the query can read from both Kafka and Tableflow-materialized Parquet data. Tableflow materializes data asynchronously, with a [table freshness](../../topics/tableflow/overview.md#cloud-tableflow-table-freshness) of approximately 5 minutes. Immediately after a compatible schema change, the Tableflow part of the query might briefly reflect an earlier schema version than the one most recently registered in Confluent Cloud Schema Registry. The remaining data that the query reads directly from Kafka always reflects the current schema. For more information, see [Schema Compatibility](../../topics/tableflow/concepts/tableflow-schemas.md#cloud-tableflow-schemas-schema-compatibility) and [Snapshot Queries and Tableflow](snapshot-queries.md#flink-sql-snapshot-queries). ## Private networking Batch and streaming queries support the same private networking options. There’s no batch-specific restriction. For more information, see [Private Networking for Flink](flink-private-networking.md#flink-sql-private-networking). ## Compute pool recommendations Snapshot queries run in batch mode and can consume a burst of compute resources before releasing them. This burst can contend with the steadier resource needs of your streaming workloads. Confluent recommends running snapshot queries in a compute pool that’s separate from the pool that runs your streaming production workloads. Run your production queries in streaming mode by default. For more information, see [Isolation and Resource Sharing](compute-pools.md#flink-sql-compute-pools). ## Examples The following code example shows a batch query. ```sql -- Count all orders in a bounded table SELECT COUNT(*) FROM orders; ``` The following code example shows a streaming query. ```sql -- Count orders per minute in an unbounded stream. SELECT window_start, window_end, COUNT(*) FROM TUMBLE(TABLE orders, DESCRIPTOR(order_time), INTERVAL '1' MINUTE) GROUP BY window_start, window_end; ``` ## Related content - [Stream Processing with Confluent Cloud for Apache Flink](../overview.md#ccloud-flink) - [Snapshot Queries](snapshot-queries.md#flink-sql-snapshot-queries) - [Time and Watermarks](timely-stream-processing.md#flink-sql-timely-stream-processing) - [Determinism in Continuous Queries](determinism.md#flink-sql-determinism) - [Window Aggregation](../reference/queries/window-aggregation.md#flink-sql-window-aggregation) - [Window Deduplication](../reference/queries/window-deduplication.md#flink-sql-window-deduplication) - [Deduplication](../reference/queries/deduplication.md#flink-sql-deduplication) - [ORDER BY Clause](../reference/queries/orderby.md#flink-sql-order-by) - [Comparing Apache Flink with Confluent Cloud](comparison-with-apache-flink.md#flink-comparison-with-open-source) - [Flink SQL Examples](../reference/sql-examples.md#flink-sql-examples) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).