5.1.0 is a major release of Confluent Platform that provides you with Apache Kafka 2.1.0, the latest stable version of Kafka.
The technical details of this release are summarized below.
- New functions for working with windowed data:
KSQL now supports
WindowEnd()functions that expose the start and end times of windows, respectively, allowing developers to easily refer to this information in other functions and in subsequent queries. Start and end times are expressed in milliseconds, like
- Metric names have changed: See KSQL upgrade notes for details on the changes.
- Sending a single, final output for windowed operations: Windowed operations can now be defined to send only a single, final output per window by suppressing intermediate outputs. For example, if an application computes five-minute averages of temperature readings, it can be told to send only one output record every 5 minutes that represents the latest 5-minute temperature average. This new feature makes it much easier to implement use cases such as alerting and metrics pipelines and to integrate Kafka Streams applications with systems that don’t support continuous streaming updates. Note that, by design, using the feature will increase the end-to-end processing latency of applications because no output records will be send until a window is closed. See the section Window Final Results in the Kafka Streams documentation and KIP-328 for more information.
- Better topology compatibility when updating an application: Repartition topics for join and aggregation operators can now be named explicitly by developers. This makes it easier to change and evolve an application while still allowing for rolling app re-deployments to prevent production outages when updating your apps. See KIP-372 for more information.
- Topology optimization:
Where possible, Kafka Streams will now merge repartitioning topics, which reduces the Kafka storage footprint of
Kafka Streams applications. Users must explicitly enable these optimizations in their application’s configuration
by setting the configuration
StreamsConfig.NO_OPTIMIZATION, i.e., disabled). See the section Optimizing Kafka Streams Topologies for more detailed information.
- Improved timestamp synchronization:
The new configuration parameter
StreamsConfig.MAX_TASK_IDLE_MS_CONFIGallows developers to tune the behavior of when Kafka Streams picks the next record to process, based on the stream-time. It allows for trade-offs between lower processing latency (lower idle settings, the default is 0 to keep pre-5.1 behavior) vs. reducing the probability of out-of-order data (higher idle settings). See KIP-353 for more information.
- Added intuitive timeouts for producer:
Adds a new configuration parameter for producers, ‘delivery.timeout.ms’, that represents a guarantee on the upper bound on when a record will either get sent, fail or expire from the point when the send returns. This timeout shields users from using workarounds that expose the internals of the producer. Notably, this setting effectively enables retries by default. Bear in mind that reordering is possible if the idempotent producer is not enabled or
max.in.flightis not set to 1. See KIP-91 for more information.
- Improved behavior for expiring committed offsets:
Consumer group offsets no longer expire as long as there are active consumers. The expiration time of offsets in a group will be when the group becomes “empty” plus retention time of
offsets.retention.minutes, unless that group becomes active again. See KIP-211 for more information.
Confluent provides high quality clients for the most popular non-Java languages. These clients are all based on the underlying librdkafka C/C++ implementation, which provides high performance and stability. The non-Java clients follow their own versioning scheme. The current clients release is 0.11.6.
Highlights from 0.11.6:
- Python client
- Adds experimental Windows support to Python client.
- Includes Confluent C3 monitoring interceptors with the Python client.
Availability and Resilience¶
- Improved unclean log truncation handling KIP-320 improves consumer handling of unclean log truncation and introduces fencing of zombie replicas. This ensures consistency of replicated data under certain failure conditions.
- ZStandard compression Kafka now supports ZStandard compression. ZStandard is a real-time compression algorithm developed by Facebook with features that make it highly performant and effective for small data. Kafka now accepts “zstd” as a new compression type for configuring producers, topics, and brokers. See KIP-110 for more information.
Apache Kafka 2.1.0-cp1¶
28 new features / config changes have been added with 161 resolved bug fixes and improvements. For a full list of changes in this release of Apache Kafka, see the 2.1.0 Release Notes.
How to Download¶
To upgrade Confluent Platform to a newer version, check the Upgrade documentation.
Supported Versions and Interoperability¶
For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.