Release Notes

5.1.0 is a major release of Confluent Platform that provides you with Apache Kafka 2.1.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Highlights

Commercial Features

Control Center

  • Improved browser support: In previous releases, Control Center only supported Chrome. Browser support in 5.1.0 has been extended to Chrome, Firefox, and Safari. For more information, see Web Browsers.
  • New theme: Control Center now uses a new, more readabable light theme

Community Features

KSQL

  • New functions for working with windowed data: KSQL now supports WindowStart() and WindowEnd() functions that expose the start and end times of windows, respectively, allowing developers to easily refer to this information in other functions and in subsequent queries. Start and end times are expressed in milliseconds, like ROWTIME.
  • Metric names have changed: See KSQL upgrade notes for details on the changes.

Kafka Streams

  • Sending a single, final output for windowed operations: Windowed operations can now be defined to send only a single, final output per window by suppressing intermediate outputs. For example, if an application computes five-minute averages of temperature readings, it can be told to send only one output record every 5 minutes that represents the latest 5-minute temperature average. This new feature makes it much easier to implement use cases such as alerting and metrics pipelines and to integrate Kafka Streams applications with systems that don’t support continuous streaming updates. Note that, by design, using the feature will increase the end-to-end processing latency of applications because no output records will be send until a window is closed. See the section Window Final Results in the Kafka Streams documentation and KIP-328 for more information.
  • Better topology compatibility when updating an application: Repartition topics for join and aggregation operators can now be named explicitly by developers. This makes it easier to change and evolve an application while still allowing for rolling app re-deployments to prevent production outages when updating your apps. See KIP-372 for more information.
  • Topology optimization: Where possible, Kafka Streams will now merge repartitioning topics, which reduces the Kafka storage footprint of Kafka Streams applications. Users must explicitly enable these optimizations in their application’s configuration by setting the configuration StreamsConfig.TOPOLOGY_OPTIMIZATION to StreamsConfig.OPTIMIZE (default: StreamsConfig.NO_OPTIMIZATION, i.e., disabled). See the section Optimizing Kafka Streams Topologies for more detailed information.
  • Improved timestamp synchronization: The new configuration parameter StreamsConfig.MAX_TASK_IDLE_MS_CONFIG allows developers to tune the behavior of when Kafka Streams picks the next record to process, based on the stream-time. It allows for trade-offs between lower processing latency (lower idle settings, the default is 0 to keep pre-5.1 behavior) vs. reducing the probability of out-of-order data (higher idle settings). See KIP-353 for more information.

Kafka Clients

  • Added intuitive timeouts for producer: Adds a new configuration parameter for producers, ‘delivery.timeout.ms’, that represents a guarantee on the upper bound on when a record will either get sent, fail or expire from the point when the send returns. This timeout shields users from using workarounds that expose the internals of the producer. Notably, this setting effectively enables retries by default. Bear in mind that reordering is possible if the idempotent producer is not enabled or max.in.flight is not set to 1. See KIP-91 for more information.
  • Improved behavior for expiring committed offsets: Consumer group offsets no longer expire as long as there are active consumers. The expiration time of offsets in a group will be when the group becomes “empty” plus retention time of offsets.retention.minutes, unless that group becomes active again. See KIP-211 for more information.

Confluent Clients

Confluent provides high quality clients for the most popular non-Java languages. These clients are all based on the underlying librdkafka C/C++ implementation, which provides high performance and stability. The non-Java clients follow their own versioning scheme. The current clients release is 0.11.6.

Highlights from 0.11.6:

  • librdkafka
    • Introduces stability fixes and performance improvements for Windows. #1930, #1980
    • Fixes several producer issues pertaining to queuing, flushing, and timeouts.
    • Fixes consumer inadvertently raising error about message size. #1472
  • Python client
    • Adds experimental Windows support to Python client.
    • Includes Confluent C3 monitoring interceptors with the Python client.

Availability and Resilience

  • Improved unclean log truncation handling KIP-320 improves consumer handling of unclean log truncation and introduces fencing of zombie replicas. This ensures consistency of replicated data under certain failure conditions.

Performance

  • ZStandard compression Kafka now supports ZStandard compression. ZStandard is a real-time compression algorithm developed by Facebook with features that make it highly performant and effective for small data. Kafka now accepts “zstd” as a new compression type for configuring producers, topics, and brokers. See KIP-110 for more information.

Apache Kafka 2.1.0-cp1

28 new features / config changes have been added with 161 resolved bug fixes and improvements. For a full list of changes in this release of Apache Kafka, see the 2.1.0 Release Notes.

How to Download

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

To upgrade Confluent Platform to a newer version, check the Upgrade documentation.

Supported Versions and Interoperability

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?

If you have questions regarding this release, feel free to reach out via the community mailing list or community Slack. Confluent customers are encouraged to contact our support directly.