Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Release Notes

Confluent Platform 5.1.4 Release Notes

This is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.1.1, the latest stable version of Kafka and additional bug fixes.

You are encouraged to upgrade to Confluent Platform 5.1.4 as it includes important bug fixes. The technical details of this release are summarized below.

Highlights

Community Features

Kafka

• PR-8205 - KAFKA-9634: Add note about thread safety in the ConfigProvider interface (#8205)
• PR-8165 - KAFKA-9601: Stop logging raw connector config values (#8165)
• PR-7870 - KAFKA-9254; Overridden topic configs are reset after dynamic default change (#7870)
• PR-7753 - KAFKA-9232; Coordinator new member timeout does not work for JoinGroup v3 and below (#7753)
• PR-6031 - KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
• PR-7532 - KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
• PR-7298 - KAFKA-8875; CreateTopic API should check topic existence before replication factor (#7298)
• PR-7545 - Fix bug in AssignmentInfo#encode and add additional logging (#7545)
• PR-7315 - KAFKA-8340, KAFKA-8819: Use PluginClassLoader while statically initializing plugins (#7315)
• PR-7524 - KAFKA-8945/KAFKA-8947 backport (#7524)
• PR-7491 - KAFKA-9014: Fix AssertionError when SourceTask.poll returns an empty list (#7491)
• PR-7427 - KAFKA-8649: send latest commonly supported version in assignment (#7427)
• PR-7442 - KAFKA-8974: Trim whitespaces in topic names in sink connector configs (#7442)
• PR-7371 - KAFKA-6290: Support casting from logical types in cast transform (#7371)
• PR-6914 - KAFKA-8523 Enabling InsertField transform to be used with tombstone events (#6914)
• PR-7373 - KAFKA-7895: Revert suppress changelog bugfix for 2.1 (#7373)
• PR-7346 - KAFKA-8913: Document topic based configs & ISR settings for Streams apps (#7346)
• PR-7223 - KAFKA-8816: Make offsets immutable to users of RecordCollector.offsets (#7223)
• PR-7207 - KAFKA-8412: Fix nullpointer exception thrown on flushing before closing producers (#7207)
• PR-6993 - KAFKA-8586: Fail source tasks when producers fail to send records (#6993)
• PR-6283 - KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread (#6283)
• PR-7197 - KAFKA-8774: Regex can be found anywhere in config value (#7197)
• PR-6959 - KAFKA-8550: Fix plugin loading of aliased converters in Connect (#6959)
• PR-7164 - KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (#7164)
• PR-7054 - KAFKA-8615: Change to track partition time breaks TimestampExtractor (#7054)
• PR-7092 - KAFKA-8602: Separate PR for 2.3 branch (#7092)
• PR-7021 - KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (#7021)
• PR-7070 - KAFKA-7157: Fix handling of nulls in TimestampConverter (#7070)
• PR-5705 - KAFKA-6605: Fix NPE in Flatten when optional Struct is null (#5705)
• PR-6678 - Fixes #8198 KStreams testing docs use non-existent method pipe (#6678)
• PR-6974 - KAFKA-8570; Grow buffer to hold down converted records if it was insufficiently sized (#6974)
• PR-6968 - KAFKA-8564; Fix NPE on deleted partition dir when no segments remain (#6968)
• PR-6791 - KAFKA-8404: Add HttpHeader to RestClient HTTP Request and Connector REST API (#6791)
• PR-6840 - KAFKA-8418: Wait until REST resources are loaded when starting a Connect Worker. (#6840)
• PR-6818 - KAFKA-8187: Add wait time for other thread in the same jvm to free the locks (#6818)
• PR-6636 - KAFKA-8290: Close producer for zombie task (#6636)
• PR-6719 - KAFKA-8347: Choose next record to process by timestamp (#6719)
• PR-6675 - KAFKA-8320 : fix retriable exception package for source connectors (#6675)
• PR-6726 - KAFKA-8363: Fix parsing bug for config providers (#6726)
• PR-6715 - KAFKA-8335; Clean empty batches when sequence numbers are reused (#6715)
• PR-5918 - KAFKA-7633: Allow Kafka Connect to access internal topics without cluster ACLs (#5918)
• PR-6707 - KAFKA-8348: Fix KafkaStreams JavaDocs (#6707)
• PR-5578 - KAFKA-6789; Handle retriable group errors in AdminClient API (#5578)
• PR-6685 - KAFKA-8240: Fix NPE in Source.equals() (#6685)
• PR-6651 - KAFKA-8304: Fix registration of Connect REST extensions (#6651)
• PR-6652 - KAFKA-8306; Initialize log end offset accurately when start offset is non-zero (#6652)
• PR-6671 - KAFKA-8289: Fix Session Expiration and Suppression (#6654) (#6671)
• PR-6568 - KAFKA-7601; Clear leader epoch cache on downgraded format in append (#6568)
• PR-6402 - KAFKA-8066; Always close the sensors in Selector.close() (#6402)
• PR-6643 - KAFKA-8298: Fix possible concurrent modification exception (#6643)

Common

• PR-244 - Disable net assembly & oss index report
• PR-195 - 3.3.x:use maven to generate archives

KSQL

• PR-4917 - fix: logic when closing ksqlEngine fixed
• PR-4857 - fix: don’t cleanup topics on engine close (#4658)
• PR-4759 - build: disable -Werror compiler argument in tests
• PR-4527 - test: data providers should be ordered (MINOR)
• PR-3807 - build: Inherit jackson from common pom
• PR-3557 - test: improve stability of tests on older branches
• PR-3104 - fix: COLLECT_LIST can now be applied to tables
• PR-3002 - Move docker image to break circular dependency
• PR-2945 - Added clarification to docs for PARTITION BY
• PR-2922 - KSQL-2236: Add paras about the WITHIN clause
• PR-2890 - KSQL-2653: Fix broken link
• PR-2885 - disable CliTest in 4.1.x
• PR-2869 - KSQL-1533: Mention in joins topic that table-table joins can be joined only on t…
• PR-2868 - KSQL-1533: Mention that table-table joins can be joined only on the key
• PR-2866 - KSQL-1560: Add link to clickstream-schema.sql in the Github repo
• PR-2861 - KSQL-2691: Fix example CREATE TABLE statement in Syntax Reference
• PR-2859 - KSQL-2619: Fix example CREATE TABLE statement in syntax reference topic

Schema Registry

• PR-1381 - CC-7027: Revert to legacy serialization of Decimal logical type on incompatible scale/precision
• PR-1268 - Add null checks
• PR-1300 - CC-7692 / ISSUE-1197: Fix configuration of basic auth credentials provider
• PR-1242 - DG-128: Allow DNS lookup policy to be configurable
• PR-1026 - CC-3759 - Regenerate ID if in use during registration
• PR-1143 - Add config for mock registry in avro serdes Part 2
• PR-1141 - Add config for mock registry in avro serdes
• PR-1124 - CC-4775: logical type preservation for hdfs connector when output format is parquet

Rest Utils

• PR-148 - Remove jackson databind version override
• PR-140 - use maven to generate archive

Confluent Cli

• No changes for Confluent Cli

Commercial Features

MQTT Proxy

• CC-3848: Support specifying the maximum allowed MQTT message size in bytes

Connectors

Elasticsearch Connector

• PR-135 - CC-6599: java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Long while setting default date value
• PR-340 - CC-6187: Cache already existing index in JestElasticsearchClient
• PR-327 - Enable Elasticsearch 7 support with adding minimum Integration test (based on TestContainer)

HDFS Connector

• PR-491 - CC-8229: HDFS2 format options are not showing all available options
• PR-460 - CC-6423: Disable FileSystem Object Caching
• PR-456 - CC-6196: fix NPE for store.url config
• PR-451 - CC-6089: Improve fault tolerance of HDFS sink connector
• PR-450 - CC-5969: Pass extra connector configs to the partitioner
• PR-442 - Adding debug logs for WAL recovery

JDBC Connector

• PR-793 - CC-7637: change from DEBUG level to TRACE level
• PR-792 - CC-8090: Validate connection.attempts property in config class
• PR-791 - CC-8087: Remove plaintext logging of task configs
• PR-715 - [jdbc-source-connector] Fixed issue in CachedConnectionProvider, to enable retries when a connection is failed
• PR-787 - Make metadata allFields ordered consistently with the valueSchema.
• PR-770 - CC-7628 Set conservative timeout for JDBC to stop trying to reach drive
• PR-785 - CC-5047: connect jdbc url field should have example text
• PR-775 - Move setFetchSize to GenericDatabaseDialect
• PR-776 - CC-7637: added logging of query to be executed at DEBUG level
• PR-758 - Fix Postgres OOM by limiting fetch size
• PR-761 - CC-7414: Fix SqlServer column converter infinite loop & add column converter tests
• PR-759 - CC-7382: use true/false as default values for boolean type
• PR-750 - CC-7265: Support SQL Server DateTimeOffset
• PR-752 - CC-7265: Add support for SQL Server DateTimeOffset type
• PR-727 - CC-6773: Abort source task if all columns were not converted correctly.
• PR-724 - CC-6747: Replace static mocking for the CachedConnectionProvider
• PR-677 - Fix ERROR Graceful stop of task failed
• PR-655 - CC-4946: changing log msgs on filtering and avoiding frequently retrying tasks when no tables
• PR-672 - Change TableQuerier log to use real class name for file

JMS Connector

• CC-8250: Revert JMS Retry Functionality
• CC-3439: Update MQ Source Connector to better handle errors and autoreconnect when MQ manager is restarted
• CC-6503: Capping batch size at 10000 for IBM MQ connector
• CC-3937: repackaging base classes
• CC-7926: Corrected logic to prevent infinite loops during poll()
• CC-7866: Added more and improved existing logging in the base JMS source task and connector
• CC-6505: Change instances of RetriableException to ConnectException in base source task
• CC-6267: Fix NPE when using permissive schemas
• CC-6267: Add permissive schema configuration
• CC-6200 : Fix NPE due to null property values
• CC-4422: JMS Connector Performance Updates

Replicator Connector

• REP-637: Ensuring timestampsCommitter is not null before closing it out
• REP-24: ZK license manager change
• REP-47 added arity to boolean cmd line options as they all default true
• REP-403: Fixed timestamp committer thread leak
• REP-26:Fixed issue where group.id was not propagated to license manager
• REP-186: Updated replicator to use delivery.timeout.ms instead of retries
• REP-20: Added a check to ensure TopicMontiorThreadWithZk is shutdown
• Add debug logs for starting consumers
• CC-2796: Pass a delta for better than ms precision
• CC-5541: Set ByteArrayConverter as header converter
• CC-5012: If topic creation fails, attempt to resize topic
• REP-22: Handle uncaught exception during offset commit
• CC-4928: During translation retries, save newer timestamps for retry

S3 Connector

• PR-311 - Add AWS IAM Assume Role credentials provider
• PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
• PR-281 - CC-6801: Retry effectively on retriable failures while closing files

Storage Common

• PR-110 - CC-5963: Improved ability to handle schema evolution with different names

Confluent Platform 5.1.3 Release Notes

This is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.1.1, the latest stable version of Kafka and additional bug fixes.

You are encouraged to upgrade to Confluent Platform 5.1.3 as it includes important bug fixes. The technical details of this release are summarized below.

Highlights

Community Features

Kafka 2.1.1

• PR-6641 - KAFKA-8254: Pass Changelog as Topic in Suppress Serdes (#6602) (#6641)
• PR-6623 - KAFKA-8204: fix Streams store flush order (#6555) (#6623)
• PR-6616 - KAFKA-7895: fix Suppress changelog restore (#6536) (#6616)
• PR-6550 - KAFKA-8277: Fix NPEs in several methods of ConnectHeaders (#6550)
• PR-6570 - KAFKA-7866; Ensure no duplicate offsets after txn index append failure (#6570)
• PR-6585 - KAFKA-8241; Handle configs without truststore for broker keystore update (#6585)
• PR-6573 - KAFKA-8210: Fix link for streams table duality (#6573)
• PR-6564 - KAFKA-8209: Wrong link for KStreams DSL in core concepts doc (#6564)
• PR-6572 - KAFKA-8208: Change paper link directly to ASM (#6572)
• PR-6574 - KAFKA-8213 - Fix typo in Streams Dev Guide (#6574)
• PR-6384 - KAFKA-8058: Fix ConnectClusterStateImpl.connectors() method (#6384)
• PR-6547 - KAFKA-8157: fix the incorrect usage of segment.index.bytes (2.2) (#6547)
• PR-6539 - KAFKA-8190; Don’t update keystore modification time during validation (#6539)
• PR-6475 - KAFKA-8126: Flaky Test org.apache.kafka.connect.runtime.WorkerTest.testAddRemoveTask (#6475)
• PR-6305 - KAFKA-7974: Avoid zombie AdminClient when node host isn’t resolvable (#6305)
• PR-6342 - KAFKA-8014: Extend Connect integration tests to add and remove workers dynamically (#6342)
• PR-6484 - KAFKA-8142: Fix NPE for nulls in Headers (#6484)
• PR-6194 - KAFKA-7855: Kafka Streams Maven Archetype quickstart fails to compile out of the box (#6194)
• PR-6373 - KAFKA-7027: Add an overload build method in scala (#6373)
• PR-6393 - KAFKA-8065: restore original input record timestamp in forward() (#6393)
• PR-6401 - KAFKA-8069; Fix early expiration of offsets due to invalid loading of expire timestamp (#6401)
• PR-6372 - KAFKA-8040: Streams handle initTransactions timeout (#6372)
• PR-6388 - KAFKA-8061; Handle concurrent ProducerId reset and call to Sender thread shutdown (#6388)
• PR-6325 - KAFKA-7895: fix stream-time reckoning for Suppress (2.1) (#6286) (#6325)
• PR-6364 - KAFKA-8002: Log dir reassignment stalls if future replica has different segment base offset (#6364)
• PR-6333 - KAFKA-8012; Ensure partitionStates have not been removed before truncating. (#6333)
• PR-6338 - KAFKA-8011: Fix for race condition causing concurrent modification exception (#6338)
• PR-6288 - KAFKA-7945: Calc refresh time correctly when token created in the past (#6288)
• PR-6256 - KAFKA-7920; Do not permit zstd produce requests until IBP is updated to 2.1 (#6256)
• PR-6274 - KAFKA-7935: UNSUPPORTED_COMPRESSION_TYPE if ReplicaManager.getLogConfig returns None (#6274)

Connectors

• Elasticsearch Connector:
• PR-302 - CC-4054: Remove the use of null_value (default values) for text and binary based fields
• HDFS Connector:
• PR-425 - CC-4318 Fix off-by-one error for offset reporting to Kafka topic
• PR-422 - CC-4124: Clear partition assignment on close
• PR-417 - CC-3950: Change the HDFS sink task to return to Connect offsets for records committed to HDFS
• JDBC Connector:
• PR-612 - CC-3812: Sqlite timestamp parse error
• S3 Connector
• PR-243 - CC-4228: Configuration option to override expect-continue in upload protocol
• JDBC Connector
• PR-593 - CC-3578: Logging query for source connector at INFO level only the first time it’s used

KSQL

• PR-2719 - fix race condition in RestApiTest shutdown: fixes flaky test in 5.0.x

Commercial Features

Replicator

• De-duplicate in timestamp interceptors; add configs for batching in Replicator
• Use committed offsets even if dest topic doesn’t exist
• Start REST server when using Replicator app
• Close admin client if constructed locally
• Allow consumer offset to be preferred over connect offset
• commit offset timestamp since it will not be invoked by the framework

Confluent Platform 5.1.2

This is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.1.1, the latest stable version of Kafka and additional bug fixes.

You are encouraged to upgrade to Confluent Platform 5.1.2 as it includes important bug fixes. The technical details of this release are summarized below.

Highlights

Community Features

Kafka 2.1.1

• PR-5986 - KAFKA-6388: Error while trying to roll a segment that already exists (#5986)
• PR-5997 - KAFKA-7697: Possible deadlock in kafka.cluster.Partition (#5997)
• PR-6049 - KAFKA-7755: Kubernetes - Kafka clients are resolving DNS entries only one time (#6049)
• PR-6215 - KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes (#6215)
• PR-6126 - KAFKA-7741: streams-scala - document dependency workaround (#6126)
• PR-6232 - KAFKA-7897: Disable leader epoch cache when older message formats are used (#6232)
• PR-6233 - KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER refresh login fails (#6233)

JDBC Connector

• PR-593 - CC-3578: Logging query for source connector at INFO level only the first time it’s used

Schema Registry

• PR-1020 - CC-3803: Corrected support for handling defaults of logical schema types
• PR-1009 - CC-3741: Remove use of IdentityHashMap

Confluent Platform 5.1.1

This is a bugfix release of Confluent Platform that provides you with Kafka 2.1.0, the latest stable version of Kafka and additional bug fixes.

You are encouraged to upgrade to Confluent Platform 5.1.1 as it includes important bug fixes. The technical details of this release are summarized below.

Highlights

Commercial Features

Confluent CLI

• Consult first JAVA_HOME and then PATH when checking java version
• Properly set JMX_PORT for each service based on ${SERVICE}_JMX_PORT prefix
• Reflect CCL change in code variables

Hub Client

• commons-collections: 4-4.0 -> 4-4.2, airline*: 2.2.0 -> 2.6.0, jackson-annotations: 2.9.0 -> 2.9.6.

JMS Connector

• Added NPE handling and info message.
• Backport changes for CC-3639 to 4.1.x and update corresponding docs
• Use client acknowledge mode to ack produced messages and maintain one inflight message

Connect Replicator

• Improve message when offset translation fails
• Fix off-by-one-error during commitSync
• Fix the number of Replicator tasks to spawn
• Ignore missing TPs when calling offsetsForTimes

Community Features

Kafka 2.1.0-cp2

• PR-6215 - KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes. (#6215)
• PR-6203 - KAFKA-7873: Always seek to beginning in KafkaBasedLog (#6203)
• PR-6202 - KAFKA-7837: Ensure offline partitions are picked up as soon as possible when shrinking ISR (#6202)
• PR-5989 - KAFKA-7693: Fix SequenceNumber overflow in producer (#5989)
• PR-5990 - KAFKA-7692: Fix ProducerStateManager SequenceNumber overflow (#5990)
• PR-6134 - KAFKA-7652: Part I; Fix SessionStore’s findSession(single-key) (#6134)
• PR-6121 - KAFKA-7741: Streams exclude javax dependency (#6121)
• PR-6070 - KAFKA-7773: Add end to end system test relying on verifiable consumer (#6070)
• PR-6101 - KAFKA-7786: Ignore OffsetsForLeaderEpoch response if epoch changed while request in flight (#6101)
• PR-6106 - KAFKA-7799: Fix flaky test RestServerTest.testCORSEnabled (#6106)
• PR-6073 - KAFKA-6833: Producer should await metadata for unknown partitions (#6073)
• PR-5470 - KAFKA-7253: The returned connector type is always null when creating connector (#5470)
• PR-6094 - KAFKA-7768: Add version to java html urls (#6094)
• PR-6049 - KAFKA-7755: Look up client host name since DNS entry may have changed (#6049)
• PR-6085 - KAFKA-6928: Refactor StreamsPartitionAssignor retry logic (#6085)
• PR-6032 - KAFKA-7734: Metrics tags should use LinkedHashMap to guarantee ordering (#6032)
• PR-5881 - KAFKA-5503: Idempotent producer ignores shutdown while fetching ProducerId (#5881)
• PR-6027 - KAFKA-3832: Connect’s JSON Converter never outputs a null value (#6027)
• PR-6051 - KAFKA-7759: Disable WADL output in the Connect REST API (#6051)
• PR-5929 - KAFKA-7655: Metadata spamming requests from Kafka Streams under some circumstances, potential DOS (#5929)
• PR-6000 - KAFKA-7705: Fix and simplify producer config in javadoc example (#6000)
• PR-5946 - KAFKA-7443: OffsetOutOfRangeException in restoring state store from changelog topic when start offset of local checkpoint is smaller than that of changelog topic (#5946)
• PR-5962 - KAFKA-7610: Proactively timeout new group members if rebalance is delayed (#5962)
• PR-5925 - KAFKA-7549: Old ProduceRequest with zstd compression does not return error to client (#5925)
• PR-6005 - KAFKA-7709: Fix ConcurrentModificationException when retrieving expired inflight batches on multiple partitions. (#6005)
• PR-5998 - KAFKA-7704: MaxLag.Replica metric is reported incorrectly (#5998)
• PR-5986 - KAFKA-6388: Recover from rolling an empty segment that already exists (#5986)
• PR-5993 - KAFKA-7678: Avoid NPE when closing the RecordCollector (#5993)
• PR-5999 - KAFKA-7697: Process DelayedFetch without holding leaderIsrUpdateLock (#5999)
• PR-5979 - KAFKA-7660: fix streams and Metrics memory leaks (#5979)
• PR-5994 - KAFKA-7702: Fix matching of prefixed ACLs to match single char prefix (#5994)
• PR-5943 - KAFKA-7389: Enable spotBugs with Java 11 and disable false positive warnings (#5943)
• PR-5959 - KAFKA-7671: Stream-Global Table join should not reset repartition flag (#5959)
• PR-5923 - KAFKA-7536: Initialize TopologyTestDriver with non-null topic (#5923)

JDBC Connector

• PR-572 - CC-3638: Added logic to optionally quote identifiers in SQL statements

Schema Registry

• PR-1008 - CC-3740: Add support for ignoring metadata during toConnectMetadata
• PR-1003 - CC-3732: Prevent SR from registering duplicate IDs
• PR-985 - CC-3562: Corrected reconstruction of default value from short and bytes

Confluent Platform 5.1.0

This is a major release of Confluent Platform that provides you with Kafka 2.1.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Highlights

Commercial Features

Control Center

• Improved browser support: In previous releases, Control Center only supported Chrome. Browser support in 5.1.0 has been extended to Chrome, Firefox, and Safari. For more information, see Web Browsers.
• New theme: Control Center now uses a new, more readabable light theme

Community Features

KSQL

• New functions for working with windowed data: KSQL now supports WindowStart() and WindowEnd() functions that expose the start and end times of windows, respectively, allowing developers to easily refer to this information in other functions and in subsequent queries. Start and end times are expressed in milliseconds, like ROWTIME.
• Metric names have changed: See KSQL upgrade notes for details on the changes.

Kafka Streams

• Sending a single, final output for windowed operations: Windowed operations can now be defined to send only a single, final output per window by suppressing intermediate outputs. For example, if an application computes five-minute averages of temperature readings, it can be told to send only one output record every 5 minutes that represents the latest 5-minute temperature average. This new feature makes it much easier to implement use cases such as alerting and metrics pipelines and to integrate Kafka Streams applications with systems that don’t support continuous streaming updates. Note that, by design, using the feature will increase the end-to-end processing latency of applications because no output records will be send until a window is closed. See the section Window Final Results in the Kafka Streams documentation and KIP-328 for more information.
• Better topology compatibility when updating an application: Repartition topics for join and aggregation operators can now be named explicitly by developers. This makes it easier to change and evolve an application while still allowing for rolling app re-deployments to prevent production outages when updating your apps. See KIP-372 for more information.
• Topology optimization: Where possible, Kafka Streams will now merge repartitioning topics, which reduces the Kafka storage footprint of Kafka Streams applications. Users must explicitly enable these optimizations in their application’s configuration by setting the configuration StreamsConfig.TOPOLOGY_OPTIMIZATION to StreamsConfig.OPTIMIZE (default: StreamsConfig.NO_OPTIMIZATION, i.e., disabled). See the section Optimizing Kafka Streams Topologies for more detailed information.
• Improved timestamp synchronization: The new configuration parameter StreamsConfig.MAX_TASK_IDLE_MS_CONFIG allows developers to tune the behavior of when Kafka Streams picks the next record to process, based on the stream-time. It allows for trade-offs between lower processing latency (lower idle settings, the default is 0 to keep pre-5.1 behavior) vs. reducing the probability of out-of-order data (higher idle settings). See KIP-353 for more information.

Kafka Clients

• Added intuitive timeouts for producer: Adds a new configuration parameter for producers, ‘delivery.timeout.ms’, that represents a guarantee on the upper bound on when a record will either get sent, fail or expire from the point when the send returns. This timeout shields users from using workarounds that expose the internals of the producer. Notably, this setting effectively enables retries by default. Bear in mind that reordering is possible if the idempotent producer is not enabled or max.in.flight is not set to 1. See KIP-91 for more information.
• Improved behavior for expiring committed offsets: Consumer group offsets no longer expire as long as there are active consumers. The expiration time of offsets in a group will be when the group becomes “empty” plus retention time of offsets.retention.minutes, unless that group becomes active again. See KIP-211 for more information.

Confluent Clients

Confluent provides high quality clients for the most popular non-Java languages. These clients are all based on the underlying librdkafka C/C++ implementation, which provides high performance and stability. The non-Java clients follow their own versioning scheme. The current clients release is 0.11.6.

Highlights from 0.11.6:

• librdkafka
  • Introduces stability fixes and performance improvements for Windows. #1930, #1980
  • Fixes several producer issues pertaining to queuing, flushing, and timeouts.
  • Fixes consumer inadvertently raising error about message size. #1472
• Python client
  • Adds experimental Windows support to Python client.
  • Includes Confluent C3 monitoring interceptors with the Python client.

Availability and Resilience

• Improved unclean log truncation handling KIP-320 improves consumer handling of unclean log truncation and introduces fencing of zombie replicas. This ensures consistency of replicated data under certain failure conditions.

Performance

• ZStandard compression Kafka now supports ZStandard compression. ZStandard is a real-time compression algorithm developed by Facebook with features that make it highly performant and effective for small data. Kafka now accepts “zstd” as a new compression type for configuring producers, topics, and brokers. See KIP-110 for more information.

Kafka 2.1.0-cp1

28 new features / config changes have been added with 161 resolved bug fixes and improvements. For a full list of changes in this release of Kafka, see the 2.1.0 Release Notes.

How to Download

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

To upgrade Confluent Platform to a newer version, check the Upgrade documentation.

Supported Versions and Interoperability

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?

If you have questions regarding this release, feel free to reach out via the community mailing list or community Slack. Confluent customers are encouraged to contact our support directly.