Confluent Platform 5.0.1 Release Notes

This release of Confluent Platform provides Confluent users with Apache Kafka 2.0.1, the latest stable version of Kafka, and additional bug fixes.

Users are encouraged to upgrade to Confluent Platform 5.0.1, as it includes important bug fixes. The technical details of this release are summarized below.

Highlights

Enterprise Features

Confluent Rebalancer

  • Warn when rack constraint relaxed.
  • CPKAFKA-1675: Relax rack constraint when already violated.

Confluent Security Plugins

  • Change Kafka REST to Confluent REST

Control Center

  • Fix the MockRecordCollectorImpl constructor
  • MMA-2031: Clarify support for 1 mil partitions
  • MMA-2871: Fix logout logic
  • ESCALATION-399: codeWithSidebar invalid rendering
  • MMA-2868: Have an end date that does not crash the UI

Hub Client

  • HUB-22: Bump up Jackson version.
  • CC-2540: Download archive using endpoint in manifest, which isn’t always the same as Plugin Registry.
  • CC-2059: Make client to work on MacOS, installed via brew cask formula.
  • CC-2287: Detect all configs passed to Connect process.

Kafka Connect Replicator

  • CC-2974: Allow intra-cluster replication when using provenance headers
  • CC-2897: Handle topic recreation when using provenance headers
  • CC-2803: Improve err msg during offset translation
  • CC-2844: Call configure on Converters.
  • CC-2884: Disable provenance headers by default
  • CC-2861: Interceptor correctly checks if internal topic exists
  • CC-2830: Add Kafka Connect Maven Plugin to earliest-packaged -post branch
  • CC-861: Improve the log messages for Replicator licenses
  • CC-1977: commit offsets to source cluster
  • CC-2795: Bump committed offset
  • CC-2424: Fix incompatibility with message.downconversion.enable
  • CC-2748: Ignore records with no timestamp
  • CC-2747: Offset translation should handle records without timestamps
  • CC-2713: conditionally check provenance header

Kafka Connect JMS

  • CC-2769: Corrected mock method call due to change in return type
  • CC-2829: Add Kafka Connect Maven Plugin to earliest-packaged -post branch
  • CC-2758: Fix build to no longer generate docs
  • CC-2607: Remove the unused HornetQ-specific connector

MQTT Proxy

  • CC-2726: MQTT Proxy creates one streams thread pool per connection
  • CC-2469: Pass prefixed ssl.client.auth configuration property.
  • CC-2482: Validate user/password for listener’s security protocol SASL_TLS.

Open Source Features

Apache Kafka 2.0.1-cp1

  • PR-5853 - KAFKA-7561: Increase stop_timeout_sec to make ConsoleConsumerTest pass (#5853)
  • PR-5833 - KAFKA-7534: Error in flush calling close may prevent underlying store from closing (#5833)
  • PR-5831 - KAFKA-7532: Clean-up controller log when shutting down brokers (#5831)
  • PR-5515 - KAFKA-7299: Batch LeaderAndIsr requests for AutoLeaderRebalance (#5515)
  • PR-5820 - KAFKA-7519 Clear pending transaction state when expiration fails (#5820)
  • PR-5763 - KAFKA-7198: Enhance KafkaStreams start method javadoc. (#5763)
  • PR-5720 - KAFKA-6914; Set parent classloader of DelegatingClassLoader same as the worker’s (#5720)
  • PR-5727 - KAFKA-7467; NoSuchElementException is raised because controlBatch is empty (#5727)
  • PR-5678 - KAFKA-7415; Persist leader epoch and start offset on becoming a leader (#5678)
  • PR-5717 - KAFKA-7459: Use thread-safe Pool for RequestMetrics.requestRateInternal (#5717)
  • PR-5712 - KAFKA-7453: Expire registered channels not selected within idle timeout (#5712)
  • PR-5713 - KAFKA-7454: Use lazy allocation for SslTransportLayer buffers and null them on close (#5713)
  • PR-5654 - KAFKA-7414; Out of range errors should never be fatal for follower (#5654)
  • PR-5673 - KAFKA-7216: Ignore unknown ResourceTypes while loading acl cache (#5673)
  • PR-5495 - KAFKA-7280; Synchronize consumer fetch request/response handling (#5495)
  • PR-5622 - KAFKA-7386: streams-scala should not cache serdes (#5622)
  • PR-5627 - KAFKA-7044; Fix Fetcher.fetchOffsetsByTimes and NPE in describe consumer group (#5627)
  • PR-5500 - KAFKA-7286; Avoid getting stuck loading large metadata records (#5500)
  • PR-5623 - KAFKA-7385; Fix log cleaner behavior when only empty batches are retained (#5623)
  • PR-5595 - KAFKA-7369; Handle retriable errors in AdminClient list groups API (#5595)
  • PR-5503 - KAFKA-7287: Set open ACL for old consumer znode path (#5503)
  • PR-5557 - KAFKA-7128; Follower has to catch up to offset within current leader epoch to join ISR (#5557)
  • PR-5489 - KAFKA-7225; Corrected system tests by generating external properties file (#5489)
  • PR-5502 - KAFKA-7301: Fix streams Scala join ambiguous overload (#5502)
  • PR-4633 - KAFKA-5891; Proper handling of LogicalTypes in Cast (#4633)
  • PR-5491 - KAFKA-7278; replaceSegments() should not call asyncDeleteSegment() for segments which have been removed from segments list (#5491)
  • PR-5518 - KAFKA-7298; Raise UnknownProducerIdException if next sequence number is unknown (#5518)
  • PR-5514 - KAFKA-7296; Handle coordinator loading error in TxnOffsetCommit (#5514)
  • PR-5509 - KAFKA-7119: Handle transient Kerberos errors on server side (#5509)
  • PR-5501 - KAFKA-7285: Create new producer on each rebalance if EOS enabled (#5501)
  • PR-5487 - KAFKA-7119: Handle transient Kerberos errors as non-fatal exceptions (#5487)
  • PR-5499 - KAFKA-7284: streams should unwrap fenced exception (#5499)
  • PR-5436 - KAFKA-7164; Follower should truncate after every missed leader epoch change (#5436)
  • PR-5484 - KAFKA-7261: Record 1.0 for total metric when Count stat is used for rate (#5484)
  • PR-5481 - KAFKA-7250: switch scala transform to TransformSupplier (#5481)
  • PR-3907 - KAFKA-4950; Fix ConcurrentModificationException on assigned-partitions metric update (#3907)
  • PR-5466 - KAFKA-7158: Add unit test for window store range queries (#5466)
  • PR-5474 - KAFKA-7080: pass segmentInterval to CachingWindowStore (#5474)
  • PR-5478 - KAFKA-7255: Fix timing issue with create/update in SimpleAclAuthorizer (#5478)
  • PR-5468 - KAFKA-7250: fix transform function in scala DSL to accept TranformerSupplier (#5468)
  • PR-5444 - KAFKA-7231; Ensure NetworkClient uses overridden request timeout (#5444)
  • PR-5390 - KAFKA-7144: Fix task assignment to be even (#5390)
  • PR-5430 - KAFKA-7192 Follow-up: update checkpoint to the reset beginning offset (#5430)
  • PR-5421 - KAFKA-7192: Wipe out if EOS is turned on and checkpoint file does not exist (#5421)

Kafka Connect Elasticsearch

  • PR-243 - CC-2827: Add Kafka Connect Maven Plugin to earliest-packaged -post branch

Kafka Connect Hdfs

  • PR-383 - CC-2797: Add trace messages for rotations
  • PR-380 - CC-2826: Add Kafka Connect Maven Plugin to earliest-packaged -post branch
  • PR-363 - CC-2515: Open temp Parquet files with OVERWRITE mode
  • PR-359 - CC-2493: Improved initialization and logging of Parquet writer
  • PR-362 - CC-2399: Fix NPE during close

Kafka Connect Jdbc

  • PR-499 - CC-2825: Add Kafka Connect Maven Plugin to earliest-packaged -post branch

Kafka Connect Storage Cloud

  • PR-209 - CC-2828: Add Kafka Connect Maven Plugin to earliest-packaged -post branch
  • PR-204 - CC-1214: Import all AvroData configs to storage connectors except enhanced avro schema support
  • PR-200 - CC-2619 Bugfix for append.late.data configuration rotating partitions
  • PR-196 - CC-2601 Add S3 Append-Late-Data option to configs
  • PR-187 - CC-2313 Handle late arriving records in storage cloud sink connectors

Kafka Connect Storage Common

  • PR-87 - CC-2313: Revert and postpone append.late.data config
  • PR-82 - CC-2213 New Config Option introduced to handle late-arriving records

Kafka Streams Examples

  • PR-169 - KSTREAMS-1761: Register JacksonFeature in client
  • PR-168 - KSTREAMS-1760: parse args for bootstrap servers and schema registry Url
  • PR-153 - KSTREAMS-1712: Add option for configuring Producers, Consumers with interceptors
  • PR-135 - KSTREAMS-1621: Initial integration of monitoring interceptors

KSQL

  • PR-2072 - KSQL-1777: Fix Create Table as Select property table
  • PR-2058 - KSQL-1609: Remove in-page TOCs
  • PR-2036 - KSQL-1602: Join Streams and Tables
  • PR-2050 - KSQL-1636: Add intro section to UDFs/UDAFs reference
  • PR-2044 - KSQL-1760: Fix text block indents and incorrect include symbols
  • PR-2026 - KSQL-1657: Update wording around KSQL Cookbook links
  • PR-2025 - KSQL-1733: Add new video on monitoring KSQL in C3
  • PR-2013 - KSQL-1724: Remove extraneous link
  • PR-1996 - KSQL-1712: Update KSQL CLI banner block to use codewithvars
  • PR-1953 - KSQL-1656: Move udf.rst to developer-guide
  • PR-1951 - KSQL-1656: Move dev topics into developer-guide directory
  • PR-1948 - KSQL-1656: Create new developer-guide directory
  • PR-1927 - KSQL-1592: Convert command snippet to codewithvars
  • PR-1926 - KSQL-1592: Add section for info endpoint
  • PR-1913 - KSQL-1587: Add links to the KSQL recipes
  • PR-1911 - KSQL-1514: Update link to STRUCT content
  • PR-1904 - KSQL-1514: Add section for STRUCT type
  • PR-1901 - KSQL-1566: Replace embedded videos with links

Schema Registry

  • PR-906 - CC-2831: Add Kafka Connect Maven Plugin to earliest-packaged -post branch
  • PR-910 - CC-2769: Document the new AvroMessageFormatter options for printing schema IDs
  • PR-911 - CC-2769: Correct getBytes to not rely on default charset.
  • PR-901 - CC-2769: Ability to optionally include in output the schema ID for key and value
  • PR-860 - HUB-13: Fix source URL for Avro Converter
  • PR-852 - CC-2011: Package Avro Converter for hosting on Confluent Hub

Support Metrics

  • PR-52 - CPKAFKA-1582: Improve metrics thread shutdown resiliency
  • PR-45 - CPKAFKA-1582: More resilient shutdown for metrics thread

Previous releases

Confluent Platform 5.0.0 Release Notes

5.0.0 is a major release of the Confluent Platform that provides Confluent users with Apache Kafka 2.0.0, the latest stable version of Kafka.

Confluent Platform users are encouraged to upgrade to 5.0.0. The technical details of this release are summarized below.

Highlights

Enterprise Features
Confluent Control Center

Confluent Control Center introduces several new features, including:

  • A new UI for KSQL: Control Center now includes a brand new graphical UI for KSQL. With this new UI, you can create KSQL streams and tables in Control Center, as well as run, view, terminate, and explain queries and statements.
  • Broker configuration: Making sure your brokers are set up just the way you want is now easy with Control Center. You can now view broker configurations across multiple Kafka clusters, quickly check specific brokers, and easily spot differences in properties.
  • Topic inspection: Control Center now makes it easy to see the actual streaming messages coming out of Kafka topics using the new topic inspection feature.
  • Consumer lag: Wondering how your consumers are performing? You can now see if any consumers are lagging using the new consumer lag feature. You can also set alerts on consumer lag, so you can monitor performance and proactively address issues whenever you think best.
  • Schema Registry support: Control Center now supports viewing the schema associated with a topic, along with its version history, and allows comparing older versions with the current one.
LDAP Authorizer Plugin

The new LDAP authorizer plugin enables group-based access control for Kafka, allowing you to use the user-group mappings in your LDAP server. The plugin works with any security protocol by mapping an authenticated user principal to one or more groups and allowing user-principal-based ACLs as well as group-based ACLs. This simplifies access control management because a smaller number of rules can be used to manage access to groups or organizations.

Replicator

Replicator already automates the replication of topic messages and related metadata across Kafka clusters. With Confluent Platform 5.0, Replicator and the new consumer timestamps interceptor also ensure that consumer offsets are available on the secondary cluster. When consumer client applications failover to the secondary cluster, Replicator handles consumer offset translation so that they can resume consuming near the last point they stopped consuming at in the original cluster. This minimizes the reprocessing that consumers needs to do in a disaster scenario without skipping messages. See the Replicator and Cross-Cluster Failover section for more detail.

To facilitate application failover and switch back in Confluent Platform 5.0, Replicator adds support for protection against circular replication. This guarantees that if two Replicator instances are configured to run, one replicating from DC1 to DC2 and the second instance configured to replicate from DC2 to DC1, Replicator will ensure that messages replicated to DC2 are not replicated back to DC1, and vice versa. As a result, Replicator safely runs in each direction.

MQTT Proxy

This new component is an MQTT 3.1.1 broker to help streamline Internet of Things (IoT) architectures with a scalable proxy similar to Confluent REST Proxy for HTTP(S) communication with Kafka. MQTT Proxy enables the replacement of third party MQTT brokers with Kafka-native MQTT proxies while avoiding intermediate storage and lag. MQTT Proxy uses Transport Layer Security (TLS) encryption and basic authentication. It also supports the widely used MQTT 3.1.1 protocol so MQTT client messages at all three MQTT quality of service levels are written directly into Kafka topics where KSQL and streaming applications can consume and process them. Future releases will add bidirectional communication and more security options.

Open Source Features
Security

Apache Kafka 2.0.0 adds support for prefixed ACLs, simplifying access control management in large, secure deployments. Prefixed ACLs enable bulk access to topics, consumer groups, or transactional ids with a prefix to be granted using a single rule. See KIP-290 for details.

Access control for topic creation has been improved to enable access to be granted to create specific topics or topics with a prefix.

KIP-255 adds a framework for authenticating to Kafka brokers using OAuth2 access tokens. Authentication is performed using SASL/OAUTHBEARER mechanism defined in RFC-7628, which enables authentication using OAuth2 bearer tokens in non-HTTP protocols. The implementation in Apache Kafka is customizable using callbacks for token retrieval and validation, providing the flexibility required for integration in different deployments.

You can now dynamically update SSL truststores without broker restart.

You can now configure security for listeners in ZooKeeper before starting brokers, including SSL keystore and truststore passwords, and JAAS configuration for SASL. With this new feature, you can store sensitive password configs in encrypted form in ZooKeeper rather than in cleartext in the broker properties file.

Host name verification is now enabled by default for SSL connections to ensure that the default SSL configuration is not susceptible to man-in-the-middle attacks. Set <code>ssl.endpoint.identification.algorithm</code> to an empty string to restore the previous behaviour.

Availability and Resilience

The replication protocol has been improved to avoid log divergence between leader and follower during fast-leader failover.

To avoid OutOfMemory errors in brokers, the memory footprint of message down-conversions has been reduced. By using message chunking, both memory usage and memory reference time have been reduced to a minimum. Down-conversion is required to support consumers that only support an older message format.

Kafka clients are now notified of throttling before any throttling is applied when quotas are enabled. This enables clients to distinguish between network errors and large throttle times when quotas are exceeded. Quota implementations can also now be customized to work in different deployments.

Consumer Enhancements

KIP-266 removes indefinite blocking in the Java consumer, adding a new configuration option to control maximum wait time. Consumer group operations have also been added to the AdminClient. New metrics have also been added to monitor the lead between consumer offset and the start of the log to track slow consumers which may lose data when lead is close to zero.

Legacy Code Cleanup

Apache Kafka 2.0.0 drops support for Java 7 and removes the previously deprecated Scala producer and consumer.

KSQL
  • Improved REST API : We have made several improvements to the KSQL REST API, including returning proper HTTP response status codes on error conditions, a cleaner hierarchy of response objects, and more details about queries in the query descriptions returned by EXPLAIN.
  • Aggregations on tables: KSQL now supports non-windowed usage of SUM and COUNT aggregation functions on tables. Previously, you could only perform such aggregations on streams.
  • Support for timestamps in more formats: Users can now specify a TIMESTAMP_FORMAT in addition to specifying the timestamp column when creating new streams and tables. This allows KSQL to use timestamp data in any format supproted by the Java DateTimeFormatter. See the KSQL Syntax Reference for details.
  • Added protection when dropping streams and tables: When you execute a DROP TABLE or DROP STREAM command, KSQL now checks whether there are any currently active queries using that stream or table. If there are, KSQL will prevent you from dropping the stream or table without first terminating the queries which depend on it. This prevents KSQL from getting into an inconsistent internal state and thus boosts stability.
  • INSERT INTO: KSQL now supports INSERT INTO, a new type of statement that lets you write the output of a query into an existing stream. See the KSQL Syntax Reference for details.
  • Confluent Platform Docker Images for KSQL: Confluent Platform Docker images are now available on docker hub for the 5.0 preview version of both the KSQL server and KSQL CLI. You can use the confluentinc/cp-ksql-server image to deploy the server in interactive (default) or headless mode. You can use the confluentinc/cp-ksql-cli image to start a CLI session inside a Docker container.
  • Topic/Schema Cleanup on DELETE: Users can now optionally delete the backing topic and its registered schema (if any) when dropping a stream or table. See the KSQL Syntax Reference for details.
  • Support for STRUCT data type: Added support for nested data using STRUCT data type. You can now declare streams and tables with columns using a STRUCT data type in your CREATE STREAM and CREATE TABLE statements, and then access the internal fields of these columns in your SELECT queries as you do in any other expression. See the KSQL Syntax Reference for details.
  • UDF/UDAF: You can bring your own custom User Defined Functions (UDFs) and User Defined Aggregate Functions (UDAFs) to KSQL and use them in your queries. UDF/UDAF open the door for many novel use cases where you need to perform custom computations over your data.
  • Stream-Stream and Table-Table Joins: Added Stream-Stream and Table-Table joins in Confluent Platform 5.0. For each of these joins, KSQL supports inner, full outer, and left join types. This means KSQL now covers all of the available join operations in Kafka Streams. See the KSQL Syntax Reference for details.
Kafka Streams

Kafka Streams now support message headers and dynamic routing. See the Kafka Streams Upgrade Guide for details.

Kafka Connect

Kafka Connect includes a number of improvements and features. You can now control how errors in connectors, transformations, and converters are handled by enabling automatic retries and controlling the number of errors that are tolerated before an exception is thrown and the connector is stopped. More contextual information can be included in the logs to help diagnose problems, and problematic messages consumed by sink connectors can be sent to a dead letter queue rather than forcing the connector to stop.

There’s also a new extension point for moving secrets outside of connector configurations. You can integrate any external key management system and will allow you to store variables or placeholders in the connector configurations and resolve them in the external key management systems. The variables are only resolved before sending the configuration to the connector, making it possible to ensure that secrets are stored and managed securely in your preferred key management system and not exposed over the REST APIs or in log files.

Confluent Clients

Confluent provides high quality clients for the most popular non-Java languages, these clients are all based on the underlying librdkafka C/C++ implementation, which provides high performance and stability. The non-Java clients follow their own versioning scheme, slightly trailing the Apache Kafka releases. The current clients release is v0.11.5.

Highlights from v0.11.5:

  • Admin API support.
  • Java compatible murmur2 partitioner.
  • Schema Registry HTTPS support for the Python client.
  • Many enhancements and bug fixes.

See the client release notes for more information:

JDBC Connector

The JDBC source and sink connectors were modified to move DBMS-specific logic into dialects, making it easier to add and support various flavors of JDBC-compliant databases. This should be transparent to users as the connector will automatically select and use the most appropriate dialect for the connection.

Elasticsearch Connector

The Elasticsearch connector is now able to use basic authentication when connecting to Elasticsearch. It also better supports keyword fields for text types.

S3 Connector

The S3 connector extends the existing SSE support to enable customer-provided server-side encryption (SSE-C) keys.

Camus

Camus was deprecated in Confluent Platform 3.0.0 and has been removed in Confluent Platform 5.0.0.

Apache Kafka 2.0.0-cp1

40 new features / config changes have been added with 246 resolved bug fixes and improvements. For a full list of changes in this release of Apache Kafka, see the 2.0.0 Release Notes.

How to Download

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

To upgrade Confluent Platform to a newer version, check the Upgrade documentation.

Supported Versions and Interoperability

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?

If you have questions regarding this release, feel free to reach out via the |cp| mailing list. Confluent customers are encouraged to contact our support directly.