Confluent Platform 6.1 Release Notes

6.1 is a major release of Confluent Platform that provides you with Apache Kafka® 2.7.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

For more information about the 6.1 release, check out the release blog and the Streaming Audio podcast.

Commercial Features

Multi-Region Clusters

Asynchronous topic-partition replicas that normally sit outside of the in-sync replica (“ISR”) list, also known as observers, can now automatically be promoted into the ISR if a partition is under-replicated or if the partition falls below the topic’s configured “min.insync.replicas” setting. To enable, in your topic’s replica placement JSON, set “version” to 2 and add a line for “observerPromotionPolicy” set to either “under-replicated” or “under-min-isr”.

Self-Balancing

Self-Balancing Clusters in Confluent Platform 6.1 expands observability by adding an additional REST API callS to monitor Self-Balancing tasks when adding a new broker.

Confluent Control Center

• ksqlDB “Add stream/table” buttons removed - These buttons previously led to a modal UI that enabled users to visually create streams and tables. This UI has been eliminated in favor of simply using SQL queries for stream and table creation.

Known Issues

• Some RBAC protected pages throughout the UI will experience a brief 404 error on page reload before the page loads as expected. This is due to a delay in permission response and will be fixed in a subsequent release.
• When configuring a connector through the UI, if a topic name and topic regex is specified an error message may be sporadically displayed. As a workaround, we recommend resubmitting your configuration changes. This will be fixed on a subsequent patch release.

Confluent Community / Apache Kafka

Confluent Platform 6.1 features Apache Kafka® 2.7. For a full list of the 23 KIPs, features, and bug fixes, take a look at the official Apache Kafka release notes, the release overview podcast, and the release blog post. Or watch Confluent’s very own Tim Berglund give an overview of Apache Kafka 2.7.

Clients

Confluent Platform 6.1 includes the latest versions of the Go (confluent-kafka-go), Python (confluent-kafka-python), and .NET (confluent-kafka-dotnet) clients, which are all based on librdkafka v1.6 (also included in the release). For more information on what’s new in the clients, check out the librdkafka v1.6 release notes

Admin REST APIs

Operators can now more easily select which APIs are exposed in the REST Proxy interface using the api.endpoints.blocklist. For example:

api.endpoints.blocklist= api.v2.partitions.list

Cluster Management

Confluent Operator

Confluent Operator 1.7.0 allows you to deploy and manage Confluent Platform versions 6.0.x and 6.1.x on Kubernetes.

There are no notable features, enhancements, or fixes added beyond what is in Confluent Operator 1.6.2.

Ansible Playbooks for Confluent Platform

New features

• You can use Ansible Playbooks for Confluent Platform to configure and deploy Confluent Replicator. Replicator allows you to easily and reliably replicate topics from one Kafka cluster to another. In addition to copying the messages, this connector will create topics as needed preserving the topic configuration in the source cluster.
• You can use Ansible Playbooks for Confluent Platform to name your clusters within the cluster registries in Confluent Platform. cluster registry provides a way to centrally register and identify Kafka clusters in the Metadata Service (MDS) to simplify the RBAC role binding process and to enable centralized audit logging.
• You can use Ansible Playbooks for Confluent Platform to update the Confluent Platform component configuration by re-running the provisioning playbook with an updated inventory file. The rolling and parallel deployment strategies supported.

Notable enhancements

• When enabling Audit Logs for RBAC, you can now specify a different destination Kafka cluster as the

  target to send Audit Logs.

• Basic authentication can now be configured for the Jolokia agent.
• Kafka health checks no longer depend on the Jolokia endpoint. You are not required to enable Jolokia.

Notable fixes

• When zookeeper_ssl_enabled: true, only the secure (TLS) port for the ZooKeeper server is enabled. The non-secure port is not enabled.

• There are no longer two log4j configs that are passed to the processes.

• When debug is enabled with the -vvv Ansible option, sensitive information, such as passwords, certificates, and keys, are printed in the output. Ansible does not provide a way to suppress sensitive information with the -vvv. Therefore, it is not recommended to use the debug mode in production environments.

  As an alternative, use the playbook with the --diff option when troubleshooting issues. With this release, Ansible Playbooks for Confluent Platform no longer prints sensitive information, such as passwords, certificates, and keys, in the output of the --diff option.

  For details, see Troubleshoot Ansible Playbooks for Confluent Platform.

Known issues

You cannot configure Replicator to read from or write to an RBAC-enabled Kafka cluster. This will be addressed in a future update release.

Kafka Brokers

Confluent Server

Known issues

After upgrading from 5.5.x to 6.1.x customers with high partition counts (greater than 100,000) may experience cluster instability. Therefore, we don’t recommend using the 2.7 inter broker protocol (IBP). Contact support if you have any questions about this known issue.

Kafka Streams

• KIP 450: Sliding windows - Adds DSL support for sliding window aggregations.
• KIP 607: RocksDB metrics - Adds additional RocksDB metrics to provide a deeper level of insight into runtime performance characteristics, particularly around RocksDB disk and memory usage.
• KIP 613: End to end latency metrics - Adds end-to-end latency metrics for records processed by Kafka Streams.
• KIP 617: State store backward iteration - Adds support for backward iteration of state stores, which in some cases improves efficiency when fetching ranges of records.

ksqlDB

• Java client support for streaming INSERT - Java client now supports streaming multiple INSERTs over a single connection.
• Java client support for admin operations - Java client now supports ksqlDB admin operations that were previously accessible via the CLI or HTTP interface.
• Java client support for DDL/DML statements - Java client now supports CREATE STREAM, CREATE TABLE, CREATE STREAM … AS SELECT, CREATE TABLE … AS SELECT.
• CREATE OR REPLACE support - Adds support for modifying persistent queries in place via CREATE OR REPLACE ... The new version of a modified persistent query will begin reading from whatever offsets the previous version left off at.
• Support ALTER STREAM | TABLE … ADD COLUMN - Adds support for modifying streams and tables using ALTER.
• Support [IF EXISTS] for DROP CONNECTOR - Adds an optional IF EXISTS clause to DROP CONNECTOR. Whenever IF EXISTS is used and the target connector does not exist, the statement will not throw an error.
• Support for DROP TYPE [IF EXISTS] - Adds an optional IF EXISTS clause to DROP TYPE. Whenever IF EXISTS is used and the target type does not exist, the statement will not throw an error.
• More robust metastore recovery semantics - Metastore recovery may now be configured to fail if any inconsistencies are detected between the existing metastore and backup metastore.
• Log more errors to the processing log - Uncaught Kafka Streams exceptions are now logged to the ksqlDB processing log.
• latest/earliest_by_offset now supports storing n latest/earliest values - latest/earliest_by_offset may now optionally return n values instead of one.
• Add utility to restore the ksqlDB command topic - Adds a utility to streamline and automate command topic restoration.
• CLI support for non-interactive command execution - Adds non-interactive command execution support to the ksqlDB CLI. The -e/--execute flags can now be passed to the CLI along with a command, causing the given command to execute without entering an interactive CLI session.
• Support for JSON key format - Row keys can now use the JSON serialization format, which can be specified via WITH (key_format=’JSON’, …), or WITH (format=’JSON’, …).
• Support for DELIMITED key format - Row keys can now use the DELIMITED serialization format, which can be specified via WITH (key_format=’DELIMITED’, …), or WITH (format=’DELIMITED’, …).
• Add stream/table buttons removed - These buttons previously led to a modal UI that enabled users to visually create streams and tables. This UI has been eliminated in favor of simply using SQL queries for stream/table creation.

Breaking changes

When mutual authentication is enabled on a ksqlDB server (with the server property ssl.client.auth=true), the browser’s client certificate selection dialog is no longer automatically triggered by Confluent Control Center. To trigger the certificate selection dialog, you should navigate to an endpoint on the ksqlDB server, such as https://my-ksqldb-server01.my-organization.net/info.

How to Download

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

Important

The Confluent Platform package includes Confluent Server by default and requires a confluent.license key in your server.properties file. Starting with Confluent Platform 5.4.x, the Confluent Server broker checks for a license during start-up. You must supply a license string in each broker’s properties file using the confluent.license property as below:

confluent.license=LICENCE_STRING_HERE_NO_QUOTES

If you want to use the Kafka broker, download the confluent-community package. The Kafka broker is the default in all Debian or RHEL and CentOS packages.

For more information about migrating to Confluent Server, see Migrate to Confluent Server.

To upgrade Confluent Platform to a newer version, check the Upgrade Confluent Platform documentation.

Supported Versions and Interoperability

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?

If you have questions regarding this release, feel free to reach out via the community mailing list or community Slack. Confluent customers are encouraged to contact our support directly.