Confluent Platform 6.1 Release Notes¶
6.1 is a major release of Confluent Platform that provides you with Apache Kafka® 2.7.0, the latest stable version of Kafka.
The technical details of this release are summarized below.
Asynchronous topic-partition replicas that normally sit outside of the in-sync replica (“ISR”) list, also known as observers, can now automatically be promoted into the ISR if a partition is under-replicated or if the partition falls below the topic’s configured “min.insync.replicas” setting. To enable, in your topic’s replica placement JSON, set “version” to 2 and add a line for “observerPromotionPolicy” set to either “under-replicated” or “under-min-isr”.
Self-Balancing Clusters in Confluent Platform 6.1 expands observability by adding an additional REST API callS to monitor Self-Balancing tasks when adding a new broker.
Confluent Control Center¶
- ksqlDB “Add stream/table” buttons removed - These buttons previously led to a modal UI that enabled users to visually create streams and tables. This UI has been eliminated in favor of simply using SQL queries for stream and table creation.
- Some RBAC protected pages throughout the UI will experience a brief 404 error on page reload before the page loads as expected. This is due to a delay in permission response and will be fixed in a subsequent release.
- When configuring a connector through the UI, if a topic name and topic regex is specified an error message may be sporadically displayed. As a workaround, we recommend resubmitting your configuration changes. This will be fixed on a subsequent patch release.
Confluent Community / Apache Kafka¶
Confluent Platform 6.1 features Apache Kafka® 2.7. For a full list of the 23 KIPs, features, and bug fixes, take a look at the official Apache Kafka release notes, the release overview podcast, and the release blog post. Or watch Confluent’s very own Tim Berglund give an overview of Apache Kafka 2.7.
Confluent Platform 6.1 includes the latest versions of the Go (confluent-kafka-go), Python (confluent-kafka-python), and .NET (confluent-kafka-dotnet) clients, which are all based on librdkafka v1.6 (also included in the release). For more information on what’s new in the clients, check out the librdkafka v1.6 release notes
Admin REST APIs¶
Operators can now more easily select which APIs are exposed in the REST Proxy interface using the
api.endpoints.blocklist. For example:
Confluent Operator 1.7.0 allows you to deploy and manage Confluent Platform versions 6.0.x and 6.1.x on Kubernetes.
There are no notable features, enhancements, or fixes added beyond what is in Confluent Operator 1.6.2.
Ansible Playbooks for Confluent Platform¶
- You can use Ansible Playbooks for Confluent Platform to configure and deploy Confluent Replicator. Replicator allows you to easily and reliably replicate topics from one Kafka cluster to another. In addition to copying the messages, this connector will create topics as needed preserving the topic configuration in the source cluster.
- You can use Ansible Playbooks for Confluent Platform to name your clusters within the cluster registries in Confluent Platform. cluster registry provides a way to centrally register and identify Kafka clusters in the Metadata Service (MDS) to simplify the RBAC role binding process and to enable centralized audit logging.
- You can use Ansible Playbooks for Confluent Platform to update the Confluent Platform component configuration by re-running the provisioning playbook with an updated inventory file. The rolling and parallel deployment strategies supported.
- When enabling Audit Logs for RBAC, you can now specify a different destination Kafka cluster as the
- target to send Audit Logs.
- Basic authentication can now be configured for the Jolokia agent.
- Kafka health checks no longer depend on the Jolokia endpoint. You are not required to enable Jolokia.
zookeeper_ssl_enabled: true, only the secure (TLS) port for the ZooKeeper server is enabled. The non-secure port is not enabled.
There are no longer two log4j configs that are passed to the processes.
When debug is enabled with the
-vvvAnsible option, sensitive information, such as passwords, certificates, and keys, are printed in the output. Ansible does not provide a way to suppress sensitive information with the
-vvv. Therefore, it is not recommended to use the debug mode in production environments.
As an alternative, use the playbook with the
--diffoption when troubleshooting issues. With this release, Ansible Playbooks for Confluent Platform no longer prints sensitive information, such as passwords, certificates, and keys, in the output of the
For details, see Troubleshoot Ansible Playbooks for Confluent Platform.
You cannot configure Replicator to read from or write to an RBAC-enabled Kafka cluster. This will be addressed in a future update release.
After upgrading from 5.5.x to 6.1.x customers with high partition counts (greater than 100,000) may experience cluster instability. Therefore, we don’t recommend using the 2.7 inter broker protocol (IBP). Contact support if you have any questions about this known issue.
- KIP 450: Sliding windows - Adds DSL support for sliding window aggregations.
- KIP 607: RocksDB metrics - Adds additional RocksDB metrics to provide a deeper level of insight into runtime performance characteristics, particularly around RocksDB disk and memory usage.
- KIP 613: End to end latency metrics - Adds end-to-end latency metrics for records processed by Kafka Streams.
- KIP 617: State store backward iteration - Adds support for backward iteration of state stores, which in some cases improves efficiency when fetching ranges of records.
- Java client support for streaming INSERT - Java client now supports streaming multiple INSERTs over a single connection.
- Java client support for admin operations - Java client now supports ksqlDB admin operations that were previously accessible via the CLI or HTTP interface.
- Java client support for DDL/DML statements -
Java client now supports
CREATE STREAM, CREATE TABLE, CREATE STREAM … AS SELECT, CREATE TABLE … AS SELECT.
- CREATE OR REPLACE support - Adds support for
modifying persistent queries in place via
CREATE OR REPLACE ... The new version of a modified persistent query will begin reading from whatever offsets the previous version left off at.
- Support ALTER STREAM | TABLE … ADD COLUMN -
Adds support for modifying streams and tables using
- Support [IF EXISTS] for DROP CONNECTOR -
Adds an optional
IF EXISTSclause to
DROP CONNECTOR. Whenever
IF EXISTSis used and the target connector does not exist, the statement will not throw an error.
- Support for DROP TYPE [IF EXISTS] - Adds an
IF EXISTSclause to DROP TYPE. Whenever
IF EXISTSis used and the target type does not exist, the statement will not throw an error.
- More robust metastore recovery semantics - Metastore recovery may now be configured to fail if any inconsistencies are detected between the existing metastore and backup metastore.
- Log more errors to the processing log - Uncaught Kafka Streams exceptions are now logged to the ksqlDB processing log.
- latest/earliest_by_offset now supports storing n latest/earliest values - latest/earliest_by_offset may now optionally return n values instead of one.
- Add utility to restore the ksqlDB command topic - Adds a utility to streamline and automate command topic restoration.
- CLI support for non-interactive command execution -
Adds non-interactive command execution support to the ksqlDB CLI. The
-e/--executeflags can now be passed to the CLI along with a command, causing the given command to execute without entering an interactive CLI session.
- Support for JSON key format - Row keys can
now use the JSON serialization format, which can be specified via WITH (
key_format=’JSON’, …), or
WITH (format=’JSON’, …).
- Support for DELIMITED key format - Row keys
can now use the DELIMITED serialization format, which can be specified via
WITH (key_format=’DELIMITED’, …), or
WITH (format=’DELIMITED’, …).
- Add stream/table buttons removed - These buttons previously led to a modal UI that enabled users to visually create streams and tables. This UI has been eliminated in favor of simply using SQL queries for stream/table creation.
When mutual authentication is enabled on a ksqlDB server (with the server property
ssl.client.auth=true), the browser’s client certificate selection dialog is no longer automatically
triggered by Confluent Control Center. To trigger the certificate selection dialog, you should navigate to an endpoint on
the ksqlDB server, such as https://my-ksqldb-server01.my-organization.net/info.
How to Download¶
The Confluent Platform package includes Confluent Server by default and requires a
confluent.license key in your
server.properties file. Starting with
Confluent Platform 5.4.x, the Confluent Server broker checks for a license during start-up. You must
supply a license string in each broker’s properties file using the
confluent.license property as below:
If you want to use the Kafka broker, download the
The Kafka broker is the default in all Debian or RHEL and CentOS packages.
For more information about migrating to Confluent Server, see Migrate to Confluent Server.
To upgrade Confluent Platform to a newer version, check the Upgrade Confluent Platform documentation.
Supported Versions and Interoperability¶
For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.