.. _release_notes: ================== |cp| Release Notes ================== |cp| 5.4.1 Release Notes ======================== 5.4.1 is a bugfix release of |cp| that provides you with |ak-tm| 2.4.0, the latest stable version of |ak|. The technical details of this bugfix release are summarized in :ref:`cp_changelog`. |cp| 5.4.0 Release Notes ======================== 5.4.0 is a major release of |cp| that provides you with |ak| 2.4.0, the latest stable version of |ak|. The technical details of this release are summarized below. Commercial Features ^^^^^^^^^^^^^^^^^^^ .. _relnotes-cli: |confluent-cli| """"""""""""""" - The |confluent-cli| is now included in the full |cp| tarball. |ccs| users must download the CLI separately and configure it to point at their installation manually. Updates are disabled for the CLI included in the full |cp| tarball to ensure compatibility. - Added support for Windows, except for ``local`` commands. - Added support for centralized ACLs. For more information, see :ref:`relnotes-security-acl`. - Added more information when listing role bindings. For more information, see :ref:`relnotes-security`. .. _relnotes-c3: |c3| """" - **Cluster overview** You can quickly navigate from the home page to an overview of your cluster and some high level metrics, including brokers, topics, and connected services. Red and orange labels denote areas of your cluster that may have issues. For more information, see :ref:`controlcenter_userguide_cluster_overview`. - **Broker metrics dashboard** A new time-series chart that allows you to easily compare metrics for Production, Consumption, Broker uptime, Partition replicas, System and Disk over a unified time frame. You can rearrange the metrics on this dashboard to personalize what metrics are most important or which are better viewed side-by-side. For more information, see :ref:`c3-brokers-metrics`. - **RBAC management** You can now view your own RBAC role assignment and permissions, and manage subordinate role bindings using |c3-short|. For more information, see :ref:`c3_rbac_manage_roles_ui`. - **Replicator monitoring** Previously in |c3-short|, the replicator monitoring was accomplished from within a |kconnect| cluster. With 5.4.0, you can now navigate directly to the **Replicators** menu to see all replicators and their status. For more information, see :ref:`controlcenter_userguide_replicators`. - **Auto-updates** You can now enable |c3-short| to automatically install the most current version, including new features, security enhancements, and critical bug fixes. For more information, see :ref:`c3_auto_update_ui`. - **Topic schema validation** You can now enable key and value schema validation when creating a topic. Under Expert mode, two settings are now visible: ``confluent_value_schema_validation`` and ``confluent_key_schema validation``. - **Topics page enhancements** - The :ref:`c3_all_topics` is updated to include metrics on under-replicated partitions, out-of-sync replicas, out-of-sync observers, and production/consumption throughput for each topic in the cluster. - **User behavior analytics** In an effort to continuously improve our products, user behavior data is now collected by default in |c3-short| starting in 5.4. You can disable collection of user behavior analytics by setting ``confluent.controlcenter.usage.data.collection.enable`` to ``false`` in the configuration. For more information, see :ref:`c3_usage_data_collection`. - **Known issues** - When RBAC is enabled, front-end updates are not available to |c3-short| users. - If RBAC is enabled, |c3-short| cannot monitor data streams that are running on |ccloud|. For more information on how to connect to non-RBAC |c3-short| to Confluent Cloud, see `Connecting Control Center to Confluent Cloud `__. - Some connector configurations are not visible in |c3-short|, for example, the ``avro.codec`` when selecting Avro as the output format in the S3 sink connector. You can instead supply these configurations by using the |kconnect| :ref:`REST API `. .. _relnotes-server: |cs| """" """""""""""""""""""""""""""""""""""""""""""" Multi-Region Clusters (Production Available) """""""""""""""""""""""""""""""""""""""""""" Multi-Region Clusters is now Production Available in CP 5.4.0. This feature allows a single cluster to be stretched across multiple datacenters. This feature centralizes Disaster Recovery operations, making DR events easier manage, monitor, and secure. For more information, see :ref:`bmrr` or check out the example: :devx-examples:`here|multiregion`. """""""""""""""""""""""" Tiered Storage (Preview) """""""""""""""""""""""" .. important:: Tiered Storage is currently in preview and is not advised to run in production environments because the interfaces are not yet stable. Tiered Storage is a major evolution of the architecture and capability of |cp|. This feature makes |cp| more elastic, while giving platform Operators the ability to cost-effectively increase retention periods, all on a per-topic basis. For more information, see :ref:`tiered-storage-preview`. """""""""""""""""""""""""""""""""""""""" Schema Validation (Production Available) """""""""""""""""""""""""""""""""""""""" Schema Validation is now certified for Production Environments. Schema Validation gives operators a centralized location to enforce data format correctness at the topic level. To enable schema validation: - Set ``confluent.schema.registry.url`` in your ``server.properties`` file. - Set ``confluent.value.schema.validation=true`` or ``confluent.key.schema.validation=true``, or both, on the topic. For more information, see :ref:`schema_validation`. |co-long| """"""""" New features - Enables configuration of schema validation. See :ref:`co-schema-validation`. - Supports installation using Helm 3 for a more secure workflow compared to Helm 2. See :ref:`co-upgrading`. - A Quick Start deployment utility is available. See :ref:`co-quickstart`. - For more information, see :ref:`operator-about-intro`. For full deployment instructions, see :ref:`co-deployment`. Upgrade considerations - The minimum supported Kubernetes version is 1.11. Make sure that your underlying Kubernetes environment is upgraded and conforms to the documented supported environments for |cp| for |cp| 5.4. - The default ports for external load balancers changed to 443 for HTTPS and 80 for HTTP traffic. - For information about upgrading to |co| for 5.4, see :ref:`co-migrate-53to54`. - Due to a known issue in |zk|, single-node |zk| clusters may experience data loss if not upgraded carefully. For more information, see :ref:`co-migrate-53to54`. - By default, upgrading to |co| 5.4 will automatically perform a rolling update of all |cp| clusters (e.g. |zk|) that are managed by |co|. For more information about how to control when individual clusters get updated, see :ref:`co-migrate-53to54`. - The default ports for the load balancers for KSQL, |kconnect|, |crep|, |sr|, and |c3| have changed in |co| 5.4. After upgrading to |co| 5.4, the load balancer ports are now the standard HTTPS or HTTP port (443 and 80, respectively). You need to update the clients that access the components through the load balancer to use the new ports. See :ref:`co-migrate-53to54` for more information on port changes. Notable enhancements - Simplifies troubleshooting for |co| by consolidating two |co| pods (“operator” and “manager”) into a single pod. - Reduced the total size of Docker images needed to deploy the product. Notable fixes - Fixes configuration of init-containers deployed by |co| so that they can no longer access data volumes of |ak| and |zk| pods. For supported |cp| features in this version of |co|, see :ref:`Interoperability `. .. _relnotes-security: Security """""""" """""""""""""""""""""""""""""""" Role-based access control (RBAC) """""""""""""""""""""""""""""""" - RBAC is now in Production Availability. This means you can run this feature in production. In 5.3, RBAC was only available in Preview mode for test and development clusters. - RBAC provides centralized and platform-wide authorization for all the |cp| components, including |ak|, |kconnect|, |sr|, KSQL, and |c3-short|. - Added support for public HTTP APIs for viewing and managing role bindings. - Added Swagger UI tooling for development testing against public RBAC and ACL APIs. For more information, see :ref:`mds-api`. - Added support in |c3-short| to view and manage role bindings. For more information, see :ref:`c3_rbac_manage_roles_ui`. - RBAC CLI improvements: - It is now easier to discover cluster ID information. - When listing role bindings for a user, transitive role bindings for that user are now listed through their group role bindings. - Privileged users can now view a list of principals (users/groups) who have a role on specific resources. - For more information, see :ref:`confluent_iam`. .. _relnotes-security-acl: """""""""""""""""""""""""""""""""""""""""""" |ak| Access Control List (ACLs) improvements """""""""""""""""""""""""""""""""""""""""""" - Centralized ACLs that create a consistent user experience for RBAC and ACLs. - A new option to persist all ACLs centrally on a topic, similar to RBAC metadata. Note that you can continue to use |zk| to store ACLs, however, it is recommended that you migrate and utilize the new centralized ACLs. - Centralized ACLs can be managed through public APIs or through |confluent-cli|. - For more information, see :ref:`authorization-acl-with-mds`. """""""""" Audit Logs """""""""" - With this feature, you can capture structured event logs that identify all of the authorization requests across the entire platform. You can monitor access, detect abnormal activity, and meet compliance requirements. - Audit logs are set using the ``server.properties`` configuration file. - Configuration using API, CLI, and GUI is planned for future releases. - For more information, see :ref:`audit-logs`. """""""""""""""""""""""""" FIPS Operational readiness """""""""""""""""""""""""" - You can now use FIPS crypto libraries, ciphers, and operating systems. - FIPS broker connections are enforced. All connections to the broker must use FIPS-compliant ciphers. - If a non-FIPS compliant cipher attempts to make a connection, it is rejected by the broker. - For more information, see :ref:`fips-operational-readiness`. Community Features ^^^^^^^^^^^^^^^^^^ .. _release_notes-clients: Clients """"""" """""""""""""""" Kafka C++ Client """""""""""""""" New features - KIP-392 - Fetch messages from closest replica / follower - Transaction aware consumer (``isolation.level=read_committed``) Upgrade considerations - Consumer property ``isolation.level=read_committed`` ensures the consumer will only read messages from successfully committed producer transactions. The default is ``read_committed``. To get the previous behavior, set the property to ``read_uncommitted``, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction. Notable enhancements - Improved authentication errors (KIP-152). - Sub-millisecond buffering (``linger.ms``) on the producer. - Optimize varint decoding, increasing consume performance by ~15%. Notable fixes - Upgrade builtin lz4 to 1.9.2 (CVE-2019-17543, #2598). - Messages were not timed out for leader-less partitions (.NET issue #1027). - Fix msgq (re)insertion code to avoid O(N^2) insert sort operations on retry (#2508). The msgq insert code now properly handles interleaved and overlapping message range inserts, which may occur during Producer retries for high-throughput applications. - Rate limit IO-based queue wakeups to ``linger.ms``, which reduces CPU load and lock contention for high-throughput producer applications (#2509). For the full librdkafka release notes, see the `librdkafka GitHub project `__. """""""""""""""""""""""""""" Python, .NET, and Go Clients """""""""""""""""""""""""""" Clients include all the improvements from the latest version. |kconnect| """""""""" - Writes log messages to a file by default in addition to writing them to the console (KIP-521). - Enables you to change log levels dynamically (KIP-495). - Adds more metrics for connectors (KIP-475). - Secures internal |kconnect| REST endpoints (KIP-507). - Connect Converter implementations can use record headers when (de)serializing record keys and values (KIP-440), although the Avro Converter and JSON Converter were not changed to use headers. - The JSON Converter supports Decimal type in JSON properly (KIP-481). Connectors """""""""" """""""""""""" JDBC Connector """""""""""""" - SQL Server’s DateTimeOffset type is handled as a |kconnect| TIMESTAMP logical type. - Includes the most recent bugfix release of the PostgreSQL JDBC driver (9.4.1212). - Notable fixes: A source task fails when it fails to read and convert any column; improved log messages; corrected PostgreSQL upsert statements when writing records to tables with no primary key columns. """"""""""""""""" S3 Sink Connector """"""""""""""""" - Now supports Parquet format, set with ``format.class=io.confluent.connect.s3.format.parquet.ParquetFormat``. - New configuration ``behavior.on.null.values`` specifies how the connector should behave when a record with a null value is consumed by this connector. - Notable fixes: Improved retry logic when failing to close files; improve behavior when evolving schemas to different names. """""""""""""""""""""" HDFS v2 Sink Connector """""""""""""""""""""" - Add support for |kconnect| logical types (time, timestamp, date, decimal) when using Parquet formatted files. When Parquet format is used with Hive, only the decimal LogicalType is supported. - File caching is disabled to be more compatible with recent HDFS versions. - Notable fixes: improved retry logic when failing to close files; improve behavior when evolving schemas to different names. """""""""""""""""""" JMS Source Connector """""""""""""""""""" - Supports permissive schemas. When ``use.permissive.schemas=true``, the messageID becomes optional in the key and value. - Notable fixes: Some versions of dependent libraries were changed to avoid versions with known CVEs. """""""""""""""""""""""""""" Elasticsearch Sink Connector """""""""""""""""""""""""""" Notable fixes: Improved INFO and DEBUG logging; improved support for Elasticsearch 7; fixed performance regression related to index checks. Ansible """"""" New features - Greatly improves cluster lifecycle management with new Ansible playbooks for in-place upgrades of |cp|. - Enables configuration of schema validation. - Enables FIPS-compliant configuration for |cp|. Notable enhancements - Enables configuration of SASL SCRAM for greater freedom to choose secure authentication configurations for |cp|. - Simplifies configuration management by making all properties configurable through variable files during initial deployment on a per-component basis. - Improves monitoring for |cp| by enabling installation of monitoring-interceptor JARs. - Standardizes logging across components for simpler troubleshooting and auditing. For supported |cp| features in in this version of |ansible|, see :ref:`Interoperability `. .. _release_notes-streams: |kstreams| """""""""" - **KIP-213: Support non-key joining in KTables** - It’s often the case that two events have a relationship in common beside their partition key. Previously, the |kstreams| DSL didn’t provide a mechanism to support joining two tables on anything but their key. This adds support for non-key table/table joins. It also supports updates from both sides of the join. - **KIP-307: Allow defining custom processor names with the KStreams DSL** - In the past, |kstreams| would automatically assign names to the processor nodes in a topology. This would make complex topologies hard to understand or debug when you examined their contents. This KIP introduces the ``.withName()`` method, allowing you to set more meaningful names to each node in the topology. - **KIP-470: TopologyTestDriver test input and output usability improvements** - The TopologyTestDriver class is a useful utility for testing |kstreams| applications in a lightweight, deterministic way. Its existing API, however, made it rather verbose to feed it input and draw output from the program being tested. This KIP adds new utilities for interacting with the input and output of topologies being tested in a more concise manner. - **KIP-471: Expose RocksDB Metrics in Kafka Streams** - RocksDB, which |kstreams| leverages internally, has functionality to collect statistics about operations over its running instances. These statistics enable users to find bottlenecks and to accordingly tune RocksDB, which can be critical for operating |kstreams| in performance-sensitive environments. Previously, |kstreams| had not surfaced these metrics. This KIP exposes a subset of RocksDB's statistics in |kstreams| metrics. .. _release_notes-ksql: KSQL """" - **Connect integration** - Create and manage external |kconnect| data sources and sinks entirely from within KSQL. External sources and sinks may be created and configured using the KSQL ``CREATE … CONNECTOR`` command. KSQL can integrate with |kconnect| in one of these ways: - **Embedded mode** - |kconnect| workers run embedded within the KSQL JVM, no external |kconnect| cluster is required. KSQL will automatically launch embedded |kconnect| workers when configured for embedded mode. - **External mode** - KSQL is pointed to an external |kconnect| cluster. KSQL executes all connector management operations against this external cluster. - **Pull queries** - Pull queries have been added to support point-in-time lookup queries against materialized views. Only single-row lookups are currently supported but query coverage will expand over time. - **Support for quoted identifiers** - Column names containing unparseable characters can now be used by enclosing them within backticks (`````). - **explode()** - User-defined table functions (UDTF) are now supported by KSQL, and ``explode()`` has been added as a new built-in UDTF. ``explode()`` takes an array as an argument, and outputs a row for each element in the array (essentially ``flatMap()``). - **New built-ins** - The following builtin functions have been added to KSQL: - **avg(v)** - Compute the average of all input values (aggregate) - **exp (v)** - Compute the exponential of ``v`` - **initcap(str)** - Capitalize the first letter of each word in the input string - **ln(v)** - Compute the natural logarithm of ``v`` - **replace(col, find, replace)** - Replace all occurrences of ``find`` with ``replace`` - **round(v [, scale])** - Round the given value optionally using the given scale (default scale is 0) - **sign(v)** - Compute the sign of ``v`` (returns 1 for positive, -1 for negative) - **sqrt(v)** - Compute the square root of ``v`` - **KAFKA serde format** - The ``KAFKA`` serialization format has been added to support |ak|-serialized keys and values. ``KAFKA`` may be assigned as a ``key_format`` and ``value_format``. For more information, see :ref:`kafka_format`. - **Client output formatting** - Output formatting improvements have been made to the KSQL CLI, and column names are now included in the output of transient queries. - **Custom type registry** - Type declarations can now be saved and reused using the ``CREATE TYPE`` command. This is particularly useful for complex, verbose type declarations that are cumbersome to reuse. Type declarations may now be named and referred to as their own types. - **User-specified delimiter characters** - You can now specify a custom delimiter when using the ``delimited`` ``value_format``. The delimiter is set with the ``value_delimiter`` ``WITH`` option. - **Decimal datatype** - A `Decimal` type has been introduced. Decimals offer significantly increased precision relative to KSQL’s `Double` type, and are thus suitable for high-precision use cases such as currency. Deprecation Warnings """""""""""""""""""" - Beginning with the next major release of |cp|, |kconnect| connectors will no longer be packaged natively with |cp|. Instead all connectors must be downloaded directly from `Confluent Hub `_. - Starting with |cp| 6.0, TLS 1.0 and 1.1 will be disabled and no longer supported in |cp|. .. _release_notes-download: How to Download ^^^^^^^^^^^^^^^ |cp| is available for download at https://www.confluent.io/download/. See the :ref:`installation` section for detailed information. .. include:: ../installation/includes/cs-license.rst For more information about migrating to |cs|, see :ref:`migrate-confluent-server`. To upgrade |cp| to a newer version, check the :ref:`upgrade` documentation. .. _release_notes-sup-version: Supported Versions and Interoperability ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For the supported versions and interoperability of |cp| and its components, see :ref:`interoperability-versions`. .. _release_notes-question: Questions? ^^^^^^^^^^ If you have questions regarding this release, feel free to reach out via the `community mailing list `_ or `community Slack `_. Confluent customers are encouraged to contact our support directly. .. toctree:: :hidden: Component Changelogs