.. _release_notes: ============= Release Notes ============= 5.3.0 is a major release of |cp| that provides you with |ak-tm| 2.3.0, the latest stable version of |ak|. The technical details of this release are summarized below. Commercial Features ^^^^^^^^^^^^^^^^^^^ .. _release_notes-cli: |confluent-cli| """"""""""""""" * The |confluent-cli| was completely rewritten to support a wider set of features and platforms. * The |confluent-cli| was removed from |cp| packaging. The |confluent-cli| is now installed separately from the |cp| package. For more information, see :ref:`cli-install`. * The |confluent-cli| development commands have been moved to `confluent local`. For example, the syntax for ``confluent start`` is now ``confluent local start``. For more information, see :ref:`confluent_local`. * Added support for managing :ref:`role-based access control (RBAC) permissions ` and role bindings. * Added support for :ref:`masking secrets ` and configuration files. * Added "self-update" capability via the :ref:`confluent_update` command so that improvements and fixes can be released out-of-band from the |cp| release schedule. * The |confluent-cli| is a commercially licensed feature, but it can be downloaded and used against both |cc| and |cp|. .. _release_notes-commercial-confluent-control-center: |c3| """" |c3-short| introduces several changes and new features, including: - **Redesigned UI**: The |c3-short| UI has been redesigned from end-to-end to provide an easier-to-use, multi-cluster-friendly, and global view of |ak| that supports working with Confluent at scale. The new homepage provides a global view of all of your |ak| clusters and connected services, including |kconnect| and KSQL. You can see a health roll-up for each cluster and use a switcher to view only the unhealthy clusters so you can focus on what needs attention. - **New broker overview**: Clicking one of the cards on the new cluster homepage takes you into the cluster and connected services, showing broker metrics or topics depending on your permissions if |rbac| is enabled. You can drill down further into broker metrics by clicking on a card, which shows time-series charts with interactive legends and shared cursors for easier metrics correlation. - **New topics pages**: The redesigned |c3-short| includes a new topics index that shows you all the topics in a cluster with key health information for each in a searchable table with sortable columns. You can click on a topic name to view an overview for it, including relevant throughput and health information, and access the message browser, schemas, and topic configuration. Topic metrics, including a topic-centric view of consumer lag, can be accessed by clicking on the metrics cards. Under consumer lag, you can select which consumer group you are interested in using a drop-down so you can focus on the specific application you care about. - **Message browser enhancements**: Message browser performance has been improved to eliminate throttling and message loss. You can also now access the full topic history by seeking, either using offset or timestamp, on a specific partition. You can also now download messages in JSON format, including using the typical Ctrl/Shift-select shortcuts to select and download more than one message. - **Connect redesign and enhancements**: The new UI also features an index page for all of the |kconnect| clusters associated with a Kafka cluster. You can search for a |kconnect| cluster by name/ID and see how many connectors are running, degraded, or failed on each cluster. Degraded connectors have at least one failed task; whereas failed connectors have failures on all of their tasks. Clicking on one of the clusters in the |kconnect| index takes you to the new |kconnect| cluster overview page. Here, you can see all the connectors running on the cluster, including their status, type, category, and number of tasks. Clicking on one of the connectors in the |kconnect| cluster overview takes you to a page showing the connector itself where you can see a breakdown of the connector's tasks and their status. This new |kconnect| experience makes it much easier to view all of the |kconnect| clusters and drill down to see how each task on a connector is performing. - **KSQL redesign and enhancements**: The new UI also features an index page for all of the KSQL clusters associated with a Kafka cluster. You can search for a KSQL cluster by name/ID, see how many queries are running, and see how many streams or tables are on the cluster. Clicking on one of the clusters in the KSQL index takes you to a page where you can access the KSQL editor, streams, tables, and persistent queries. The editor itself features similar improvements as the message browser, including performance improvements to eliminate throttling and message loss and JSON downloading of results. In addition, the KSQL editor area is resizeable to accommodate larger queries and statements. It also includes a data discovery side-panel, making it much easier to find streams and tables to run KSQL against. - **Consumers index**: |c3-short| also includes a new index to show you all the consumer groups associated with a selected Kafka cluster, including the number of consumers per group and the number of topics being consumed. Clicking on a consumer group takes you to a page where you can view Consumer Lag across all relevant topics and a redesigned Streams Monitoring page, making it much easier to understand consumption metrics in context. - **Alerting enhancements**: |c3-short| features several highly requested alerting enhancements. To date, |c3-short| only supported email-based alert actions. In |cp| 5.3.0, you can now use webhooks for Slack and PagerDuty, making it much easier to integrate |c3-short| with a range of monitoring approaches. There is also a new Consumer Lead alert, which you can customize to fire when a consumer is within a certain number of offsets of the partition tail. You can also now pause and resume actions globally, silencing alerts when needed. Finally, the alert messages themselves have become more informative and now include better contextual information about the alert, including the cluster, metric, condition, and trigger. - **Feature flagging for System Health and Data Streams**: While |cp| 5.3.0 introduces a significantly redesigned |c3-short|, a feature flag allows you to view the older System Health and Data Streams pages. .. note:: These pages cannot be viewed if |rbac| is enabled. - **RBAC enforcement**: As part of the introduction of |rbac| to |cp|, |c3-short| respects the rolebindings assigned using the CLI, so you have platform-wide consistency for your security strategy. For more information, see :ref:`controlcenter_security_rbac`. For more information, see the :ref:`documentation `. .. _known_issues: """"""""""""""""""""""" |c3-short| Known Issues """"""""""""""""""""""" - **Multi-cluster alert triggers:** You can currently select more than one cluster when defining a Cluster or Broker alert trigger in |c3-short| but doing so results in a trigger firing against all of the associated clusters rather than the one that actually meets the alert definition. To avoid this issue in the interim, Cluster and Broker alerts should be defined against individual clusters. - **Direct access to each external cluster is required under RBAC:** When |rbac| is enabled, |c3-short| requires management to be enabled on each cluster. This means that metrics reporters and interceptors must have a direct connection for |c3-short| to operate properly in an |rbac| environment at this time. See :ref:`c3_KI_cluster connections` for more details. - **Connect and KSQL require at least minimal user access to the underlying Kafka clusters:** See :ref:`c3_KI_ksql_connect` for details. - **The service principal for Control Center requires SystemAdmin access:** The |c3-short| user must be set up as a privileged user ``SystemAdmin`` on each cluster. Due to the underlying architecture for consumer lag, elevated privileges are required to guarantee users access to all consumer groups and continued support for :ref:`consumer group alerts for consumer lag `. The consumer lag offsets are currently not obtained from metrics reporters. - **Prefix search in the message browser cannot currently search column names that contain a period.** - **There is a mismatch between products in enforced default values for max message bytes:** When connecting |c3-short| to |ccloud|, the ``confluent.metrics.topic.max.message.bytes`` must be set to ``8388608`` rather than the current |c3-short| default of ``10485760``. For more information, see :ref:`c3_connect_ccloud_max_bytes` and `Connecting Kafka Streams to Confluent Cloud `__. .. MMA-3564 max msg bytes Security """""""" """""""""""""""""""""""""""""""" Role-based access control (RBAC) """""""""""""""""""""""""""""""" .. include:: ../includes/rbac-preview.rst * Provides the secure authorization of access to resources by users and groups. * Uses a set of predefined roles that give you depth and granularity in its configuration of managing access. * Provides centralized authorization for all the components including |c3-short|, |kconnect|, |sr|, |crest|, and KSQL. * Provides connector level access control. For more information, see the RBAC documentation: - :ref:`Overview of RBAC in Confluent Platform ` - :ref:`RBAC in Control Center ` - :ref:`RBAC in KSQL ` - :ref:`RBAC in Connect ` - :ref:`RBAC in Schema Registry ` """""""""""""""""""" AD/LDAP enhancements """""""""""""""""""" * Centralized configuration. * Centralized authentication for |c3-short|, |kconnect|, |sr|, |crest|, and KSQL. For more information, see :ref:`kafka_ldap_authorizer`. """"""""""""""""" Secret Protection """"""""""""""""" * To enable masking of sensitive information in the configuration file and logs. * Across the whole platform including |ak|, |kconnect|, KSQL, |sr|, |crest|, and |c3-short|. * Utilize the :ref:`new CLI ` to create/manage secrets. For more information, see :ref:`secrets`. Community Features ^^^^^^^^^^^^^^^^^^ .. _release_notes-clients: Clients """"""" librdkafka v1.1.0 is a security feature release with added support for: * OAUTHBEARER SASL authentication. * In-memory SSL certificates. * Windows Root Certificate store. * Pluggable broker SSL certificate verification callback. * Improved SASL GSSAPI/Kerberos ticket refresh. **Upgrade considerations:** * Windows SSL users will no longer need to specify a CA certificate file/directory (``ssl.ca.location``), librdkafka will load the CA certs by default from the Windows Root Certificate Store. * SSL peer (broker) certificate verification is now enabled by default (disable with ``enable.ssl.certificate.verification=false``) * ``%{broker.name}`` is no longer supported in ``sasl.kerberos.kinit.cmd`` since kinit refresh is no longer executed per broker, but per client instance. See the full `librdkafka v1.1.0 release notes `_ for more information. """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" confluent-kafka-python, confluent-kafka-dotnet, confluent-kafka-go """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Clients now use librdkafka v1.1.0 and include all the improvements from the latest version. |kconnect| """""""""" * `PR-6363 `_ - KAFKA-5505: Incremental cooperative rebalancing in |kconnect| (KIP-415) (#6363) When a rebalance happened in |ak| 2.2 or earlier, it stops all tasks in a |kconnect| cluster and restarts them. This can be a hard stop for users who run multiple connectors in a |kconnect| cluster. With KIP-415, a rebalance happens more gracefully. It stops only the tasks that need to move between workers (if any), leaving the rest running on their assigned worker * `PR-5743 `_ - KAFKA-3816: Add MDC logging to |kconnect| runtime (#5743) With KIP-449, The SLF4J API includes "Mapped Diagnostic Contexts" (MDC) that allow injection of a series of parameters that can be included in every log message written using that thread, regardless of how the SLF4J Logger instance was obtained. * `PR-6584 `_ - KAFKA-8231: Expansion of ConnectClusterState interface (#6584) * `PR-6789 `_ - KAFKA-8407: Fix validation of class and list configs in connector client overrides (#6789) * `PR-6658 `_ - KAFKA-8309: Add Consolidated Connector Endpoint to |kconnect| REST API (#6658) Connectors """""""""" * **HDFS Connector:** HDFS connector is removed from the |cp| package. However, the connector will continue to be downloadable from `Confluent Hub `_ and supported by Confluent. * `CVE-2015-2156 `_ * `CVE-2016-4970 `_ * `CVE-2015-3253 `_ * `CVE-2016-6814 `_ * **JDBC Connector:** `PR-641 `_ - CC-349: Added delete support for sink. (#641) A record with null value is considered a tombstone event and result in deleting the corresponding row in the destination table. * **Elasticsearch Connector:** `PR-239 `_ - Support for document upsert. (#239) In some cases, records in Kafka topics consist only subset of fields ElasticSearch needs. With this feature, ElasticSearch Sink Connector can perform an upsert operation. .. _release_notes-ak-release: |ak| 2.3.0 """"""""""" * **KIP-339: A new IncrementalAlterConfigs API in AdminClient** Changes configurations incrementally, only modifying the configuration values that are specified to prevent lost updates. AlterConfigs API has been marked for deprecation. For more information, see `KIP-339 `_ * **KIP-341: Update Sticky Assignor’s User Data Protocol** Improves stability of consumer partition assignments by preventing assigning the same partition to multiple consumers in the same consumer group. For more information, see `KIP-341 `_ * **KIP-351: Add --under-min-isr option to describe topics command** Adds the ``--under-min-isr`` option in the describe topics command, which allows users to see precisely which topic partitions are below ``min.insync.replicas`` need to be addressed. For more information, see `KIP-351 `_ * **KIP-354: Maximum Log Compaction Lag** Compaction allows Kafka to remove messages that are are older than ``min.compaction.lag.ms``, which ensures that a segment is rolled and remains uncompacted for a given period ( “lag”). Regulations such as GDPR require that data is deleted in a timely manner. ``max.compaction.lag.ms`` sets the maximum lag time for which a segment may remain uncompacted, after which the corresponding log partition becomes eligible for log compaction. For more information, see `KIP-354 `_ * **KIP-361: Add Consumer Configuration to Disable Auto Topic Creation** Both auto.create.topics.enable on the broker and allow.auto.create.topics on the consumer need to be set for auto-topic creation to happen. For more information, see `KIP-361 `_ * **KIP-402: Improve fairness in SocketServer processors** The first part of KIP-402 was introduced in |ak| 2.2, which changed how connections were prioritized. Now existing connections are given more priority over new connection requests. The KIP introduces max.connections, which limits the total number of connections that may be active on the broker at any time. This is in addition to the existing ``max.connections.per.ip`` configuration that will continue to limit the number of connections from each host IP address. For more information, see `KIP-402 `_ * **KIP-417: Allow JMXTool to connect to to a secure RMI port** Adds ``--jmx-ssl-enable`` and ``--jmx-auth-prop`` to allow connecting to a secure Java VM. For more information, see `KIP-417 `_ * **KIP-421: Support resolving externalized secrets in AbstractConfig** Enhances the AbstractConfig base class to automatically resolve variables of the form specified in `KIP-297 `_. For more information, see `KIP-421 `_ * **KIP-425: Add Log4J Kafka Appender Properties to Producing to Secure Brokers** Extends the Log4J Kafka appender to support PLAIN mechanism and configuration of jaas via a property passed to the producer. For more information, see `KIP-425 `_ * **KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)** Introduces a new topic partition category in the metrics group between ``UnderReplicated`` and ``UnderMinIsr`` called ``AtMinIsr``. For more information, see `KIP-427 `_ * **KIP-430 - Return Authorized Operations in Describe Responses** The AdminClient now allows users to determine what operations they are authorized to perform on topics. For more information, see `KIP-430 `_ * **KIP-436: Add a metric indicating start time** Useful for detecting restarts. For more information, see `KIP-436 `_ * **KIP-461: Improve Replica Fetcher Behavior at handling partition failure** In previous versions, if a partition fails, the replica fetcher thread associated with that partition will terminate. Because the replica fetcher threads handle multiple partitions, this led to under-replicated partitions. Now, whenever a partition crashes, the concerned thread stops tracking the crashed parition and continues handling rest of the partitions. This all reduces the chance of under-replicated partitions and improves cluster stability. For more information, see `KIP-461 `_ The release includes 157 new features, improvements, and fixes. For a full list of changes in this release of |ak|, see the `Apache Kafka 2.3.0 Release Notes `_. Also, see the blog post covering `What's New in Apache Kafka 2.3 `_ or this `video `_. .. _release_notes-streams: |kstreams| """""""""" * **Stored Record Timestamps in RocksDB:** To offer better timestamp processing semantics and introduce new time-related KTable features, more information about the timeliness of records must be persisted. |kstreams| now stores the timestamps for each record that contributes to state stores inside RocksDB. These timestamps are now accessible through Interactive Queries, too. This functionality paves the way for potential future features, like KTable TTLs and support for out-of-order KTable data. See `KIP-258 `_ for more information. * **In-memory window and session stores:** |kstreams| supports pluggable storage of its tables. Until this release, |kstreams| offered only durable versions out of the box of its window and session store abstractions. |kstreams| now ships with in-memory versions implemented to support high performance, transient operations. See `KIP-428 `_ and `KIP-445 `_ for more information. * **New flatTransformValues method:** |kstreams| now supports a new method in its API, flatTransformValues. It is the equivalent of flatMapValues for the Processor API. Unlike the traditional transformValues method, this method is able to ensure strong typing by specifying in its signature a list of key-value pairs (i.e. Iterable) as output records for each input record. See `KIP-313 `_ for more information. * **Default implementation to close() and configure() for Serializer, Deserializer and Serde:** When serializers, deserializers, and serdes are authored, the close and configure methods are typically implemented with no operation behind them. |kstreams| now leverages Java's new default interface inheritance feature, and provides a sensible, default implementation for these methods. See `KIP-331 `_ for more information. * **Better defaults for increased stability:** To better improve footprint and threading stability, a number of defaults have been changed to more sensible values. These include `max.poll.interval.ms`, `segment.ms`, and `segment.index.bytes`. See `KIP-442 `_ and `KIP-443 `_ for more information. * **New close() method on RocksDBConfigSetter:** A new ``close()`` method has been added to support cleanup of RocksDB configs. This helps to evade inadvertent memory leaks. See `KIP-453 `_ for more information. .. _release_notes-ksql: KSQL """" * **KEY no longer required for CREATE TABLE:** Tables can now be created without manually specifying a ``KEY`` parameter. For example, ``CREATE TABLE T1 (ID INT, OTHER INT)``. ``KEY`` can still be specified as a hint, but is no longer required (`#2745 `_). * **CREATE STREAM/TABLE now creates topics if they don't exist:** Previously a topic underlying a stream or table had to exist already to create a stream or table over it. KSQL now creates the underlying topic automatically if it doesn't already exist (`#2771 `_). * **Improved UDF interfaces:** Two of the four components of `KLIP-1 `_ have made it into this release: UDFs can now be defined with variable-length arguments as well as custom Struct argument types (`#2503 `_). * **Data can now be written to streams via INSERT INTO:** Users can now write events directly to streams from within KSQL using standard SQL ``INSERT INTO`` syntax. While not designed for high-performance production usage, simplifying the process of getting data into KSQL should greatly improve the initial KSQL experience and make it much easier for new users to experiment with the system (`KLIP-2 `_, `#2723 `_). * **KSQL test runner tool:** The ``ksql-test-runner`` tool is a command line utility that enables testing KSQL statements without requiring any infrastructure, which means that Kafka and KSQL don't need to be running. It’s a lightweight way to design and iterate on your KSQL-based applications and ensure that the expected results are generated (`#2802 `_). * **CLI sessions can now be recorded to text files:** ``SPOOL`` is a new CLI command that allows users to record a KSQL CLI session’s inputs and outputs, writing the session content out to a file (`#2789 `_). * **ELT and FIELD functions:** Both the ``ELT`` and ``FIELD`` functions are based on the SQL standard and are available in many other SQL-based systems (`#2627 `_). ``ELT (N, x, y, ...)`` returns the Nth element of the given list of strings. This is the complement to ``FIELD``. ``FIELD (string, str1, str2, ...)`` returns the index of string within the list of given strings. This is the complement to ``ELT``. * **More date/timestamp formats now supported:** KSQL’s date/timestamp parser is now more robust and understands more date/timestamp formats (`#2499 `_). * **Quotes in string literals now escaped:** This protects KSQL from malicious Java code injection attacks (`#2545 `_). * **Bug fixes**: * Fixed GEO_DISTANCE function (`#2700 `_). * Fixed bug causing tabs in multi-line input strings to cause unnecessary errors (`#2734 `_). * Fixed bug that caused command topic replay during KSQL startup to recreate deleted topics unintentionally (`#2329 `_). * Fixed bug with performing joins on ROWKEY (`#2735 `_). * Fixed bug causing potential crash during KSQL REST server shutdown (`#2507 `_). .. _release_notes-confluent-rest-proxy: |crest| """"""" * FIX: |crest| now responds with a 401/403 for authentication/authorization errors. |rest-utils| """""""""""" The default value for ``ssl.endpoint.identification.algorithm`` was changed to ``https``. This setting performs hostname verification to prevent man-in-the-middle attacks. You can set ``ssl.endpoint.identification.algorithm`` to an empty string to restore the previous behaviour. |sr| """" * `PR-1124 `_ - CC-4775: logical type preservation for hdfs connector when output format is parquet (#1124) Other Improvements & Changes """""""""""""""""""""""""""" * **KAFKA-8336: Enable dynamic update of client-side SSL factory in brokers** When mutual authentication is enabled for inter-broker-communication (``ssl.client.auth=required``), broker restarts are no longer needed when updating client-side keystores. * **FIX: Added better open file limit to systemd files** Deprecation Warnings """""""""""""""""""" * OS support for RHEL 6, Ubuntu 12.04, Ubuntu 14.04, and Debian 7 * Confluent Rest Proxy v1 API .. _release_notes-download: How to Download ^^^^^^^^^^^^^^^ |cp| is available for download at https://www.confluent.io/download/. See the :ref:`installation` section for detailed information. To upgrade |cp| to a newer version, check the :ref:`upgrade` documentation. .. _release_notes-sup-version: Supported Versions and Interoperability ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For the supported versions and interoperability of |cp| and its components, see :ref:`interoperability-versions`. .. _release_notes-question: Questions? ^^^^^^^^^^ If you have questions regarding this release, feel free to reach out via the `community mailing list `__ or `community Slack `__. Confluent customers are encouraged to contact our support directly. .. toctree:: :hidden: changelog 5-4-preview