.. _release_notes: |cp| |release| Release Notes ============================ |release| is a bug fix release of |cp| that provides you with |ak-tm| |kafka_upstream_release|, the latest stable version of |ak|. Changelogs for the patch releases can be found in :ref:`Confluent Platform Component Changelogs `. The technical details of this release are summarized below. For more information about the 5.5.0 release, check out the `release blog `__ and the `Streaming Audio podcast `__. .. raw:: html Commercial Features ^^^^^^^^^^^^^^^^^^^ .. _relnotes-c3: |c3| """" New |ksqldb| topology viewer |ksqldb| Applications can now be represented as a topology graph. With this view you can quickly see and make sense of how data flows across your |ksqldb| application and how existing queries interact with each other. Each node in the graph represents either a stream, table or query while each connecting line (or edge) in the graph represents one node’s input source and another node’s output destination. Selecting a node allows you to view real-time messages and schemas. UI Rebrand The |c3-short| UI has a brand new look! There are new colors and fonts in the UI, and more emphasis on the most important and relevant pieces of information. The fonts have been updated to make the metrics more legible and improve the hierarchy of data, to make it easier to find what you need. For more information, see :ref:`controlcenter_userguide`. Protobuf and JSON support in message viewer With the addition of Protocol Buffers and JSON Schema support in |sr|, the |c3-short| schema viewer has been updated allowing you to create, inspect, edit and version these additional schema types. Seek to human readable time/date in message viewer Human-readable timestamps have been added to the topic message viewer UI. This makes it easier to jump backwards to a date and time on a given topic and see messages from that point forward. Use the feature to easily inspect the messages flowing through a topic without having to know a particular offset or the epoch timestamp “Messages behind” added to Consumer Group table Back by popular demand, the “Messages behind” has been added as a column in our Consumer Groups overview table. This column allows you to see the total sum of messages behind for all the consumers in a given consumer group without having to drill into each one. .. _relnotes-server: |cs| """" - Multi-Region Clusters improvements: - Default replica placement constraints with ``confluent.log.placement.constraints`` can now be set on the brokers to make it easier to use multi-region with |ksqldb| and |kstreams|. You are still required to manually set replica placement constraints when setting up a multi-region cluster on ``__consumer_offsets`` and ``__transaction``. - The ``kafka-topic`` command now has an ``--invalid-replica-placement-partitions`` flag that allows you to quickly see all partitions that are not meeting their replication placement constraints. - |cadb-full| now includes ``-topics`` and ``--replica-placement-only`` flags that make it easier to change replica placement constraints. This procedure is sometimes required to properly sequence disaster recovery failback procedures. - Tiered Storage (Preview) now supports inter-broker SSL. .. _relnotes-security: Security """""""" - Added support for JSON and JAAS file types for secret protection. For more information, see :ref:`secrets-encrypt-jaas`. .. important:: Only embedded JAAS is supported. Static JAAS files are not supported. Static JAAS files are only used by |zk| clients. - Enabled TLS to |zk| so that all the network communication to |zk| is secure. For more information, see :ref:`zk-authentication-overview`. - Added support for AD/LDAP based authentication for clients when SASL/PLAIN is used. For more information, see :ref:`client-auth-with-ldap`. .. _known-secrets-jaas: """"""""""" Known issue """"""""""" You cannot encrypt multiple passwords in the same JAAS configuration using the |confluent-cli| secrets encryption command for encrypting the JAAS, otherwise some of the encrypted configuration get replaced with empty string. Symptoms You specify multiple passwords in the same JAAS configuration: :: listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret"; When you try to encrypt password and then encrypt ``user_admin``, the encrypted configuration looks like this (an empty value for ``password``): :: listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="" \ user_admin=$ {securepass:/etc/secrets/secrets.properties:server.properties/listener.name.sasl_ssl.plain.sasl.jaas.config/org.apache.kafka.common.security.plain.PlainLoginModule/user_admin} ; Solution If you encounter any empty password strings, after encrypting the JAAS configuration, follow these steps to fix: #. Open the ``secrets.properties`` file located at ``/etc/secrets/secrets.properties``. #. Find the key for the empty password in ``/etc/secrets/secrets.properties``. For example, ``listener.name.sasl_ssl.plain.sasl.jaas.config/org.apache.kafka.common.security.plain.PlainLoginModule/password``. #. Replace the empty string in ``server.properties`` file with tuple ``{securepass:/path/to/secrets-remote.txt:key}``. For example: :: listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password=$ {securepass:/etc/secrets/secrets.properties:server.properties/listener.name.sasl_ssl.plain.sasl.jaas.config/org.apache.kafka.common.security.plain.PlainLoginModule/password} \ user_admin=$ {securepass:/etc/secrets/secrets.properties:server.properties/listener.name.sasl_ssl.plain.sasl.jaas.config/org.apache.kafka.common.security.plain.PlainLoginModule/user_admin} ; This issue will be fixed in the next |confluent-cli| release. |co-long| """"""""" """""""""""" New features """""""""""" - Secures the platform by enabling |rbac-long| across |cp| for greenfield installations via the |co|, including partial automation of the full configuration and bootstrapping. For more information, see :ref:`co-rbac`. - Secures the platform by enabling configuration of TLS for encryption and mTLS for authentication on all JMX and Jolokia endpoints associated with each CP component. - Integrates better with Kubernetes-native workflows by enabling users to set Kubernetes Pod Annotations on all |cp| components managed by |co|. - Simplifies platform configuration management by allowing users to specify the Kubernetes Storage Class to be used for the Persistent Volumes associated with each |cp| component deployed by the |co|, and automatically uses the default Storage Class if none is specified. """""""""""""""""""""" Upgrade considerations """""""""""""""""""""" - For this release of |co|, the minimum supported Kubernetes version is now 1.13. Ensure that your underlying Kubernetes environment is upgraded and conforms to the :ref:`documented supported environments ` for |co|. - By default, upgrading to |co| to this latest release will automatically perform a rolling update of all |cp| clusters (e.g. |zk|) managed by the |co|. If you need greater control over when individual clusters get updated, and don’t want all clusters to be updated upon upgrading |co|, see the documentation for :ref:`upgrading to Operator for 5.5 `. - Upgrading KSQL 5.4 to |ksqldb| 5.5 in-place is not supported. See :ref:`co-upgrading` for the steps to migrate KSQL 5.4 to |ksqldb| 5.5. - The name of the |ksqldb| Docker Hub image changed to ``cp-ksqldb-server-operator``. - Confluent has deprecated support for Docker images based on the Debian and Alpine operating systems, and provides images based on the Red Hat Universal Base Image (UBI) instead. After July 1, 2020, Confluent will not release new versions of the Debian- and Alpine-based images. For more information about this change, see this `Knowledge Base article (requires login) `__. For instructions on how to use the UBI-based images with your |co| deployments, see :ref:`co-custom-image-tags`. """""""""""""""""""" Notable enhancements """""""""""""""""""" - Adds support for Kubernetes 1.16 and 1.17 (and will likely support newer Kubernetes versions as they’re released). - Enables you to deploy |cp| components via the |co| without the requirement to have cluster-level user permissions. - Enables you to install the |co| without the requirement to grant the |co| cluster-level service account permissions. - Improves the management of sensitive credentials, such as allowing users to specify a reference to a secret object for the Confluent license key, rather than passing the value in directly when running Helm commands. |cadb-full| """"""""""" - |cadb-full| (ADB) now includes a ``--topics`` flag that will limit the rebalance plan to a single topic or set of topics. This will not find a globally optimal rebalance solution, but will allow for a faster rebalance on the topics selected. - ADB includes a ``--replica-placement-only`` flag that you can use to limit a rebalance plan to topics that have changed their replica placement constraints, and are not satisfying this constraint under their current location. For more information, see :ref:`rebalancer`. .. _relnotes-crep: |crep| """""" A new |crep| command line tool is added that can identify issues with a |crep| configuration and recommend remedial actions. For more information, see :ref:`replicator_verifier`. Community Features ^^^^^^^^^^^^^^^^^^ Apache Kafka """""""""""" |cp| 5.5 includes |ak| 2.5. For a full list of |ak| KIPs, features, and bug fixes, see the |ak| `release notes `__. Noteable KIPs include: - `KIP-515 `__ enables |zk-full| to Broker TLS authentication and |ak| now ships with |zk| 3.5.7. - `KIP-447 `__ saw significant producer scalability when using exactly once semantics. - `KIP-541 `__ added a ``fetch.max.bytes`` configuration for the broker that makes it easier for |ak| operators to enforce client behavior centrally across the cluster. Ansible """"""" """""""""""" New features """""""""""" - Supports resilient and performant multi-site architectures of |cp| by allowing users to configure a logical “rack” property per broker, unlocking the ability to leverage rack awareness across the platform. This feature is also now available on |cp| 5.4.x. - Simplifies the management of multiple |ksqldb| clusters and multiple |kconnect| clusters within a single instance of the platform, all within a single Ansible inventory file. This feature is also now available on |cp| 5.4.x. - Secures the platform by enabling |rbac-long| and Access Control Lists (ACLs) across |cp| for greenfield installations via the Ansible Playbooks, including automation of the full configuration and bootstrapping. This release adds support for ACLs based on the |mds-long|, rather than |zk|-based ACLs. For more information, see :ref:`ansible-authz-rbac`. """""""""""""""""""""" Upgrade considerations """""""""""""""""""""" - Upgrading KSQL 5.4 to |ksqldb| 5.5 in-place is not supported in Ansible. Follow the instructions in :ref:`upgrading-ksql` to migrate previous versions of KSQL to |ksqldb| 5.5. """""""""""""""""""" Notable enhancements """""""""""""""""""" - Allows you to install |cp| components individually, rather than all in a single command. - Allows you to specify the Linux users that |cp| component processes should run as. |sr| """" - Support for Protocol Buffers and JSON Schema has been added in |sr| and throughout |cp|. - |sr| is now pluggable with the ability to add new schema types via schema plugins. - Confluent provides three out-of-the box schema plugins for Avro, Protobuf and JSON Schema schema types. - Proto 3 version is supported for Protocol Buffers (version 2 is partially supported). - Draft 7 spec is supported for JSON Schema. - Other Confluent components upgraded to support Protobuf and JSON, including: |c3-short|, :ref:`Confluent CLI `, :ksqldb-docs:`ksqlDB|operate-and-deploy/installation/server-config/avro-schema/`, :ref:`Streams `, :ref:`Connect `, :ref:`REST Proxy `, and |ak| :ref:`Clients (Java, Python, .NET) `. For more information, see :ref:`serializer_and_formatter`. - :ref:`confluentsecurityplugins_schema_registry_security_plugin` trial licenses are not supported with TLS-enabled |zk| quorums. .. _release_notes-clients: Clients """"""" - librdkafka now has complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (in this release). This enables developers to create Exactly-Once applications with |ak|. For a complete transactional application example, see the `librdkafka transactions example `__. - librdkafka 1.4 adds support for `KIP-345 `__ (static consumer group membership) and `KIP-511 `__ (report client software name and version to broker). - librdkafka 1.4 includes a number of fixes and enhancements. For more details, see the `librdkafka release notes `__. |crest| """"""" |crest-long| API v3 preview incorporates a new v3 namespace that will enable full coverage of all |ak| APIs, including AdminClient APIs. In this release, support is added for common operations such as listing topics, creating topics, listing brokers, and describing topic configs. For a full list of supported operations, see :ref:`Kafka REST Proxy API reference `. |kconnect| """""""""" `KIP-558 `__ - track the set of actively used topics by connectors in |kconnect|. During runtime it’s not easy to know the topics a sink connector reads records from when a regex is used for topic selection. For a source connector, bookkeeping of active topics requires some sort of external tracking by the connector or its user. Connectors """""""""" """""""""""""" JDBC Connector """""""""""""" - Enables retries when a connection fails. - Adds an option to start an initial query with a specific timestamp using the ``timestamp.initial`` configuration property in JDBC source. - Supports a suffix query config for DB2. - JDBC source logs at the trace level queries executed by the connector. - Fixes OOM for Postgres by limiting the fetch size. - Packages the latest version, 42.2.10, of Postgres JDBC driver. - Upgrades derby to 10.14.2.0. """""""""""""""""""""""" Amazon S3 Sink Connector """""""""""""""""""""""" - Introduces configuration properties to easily define key and secret credentials per connector instance. - Adds AWS IAM Assume Role credentials provider to enable cross-account authorization. - Allow setting a compression level for gzip in Json and ByteArray formats. - Upgrades AWS SDK to 1.11.725. - `CVE-2019-10172 `__: Upgrades jackson-mapper-asl to 1.11.0 .. _release_notes-streams: |kstreams| """""""""" - `KIP-150 `__: Add Cogroup to the DSL: In previous versions, aggregating multiple streams into one could be complicated and error-prone. It generally requires you to group and aggregate all of the streams into tables, then make multiple outer join calls. Using the new ``co-group`` operator will clean up the syntax of your programs and reduce the number of state store invocations, which ultimately increases performance. For more information, see :ref:`kstreams-co-group`. - `KIP-523 `__: Add ``toTable()`` to the DSL: Sometimes you want to interpret a stream of events as a changelog and materialize a table over it. The new ``toTable`` can be applied to a stream and will materialize the latest value per key. Any null values will be interpreted as deletes for that key (tombstones). For more information, see :ref:`streams_upgrade-guide_5.5.x-api-changes`. - `KIP-535 `__: Allow state stores to serve stale reads during rebalance: Previously, interactive queries (IQ) against state stores would fail during the time period where there is a rebalance in progress. This hurts the uptime of applications that depend on the ability to query Kafka Streams’ tables of state. Applications can now query any replica of a state store and observe how far each replica is lagging behind the primary. For more information, see :ref:`kstreams-stale-data-standby`. .. _release_notes-ksql: |ksqldb| """""""" - KSQL has been renamed to |ksqldb| - “KSQL” has been renamed to “ksqlDB” in |cp| 5.5.x. This rename affects the release package and Docker image names as well as artifact names. Run scripts and configuration parameter names have not been changed. - |ksqldb| in |cp| 5.5 is not backward compatible with |cp| 5.4 - In-place upgrades on existing |cp| deployments will not be possible. To run existing workloads using |ksqldb| 5.5.x, a migration will be required. Migrations can be performed using :ref:`this migration guide `. - Topology viewer - |cp| 5.5.x includes a topology viewer for |ksqldb|, which visualizes your end-to-end workload as a directed acyclic graph of continuous queries. Individual nodes (continuous queries) in the topology may be inspected to determine node throughput, schema, and query definition. - Protobuf serialization format - A new serialization format has been added to support messages serialized using protocol buffers. The protocol buffers serialization format may be used by specifying ``PROTOBUF`` as the ``value_format``. - Highly available pull queries - Pull queries can now be configured such that they continue to work during cluster rebalances and node failures. If a target node is down, a standby node will be chosen to obtain the pull query result from. |ksqldb| can be configured to select the standby with the least amount of lag in these cases. - Primitive keys - Primitive row keys (of supported types) may be worked with directly instead of parsing them out of strings. You can now serialize ROWKEYs using the following types: ``INT``, ``BIGINT``, ``DOUBLE``, and ``STRING``. - JOIN expression support - Arbitrary expressions may now be used in JOIN clauses: :: SELECT * FROM s1 JOIN s2 ON s1.str = SUBSTRING(s2.str, 3) EMIT CHANGES; - PARTITION BY expression support - Arbitrary expressions may now be used in PARTITION BY clauses: :: SELECT x, y, z FROM s1 PARTITION BY MY_FUNCTION(x, 2) EMIT CHANGES; - New builtins - The following new builtin functions have been added: - ``COUNT_DISTINCT(v)`` - Aggregate function, computes the number of distinct values of v. - ``CUBE_EXPLODE(a)`` - Tabular function, outputs a row for each unique combination of elements in array a. - Native constructors for Arrays - Arrays can now be constructed in SELECT expression list: :: SELECT ARRAY[1, 2] FROM s1 EMIT CHANGES; - Native constructors for Maps - Maps can now be constructed in SELECT expression list: :: SELECT MAP(k1:=v1, k2:=v1*2) FROM s1 EMIT CHANGES; - Native constructors for Structs - Structs can now be constructed in SELECT expression list: :: SELECT {f1 v1, f2 v2} FROM s1 EMIT CHANGES; Deprecation Warnings """""""""""""""""""" - Beginning with the next major release of |cp|, |kconnect| connectors will no longer be packaged natively with |cp|. All formerly packaged connectors must be downloaded directly from `Confluent Hub `__. - ../includes/restproxyv1.rst .. _release_notes-download: How to Download ^^^^^^^^^^^^^^^ |cp| is available for download at https://www.confluent.io/download/. See the :ref:`installation` section for detailed information. .. include:: ../installation/includes/cs-license.rst For more information about migrating to |cs|, see :ref:`migrate-confluent-server`. To upgrade |cp| to a newer version, check the :ref:`upgrade` documentation. .. _release_notes-sup-version: Supported Versions and Interoperability ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For the supported versions and interoperability of |cp| and its components, see :ref:`interoperability-versions`. .. _release_notes-question: Questions? ^^^^^^^^^^ If you have questions regarding this release, feel free to reach out via the `community mailing list `_ or `community Slack `_. Confluent customers are encouraged to contact our support directly. .. toctree:: :hidden: Component Changelogs