Release Notes

Confluent Platform 5.3.1 Release Notes

5.3.1 is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.3.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features

Confluent Enterprise Kafka

  • PR-7287 - MINOR: Add unit test for KAFKA-8676 to guard against unrequired task restarts (#7287)
  • PR-696 - Hotfix KSTORAGE-228 by removing tiered storage replication code
  • PR-7256 - MINOR: Only send delete request if there are offsets in map (#7256)
  • PR-7207 - KAFKA-8412: Fix nullpointer exception thrown on flushing before closing producers (#7207)
  • PR-6993 - KAFKA-8586: Fail source tasks when producers fail to send records (#6993)
  • PR-7235 - KAFKA-8824: bypass value serde on null (#7235)
  • PR-7176 - KAFKA-8325; Remove batch from in-flight requests on MESSAGE_TOO_LARGE errors (#7176)
  • PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
  • PR-7219 - MINOR: Fixing log format typo (#7219)
  • PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
  • PR-7212 - KAFKA-8802: ConcurrentSkipListMap shows performance regression in cache and in-memory store (#7212)
  • PR-7163 - MINOR: Fix bugs in handling zero-length ImplicitLinkedHashCollections (#7163)
  • PR-7192 - KAFKA-8788: Optimize client metadata handling with a large number of partitions (#7192)
  • PR-7210 - MINOR: Correct typo in test name TimetampedSegmentsTest (#7210)
  • PR-7201 - KAFKA-8791: RocksDBTimestampedStore should open in regular mode by default (#7201)
  • PR-7177 - KAFKA-8736: Track size in InMemoryKeyValueStore (#7177)
  • PR-6283 - KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread (#6283)
  • PR-7197 - KAFKA-8774: Regex can be found anywhere in config value (#7197)
  • PR-632 - CONFLUENT: Ignore the return value from parallel stages in Jenkinsfile
  • PR-216 - CPKAFKA-2585: Configurable soak clients task type counts
  • PR-6959 - KAFKA-8550: Fix plugin loading of aliased converters in Connect (#6959)
  • PR-618 - CONFLUENT: Build cp-downstream-builds as part of Jenkins PR Job
  • PR-7125 - MINOR: Upgrade jackson-databind to 2.9.9.3 (#7125)
  • PR-7164 - KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (#7164)
  • PR-7143 - MINOR: Avoid dividing by zero (#7143)
  • PR-7143 - MINOR: Avoid dividing by zero (#7143)
  • PR-7143 - MINOR: Avoid dividing by zero (#7143)
  • PR-7132 - KAFKA-8731: InMemorySessionStore throws NullPointerException on startup (#7132)
  • PR-602 - CONFLUENT: Set default Kerberos service name for LDAP
  • PR-7116 - KAFKA-8715; Fix buggy reliance on state timestamp in static member.id generation (#7116)
  • PR-7101 - KAFKA-8678; Fix leave group protocol bug in throttling and error response (#7101)
  • PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
  • PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
  • PR-531 - CONFLUENT: Jenkins gradle exit 1 on executor failure, 0 on test failure
  • PR-490 - CONFLUENT: Jenkinsfile should not swallow test return status codes
  • PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
  • PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
  • PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
  • PR-6844 - MINOR: Remove stale streams producer retry default docs. (#6844)
  • PR-588 - OAUTHBEARER sasl mechanism constant
  • PR-7085 - KAFKA-8635; Skip client poll in Sender loop when no request is sent (#7085)
  • PR-7054 - KAFKA-8615: Change to track partition time breaks TimestampExtractor (#7054)
  • PR-7094 - KAFKA-8670; Fix exception for kafka-topics.sh --describe without --topic mentioned (#7094)
  • PR-7092 - KAFKA-8602: Separate PR for 2.3 branch (#7092)
  • PR-6928 - KAFKA-8530; Check for topic authorization errors in OffsetFetch response (#6928)
  • PR-7086 - KAFKA-8662; Fix producer metadata error handling and consumer manual assignment (#7086)
  • PR-7050 - KAFKA-8637: WriteBatch objects leak off-heap memory (#7050)
  • PR-7021 - KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (#7021)
  • PR-7076 - HOT FIX: close RocksDB objects in correct order (#7076)
  • PR-7070 - KAFKA-7157: Fix handling of nulls in TimestampConverter (#7070)
  • PR-5705 - KAFKA-6605: Fix NPE in Flatten when optional Struct is null (#5705)
  • PR-6678 - Fixes #8198 KStreams testing docs use non-existent method pipe (#6678)
  • PR-7030 - KAFKA-5998: fix checkpointableOffsets handling (#7030)
  • PR-7072 - KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 (#7072)
  • PR-6991 - KAFKA-8591; WorkerConfigTransformer NPE on connector configuration reloading (#6991)
  • PR-7013 - MINOR: add upgrade text (#7013)

Confluent Security Plugins

  • PR-192 - Allow access to KSQL /info endpoint to authenticated users
  • PR-187 - Remove jackson databind version override

Control Center

  • PR-1888 - [MMA-4870] adding token-scoped SR clients
  • PR-1898 - [MMA-2901] fix file limit issue on 5.2.x
  • PR-1877 - inject RBAC configs for rock too
  • PR-746 - bump version to v3.35.21-cp-5.3.x.0
  • PR-745 - chore: update cp version to 3.35.20-cp-5.3.x.0
  • PR-743 - upgrade to 3.10.20-cp-5.2.x.0
  • PR-727 - upgrade to 3.35.19-cp-5.3.x.0
  • PR-726 - upgrade to 3.10.19-cp-5.2.x.0

Metadata Service

  • PR-113 - CLI-210 - Fix lookupResourcesForPrincipal to accept User or Group principal
  • PR-112 - CLI-210 - Fix swagger return type of LookupResourcesForPrincipal
  • PR-109 - SEC-233 - MDS Test Infra : Find / build a better testing LDAP
  • PR-108 - SEC-233 MDS Test Infra : Find / build a better testing LDAP
  • PR-106 - SEC-319 Add endpoint to retrieve all resources for a user, including …
  • PR-101 - SEC-265 Add endpoints to lookup which principals have particular roles
  • PR-103 - Minor - fix version of new modules for 5.3.x
  • PR-102 - SEC-233 - MDS Test Infra : Find / build a better testing LDAP (#98)
  • PR-99 - SEC-314 - Remove unnecessary overrides that might mask upstream updates (#97)

Kafka Connect

  • PR-337 - CC-4886: Fix the SecurityIT integration test
  • PR-334 - CC-4886: Temporarily ignore SecurityIT
  • PR-335 - MINOR: Upgrade Mockito to work on Java 11
  • PR-327 - Enable Elasticsearch 7 support with adding minimum Integration test (based on TestContainer)
  • PR-324 - CC-5605: Log version conflicts as warnings.
  • PR-451 - CC-6089: Improve fault tolerance of HDFS sink connector
  • PR-450 - CC-5969: Pass extra connector configs to the partitioner
  • PR-442 - Adding debug logs for WAL recovery
  • PR-440 - Remove jackson databind version override
  • PR-677 - Fix ERROR Graceful stop of task failed
  • PR-686 - MINOR: Upgrade postgress jdbc driver to most recent bugfix release (9.4.1212)
  • PR-655 - CC-4946: changing log msgs on filtering and avoiding frequently retrying tasks when no tables
  • PR-672 - Change TableQuerier log to use real class name for file
  • PR-79 - MINOR: Revert to using the older version of jackson-databind to fix the build
  • PR-77 - CC-5671: fix test failures caused by outdated dependencies
  • PR-76 - CC-5664: Add ' Source' to end of Confluent Hub titles for connectors
  • PR-237 - REP-30: Try to stop flaky build
  • PR-236 - REP-20: Added a check to ensure TopicMontiorThreadWithZk is shutdown
  • PR-235 - REP - 30: Change to stop flaky builds
  • PR-232 - REP-30: Change to stop flaky builds
  • PR-228 - REP-30: MINOR: Fix Jenkins build failures due to the maven-surefire plugin
  • PR-226 - REP-30: MINOR: Fix Jenkins build failures due to the maven-surefire plugin
  • PR-225 - MINOR: Updated Jenkinsfile to reference new Slack channel
  • PR-219 - Add debug logs for starting consumers
  • PR-222 - MINOR: update slack channel for Jenkins notifications, to #replicator…
  • PR-220 - CC-5818: remove a test that isn't valid for 5.1.x onwards.
  • PR-218 - MINOR: Subsume provenance header addition into existing utility metho…
  • PR-217 - CC-2796: Pass a delta for better than ms precision
  • PR-214 - CC-5541: Set ByteArrayConverter as header converter
  • PR-209 - CC-5012: If topic creation fails, attempt to resize topic
  • PR-263 - Remove jackson databind version override
  • PR-52 - Remove jackson databind version override
  • PR-112 - MINOR: Improve the locale and timezone config descriptions
  • PR-110 - CC-5963: Improved ability to handle schema evolution with different names

Confluent Operator

  • To obtain the latest fixes for Confluent Operator supported for Confluent Platform 5.3.1, download the Operator Helm Charts (confluent-operator-20190912-v0.65.1) here.
  • To upgrade Confluent Operator and Confluent Platform with these Helm Charts, download the Helm Charts and follow the instructions here.
  • This version fixes the issue seen with KSQL where crashloopbacks are observed when the interceptor is enabled.
  • This version supports non-alphanumeric characters for SASL_PLAIN Authentication passwords.
  • This version fixes the Helm Chart example default configurations in the providers folder where it was missing Kubernetes resource request configurations.
  • This version updates the Operator image to 0.176.1 to fix a Confluent Operator license key issue. These changes will interfere with CP upgrade for existing deployment (deployed through an older version of Operator). To fix the issue, complete the following steps:
    • This licensing issue only affects Confluent Operator licensing, and not other CP components like Control Center
    • This issue is only applicable for users upgrading to this Confluent Platform 5.3.1 Operator patch that are still using the free 30 day trial for Confluent Platform.
    • Users that are still on the free 30 day trial will need to run the following steps before upgrading using Helm:
      • Zero out the licensing key section within the “operator-licensing” secret to generating a new valid trial on a rolling upgrade.
      • Specifically, run the command kubectl edit secrets -n <namespace> operator-licensing. Then, zero out the licenseKey key in the data map (e.g. licenseKey: “” will suffice). Make sure you don’t delete the licenseKey key itself, as the operator would not generate a license key, key-value.

Community Features

Confluent Platform Ansible

  • PR-124 - Roles and groups now match service names, only 1 playbook, moved sample inventories into new directory
  • PR-123 - Corrected link to Contribution documentation.
  • PR-121 - adding files
  • PR-120 - Revert "Merge pull request #118 from confluentinc/5.3.0-pre"
  • PR-118 - 5.30 Release.
  • PR-119 - Merge pull request #118 from confluentinc/5.3.0-pre
  • PR-118 - 5.30 Release.
  • PR-117 - Revert "Release for CP 5.3.0"
  • PR-114 - Release for CP 5.3.0
  • PR-116 - kerberos realm hardcoded to confluent.example.com

Kafka

  • PR-7287 - MINOR: Add unit test for KAFKA-8676 to guard against unrequired task restarts (#7287)
  • PR-7097 - Changed for updatedTasks, avoids stopping and starting of unnecessary tasks (#7097)
  • PR-7256 - MINOR: Only send delete request if there are offsets in map (#7256)
  • PR-7207 - KAFKA-8412: Fix nullpointer exception thrown on flushing before closing producers (#7207)
  • PR-6993 - KAFKA-8586: Fail source tasks when producers fail to send records (#6993)
  • PR-7235 - KAFKA-8824: bypass value serde on null (#7235)
  • PR-7176 - KAFKA-8325; Remove batch from in-flight requests on MESSAGE_TOO_LARGE errors (#7176)
  • PR-7219 - MINOR: Fixing log format typo (#7219)
  • PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
  • PR-7212 - KAFKA-8802: ConcurrentSkipListMap shows performance regression in cache and in-memory store (#7212)
  • PR-7163 - MINOR: Fix bugs in handling zero-length ImplicitLinkedHashCollections (#7163)
  • PR-7192 - KAFKA-8788: Optimize client metadata handling with a large number of partitions (#7192)
  • PR-7210 - MINOR: Correct typo in test name TimetampedSegmentsTest (#7210)
  • PR-7201 - KAFKA-8791: RocksDBTimestampedStore should open in regular mode by default (#7201)
  • PR-7177 - KAFKA-8736: Track size in InMemoryKeyValueStore (#7177)
  • PR-6283 - KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread (#6283)
  • PR-7197 - KAFKA-8774: Regex can be found anywhere in config value (#7197)
  • PR-216 - CONFLUENT: Build cp-downstream-builds as part of Jenkins PR Job
  • PR-6959 - KAFKA-8550: Fix plugin loading of aliased converters in Connect (#6959)
  • PR-7125 - MINOR: Upgrade jackson-databind to 2.9.9.3 (#7125)
  • PR-7164 - KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (#7164)
  • PR-7143 - MINOR: Avoid dividing by zero (#7143)
  • PR-7143 - MINOR: Avoid dividing by zero (#7143)
  • PR-7132 - KAFKA-8731: InMemorySessionStore throws NullPointerException on startup (#7132)
  • PR-7116 - KAFKA-8715; Fix buggy reliance on state timestamp in static member.id generation (#7116)
  • PR-7101 - KAFKA-8678; Fix leave group protocol bug in throttling and error response (#7101)
  • PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
  • PR-531 - CONFLUENT: Jenkins gradle exit 1 on executor failure, 0 on test failure (#531)
  • PR-490 - CONFLUENT: Jenkinsfile should not swallow test return status codes (#490)
  • PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
  • PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
  • PR-6844 - MINOR: Remove stale streams producer retry default docs. (#6844)
  • PR-7085 - KAFKA-8635; Skip client poll in Sender loop when no request is sent (#7085)
  • PR-7054 - KAFKA-8615: Change to track partition time breaks TimestampExtractor (#7054)
  • PR-7094 - KAFKA-8670; Fix exception for kafka-topics.sh --describe without --topic mentioned (#7094)
  • PR-7092 - KAFKA-8602: Separate PR for 2.3 branch (#7092)
  • PR-6928 - KAFKA-8530; Check for topic authorization errors in OffsetFetch response (#6928)
  • PR-7086 - KAFKA-8662; Fix producer metadata error handling and consumer manual assignment (#7086)
  • PR-7050 - KAFKA-8637: WriteBatch objects leak off-heap memory (#7050)
  • PR-7021 - KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (#7021)
  • PR-7076 - HOT FIX: close RocksDB objects in correct order (#7076)
  • PR-7070 - KAFKA-7157: Fix handling of nulls in TimestampConverter (#7070)
  • PR-5705 - KAFKA-6605: Fix NPE in Flatten when optional Struct is null (#5705)
  • PR-6678 - Fixes #8198 KStreams testing docs use non-existent method pipe (#6678)
  • PR-7030 - KAFKA-5998: fix checkpointableOffsets handling (#7030)
  • PR-7072 - KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 (#7072)
  • PR-6991 - KAFKA-8591; WorkerConfigTransformer NPE on connector configuration reloading (#6991)
  • PR-7013 - MINOR: add upgrade text (#7013)

KSQL

  • PR-3225 - Fix quoted reserved identifier 5.3
  • PR-3197 - fix: validate CLI using REST / instead of REST /info
  • PR-3176 - fix: default timestamp extractor override is not working
  • PR-3154 - refactor: allow for more specific error messages to be sent when the server is unavailable
  • PR-3111 - fix: add ksql-functional-tests to the ksql package
  • PR-3147 - fix: filter null entries before creating KafkaConfigStore
  • PR-3079 - feat: add config for enabling topic access validator
  • PR-3069 - feat(rest server): add warnings when listing fails with kafka error
  • PR-3104 - fix: COLLECT_LIST can now be applied to tables
  • PR-2997 - feat: add extension for custom metrics (5.3.x)
  • PR-2996 - feat: add config for custom metrics tags (5.3.x)

Newwave

  • PR-42 - Remove jackson databind version override

Rest Utils

  • PR-148 - Remove jackson databind version override

Schema Registry

  • PR-855 - #854 - mvn plugin - allow schemas composition
  • PR-1182 - MINOR: Update slack channel to #schema-registry-eng
  • PR-1174 - MINOR: fix NPE by ignoring tombstones
  • PR-1166 - MINOR: Add missing reset method to mock client

Confluent Platform 5.3.0 Release Notes

5.3.0 is a major release of Confluent Platform that provides you with Apache Kafka® 2.3.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features

Confluent CLI

  • The Confluent CLI was completely rewritten to support a wider set of features and platforms.
  • The Confluent CLI was removed from Confluent Platform packaging. The Confluent CLI is now installed separately from the Confluent Platform package. For more information, see Installing and Configuring the CLI.
  • The Confluent CLI development commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local start. For more information, see confluent local.
  • Added support for managing role-based access control (RBAC) permissions and role bindings.
  • Added support for masking secrets and configuration files.
  • Added "self-update" capability via the confluent update command so that improvements and fixes can be released out-of-band from the Confluent Platform release schedule.
  • The Confluent CLI is a commercially licensed feature, but it can be downloaded and used against both Confluent Community and Confluent Platform.

Confluent Control Center

Control Center introduces several changes and new features, including:

  • Redesigned UI: The Control Center UI has been redesigned from end-to-end to provide an easier-to-use, multi-cluster-friendly, and global view of Kafka that supports working with Confluent at scale. The new homepage provides a global view of all of your Kafka clusters and connected services, including Connect and KSQL. You can see a health roll-up for each cluster and use a switcher to view only the unhealthy clusters so you can focus on what needs attention.

  • New cluster overview: Clicking one of the cards on the new homepage takes you into the cluster and connected services, showing broker metrics or topics depending on your permissions if RBAC is enabled. You can drill down further into broker metrics by clicking on a card, which shows time-series charts with interactive legends and shared cursors for easier metrics correlation.

  • New topics pages: The redesigned Control Center includes a new topics index that shows you all the topics in a cluster with key health information for each in a searchable table with sortable columns. You can click on a topic name to view an overview for it, including relevant throughput and health information, and access the message browser, schemas, and topic configuration. Topic metrics, including a topic-centric view of consumer lag, can be accessed by clicking on the metrics cards. Under consumer lag, you can select which consumer group you are interested in using a drop-down so you can focus on the specific application you care about.

  • Message browser enhancements: Message browser performance has been improved to eliminate throttling and message loss. You can also now access the full topic history by seeking, either using offset or timestamp, on a specific partition. You can also now download messages in JSON format, including using the typical Ctrl/Shift-select shortcuts to select and download more than one message.

  • Connect redesign and enhancements: The new UI also features an index page for all of the Connect clusters associated with a Kafka cluster. You can search for a Connect cluster by name/ID and see how many connectors are running, degraded, or failed on each cluster. Degraded connectors have at least one failed task; whereas failed connectors have failures on all of their tasks. Clicking on one of the clusters in the Connect index takes you to the new Connect cluster overview page. Here, you can see all the connectors running on the cluster, including their status, type, category, and number of tasks. Clicking on one of the connectors in the Connect cluster overview takes you to a page showing the connector itself where you can see a breakdown of the connector's tasks and their status. This new Connect experience makes it much easier to view all of the Connect clusters and drill down to see how each task on a connector is performing.

  • KSQL redesign and enhancements: The new UI also features an index page for all of the KSQL clusters associated with a Kafka cluster. You can search for a KSQL cluster by name/ID, see how many queries are running, and see how many streams or tables are on the cluster. Clicking on one of the clusters in the KSQL index takes you to a page where you can access the KSQL editor, streams, tables, and persistent queries. The editor itself features similar improvements as the message browser, including performance improvements to eliminate throttling and message loss and JSON downloading of results. In addition, the KSQL editor area is resizeable to accommodate larger queries and statements. It also includes a data discovery side-panel, making it much easier to find streams and tables to run KSQL against.

  • Consumers index: Control Center also includes a new index to show you all the consumer groups associated with a selected Kafka cluster, including the number of consumers per group and the number of topics being consumed. Clicking on a consumer group takes you to a page where you can view Consumer Lag across all relevant topics and a redesigned Streams Monitoring page, making it much easier to understand consumption metrics in context.

  • Alerting enhancements: Control Center features several highly requested alerting enhancements. To date, Control Center only supported email-based alert actions. In Confluent Platform 5.3.0, you can now use webhooks for Slack and PagerDuty, making it much easier to integrate Control Center with a range of monitoring approaches. There is also a new Consumer Lead alert, which you can customize to fire when a consumer is within a certain number of offsets of the partition tail. You can also now pause and resume actions globally, silencing alerts when needed. Finally, the alert messages themselves have become more informative and now include better contextual information about the alert, including the cluster, metric, condition, and trigger.

  • Feature flagging for System Health and Data Streams: While Confluent Platform 5.3.0 introduces a significantly redesigned Control Center, a feature flag allows you to view the older System Health and Data Streams pages.

    Note

    These pages cannot be viewed if RBAC is enabled.

  • RBAC enforcement: As part of the introduction of RBAC to Confluent Platform, Control Center respects the rolebindings assigned using the CLI, so you have platform-wide consistency for your security strategy. For more information, see Role-based access control (RBAC) in Confluent Control Center.

For more information, see the documentation.

Control Center Known Issues
  • Multi-cluster alert triggers: You can currently select more than one cluster when defining a Cluster or Broker alert trigger in Control Center but doing so results in a trigger firing against all of the associated clusters rather than the one that actually meets the alert definition. To avoid this issue in the interim, Cluster and Broker alerts should be defined against individual clusters.
  • Direct access to each external cluster is required under RBAC: When RBAC is enabled, Control Center requires management to be enabled on each cluster. This means that metrics reporters and interceptors must have a direct connection for Control Center to operate properly in an RBAC environment at this time. See Control Center connections to external clusters for more details.
  • Connect and KSQL require at least minimal user access to the underlying Kafka clusters: See Connect and KSQL clusters user access for details.
  • The service principal for Control Center requires SystemAdmin access: The Control Center user must be set up as a privileged user SystemAdmin on each cluster. Due to the underlying architecture for consumer lag, elevated privileges are required to guarantee users access to all consumer groups and continued support for consumer group alerts for consumer lag. The consumer lag offsets are currently not obtained from metrics reporters.
  • Prefix search in the message browser cannot currently search column names that contain a period.
  • There is a mismatch between products in enforced default values for max message bytes: When connecting Control Center to Confluent Cloud, the confluent.metrics.topic.max.message.bytes must be set to 8388608 rather than the current Control Center default of 10485760. For more information, see Control Center cannot connect to Confluent Cloud and Connecting Control Center to Confluent Cloud.

Security

Role-based access control (RBAC)

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is being introduced to gain early feedback from developers. This feature can be used for evaluation and non-production testing purposes or to provide feedback to Confluent.

  • Provides the secure authorization of access to resources by users and groups.
  • Uses a set of predefined roles that give you depth and granularity in its configuration of managing access.
  • Provides centralized authorization for all the components including Control Center, Connect, Schema Registry, REST Proxy, and KSQL.
  • Provides connector level access control.

For more information, see the RBAC documentation:

AD/LDAP enhancements
  • Centralized configuration.
  • Centralized authentication for Control Center, Connect, Schema Registry, REST Proxy, and KSQL.

For more information, see Confluent LDAP Authorizer.

Secret Protection
  • To enable masking of sensitive information in the configuration file and logs.
  • Across the whole platform including Kafka, Connect, KSQL, Schema Registry, REST Proxy, and Control Center.
  • Utilize the new CLI to create/manage secrets.

For more information, see Secrets.

Community Features

Clients

librdkafka v1.1.0 is a security feature release with added support for:
  • OAUTHBEARER SASL authentication.
  • In-memory SSL certificates.
  • Windows Root Certificate store.
  • Pluggable broker SSL certificate verification callback.
  • Improved SASL GSSAPI/Kerberos ticket refresh.

Upgrade considerations:

  • Windows SSL users will no longer need to specify a CA certificate file/directory (ssl.ca.location), librdkafka will load the CA certs by default from the Windows Root Certificate Store.
  • SSL peer (broker) certificate verification is now enabled by default (disable with enable.ssl.certificate.verification=false)
  • %{broker.name} is no longer supported in sasl.kerberos.kinit.cmd since kinit refresh is no longer executed per broker, but per client instance.

See the full librdkafka v1.1.0 release notes for more information.

confluent-kafka-python, confluent-kafka-dotnet, confluent-kafka-go

Clients now use librdkafka v1.1.0 and include all the improvements from the latest version.

Connect

  • PR-6363 - KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) (#6363) When a rebalance happened in Kafka 2.2 or earlier, it stops all tasks in a Connect cluster and restarts them. This can be a hard stop for users who run multiple connectors in a Connect cluster. With KIP-415, a rebalance happens more gracefully. It stops only the tasks that need to move between workers (if any), leaving the rest running on their assigned worker
  • PR-5743 - KAFKA-3816: Add MDC logging to Connect runtime (#5743) With KIP-449, The SLF4J API includes "Mapped Diagnostic Contexts" (MDC) that allow injection of a series of parameters that can be included in every log message written using that thread, regardless of how the SLF4J Logger instance was obtained.
  • PR-6584 - KAFKA-8231: Expansion of ConnectClusterState interface (#6584)
  • PR-6789 - KAFKA-8407: Fix validation of class and list configs in connector client overrides (#6789)
  • PR-6658 - KAFKA-8309: Add Consolidated Connector Endpoint to Connect REST API (#6658)

Connectors

  • HDFS Connector:
    HDFS connector is removed from the Confluent Platform package. However, the connector will continue to be downloadable from Confluent Hub and supported by Confluent.
  • JDBC Connector:
    PR-641 - CC-349: Added delete support for sink. (#641) A record with null value is considered a tombstone event and result in deleting the corresponding row in the destination table.
  • Elasticsearch Connector:
    PR-239 - Support for document upsert. (#239) In some cases, records in Kafka topics consist only subset of fields ElasticSearch needs. With this feature, ElasticSearch Sink Connector can perform an upsert operation.

Kafka 2.3.0

  • KIP-339: A new IncrementalAlterConfigs API in AdminClient Changes configurations incrementally, only modifying the configuration values that are specified to prevent lost updates. AlterConfigs API has been marked for deprecation. For more information, see KIP-339
  • KIP-341: Update Sticky Assignor’s User Data Protocol Improves stability of consumer partition assignments by preventing assigning the same partition to multiple consumers in the same consumer group. For more information, see KIP-341
  • KIP-351: Add --under-min-isr option to describe topics command Adds the --under-min-isr option in the describe topics command, which allows users to see precisely which topic partitions are below min.insync.replicas need to be addressed. For more information, see KIP-351
  • KIP-354: Maximum Log Compaction Lag Compaction allows Kafka to remove messages that are are older than min.compaction.lag.ms, which ensures that a segment is rolled and remains uncompacted for a given period ( “lag”). Regulations such as GDPR require that data is deleted in a timely manner. max.compaction.lag.ms sets the maximum lag time for which a segment may remain uncompacted, after which the corresponding log partition becomes eligible for log compaction. For more information, see KIP-354
  • KIP-361: Add Consumer Configuration to Disable Auto Topic Creation Both auto.create.topics.enable on the broker and allow.auto.create.topics on the consumer need to be set for auto-topic creation to happen. For more information, see KIP-361
  • KIP-402: Improve fairness in SocketServer processors The first part of KIP-402 was introduced in Kafka 2.2, which changed how connections were prioritized. Now existing connections are given more priority over new connection requests. The KIP introduces max.connections, which limits the total number of connections that may be active on the broker at any time. This is in addition to the existing max.connections.per.ip configuration that will continue to limit the number of connections from each host IP address. For more information, see KIP-402
  • KIP-417: Allow JMXTool to connect to to a secure RMI port Adds --jmx-ssl-enable and --jmx-auth-prop to allow connecting to a secure Java VM. For more information, see KIP-417
  • KIP-421: Support resolving externalized secrets in AbstractConfig Enhances the AbstractConfig base class to automatically resolve variables of the form specified in KIP-297. For more information, see KIP-421
  • KIP-425: Add Log4J Kafka Appender Properties to Producing to Secure Brokers Extends the Log4J Kafka appender to support PLAIN mechanism and configuration of jaas via a property passed to the producer. For more information, see KIP-425
  • KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option) Introduces a new topic partition category in the metrics group between UnderReplicated and UnderMinIsr called AtMinIsr. For more information, see KIP-427
  • KIP-430 - Return Authorized Operations in Describe Responses The AdminClient now allows users to determine what operations they are authorized to perform on topics. For more information, see KIP-430
  • KIP-436: Add a metric indicating start time Useful for detecting restarts. For more information, see KIP-436
  • KIP-461: Improve Replica Fetcher Behavior at handling partition failure In previous versions, if a partition fails, the replica fetcher thread associated with that partition will terminate. Because the replica fetcher threads handle multiple partitions, this led to under-replicated partitions. Now, whenever a partition crashes, the concerned thread stops tracking the crashed parition and continues handling rest of the partitions. This all reduces the chance of under-replicated partitions and improves cluster stability. For more information, see KIP-461

The release includes 157 new features, improvements, and fixes. For a full list of changes in this release of Kafka, see the Apache Kafka 2.3.0 Release Notes.

Also, see the blog post covering What's New in Apache Kafka 2.3 or this video.

Kafka Streams

  • Stored Record Timestamps in RocksDB: To offer better timestamp processing semantics and introduce new time-related KTable features, more information about the timeliness of records must be persisted. Kafka Streams now stores the timestamps for each record that contributes to state stores inside RocksDB. These timestamps are now accessible through Interactive Queries, too. This functionality paves the way for potential future features, like KTable TTLs and support for out-of-order KTable data. See KIP-258 for more information.
  • In-memory window and session stores: Kafka Streams supports pluggable storage of its tables. Until this release, Kafka Streams offered only durable versions out of the box of its window and session store abstractions. Kafka Streams now ships with in-memory versions implemented to support high performance, transient operations. See KIP-428 and KIP-445 for more information.
  • New flatTransformValues method: Kafka Streams now supports a new method in its API, flatTransformValues. It is the equivalent of flatMapValues for the Processor API. Unlike the traditional transformValues method, this method is able to ensure strong typing by specifying in its signature a list of key-value pairs (i.e. Iterable) as output records for each input record. See KIP-313 for more information.
  • Default implementation to close() and configure() for Serializer, Deserializer and Serde: When serializers, deserializers, and serdes are authored, the close and configure methods are typically implemented with no operation behind them. Kafka Streams now leverages Java's new default interface inheritance feature, and provides a sensible, default implementation for these methods. See KIP-331 for more information.
  • Better defaults for increased stability: To better improve footprint and threading stability, a number of defaults have been changed to more sensible values. These include max.poll.interval.ms, segment.ms, and segment.index.bytes. See KIP-442 and KIP-443 for more information.
  • New close() method on RocksDBConfigSetter: A new close() method has been added to support cleanup of RocksDB configs. This helps to evade inadvertent memory leaks. See KIP-453 for more information.

KSQL

  • KEY no longer required for CREATE TABLE: Tables can now be created without manually specifying a KEY parameter. For example, CREATE TABLE T1 (ID INT, OTHER INT). KEY can still be specified as a hint, but is no longer required (#2745).
  • CREATE STREAM/TABLE now creates topics if they don't exist: Previously a topic underlying a stream or table had to exist already to create a stream or table over it. KSQL now creates the underlying topic automatically if it doesn't already exist (#2771).
  • Improved UDF interfaces: Two of the four components of KLIP-1 have made it into this release: UDFs can now be defined with variable-length arguments as well as custom Struct argument types (#2503).
  • Data can now be written to streams via INSERT INTO: Users can now write events directly to streams from within KSQL using standard SQL INSERT INTO syntax. While not designed for high-performance production usage, simplifying the process of getting data into KSQL should greatly improve the initial KSQL experience and make it much easier for new users to experiment with the system (KLIP-2, #2723).
  • KSQL test runner tool: The ksql-test-runner tool is a command line utility that enables testing KSQL statements without requiring any infrastructure, which means that Kafka and KSQL don't need to be running. It’s a lightweight way to design and iterate on your KSQL-based applications and ensure that the expected results are generated (#2802).
  • CLI sessions can now be recorded to text files: SPOOL is a new CLI command that allows users to record a KSQL CLI session’s inputs and outputs, writing the session content out to a file (#2789).
  • ELT and FIELD functions: Both the ELT and FIELD functions are based on the SQL standard and are available in many other SQL-based systems (#2627). ELT (N, x, y, ...) returns the Nth element of the given list of strings. This is the complement to FIELD. FIELD (string, str1, str2, ...) returns the index of string within the list of given strings. This is the complement to ELT.
  • More date/timestamp formats now supported: KSQL’s date/timestamp parser is now more robust and understands more date/timestamp formats (#2499).
  • Quotes in string literals now escaped: This protects KSQL from malicious Java code injection attacks (#2545).
  • Bug fixes:
    • Fixed GEO_DISTANCE function (#2700).
    • Fixed bug causing tabs in multi-line input strings to cause unnecessary errors (#2734).
    • Fixed bug that caused command topic replay during KSQL startup to recreate deleted topics unintentionally (#2329).
    • Fixed bug with performing joins on ROWKEY (#2735).
    • Fixed bug causing potential crash during KSQL REST server shutdown (#2507).

REST Proxy

  • FIX: REST Proxy now responds with a 401/403 for authentication/authorization errors.

REST Utils

The default value for ssl.endpoint.identification.algorithm was changed to https. This setting performs hostname verification to prevent man-in-the-middle attacks. You can set ssl.endpoint.identification.algorithm to an empty string to restore the previous behaviour.

Schema Registry

  • PR-1124 - CC-4775: logical type preservation for hdfs connector when output format is parquet (#1124)

Other Improvements & Changes

  • KAFKA-8336: Enable dynamic update of client-side SSL factory in brokers When mutual authentication is enabled for inter-broker-communication (ssl.client.auth=required), broker restarts are no longer needed when updating client-side keystores.
  • FIX: Added better open file limit to systemd files

Deprecation Warnings

  • OS support for RHEL 6, Ubuntu 12.04, Ubuntu 14.04, and Debian 7
  • Confluent Rest Proxy v1 API

How to Download

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

To upgrade Confluent Platform to a newer version, check the Upgrade documentation.

Supported Versions and Interoperability

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?

If you have questions regarding this release, feel free to reach out via the community mailing list or community Slack. Confluent customers are encouraged to contact our support directly.