Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Release Notes¶

Confluent Platform 5.3.2 Release Notes¶

5.3.2 is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.3.1, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features¶

Confluent Server¶

PR-7805 - KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795) (#7805)
PR-7791 - KAFKA-7489: Fix ConnectDistributedTest system test to use KafkaVersion (backport) (#7791)
PR-7783 - KAFKA-9184: (port on 2.3): Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771) (#7783)
PR-7763 - KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
PR-7723 - KAFKA-9190: Close connections with expired authentication sessions (#7723)
PR-7773 - KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
PR-6847 - KAFKA-8448: Cancel PeriodicProducerExpirationCheck when closing a Log instance (#6847)
PR-7772 - KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
PR-7630 - KAFKA-9073: check assignment in requestFailed to avoid NPE (#7630)
PR-7769 - KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
PR-7760 - KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
PR-1026 - KSTORAGE-329: Remove log metrics on deletion
PR-6031 - KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
PR-1002 - CPKAFKA-4090: License manager should never configure metrics reporter
PR-7532 - KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
PR-7704 - KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
PR-7662 - KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
PR-7511 - KAKFA-8950: Fix KafkaConsumer Fetcher breaking on concurrent disconnect (#7511)
PR-7315 - KAFKA-8340, KAFKA-8819: Use PluginClassLoader while statically initializing plugins (#7315)
PR-7392 - KAFKA-8945/KAFKA-8947: Fix bugs in Connect REST extension API (#7392)
PR-7491 - KAFKA-9014: Fix AssertionError when SourceTask.poll returns an empty list (#7491)
PR-7457 - KAFKA-8262, KAFKA-8263: Fix flaky test MetricsIntegrationTest (#6922) (#7457)
PR-7425 - KAFKA-8649: send latest commonly supported version in assignment (#7425)
PR-7442 - KAFKA-8974: Trim whitespaces in topic names in sink connector configs (#7442)
PR-7371 - KAFKA-6290: Support casting from logical types in cast transform (#7371)
PR-6914 - KAFKA-8523: Enabling InsertField transform to be used with tombstone events (#6914)
PR-7377 - KAFKA-8896: Check group state before completing delayed heartbeat (#7377)
PR-226 - RBAC: - Add Roll definitions for Connect and Schema Registry
PR-7346 - KAFKA-8913: Document topic based configs & ISR settings for Streams apps (#7346)
PR-7298 - KAFKA-8875: CreateTopic API should check topic existence before replication factor (#7298)
PR-7223 - KAFKA-8816: Make offsets immutable to users of RecordCollector.offsets (#7223)
PR-7264 - KAFKA-8724: Improve range checking when computing cleanable partitions (#7264)
PR-7281 - KAFKA-8861: Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic (#7281)

Kafka Connect¶

PR-88 - CC-6505: Change instances of RetriableException to ConnectException in base source task
PR-83 - CC-6267: Fix NPE when using permissive schemas
PR-81 - CC-6267: Add permissive schema configuration
PR-80 - CC-6200: Fix NPE due to null property values
PR-276 - REP-30: BACKPORT Refactor the “admin-client-with-zookeeper” test to NOT have a Kafka binary dependency.
PR-266 - REP-47: added arity to boolean cmd line options as they all default true
PR-249 - REP-403: Fixed timestamp committer thread leak
PR-248 - REP-386: changed describeTopics behaviour in topic monitor thread
PR-244 - REP-26:Fixed issue where group.id was not propogated to license manager
PR-242 - REP-30: Flaky Jenkins failure with maven-surefire-plugin
PR-240 - REP-31: Added code to specify default destination security settings
PR-238 - REP-186: Updated replicator to use delivery.timeout.ms instead of retries

Metadata Service¶

PR-129 - SEC-436: migrate mds jenkins bot to mds-alerts

Control Center¶

PR-1996 - MMA-6043: fix MDS client always points at the same url
PR-1966 - MMA-5608: Allow proxy servlet to proxy to endpoints with paths

Community Features¶

Kafka¶

PR-7805 - KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795) (#7805)
PR-7791 - KAFKA-7489: Fix ConnectDistributedTest system test to use KafkaVersion (backport) (#7791)
PR-7783 - KAFKA-9184: (port on 2.3): Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771) (#7783)
PR-7763 - KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
PR-7723 - KAFKA-9190: Close connections with expired authentication sessions (#7723)
PR-7773 - KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
PR-6847 - KAFKA-8448: Cancel PeriodicProducerExpirationCheck when closing a Log instance (#6847)
PR-7772 - KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
PR-7630 - KAFKA-9073: check assignment in requestFailed to avoid NPE (#7630)
PR-7769 - KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
PR-7760 - KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
PR-6031 - KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
PR-7532 - KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
PR-7704 - KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
PR-7662 - KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
PR-7511 - KAKFA-8950: Fix KafkaConsumer Fetcher breaking on concurrent disconnect (#7511)
PR-7315 - KAFKA-8340, KAFKA-8819: Use PluginClassLoader while statically initializing plugins (#7315)
PR-7392 - KAFKA-8945/KAFKA-8947: Fix bugs in Connect REST extension API (#7392)
PR-7491 - KAFKA-9014: Fix AssertionError when SourceTask.poll returns an empty list (#7491)
PR-7457 - KAFKA-8262, KAFKA-8263: Fix flaky test MetricsIntegrationTest (#6922) (#7457)
PR-7425 - KAFKA-8649: send latest commonly supported version in assignment (#7425)
PR-7442 - KAFKA-8974: Trim whitespaces in topic names in sink connector configs (#7442)
PR-7371 - KAFKA-6290: Support casting from logical types in cast transform (#7371)
PR-6914 - KAFKA-8523: Enabling InsertField transform to be used with tombstone events (#6914)
PR-7377 - KAFKA-8896: Check group state before completing delayed heartbeat (#7377)
PR-7346 - KAFKA-8913: Document topic based configs & ISR settings for Streams apps (#7346)
PR-7298 - KAFKA-8875: CreateTopic API should check topic existence before replication factor (#7298)
PR-7223 - KAFKA-8816: Make offsets immutable to users of RecordCollector.offsets (#7223)
PR-7264 - KAFKA-8724: Improve range checking when computing cleanable partitions (#7264)
PR-7281 - KAFKA-8861: Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic (#7281)

Elasticsearch Connector¶

PR-364 - CC-7367: Add a lot more logging
PR-135 - CC-6599: java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Long while setting default date value
PR-340 - CC-6187: Cache already existing index in JestElasticsearchClient

HDFS Connector¶

PR-460 - CC-6423: Disable FileSystem Object Caching
PR-456 - CC-6196: fix NPE for store.url config

JDBC Connector¶

PR-761 - CC-7414: Fix SqlServer column converter infinite loop & add column converter tests
PR-759 - CC-7382: use true/false as default values for boolean type
PR-757 - CC-7332: Changed one log message from INFO to DEBUG to reduce verbocity
PR-750 - CC-7265: Support SQL Server DateTimeOffset
PR-752 - CC-7265: Add support for SQL Server DateTimeOffset type
PR-727 - CC-6773: Abort source task if all columns were not converted correctly.
PR-724 - CC-6747: Replace static mocking for the CachedConnectionProvider

S3 Connector¶

PR-284 - CC-6801: Retry effectively on retriable failures (backport to java 7 branches)
PR-281 - CC-6801: Retry effectively on retriable failures while closing files

Common Modules for Connectors¶

PR-114 - CC-147: logical types support for HiveSchemaConverter

Kafka Streams Examples¶

PR-296 - CustomStreamTableJoinIntegrationTest: Correctly read and validate test output
PR-275 - Update Probabilistic Counting example to use Scala test utils

KSQL¶

PR-3167 - feat: add KsqlRocksDBConfigSetter to bound memory and set num threads

Confluent Platform 5.3.1 Release Notes¶

5.3.1 is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.3.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features¶

Confluent Server¶

PR-7287 - MINOR: Add unit test for KAFKA-8676 to guard against unrequired task restarts (#7287)
PR-696 - Hotfix KSTORAGE-228 by removing tiered storage replication code
PR-7256 - MINOR: Only send delete request if there are offsets in map (#7256)
PR-7207 - KAFKA-8412: Fix nullpointer exception thrown on flushing before closing producers (#7207)
PR-6993 - KAFKA-8586: Fail source tasks when producers fail to send records (#6993)
PR-7235 - KAFKA-8824: bypass value serde on null (#7235)
PR-7176 - KAFKA-8325; Remove batch from in-flight requests on MESSAGE_TOO_LARGE errors (#7176)
PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
PR-7219 - MINOR: Fixing log format typo (#7219)
PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
PR-7212 - KAFKA-8802: ConcurrentSkipListMap shows performance regression in cache and in-memory store (#7212)
PR-7163 - MINOR: Fix bugs in handling zero-length ImplicitLinkedHashCollections (#7163)
PR-7192 - KAFKA-8788: Optimize client metadata handling with a large number of partitions (#7192)
PR-7210 - MINOR: Correct typo in test name TimetampedSegmentsTest (#7210)
PR-7201 - KAFKA-8791: RocksDBTimestampedStore should open in regular mode by default (#7201)
PR-7177 - KAFKA-8736: Track size in InMemoryKeyValueStore (#7177)
PR-6283 - KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread (#6283)
PR-7197 - KAFKA-8774: Regex can be found anywhere in config value (#7197)
PR-632 - CONFLUENT: Ignore the return value from parallel stages in Jenkinsfile
PR-216 - CPKAFKA-2585: Configurable soak clients task type counts
PR-6959 - KAFKA-8550: Fix plugin loading of aliased converters in Connect (#6959)
PR-618 - CONFLUENT: Build cp-downstream-builds as part of Jenkins PR Job
PR-7125 - MINOR: Upgrade jackson-databind to 2.9.9.3 (#7125)
PR-7164 - KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (#7164)
PR-7143 - MINOR: Avoid dividing by zero (#7143)
PR-7143 - MINOR: Avoid dividing by zero (#7143)
PR-7143 - MINOR: Avoid dividing by zero (#7143)
PR-7132 - KAFKA-8731: InMemorySessionStore throws NullPointerException on startup (#7132)
PR-602 - CONFLUENT: Set default Kerberos service name for LDAP
PR-7116 - KAFKA-8715; Fix buggy reliance on state timestamp in static member.id generation (#7116)
PR-7101 - KAFKA-8678; Fix leave group protocol bug in throttling and error response (#7101)
PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
PR-531 - CONFLUENT: Jenkins gradle exit 1 on executor failure, 0 on test failure
PR-490 - CONFLUENT: Jenkinsfile should not swallow test return status codes
PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
PR-6844 - MINOR: Remove stale streams producer retry default docs. (#6844)
PR-588 - OAUTHBEARER sasl mechanism constant
PR-7085 - KAFKA-8635; Skip client poll in Sender loop when no request is sent (#7085)
PR-7054 - KAFKA-8615: Change to track partition time breaks TimestampExtractor (#7054)
PR-7094 - KAFKA-8670; Fix exception for kafka-topics.sh –describe without –topic mentioned (#7094)
PR-7092 - KAFKA-8602: Separate PR for 2.3 branch (#7092)
PR-6928 - KAFKA-8530; Check for topic authorization errors in OffsetFetch response (#6928)
PR-7086 - KAFKA-8662; Fix producer metadata error handling and consumer manual assignment (#7086)
PR-7050 - KAFKA-8637: WriteBatch objects leak off-heap memory (#7050)
PR-7021 - KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (#7021)
PR-7076 - HOT FIX: close RocksDB objects in correct order (#7076)
PR-7070 - KAFKA-7157: Fix handling of nulls in TimestampConverter (#7070)
PR-5705 - KAFKA-6605: Fix NPE in Flatten when optional Struct is null (#5705)
PR-6678 - Fixes #8198 KStreams testing docs use non-existent method pipe (#6678)
PR-7030 - KAFKA-5998: fix checkpointableOffsets handling (#7030)
PR-7072 - KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 (#7072)
PR-6991 - KAFKA-8591; WorkerConfigTransformer NPE on connector configuration reloading (#6991)
PR-7013 - MINOR: add upgrade text (#7013)

Confluent Security Plugins¶

PR-192 - Allow access to KSQL /info endpoint to authenticated users
PR-187 - Remove jackson databind version override

Control Center¶

PR-1888 - [MMA-4870] adding token-scoped SR clients
PR-1898 - [MMA-2901] fix file limit issue on 5.2.x
PR-1877 - inject RBAC configs for rock too
PR-746 - bump version to v3.35.21-cp-5.3.x.0
PR-745 - chore: update cp version to 3.35.20-cp-5.3.x.0
PR-743 - upgrade to 3.10.20-cp-5.2.x.0
PR-727 - upgrade to 3.35.19-cp-5.3.x.0
PR-726 - upgrade to 3.10.19-cp-5.2.x.0

Metadata Service¶

PR-113 - CLI-210 - Fix lookupResourcesForPrincipal to accept User or Group principal
PR-112 - CLI-210 - Fix swagger return type of LookupResourcesForPrincipal
PR-109 - SEC-233 - MDS Test Infra : Find / build a better testing LDAP
PR-108 - SEC-233 MDS Test Infra : Find / build a better testing LDAP
PR-106 - SEC-319 Add endpoint to retrieve all resources for a user, including …
PR-101 - SEC-265 Add endpoints to lookup which principals have particular roles
PR-103 - Minor - fix version of new modules for 5.3.x
PR-102 - SEC-233 - MDS Test Infra : Find / build a better testing LDAP (#98)
PR-99 - SEC-314 - Remove unnecessary overrides that might mask upstream updates (#97)

Kafka Connect¶

PR-337 - CC-4886: Fix the SecurityIT integration test
PR-334 - CC-4886: Temporarily ignore SecurityIT
PR-335 - MINOR: Upgrade Mockito to work on Java 11
PR-327 - Enable Elasticsearch 7 support with adding minimum Integration test (based on TestContainer)
PR-324 - CC-5605: Log version conflicts as warnings.
PR-451 - CC-6089: Improve fault tolerance of HDFS sink connector
PR-450 - CC-5969: Pass extra connector configs to the partitioner
PR-442 - Adding debug logs for WAL recovery
PR-440 - Remove jackson databind version override
PR-677 - Fix ERROR Graceful stop of task failed
PR-686 - MINOR: Upgrade postgress jdbc driver to most recent bugfix release (9.4.1212)
PR-655 - CC-4946: changing log msgs on filtering and avoiding frequently retrying tasks when no tables
PR-672 - Change TableQuerier log to use real class name for file
PR-79 - MINOR: Revert to using the older version of jackson-databind to fix the build
PR-77 - CC-5671: fix test failures caused by outdated dependencies
PR-76 - CC-5664: Add ‘ Source’ to end of Confluent Hub titles for connectors
PR-237 - REP-30: Try to stop flaky build
PR-236 - REP-20: Added a check to ensure TopicMontiorThreadWithZk is shutdown
PR-235 - REP - 30: Change to stop flaky builds
PR-232 - REP-30: Change to stop flaky builds
PR-228 - REP-30: MINOR: Fix Jenkins build failures due to the maven-surefire plugin
PR-226 - REP-30: MINOR: Fix Jenkins build failures due to the maven-surefire plugin
PR-225 - MINOR: Updated Jenkinsfile to reference new Slack channel
PR-219 - Add debug logs for starting consumers
PR-222 - MINOR: update slack channel for Jenkins notifications, to #replicator…
PR-220 - CC-5818: remove a test that isn’t valid for 5.1.x onwards.
PR-218 - MINOR: Subsume provenance header addition into existing utility metho…
PR-217 - CC-2796: Pass a delta for better than ms precision
PR-214 - CC-5541: Set ByteArrayConverter as header converter
PR-209 - CC-5012: If topic creation fails, attempt to resize topic
PR-263 - Remove jackson databind version override
PR-52 - Remove jackson databind version override
PR-112 - MINOR: Improve the locale and timezone config descriptions
PR-110 - CC-5963: Improved ability to handle schema evolution with different names

Confluent Operator¶

To obtain the latest fixes for Confluent Operator supported for Confluent Platform 5.3.1, download the Operator Helm Charts (confluent-operator-20190912-v0.65.1) here.
To upgrade Confluent Operator and Confluent Platform with these Helm Charts, download the Helm Charts and follow the instructions here.
This version fixes the issue seen with KSQL where crashloopbacks are observed when the interceptor is enabled.
This version supports non-alphanumeric characters for SASL_PLAIN Authentication passwords.
This version fixes the Helm Chart example default configurations in the providers folder where it was missing Kubernetes resource request configurations.
This version updates the Operator image to 0.176.1 to fix a Confluent Operator license key issue. These changes will interfere with CP upgrade for existing deployment (deployed through an older version of Operator). To fix the issue, complete the following steps:
- This licensing issue only affects Confluent Operator licensing, and not other CP components like Control Center
- This issue is only applicable for users upgrading to this Confluent Platform 5.3.1 Operator patch that are still using the free 30 day trial for Confluent Platform.
- Users that are still on the free 30 day trial will need to run the following steps before upgrading using Helm:
  - Zero out the licensing key section within the “operator-licensing” secret to generating a new valid trial on a rolling upgrade.
  - Specifically, run the command kubectl edit secrets -n <namespace> operator-licensing. Then, zero out the licenseKey key in the data map (e.g. licenseKey: “” will suffice). Make sure you don’t delete the licenseKey key itself, as the operator would not generate a license key, key-value.

MQTT Proxy¶

PR-42 - Remove jackson databind version override

Community Features¶

Confluent Platform Ansible¶

PR-124 - Roles and groups now match service names, only 1 playbook, moved sample inventories into new directory
PR-123 - Corrected link to Contribution documentation.
PR-121 - adding files
PR-120 - Revert “Merge pull request #118 from confluentinc/5.3.0-pre”
PR-118 - 5.30 Release.
PR-119 - Merge pull request #118 from confluentinc/5.3.0-pre
PR-118 - 5.30 Release.
PR-117 - Revert “Release for CP 5.3.0”
PR-114 - Release for CP 5.3.0
PR-116 - kerberos realm hardcoded to confluent.example.com

Kafka¶

PR-7287 - MINOR: Add unit test for KAFKA-8676 to guard against unrequired task restarts (#7287)
PR-7097 - Changed for updatedTasks, avoids stopping and starting of unnecessary tasks (#7097)
PR-7256 - MINOR: Only send delete request if there are offsets in map (#7256)
PR-7207 - KAFKA-8412: Fix nullpointer exception thrown on flushing before closing producers (#7207)
PR-6993 - KAFKA-8586: Fail source tasks when producers fail to send records (#6993)
PR-7235 - KAFKA-8824: bypass value serde on null (#7235)
PR-7176 - KAFKA-8325; Remove batch from in-flight requests on MESSAGE_TOO_LARGE errors (#7176)
PR-7219 - MINOR: Fixing log format typo (#7219)
PR-7211 - KAFKA-8800: Increase poll timeout in poll[Records]UntilTrue (#7211)
PR-7212 - KAFKA-8802: ConcurrentSkipListMap shows performance regression in cache and in-memory store (#7212)
PR-7163 - MINOR: Fix bugs in handling zero-length ImplicitLinkedHashCollections (#7163)
PR-7192 - KAFKA-8788: Optimize client metadata handling with a large number of partitions (#7192)
PR-7210 - MINOR: Correct typo in test name TimetampedSegmentsTest (#7210)
PR-7201 - KAFKA-8791: RocksDBTimestampedStore should open in regular mode by default (#7201)
PR-7177 - KAFKA-8736: Track size in InMemoryKeyValueStore (#7177)
PR-6283 - KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread (#6283)
PR-7197 - KAFKA-8774: Regex can be found anywhere in config value (#7197)
PR-216 - CONFLUENT: Build cp-downstream-builds as part of Jenkins PR Job
PR-6959 - KAFKA-8550: Fix plugin loading of aliased converters in Connect (#6959)
PR-7125 - MINOR: Upgrade jackson-databind to 2.9.9.3 (#7125)
PR-7164 - KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (#7164)
PR-7143 - MINOR: Avoid dividing by zero (#7143)
PR-7143 - MINOR: Avoid dividing by zero (#7143)
PR-7132 - KAFKA-8731: InMemorySessionStore throws NullPointerException on startup (#7132)
PR-7116 - KAFKA-8715; Fix buggy reliance on state timestamp in static member.id generation (#7116)
PR-7101 - KAFKA-8678; Fix leave group protocol bug in throttling and error response (#7101)
PR-7093 - MINOR: kafkatest - adding whitelist for interbroker sasl configs (#7093)
PR-531 - CONFLUENT: Jenkins gradle exit 1 on executor failure, 0 on test failure (#531)
PR-490 - CONFLUENT: Jenkinsfile should not swallow test return status codes (#490)
PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
PR-7099 - MINOR: Update documentation for enabling optimizations (#7099)
PR-6844 - MINOR: Remove stale streams producer retry default docs. (#6844)
PR-7085 - KAFKA-8635; Skip client poll in Sender loop when no request is sent (#7085)
PR-7054 - KAFKA-8615: Change to track partition time breaks TimestampExtractor (#7054)
PR-7094 - KAFKA-8670; Fix exception for kafka-topics.sh –describe without –topic mentioned (#7094)
PR-7092 - KAFKA-8602: Separate PR for 2.3 branch (#7092)
PR-6928 - KAFKA-8530; Check for topic authorization errors in OffsetFetch response (#6928)
PR-7086 - KAFKA-8662; Fix producer metadata error handling and consumer manual assignment (#7086)
PR-7050 - KAFKA-8637: WriteBatch objects leak off-heap memory (#7050)
PR-7021 - KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (#7021)
PR-7076 - HOT FIX: close RocksDB objects in correct order (#7076)
PR-7070 - KAFKA-7157: Fix handling of nulls in TimestampConverter (#7070)
PR-5705 - KAFKA-6605: Fix NPE in Flatten when optional Struct is null (#5705)
PR-6678 - Fixes #8198 KStreams testing docs use non-existent method pipe (#6678)
PR-7030 - KAFKA-5998: fix checkpointableOffsets handling (#7030)
PR-7072 - KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 (#7072)
PR-6991 - KAFKA-8591; WorkerConfigTransformer NPE on connector configuration reloading (#6991)
PR-7013 - MINOR: add upgrade text (#7013)

KSQL¶

PR-3225 - Fix quoted reserved identifier 5.3
PR-3197 - fix: validate CLI using REST / instead of REST /info
PR-3176 - fix: default timestamp extractor override is not working
PR-3154 - refactor: allow for more specific error messages to be sent when the server is unavailable
PR-3111 - fix: add ksql-functional-tests to the ksql package
PR-3147 - fix: filter null entries before creating KafkaConfigStore
PR-3079 - feat: add config for enabling topic access validator
PR-3069 - feat(rest server): add warnings when listing fails with kafka error
PR-3104 - fix: COLLECT_LIST can now be applied to tables
PR-2997 - feat: add extension for custom metrics (5.3.x)
PR-2996 - feat: add config for custom metrics tags (5.3.x)

Rest Utils¶

PR-148 - Remove jackson databind version override

Schema Registry¶

PR-855 - #854 - mvn plugin - allow schemas composition
PR-1182 - MINOR: Update slack channel to #schema-registry-eng
PR-1174 - MINOR: fix NPE by ignoring tombstones
PR-1166 - MINOR: Add missing reset method to mock client

Confluent Platform 5.3.0 Release Notes¶

5.3.0 is a major release of Confluent Platform that provides you with Apache Kafka® 2.3.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features¶

Confluent CLI¶

The Confluent CLI was completely rewritten to support a wider set of features and platforms.
The Confluent CLI was removed from Confluent Platform packaging. The Confluent CLI is now installed separately from the Confluent Platform package. For more information, see Installing and Configuring the CLI.
The Confluent CLI development commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local start. For more information, see confluent local.
Added support for managing role-based access control (RBAC) permissions and role bindings.
Added support for masking secrets and configuration files.
Added “self-update” capability via the confluent update command so that improvements and fixes can be released out-of-band from the Confluent Platform release schedule.
The Confluent CLI is a commercially licensed feature, but it can be downloaded and used against both Confluent Community and Confluent Platform.

Confluent Control Center¶

Control Center introduces several changes and new features, including:

Redesigned UI: The Control Center UI has been redesigned from end-to-end to provide an easier-to-use, multi-cluster-friendly, and global view of Kafka that supports working with Confluent at scale. The new homepage provides a global view of all of your Kafka clusters and connected services, including Connect and KSQL. You can see a health roll-up for each cluster and use a switcher to view only the unhealthy clusters so you can focus on what needs attention.
New broker overview: Clicking one of the cards on the new cluster homepage takes you into the cluster and connected services, showing broker metrics or topics depending on your permissions if RBAC is enabled. You can drill down further into broker metrics by clicking on a card, which shows time-series charts with interactive legends and shared cursors for easier metrics correlation.
New topics pages: The redesigned Control Center includes a new topics index that shows you all the topics in a cluster with key health information for each in a searchable table with sortable columns. You can click on a topic name to view an overview for it, including relevant throughput and health information, and access the message browser, schemas, and topic configuration. Topic metrics, including a topic-centric view of consumer lag, can be accessed by clicking on the metrics cards. Under consumer lag, you can select which consumer group you are interested in using a drop-down so you can focus on the specific application you care about.
Message browser enhancements: Message browser performance has been improved to eliminate throttling and message loss. You can also now access the full topic history by seeking, either using offset or timestamp, on a specific partition. You can also now download messages in JSON format, including using the typical Ctrl/Shift-select shortcuts to select and download more than one message.
Connect redesign and enhancements: The new UI also features an index page for all of the Connect clusters associated with a Kafka cluster. You can search for a Connect cluster by name/ID and see how many connectors are running, degraded, or failed on each cluster. Degraded connectors have at least one failed task; whereas failed connectors have failures on all of their tasks. Clicking on one of the clusters in the Connect index takes you to the new Connect cluster overview page. Here, you can see all the connectors running on the cluster, including their status, type, category, and number of tasks. Clicking on one of the connectors in the Connect cluster overview takes you to a page showing the connector itself where you can see a breakdown of the connector’s tasks and their status. This new Connect experience makes it much easier to view all of the Connect clusters and drill down to see how each task on a connector is performing.
KSQL redesign and enhancements: The new UI also features an index page for all of the KSQL clusters associated with a Kafka cluster. You can search for a KSQL cluster by name/ID, see how many queries are running, and see how many streams or tables are on the cluster. Clicking on one of the clusters in the KSQL index takes you to a page where you can access the KSQL editor, streams, tables, and persistent queries. The editor itself features similar improvements as the message browser, including performance improvements to eliminate throttling and message loss and JSON downloading of results. In addition, the KSQL editor area is resizeable to accommodate larger queries and statements. It also includes a data discovery side-panel, making it much easier to find streams and tables to run KSQL against.
Consumers index: Control Center also includes a new index to show you all the consumer groups associated with a selected Kafka cluster, including the number of consumers per group and the number of topics being consumed. Clicking on a consumer group takes you to a page where you can view Consumer Lag across all relevant topics and a redesigned Streams Monitoring page, making it much easier to understand consumption metrics in context.
Alerting enhancements: Control Center features several highly requested alerting enhancements. To date, Control Center only supported email-based alert actions. In Confluent Platform 5.3.0, you can now use webhooks for Slack and PagerDuty, making it much easier to integrate Control Center with a range of monitoring approaches. There is also a new Consumer Lead alert, which you can customize to fire when a consumer is within a certain number of offsets of the partition tail. You can also now pause and resume actions globally, silencing alerts when needed. Finally, the alert messages themselves have become more informative and now include better contextual information about the alert, including the cluster, metric, condition, and trigger.
Feature flagging for System Health and Data Streams: While Confluent Platform 5.3.0 introduces a significantly redesigned Control Center, a feature flag allows you to view the older System Health and Data Streams pages.

Note

These pages cannot be viewed if RBAC is enabled.
RBAC enforcement: As part of the introduction of RBAC to Confluent Platform, Control Center respects the rolebindings assigned using the CLI, so you have platform-wide consistency for your security strategy. For more information, see Role-based access control (RBAC) in Confluent Control Center.

For more information, see the documentation.

Control Center Known Issues¶

Multi-cluster alert triggers: You can currently select more than one cluster when defining a Cluster or Broker alert trigger in Control Center but doing so results in a trigger firing against all of the associated clusters rather than the one that actually meets the alert definition. To avoid this issue in the interim, Cluster and Broker alerts should be defined against individual clusters.
Direct access to each external cluster is required under RBAC: When RBAC is enabled, Control Center requires management to be enabled on each cluster. This means that metrics reporters and interceptors must have a direct connection for Control Center to operate properly in an RBAC environment at this time. See Control Center connections to external clusters for more details.
Connect and KSQL require at least minimal user access to the underlying Kafka clusters: See Connect and KSQL clusters user access for details.
The service principal for Control Center requires SystemAdmin access: The Control Center user must be set up as a privileged user SystemAdmin on each cluster. Due to the underlying architecture for consumer lag, elevated privileges are required to guarantee users access to all consumer groups and continued support for consumer group alerts for consumer lag. The consumer lag offsets are currently not obtained from metrics reporters.
Prefix search in the message browser cannot currently search column names that contain a period.
There is a mismatch between products in enforced default values for max message bytes: When connecting Control Center to Confluent Cloud, the confluent.metrics.topic.max.message.bytes must be set to 8388608 rather than the current Control Center default of 10485760. For more information, see Control Center cannot connect to Confluent Cloud and Connecting Control Center to Confluent Cloud.

Security¶

Role-based access control (RBAC)¶

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is being introduced to gain early feedback from developers. This feature can be used for evaluation and non-production testing purposes or to provide feedback to Confluent.

Provides the secure authorization of access to resources by users and groups.
Uses a set of predefined roles that give you depth and granularity in its configuration of managing access.
Provides centralized authorization for all the components including Control Center, Connect, Schema Registry, REST Proxy, and KSQL.
Provides connector level access control.

For more information, see the RBAC documentation:

AD/LDAP enhancements¶

Centralized configuration.
Centralized authentication for Control Center, Connect, Schema Registry, REST Proxy, and KSQL.

For more information, see Confluent LDAP Authorizer.

Secret Protection¶

To enable masking of sensitive information in the configuration file and logs.
Across the whole platform including Kafka, Connect, KSQL, Schema Registry, REST Proxy, and Control Center.
Utilize the new CLI to create/manage secrets.

For more information, see Secrets.

Community Features¶

Clients¶

librdkafka v1.1.0 is a security feature release with added support for:

OAUTHBEARER SASL authentication.
In-memory SSL certificates.
Windows Root Certificate store.
Pluggable broker SSL certificate verification callback.
Improved SASL GSSAPI/Kerberos ticket refresh.

Upgrade considerations:

Windows SSL users will no longer need to specify a CA certificate file/directory (ssl.ca.location), librdkafka will load the CA certs by default from the Windows Root Certificate Store.
SSL peer (broker) certificate verification is now enabled by default (disable with enable.ssl.certificate.verification=false)
%{broker.name} is no longer supported in sasl.kerberos.kinit.cmd since kinit refresh is no longer executed per broker, but per client instance.

See the full librdkafka v1.1.0 release notes for more information.

confluent-kafka-python, confluent-kafka-dotnet, confluent-kafka-go¶

Clients now use librdkafka v1.1.0 and include all the improvements from the latest version.

Connect¶

PR-6363 - KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) (#6363) When a rebalance happened in Kafka 2.2 or earlier, it stops all tasks in a Connect cluster and restarts them. This can be a hard stop for users who run multiple connectors in a Connect cluster. With KIP-415, a rebalance happens more gracefully. It stops only the tasks that need to move between workers (if any), leaving the rest running on their assigned worker
PR-5743 - KAFKA-3816: Add MDC logging to Connect runtime (#5743) With KIP-449, The SLF4J API includes “Mapped Diagnostic Contexts” (MDC) that allow injection of a series of parameters that can be included in every log message written using that thread, regardless of how the SLF4J Logger instance was obtained.
PR-6584 - KAFKA-8231: Expansion of ConnectClusterState interface (#6584)
PR-6789 - KAFKA-8407: Fix validation of class and list configs in connector client overrides (#6789)
PR-6658 - KAFKA-8309: Add Consolidated Connector Endpoint to Connect REST API (#6658)

Connectors¶

HDFS Connector:
HDFS connector is removed from the Confluent Platform package. However, the connector will continue to be downloadable from Confluent Hub and supported by Confluent.
JDBC Connector:

PR-641 - CC-349: Added delete support for sink. (#641) A record with null value is considered a tombstone event and result in deleting the corresponding row in the destination table.
Elasticsearch Connector:

PR-239 - Support for document upsert. (#239) In some cases, records in Kafka topics consist only subset of fields ElasticSearch needs. With this feature, ElasticSearch Sink Connector can perform an upsert operation.

Kafka 2.3.0¶

KIP-339: A new IncrementalAlterConfigs API in AdminClient Changes configurations incrementally, only modifying the configuration values that are specified to prevent lost updates. AlterConfigs API has been marked for deprecation. For more information, see KIP-339
KIP-341: Update Sticky Assignor’s User Data Protocol Improves stability of consumer partition assignments by preventing assigning the same partition to multiple consumers in the same consumer group. For more information, see KIP-341
KIP-351: Add –under-min-isr option to describe topics command Adds the --under-min-isr option in the describe topics command, which allows users to see precisely which topic partitions are below min.insync.replicas need to be addressed. For more information, see KIP-351
KIP-354: Maximum Log Compaction Lag Compaction allows Kafka to remove messages that are are older than min.compaction.lag.ms, which ensures that a segment is rolled and remains uncompacted for a given period ( “lag”). Regulations such as GDPR require that data is deleted in a timely manner. max.compaction.lag.ms sets the maximum lag time for which a segment may remain uncompacted, after which the corresponding log partition becomes eligible for log compaction. For more information, see KIP-354
KIP-361: Add Consumer Configuration to Disable Auto Topic Creation Both auto.create.topics.enable on the broker and allow.auto.create.topics on the consumer need to be set for auto-topic creation to happen. For more information, see KIP-361
KIP-402: Improve fairness in SocketServer processors The first part of KIP-402 was introduced in Kafka 2.2, which changed how connections were prioritized. Now existing connections are given more priority over new connection requests. The KIP introduces max.connections, which limits the total number of connections that may be active on the broker at any time. This is in addition to the existing max.connections.per.ip configuration that will continue to limit the number of connections from each host IP address. For more information, see KIP-402
KIP-417: Allow JMXTool to connect to to a secure RMI port Adds --jmx-ssl-enable and --jmx-auth-prop to allow connecting to a secure Java VM. For more information, see KIP-417
KIP-421: Support resolving externalized secrets in AbstractConfig Enhances the AbstractConfig base class to automatically resolve variables of the form specified in KIP-297. For more information, see KIP-421
KIP-425: Add Log4J Kafka Appender Properties to Producing to Secure Brokers Extends the Log4J Kafka appender to support PLAIN mechanism and configuration of jaas via a property passed to the producer. For more information, see KIP-425
KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option) Introduces a new topic partition category in the metrics group between UnderReplicated and UnderMinIsr called AtMinIsr. For more information, see KIP-427
KIP-430 - Return Authorized Operations in Describe Responses The AdminClient now allows users to determine what operations they are authorized to perform on topics. For more information, see KIP-430
KIP-436: Add a metric indicating start time Useful for detecting restarts. For more information, see KIP-436
KIP-461: Improve Replica Fetcher Behavior at handling partition failure In previous versions, if a partition fails, the replica fetcher thread associated with that partition will terminate. Because the replica fetcher threads handle multiple partitions, this led to under-replicated partitions. Now, whenever a partition crashes, the concerned thread stops tracking the crashed parition and continues handling rest of the partitions. This all reduces the chance of under-replicated partitions and improves cluster stability. For more information, see KIP-461

The release includes 157 new features, improvements, and fixes. For a full list of changes in this release of Kafka, see the Apache Kafka 2.3.0 Release Notes.

Also, see the blog post covering What’s New in Apache Kafka 2.3 or this video.

Kafka Streams¶

Stored Record Timestamps in RocksDB: To offer better timestamp processing semantics and introduce new time-related KTable features, more information about the timeliness of records must be persisted. Kafka Streams now stores the timestamps for each record that contributes to state stores inside RocksDB. These timestamps are now accessible through Interactive Queries, too. This functionality paves the way for potential future features, like KTable TTLs and support for out-of-order KTable data. See KIP-258 for more information.
In-memory window and session stores: Kafka Streams supports pluggable storage of its tables. Until this release, Kafka Streams offered only durable versions out of the box of its window and session store abstractions. Kafka Streams now ships with in-memory versions implemented to support high performance, transient operations. See KIP-428 and KIP-445 for more information.
New flatTransformValues method: Kafka Streams now supports a new method in its API, flatTransformValues. It is the equivalent of flatMapValues for the Processor API. Unlike the traditional transformValues method, this method is able to ensure strong typing by specifying in its signature a list of key-value pairs (i.e. Iterable) as output records for each input record. See KIP-313 for more information.
Default implementation to close() and configure() for Serializer, Deserializer and Serde: When serializers, deserializers, and serdes are authored, the close and configure methods are typically implemented with no operation behind them. Kafka Streams now leverages Java’s new default interface inheritance feature, and provides a sensible, default implementation for these methods. See KIP-331 for more information.
Better defaults for increased stability: To better improve footprint and threading stability, a number of defaults have been changed to more sensible values. These include max.poll.interval.ms, segment.ms, and segment.index.bytes. See KIP-442 and KIP-443 for more information.
New close() method on RocksDBConfigSetter: A new close() method has been added to support cleanup of RocksDB configs. This helps to evade inadvertent memory leaks. See KIP-453 for more information.

KSQL¶

KEY no longer required for CREATE TABLE: Tables can now be created without manually specifying a KEY parameter. For example, CREATE TABLE T1 (ID INT, OTHER INT). KEY can still be specified as a hint, but is no longer required (#2745).
CREATE STREAM/TABLE now creates topics if they don’t exist: Previously a topic underlying a stream or table had to exist already to create a stream or table over it. KSQL now creates the underlying topic automatically if it doesn’t already exist (#2771).
Improved UDF interfaces: Two of the four components of KLIP-1 have made it into this release: UDFs can now be defined with variable-length arguments as well as custom Struct argument types (#2503).
Data can now be written to streams via INSERT INTO: Users can now write events directly to streams from within KSQL using standard SQL INSERT INTO syntax. While not designed for high-performance production usage, simplifying the process of getting data into KSQL should greatly improve the initial KSQL experience and make it much easier for new users to experiment with the system (KLIP-2, #2723).
KSQL test runner tool: The ksql-test-runner tool is a command line utility that enables testing KSQL statements without requiring any infrastructure, which means that Kafka and KSQL don’t need to be running. It’s a lightweight way to design and iterate on your KSQL-based applications and ensure that the expected results are generated (#2802).
CLI sessions can now be recorded to text files: SPOOL is a new CLI command that allows users to record a KSQL CLI session’s inputs and outputs, writing the session content out to a file (#2789).
ELT and FIELD functions: Both the ELT and FIELD functions are based on the SQL standard and are available in many other SQL-based systems (#2627). ELT (N, x, y, ...) returns the Nth element of the given list of strings. This is the complement to FIELD. FIELD (string, str1, str2, ...) returns the index of string within the list of given strings. This is the complement to ELT.
More date/timestamp formats now supported: KSQL’s date/timestamp parser is now more robust and understands more date/timestamp formats (#2499).
Quotes in string literals now escaped: This protects KSQL from malicious Java code injection attacks (#2545).
Bug fixes:
- Fixed GEO_DISTANCE function (#2700).
- Fixed bug causing tabs in multi-line input strings to cause unnecessary errors (#2734).
- Fixed bug that caused command topic replay during KSQL startup to recreate deleted topics unintentionally (#2329).
- Fixed bug with performing joins on ROWKEY (#2735).
- Fixed bug causing potential crash during KSQL REST server shutdown (#2507).

REST Proxy¶

FIX: REST Proxy now responds with a 401/403 for authentication/authorization errors.

REST Utils¶

The default value for ssl.endpoint.identification.algorithm was changed to https. This setting performs hostname verification to prevent man-in-the-middle attacks. You can set ssl.endpoint.identification.algorithm to an empty string to restore the previous behaviour.

Schema Registry¶

PR-1124 - CC-4775: logical type preservation for hdfs connector when output format is parquet (#1124)

Other Improvements & Changes¶

KAFKA-8336: Enable dynamic update of client-side SSL factory in brokers When mutual authentication is enabled for inter-broker-communication (ssl.client.auth=required), broker restarts are no longer needed when updating client-side keystores.
FIX: Added better open file limit to systemd files

Deprecation Warnings¶

OS support for RHEL 6, Ubuntu 12.04, Ubuntu 14.04, and Debian 7
Confluent Rest Proxy v1 API

How to Download¶

Confluent Platform is available for download at https://www.confluent.io/download/. See the On-Premises Deployments section for detailed information.

To upgrade Confluent Platform to a newer version, check the Upgrade documentation.

Supported Versions and Interoperability¶

For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.

Questions?¶

If you have questions regarding this release, feel free to reach out via the community mailing list or community Slack. Confluent customers are encouraged to contact our support directly.