Confluent Platform Component Changelogs

This topic provides changelogs for the individual Confluent Platform components.

Confluent Platform Changelog

5.4.1

5.4.1 is a bugfix release of Confluent Platform that provides you with Apache Kafka® 2.4.0, the latest stable version of Kafka.

Commercial Features

Confluent Server

  • KAFKA-9601: Stop logging raw connector config values (#8165)
  • CONFLUENT: Fix exception on trying to retrieve tierable segments
  • KAFKA-9599 create unique sensor to record group rebalance (#8159)
  • CPKAFKA-2705: Added retention to soak topics
  • KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset … (#8136)
  • KAFKA-9533: ValueTransform forwards null values (#8108)
  • KAFKA-8025: Fix flaky RocksDB test (#8126)
  • CONFLUENT: Add KeyPair PEM reader/writer to PemUtils
  • KAFKA-8245: Fix Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups (#8032)
  • KAFKA-9515: Upgrade ZooKeeper to 3.5.7 (#8125)
  • KAFKA-9535; Update metadata before retrying partitions when fetching offsets (#8088)
  • CONFLUENT: Migrate from Nexus to Artifactory
  • KAFKA-9355: Fix bug that removed RocksDB metrics after failure in EOS (#7996)
  • ST-3015: Switching from nexus to artifactory repo
  • allow ReplaceField SMT to handle tombstone records (#7731)
  • KAFKA-9192: fix NPE when for converting optional json schema in structs (#7733)
  • KAFKA-9181; Maintain clean separation between local and group subscriptions in consumer’s SubscriptionState (#7941)
  • KAFKA-9500: Fix FK Join Topology (#8015)
  • Kafka-9503: Fix TopologyTestDriver output order (#8065)
  • KAFKA-9390: Make serde pseudo-topics unique (#8054)
  • KAFKA-9540: Move “Could not find the standby task while closing it” log to debug level (#8092)
  • KAFKA-7052 Avoiding NPE in ExtractField SMT in case of non-existent fields (#8059)
  • KAFKA-9517: Fix default serdes with FK join (#8061)
  • KAFKA-9507; AdminClient should check for missing committed offsets (#8057)
  • Initial commit to migrate Docker build from cc-kafka
  • SEC-713 Refresh Audit Log topic metadata
  • KAFKA-9491; Increment high watermark after full log truncation (#8037)
  • KAFKA-9074: Correct Connect’s Values.parseString to properly parse a time and timestamp literal (#7568)
  • KAFKA-9492; Ignore record errors in ProduceResponse for older versions (#8030)
  • KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning (#7932)
  • KSTORAGE-409: lazily initialize consumer lag histogram to address all…
  • CONFLUENT: Bump up version to 5.4.1
  • CPKAFKA-2388; Add RBAC system test to verify role-based authorization
  • KAFKA-9254; Overridden topic configs are reset after dynamic default change (#7870)
  • KAFKA-9143: Log task reconfiguration error only when it happened (#7648)
  • KAFKA-9024: Better error message when field specified does not exist (#7819)
  • KAFKA-9083: Various fixes/improvements for Connect’s Values class (#7593)
  • KAFKA-9449; Adds support for closing the producer’s BufferPool. (#7967)
  • KAFKA-9235; Ensure transaction coordinator is stopped after replica deletion (#7963)
  • KAFKA-9144; Track timestamp from txn markers to prevent early producer expiration (#7687)
  • KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore (#7908)
  • KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
  • KAFKA-9065; Fix endless loop when loading group/transaction metadata (#7554)
  • KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
  • HOTFIX: fix system test race condition (#7836)
  • KAFKA-9293; Fix NPE in DumpLogSegments offsets parser and display tombstone keys (#7820)
  • KAFKA-8933; Fix NPE in DefaultMetadataUpdater after authentication failure (#7682)
  • KAFKA-9232; Coordinator new member timeout does not work for JoinGroup v3 and below (#7753)
  • KAFKA-9307; Make transaction metadata loading resilient to previous errors (#7840)
  • fix rows (#7837)
  • KAFKA-9011: Removed multiple calls to supplier.get() in order to avoid multiple transformer instances being created. (#7685)
  • KAFKA-9011: Scala bindings for flatTransform and flatTransformValues in KStream (#7520)
  • KAFKA-9310: Handle UnknownProducerId from RecordCollector.send (#7845)
  • KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
  • KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
  • KAFKA-8705: Remove parent node after leaving loop to prevent NPE (#7117)
  • port paragrpah from CP docs (#7808)
  • fix typo in processor-api developer guide docs (#7689)

Control Center

  • [MMA-6375, MMA-6372] update license viewing permissions for OTA
  • MMA-6007: Fix for invalid follower and observer count values
  • [MMA-5879] smart proxy via service healthchecks
  • Sort SR urls for multi-cluster SR hash
  • support for multiple SR clients
  • refactor kafka-rest proxy to handle async registration
  • actually use caching SR client supplier
  • chore: update @confluent/control-center to 3.91.43-cp-5.4.x.0
  • chore: update @confluent/control-center to 3.91.41-cp-5.4.x.0
  • chore: update @confluent/control-center to 3.91.40-cp-5.4.x.0
  • chore: update @confluent/control-center to 3.91.39-cp-5.4.x.0
  • chore: update @confluent/control-center to 3.35.31-cp-5.3.x.0
  • chore: update @confluent/control-center to 3.91.38-cp-5.4.x.0
  • update @confluent/control-center to 3.91.37-cp-5.4.x.0
  • chore: upgrade @confluent/control-center to v3.91.34-cp-5.4.x.0
  • chore: upgrade @confluent/control-center to v3.91.33-cp-5.4.x.0
  • chore: update @confluent/control-center to v3.91.32-cp-5.4.x.0

Confluent Hub Client

  • Update upstream to ce-kafka

Kafka Connect Elasticsearch

  • CC-7147: set the max connection per route config for client

Kafka Connect JDBC Connector

  • CC-7637: change from DEBUG level to TRACE level
  • CC-8090: Validate connection.attempts property in config class
  • CC-8087: Remove plaintext logging of task configs
  • [jdbc-source-connector] Fixed issue in CachedConnectionProvider, to enable retries when a connection is failed
  • Make metadata allFields ordered consistently with the valueSchema.
  • CC-7628 Set conservative timeout for JDBC to stop trying to reach drive
  • CC-5047: connect jdbc url field should have example text
  • Move setFetchSize to GenericDatabaseDialect
  • CC-7637: added logging of query to be executed at DEBUG level
    • Handling deletes with schema !null
  • Fix Postgres OOM by limiting fetch size

Kafka Connect JMS Source

  • MINOR: Fix Kafka Scala version in all modules
  • CC-3439: Update MQ Source Connector to better handle errors and autoreconnect when MQ manager is restarted
  • CC-6503: Capping batch size at 10000 for IBM MQ connector
  • MINOR: maven compiler plugin no longer requires Java 8
  • fix dependency to not use hardcoded scala
  • CC-7957: corrected pom version and confluent-licensing-new version
  • CC-8052: corrected pom version and confluent-licensing-new version
  • CC-7984: corrected pom version
  • Revert “Revert “CC-3937: repackaging base classes””
  • Revert “CC-3937: repackaging base classes”
  • CC-3937: repackaging base classes
  • CC-7926: Corrected logic to prevent infinite loops during poll()
  • CC-7866: Added more and improved existing logging in the base JMS source task and connector
  • CC-7866: Added more and improved existing logging in the base JMS source task and connector
  • Update upstream to ce-kafka

Confluent Replicator

  • REP-638: Ensuring translatorMonitor is not null before closing it out
  • REP-637: Ensuring timestampsCommitter is not null before closing it out

Community Features

Kafka

  • PR-8188 - throttle consumer timeout increase (#8188)
  • PR-8165 - KAFKA-9601: Stop logging raw connector config values (#8165)
  • PR-270 - Update to jackson databind 2.9.10.3
  • PR-8159 - KAFKA-9599 create unique sensor to record group rebalance (#8159)
  • PR-263 - Changes to migrate to Artifactory
  • PR-8136 - KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset … (#8136)
  • PR-8108 - KAFKA-9533: ValueTransform forwards null values (#8108)
  • PR-8126 - KAFKA-8025: Fix flaky RocksDB test (#8126)
  • PR-259 - Update to jackson databind 2.9.10.2
  • PR-8032 - KAFKA-8245: Fix Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups (#8032)
  • PR-8125 - KAFKA-9515: Upgrade ZooKeeper to 3.5.7 (#8125)
  • PR-8088 - KAFKA-9535; Update metadata before retrying partitions when fetching offsets (#8088)
  • PR-7996 - KAFKA-9355: Fix bug that removed RocksDB metrics after failure in EOS (#7996)
  • PR-7731 - allow ReplaceField SMT to handle tombstone records (#7731)
  • PR-7733 - KAFKA-9192: fix NPE when for converting optional json schema in structs (#7733)
  • PR-7941 - KAFKA-9181; Maintain clean separation between local and group subscriptions in consumer’s SubscriptionState (#7941)
  • PR-8015 - KAFKA-9500: Fix FK Join Topology (#8015)
  • PR-8065 - Kafka-9503: Fix TopologyTestDriver output order (#8065)
  • PR-8054 - KAFKA-9390: Make serde pseudo-topics unique (#8054)
  • PR-8092 - KAFKA-9540: Move “Could not find the standby task while closing it” log to debug level (#8092)
  • PR-8059 - KAFKA-7052 Avoiding NPE in ExtractField SMT in case of non-existent fields (#8059)
  • PR-8061 - KAFKA-9517: Fix default serdes with FK join (#8061)
  • PR-8057 - KAFKA-9507; AdminClient should check for missing committed offsets (#8057)
  • PR-1 - Kafka 2257
  • PR-8037 - KAFKA-9491; Increment high watermark after full log truncation (#8037)
  • PR-7568 - KAFKA-9074: Correct Connect’s Values.parseString to properly parse a time and timestamp literal (#7568)
  • PR-8030 - KAFKA-9492; Ignore record errors in ProduceResponse for older versions (#8030)
  • PR-7932 - KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning (#7932)
  • PR-248 - CONFLUENT: Bump up version to 5.4.1
  • PR-7870 - KAFKA-9254; Overridden topic configs are reset after dynamic default change (#7870)
  • PR-7648 - KAFKA-9143: Log task reconfiguration error only when it happened (#7648)
  • PR-7819 - KAFKA-9024: Better error message when field specified does not exist (#7819)
  • PR-7593 - KAFKA-9083: Various fixes/improvements for Connect’s Values class (#7593)
  • PR-7967 - KAFKA-9449; Adds support for closing the producer’s BufferPool. (#7967)
  • PR-7963 - KAFKA-9235; Ensure transaction coordinator is stopped after replica deletion (#7963)
  • PR-7687 - KAFKA-9144; Track timestamp from txn markers to prevent early producer expiration (#7687)
  • PR-7908 - KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore (#7908)
  • PR-7554 - KAFKA-9065; Fix endless loop when loading group/transaction metadata (#7554)
  • PR-7904 - KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
  • PR-7836 - HOTFIX: fix system test race condition (#7836)
  • PR-7820 - KAFKA-9293; Fix NPE in DumpLogSegments offsets parser and display tombstone keys (#7820)
  • PR-7682 - KAFKA-8933; Fix NPE in DefaultMetadataUpdater after authentication failure (#7682)
  • PR-7753 - KAFKA-9232; Coordinator new member timeout does not work for JoinGroup v3 and below (#7753)
  • PR-7840 - KAFKA-9307; Make transaction metadata loading resilient to previous errors (#7840)
  • PR-7837 - fix rows (#7837)
  • PR-7685 - KAFKA-9011: Removed multiple calls to supplier.get() in order to avoid multiple transformer instances being created. (#7685)
  • PR-7520 - KAFKA-9011: Scala bindings for flatTransform and flatTransformValues in KStream (#7520)
  • PR-7845 - KAFKA-9310: Handle UnknownProducerId from RecordCollector.send (#7845)
  • PR-7839 - KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
  • PR-7117 - KAFKA-8705: Remove parent node after leaving loop to prevent NPE (#7117)
  • PR-7808 - port paragrpah from CP docs (#7808)
  • PR-7689 - fix typo in processor-api developer guide docs (#7689)

KSQL

  • PR-4662 - fix: backport fixes from query close
  • PR-4626 - docs: document healthcheck for ksqlDB headless containers (DOCS-3066)
  • PR-4630 - docs: add VALUE_DELIMITER property to serialization topic (DOCS-3320)
  • PR-4613 - docs: fix broken link in ksqlDB architecture topic (DOCS-2864)
  • PR-4527 - test: data providers should be ordered (MINOR)
  • PR-4544 - docs: pin Pygments version to 2.4.2 (DOCS-3389)
  • PR-4520 - docs: add new config settings (DOCS-3380)
  • PR-4519 - docs: link to MD blog post on ksqlDB tutorials page (DOCS-3379)
  • PR-4470 - docs: clarify that DROP STREAM/TABLE deletes source topic (DOCS-2386)
  • PR-4446 - docs: fix typo on ksqlDB installation index page (DOCS-3332)
  • PR-4438 - docs: update tutorial stack file for v0.6.0/5.4.0 (DOCS-3330)
  • PR-4438 - docs: update tutorial stack file for v0.6.0/5.4.0 (DOCS-3330)
  • PR-4431 - docs: add note about KSQL headless mode and Avro schemas (DOCS-1487)
  • PR-4361 - docs: add note about ksqlDB headless mode and Avro schemas (DOCS-1487)
  • PR-4272 - fix: add ksql-test-runner deps to ksql package lib
  • PR-4314 - 5.4 pending shutdown fix
  • PR-4271 - fix: pull queries should work across nodes (#4169)
  • PR-4377 - docs: add missing KSQL/CCloud Docker parameter (DOCS-2550)
  • PR-4350 - docs: add workaround to docs for ksql-test-runner jars (DOCS-3266)
  • PR-4343 - docs: add CP release version token to markdown docs (DOCS-3259)
  • PR-4342 - docs: update release branch token in ksqlDB markdown docs (DOCS-3258)
  • PR-4341 - docs: remove inline struct content from 5.4.x branch (DOCS-3256)
  • PR-4340 - docs: remove inline struct content from 5.4.0-post branch (DOCS-3256)
  • PR-4338 - docs: add readthedocs.yml to 5.4.0-post branch (DOCS-3255)
  • PR-4337 - docs: add docs-md directory to the new 5.4.0-post branch (DOCS-3254)
  • PR-4297 - fix: deadlock when closing transient push query
  • PR-4299 - build: Do not run twist lock scan and other docker image operations since we no longer build images in this repo

MQTT Proxy

  • Update upstream to ce-kafka

Schema Registry

  • PR-1357 - DG-263: Fix Avro defaults of type record
  • PR-1268 - Add null checks
  • PR-1324 - Update Metadata::bootstrap signature
  • PR-1300 - CC-7692 / ISSUE-1197: Fix configuration of basic auth credentials provider
  • PR-1296 - Adds always-true hostname verifier if ssl.endpoint.identification.algorithm is set to null or empty string

5.4.0

5.4.0 is a major release of Confluent Platform that provides you with Apache Kafka® 2.4.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

Commercial Features

Confluent Server

  • KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
  • CPKAFKA-4272: revert KafkaServer shutdown order to match AK
  • CPKAFKA-4178: replication system test should wait for hotset retention on local segments
  • CPKAFKA-4247: resign as controller on KafkaController.shutdown
  • KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
  • SEC-701: Change confluent.http.server.listeners default host to 0.0.0.0
  • CPKAFKA-4236: Check for empty observer list in addition to None (#1082)
  • SEC-699: Audit Log timestamps always include milliseconds
  • KSTORAGE-339: Tiered storage system tests flaky due to tiering_completed check (#1050)
  • MINOR: Hardcoded fix for us-east-1 buckets returning value of “US” (#…
  • SEC-698 Change audit log topic prefix to not be internal
  • CPKAFKA-4189: Add trogdor test classes for Acl performance tests
  • KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795)
  • KSTORAGE-349: Synchronize log deletion and tier partition state flush
  • MINOR: Bump system test version from 2.2.1 to 2.2.2 (#7765)
  • MINOR: Increase the timeout in one of Connect’s distributed system tests (#7789)
  • CPKAFKA-4202: Use older AdminClient API in LicenseStore for connector compatibility
  • MINOR: Convert last streams join test to TTD (#7777)
  • KAFKA-9184: Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771)
  • KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
  • MINOR: Convert Stream-StreamJoin Integration Test to TTD (#7752)
  • KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
  • KAFKA-9190: Close connections with expired authentication sessions (#7723)
  • KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
  • KAFKA-9258: Check Connect Metrics non-null in task stop (#7768)
  • KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
  • KAFKA-9231: Streams Threads may die from recoverable errors with EOS enabled (#7748)
  • CPKAFKA-4114: Improve ReplicaStatusCommand output.
  • KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
  • CPKAFKA-4167: Pass custom configs to the ChannelBuilder implementations
  • KSTORAGE-319 Expose tiered storage configs in docs
  • KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
  • KAFKA-9244: Update FK reference should unsubscribe old FK (#7758)
  • CPKAFKA-4165: Propagate schema registry URL to metadata server
  • SEC-595: Revert the changes related to adding new prefix for RestClientConfig
  • KAFKA-9123: Test a large number of replicas (#7621)
  • KAFKA-8981: Add rate limiting to NetworkDegradeSpec (#7446)
  • CPKAFKA-4107: Make client configs for license and RBAC consistent
  • CPKAFKA-4127: Make MDS host check in brokers reliable
  • SEC-595: Pass RestClientConfig to the token callback handler
  • CPKAFKA-4024: Make SslEngineBuilder usable without netty
  • KSTORAGE-329: Remove log metrics on deletion
  • KAFKA-8981: Add rate limiting to NetworkDegradeSpec (#7446)
  • MINOR: Remove unused import in ReplicaStatus
  • CPKAFKA-3945: Remove deprecated ReplicaStatus API
  • MINOR: Updated StreamTableJoinIntegrationTest to use TTD (#7722)
  • CPKAFKA-4119: Ensure state is properly reset on writer rebalance
  • MINOR: Fix –enable-autocommit flag in verifiable consumer (#7743)
  • CPKAFKA-4111: Detaches the MetadataServer life-cycle from RBAC provider
  • KAFKA-8509: Add downgrade system test (#7724)
  • KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
  • CPKAFKA-3865: Clean up AdminClient changes in CE Kafka
  • KSTORAGE-311: Ensure offset cache is maintained accurately when batches are skipped
  • KSTORAGE-323: invalid handling of SegmentDeleteInitiate for fenced segments
  • MINOR: Fix Streams EOS system tests by adding clean-up of state dir (#7693)
  • KSTORAGE-323: metadata read should only return valid segments (#1008)
  • KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
  • CPKAFKA-4080: Don’t flag ReplicaStatus as an internal API
  • CPKAFKA-4090: License manager should never configure metrics reporter
  • HOTFIX: safely clear all active state in onPartitionsLost (#7691)
  • MINOR: Ignore avro-generated support-metrics-fullcollector files
  • CPKAFKA-4072: Make sure audit log exporter has metadata (simpler version)
  • HOTFIX: safely clear all active state in onPartitionsLost (#7691)
  • CPKAFKA-4086: Fix configuration in reassign_partitions_test.py
  • DG-120: Move the step of retrieving SR URL to copyKafkaConfigToLog to address 5.4 blocker (#967)
  • CONFLUENT: Assert bucket access via more constrained S3 policy action
  • HOTFIX: Fix unit tests that failed when executed from IDE (#7707)
  • KAFKA-9198: Complete purgatory operations on receiving StopReplica (#7701)
  • MINOR: Fetch only from leader should be respected in purgatory (#7650)
  • KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
  • KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
  • KAFKA-9196: Update high watermark metadata after segment roll (#7695)
  • CPKAFKA-3866: Simplify tests to remove order-based flakiness
  • KSTORAGE-309: invalid handling of full fetch sessions for tiered fetches (#941)
  • KAFKA-9072: Add Topology naming to the dev guide (#7629)
  • CONFLUENT: CC-6522: Use non-null default Kafka cluster ID if herder cannot deduce one
  • KAFKA-9175: Update MirrorMaker 2 topic/partition metrics (#7688)
  • CPKAFKA-4039: Include authorizationInfo subtypes in Audit Log Entry
  • CPKAFKA-3996: Fix LogDirFailureTest to work when license topic is present
  • MINOR: refactor replica last sent HW updates due to performance regression (#7671)
  • KSTORAGE-252: change to info log level (from debug) when endoffset materialized and…
  • KSTORAGE-252: update partition state endOffset on deleteInitiate (#820)
  • KAFKA-9102: Increase default zk session timeout and replica max lag [KIP-537] (#7596)
  • CPKAFKA-4034: Retry on UnsupportedVersionException for license topic create
  • KAFKA-9171: Handle ReplicaNotAvailableException during DelayedFetch (#7678)
  • CPKAFKA-3999: Disable hostname validation for license client in error system test
  • CPKAFKA-3894: Fix flaky test KafkaAuthWriterTest.testWriterElection
  • CONFLUENT: Check if license topic exists before attempting to create
  • MINOR: Change topic-exists log for CreateTopics from INFO to DEBUG (#7666)
  • KSTORAGE-232: use ListOffsetRequest rather than TierListOffsetRequest when tiering enabled (#934)
  • CONFLUENT: Remove usage of ThrowingRunnable
  • MINOR: Update Epoch field descrition in FetchRequest api doc (#7673)
  • CPKAFKA-3848: Add reassign validation for observer and topic placement
  • KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
  • SEC-658 Disable metrics reporter for audit log producer
  • DG-65: Integrate KIP 467 changes in AK
  • CONFLUENT: Add static method to NewPartitionReassignment for Observers
  • KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
  • CPKAFKA-3949: Reset metadata writer executor after writer rebalance
  • CPKAFKA-3985: Rename RBAC and ACL providers to CONFLUENT and ZK_ACL
  • KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
  • CPKAFKA-3955: Validate placement constraint when altering configs
  • MINOR: Return null in key mapping of committed (#7659)
  • CNKAF-103: KIP-496 does not work in multi-tenant clusters
  • KSTORAGE-286: Better handling of Archiver Exceptions from race conditi…
  • CONFLUENT: Add unit tests to check observers tagged field
  • CPKAFKA-3932: Use IBP to gate observer usage and remove feature flag
  • CPKAFKA-3951: Change audit log config name to “enabled” for consistency
  • KSTORAGE-307: memory leak in Tiered Storage CancellationContext(s)
  • CPKAFKA-3942: Support multi-login-module static JAAS for license clients in system tests
  • CPKAFKA-3859: Improve ReplicaStatus information
  • DG-66: Throw an error when broker-side schema validation is configured improperly
  • KAFKA-9140: Also reset join future when generation was reset in order to re-join (#7647)
  • CONFLUENT: Add broker ACLs for license topic in system tests
  • MINOR: Rework NewPartitionReassignment public API (#7638)
  • HOTFIX: remove reference to unused Assignment error code (#7645)
  • MINOR: Rework NewPartitionReassignment public API (#7638)
  • CPKAFKA-3853: Use Kafka protocol for centralized ACL update from brokers
  • HOTFIX: fix bug in VP test where it greps for the wrong log message (#7642)
  • CONFLUENT: Quote sasl.jaas.config option names used for license client
  • CPKAFKA-3904 Route Audit Log messages without building CloudEvent
  • CPKAFKA-3901 Audit Log Router cache should store negative result
  • HOTFIX: Remove from restoringByPartition once restored (#7631)
  • HOTFIX: Remove from restoringByPartition once restored (#7631)
  • CPKAFKA-3783: Set observers in CreatePartitons call (#879)
  • KAFKA-8972 (2.4 blocker): TaskManager state should always be updated after rebalance (#7620)
  • KSTORAGE-288: fix dynamic updating of proprietary log configs (#885)
  • KSTORAGE-290: retry archiving for general S3 AmazonClientException (#862)
  • CPKAFKA-3877: Add backoff and timeout for license retry, set replication factor for tests
  • KAFKA-8972 (2.4 blocker): bug fix for restoring task (#7617)
  • CPKAFKA-3831: Fix ReplicaStatus IsInSync reporting incorrect value
  • CPKAFKA-3855: Don’t allow produce logging to the audit log topic
  • KSTORAGE-230: TierFetcher offset/batch size cache (#719)
  • KAFKA-8972 (2.4 blocker): clear all state for zombie task on TaskMigratedException (#7608)
  • MMA-5596]: Add ‘ObserverReplicasCount’ to monitored metrics list
  • CPKAFKA-3718: Add ObserverReplicasCount metric
  • CPKAFKA-3849: Update leader for static observers
  • KSTORAGE-82: Tiered storage topic deletion support
  • KAFKA-8972 (2.4 blocker): correctly release lost partitions during consumer.unsubscribe() (#7441)
  • MINOR: improve logging of tasks on shutdown (#7597)
  • CPKAFKA-3718: Add ObserverReplicasCount metric
  • CPKAFKA-3843: Observers should be passed through in ListPartitionReassignment
  • CONFLUENT: Update observers_test for static observers
  • CPKAFKA-3719: Observer support in the controller
  • CPKAFKA-3721: Update topic describe to use metadata for observers
  • CPKAFKA-3816: Use CaughtUpReplicasCount for consistency
  • CPKAFKA-3552: Move license validator out of authorizer and make broker wide
  • CPKAFKA-3852: Add/remove listener metrics during dynamic listener update
  • KAFKA-9105: Add back truncateHead method to ProducerStateManager (#7599)
  • MINOR: disable use of ConfluentLeaderAndIsrRequest in 5.4.x
  • KAFKA-8992: Redefine RemoveMembersFromGroup interface on AdminClient (#7478)
  • KAFKA-9038: Allow creating partitions while a reassignment is in progress (#7582)
  • KAFKA-8972: Need to flush state even on unclean close (#7589)
  • KAFKA-9089: Reassignment should be resilient to unexpected errors (#7562)
  • KAFKA-8729: Add upgrade docs for KIP-467 on augmented produce response (#7522)
  • MINOR: Re-implement NewPartitionReassignment#of() (#7592)
  • MINOR: Add documentation for foreign-key joins (KIP-213) (#7535)
  • METRICS-734: Categorize Kafka API calls for Audit Logs
  • CONFLUENT: Disable audit logger for CCloud
  • METRICS-736: Log Error for audit log of consumption on audit log topic
  • MINOR: Use LeaderAndIsr instead of ConfluentLeaderAndIsr when tiered is enabled
  • CPKAFKA-3830: Include observer information in MetadataResponse
  • CPKAFKA-3730: Fix createTopicPolicy incompatibility in ce-kafka
  • CPKAFKA-3723: Message changes to add observer tagged field

Confluent Cloud Plugins

  • CC-6911: Fix healthcheck integration tests by replacing deprecated configs
  • MINOR: fix shaded use of guava
  • DG-78: Clear quota manager during IMPORT
  • MINOR: Add maxId to TenantIncrementalIdGenerator
  • CC-4841: Use CE Kafka across the repo
  • CC-4345: Implement multitenant store

Confluent Rebalancer

  • CPKAFKA-3550: Add topic placement constraint support (#21)
  • MINOR: Force broker config updates when using rebalancer’s KafkaAdmin…
  • MINOR: Use latest NewPartitionReassignment API
  • MINOR: Update usage of NewPartitionReassignment class
  • CPKAFKA-3814 (3/3): Implements rebalancer’s KafkaAdminClient
  • CPKAFKA-3814 (1/3): Adds KafkaAdmin interface to abstract admin imple…
  • CPKAFKA-3292: Remove usages of ZkUtils and org.I0Itec.ZkClient in favor of KafkaZkClient, AdminZkClient

Confluent Security Plugins

  • MMA-6119: Fixed error that C3 cannot view topic messages through UX
  • SEC-672: Removed authentication for endpoint /v1/metadata/id
  • SEC-693: Added ServerMetadataResource to initializeSchemaRegistryResourceActio…
  • MINOR: Modify schema validation integration tests to use correct exception
  • CC-7272: Corrected ConnectSecurityExtensionConfig to also support old prefixes
  • MINOR: Pass custom metadata configs to underlyring Rest clients
  • SEC-595: Pass all configs to underlying RestClient
  • CPKAFKA-4100: Update the cached KafkaRestContext.getConfig() with latest token details
  • CC-6522: Use Kafka cluster of ‘UNKNOWN’ if one cannot be found by the herder
  • DG-118: Catch exception when constructing error message
  • SEC-649: Added BC non-FIPS into package of package-schema-registry-security-pl…
  • MINOR: Add connect-runtime as test dependency required by LicenseManager
  • MINOR: Change Kafka test dependencies from CCS to CE
  • MINOR: Fix SslPrincipalAliasTest by setting replication factor to 1 for the license topic
  • MINOR: fix shaded use of guava
  • CC-6671: Serve JSON-parseable error messages instead of raw text
  • CC-6154: Add RBAC credentials injection to Connect security extension
  • SEC-614: add debug logging to jwtloginservice
  • DG-72: Stop DDoS from a batch of bad records to schema validation plugin
  • DG-46: Add /v1/metadata/id endpoint to display SR cluster information
  • Sec-237: Remove www-authenticate from MDS authenticate responses
  • SEC-366: Excluded Bouncy Castle FIPS jars in every module under repo confluent-security-plugins
  • MINOR: Override getAllSubjectsById in AbstractSchemaRegistryClient
  • MINOR: Test to check if describeConfigs doesn’t include broker-related configs for a topic
  • MINOR: distinguish between REST client and server errors
  • SEC-323: Instantiate JwtLoginService with preloaded PublicKey
  • CPKAFKA-3170, pt. 2: Add integration tests for schema validation on broker side
  • CPKAFKA-3166: Check the configs for validate key and / or value
  • MINOR: use configs from the broker
  • CC-5404: Change 401 responses to 403 in Schema Registry security plugin
  • CPKAFKA-3169: Add a broker interceptor for schema validation
  • CPKAFKA-3239: Set token credentials in MdsBearerAuthProvider Initialization
  • MINOR: Remove kafka-security-plugins since this is now included in confluent-server
  • CC-5038: Add additional args to account for upstream changes

Control Center

  • MMA-6158: use jetty HttpClient instead of jersey Client
  • MMA-6196: Call AdminClient.close in a separate thread
  • MMA-6165: Remove multi-cluster SR associated default for 5.4.x release
  • MMA-5947: support for SR url kafka rest api
  • CPKAFKA-4127: Changes to match broker config update for metadata apps
  • MMA-5925: only add schema registry config sets with valid url field
  • MMA-5938: Revert “change the min.isr from 2 to 1 for internal topics (#1948)”
  • MMA-4739: add RBAC props to each properties file
  • MMA-5602: alerting endpoints for topic/group access lists
  • CPKAFKA-3580: Add get replica status REST endpoint
  • METRICS-315: change min.isr from 2 to 1 for internal topics
  • MMA-5424: MDS visibility endpoint was moved
  • MMA-5422: Add ui autoupdate default c3 config property
  • MMA-5408: Add a flag for replicator monitoring feature
  • MMA-5300: Make the cluster list page usable for large # of clusters
  • MMA-4991: Show offsets and size per partition
  • MMA-4990: Fetch under replicated metric time-series data as well
  • MMA-4014:(Broker metrics): Expose new endpoint for broker partition and replicas
  • MMA-4094: Topic replica and partition time series data
  • MMA-4016:(feat) Broker metrics page - System
  • MMA-4015:(feat) Broker metrics page - Disk usage
  • MINOR: Use confluent-security-plugins as the parent pom to use CE Kafka

Confluent Platform Documentation

  • DOCS-3265 - Fixed bad anchors in Upgrade topic
  • DOCS-3257 - Fix typos on Control Center Manage Schemas for Topics
  • Remove RBAAC “coming soon” from Cloud limits
  • DOCS-3229 - Update docs and changelog for GCP, Azure - functions connectors
  • DOCS-3252 - Operator 5.4 updates (upgrades, migrating Helm 2.0 to 3.0, migrating 5.3 to 5.4, Quick Start)
  • DOCS-3211 - Remove warnings re: License tab availability for expired trial license 5.4.1 release
  • DOCS-2290 - Update Confluent Control Center alerts nav
  • Fix indentation for Operator
  • Update RBAC quick start
  • Add correct upstream Kafka version
  • Updated confluent CLI requirement
  • DOCS-3173 - More MRR failover and replica placement details
  • DOCS-2990 - 5.4.0 release notes
  • 5.4.0 changelog
  • DOCS-3215 - Adding details around the 5.4 Kubernetes Operator and tiered storage support
  • DOCS-3198 - Add RBAC perms for tables created with persistent query
  • DOCS-1120 - Rev’d title and cleaned up some grammar, spelling error
  • DOCS-1845 - Fixed typo in include file
  • DOCS-3188 - Added example to config audit log to another cluster
  • DOCS-3114 - Update ZooKeeper upgrade docs
  • DOCS-3181 - Attribute name correction
  • Update screenshot and description of metrics
  • Adding changelog for Replicator for 5.4
  • Fixed dates on site footer
  • Centralized ACL limitations
  • Fix include path for 5.4.x
  • Add CLI Windows support info to docs
  • Import Kafka 2.4 configs
  • Update changelogs for Confluent Platform connectors
  • Add 5.4 changelog for KSQL
  • Updated interoperability information for 5.4
  • Add info to clients doc about producer
  • Update quick starts for 5.4.x
  • mTLS plus RBAC, update confluent authorizer values
  • Add Control Center changelog for 5.4.0
  • Schema Registry Control Center screen snaps, workflow update for 5.4.x
  • 5.4.x Upgrade Doc
  • Control Center Connector RBAC Security tab Creds caveats
  • Add 5.4.0 changelog for Schema Registry
  • In https MDS config how certs are passed in CLI login
  • Note not to use keytool to gen PEM files
  • Add summary for Streams API changes
  • REST Proxy SASL should use SASL_SSL in bootstrap.servers
  • Revised ACL section in Security tutorial for centralized ACLs
  • Tiered Storage Preview Docs
  • Update license screenshots for version # and new OTA updates tab
  • Topic Viewer Audit Log search needs to use User:<username> instead of <username>
  • Remove preview from RBAC entry in license graphic
  • Added MDS-related error code
  • Control Center RBAC Manage Roles
  • RBAC limitations
  • New troubleshooting token auth section in MDS config
  • Add link from Control Center configs to Replicator monitoring topic in Control Center
  • Fix docs formatting Streams
  • Port over of Kafka docs naming DSL topologies for Streams
  • Add upgrade from docs for Kafka 2.4
  • Audit logs
  • Config Kafka to connect to MDS
  • MDS - Adapt Swagger spec for Sphinx doc tool
  • Clarify source and replicated topics, origin/destination, respectively
  • Replicator fix sed commands and port numbers in tutorial

Kafka Connect

  • REP-609: Fixing potential null pointer exception
  • REP-600: Update admin client to support default number of partitions and replication factors when creating topics
  • REP-24: ZK license manager change
  • REP-381: Replicator should not automatically create source topics
  • REP-475: do not add connect worker configurations if dest.kafka.bootstrap.servers is peresent
  • REP-462: Resolve Replicator Master build failure
  • REP-450: filters default configs out when applying topic configs to dest
  • REP-30: Refactor the “admin-client-with-zookeeper” test to NOT have a Kafka binary dependency
  • REP-379: added configuration provider that can infer dest.kafka.* config from the Connect Worker
  • REP-436: refactor non timestamp specific class to a new replicator-common module
  • REP-30: Better identify errors during startup()/shutdown() from KafkaServerStartable within test
  • REP-322: REP-323 REST extensions for Replicator monitoring
  • REP-231: Refactor ConsumerTimestampInterceptor out into a separate JAR
  • REP-321: Initial commit for JMX metrics implementation
  • CC-5364: Remove all package excludes except connect-api
  • CC-5364: Exclude core kafka from package
  • CC-4930: Adjust expected calls in case schedulePeriodicRefreshMetadata gets scheduled by NewReplicatorAdminClient
  • MINOR: Upgrade bugfix versions of dependencies to avoid CVEs
  • MINOR: Pin hadoop-common version to 2.7.7 and exclude htrace-core dependency
  • CC-6565: Use parquet.codec and add compression options for Parquet format
  • CC-6766 Handle null-valued records in all three output types
  • CC-4969 MINOR: Remove Unused S3Storage object from TopicPartitionWriter
  • CC-6801: Retry effectively on retriable failures
  • CC-6872: Modify GCS credentials logic to follow same pattern as BigQuery
  • CC-6766 Handle null-valued records in all three output types
  • CC-5186: Uncaught errors breaking connector config
  • CC-4686: Add credentials provider method to GCS config
  • CC-6565: Add parquet.codec property to enable compression in storage sink connectors

Metadata Service

  • SEC-701: Change confluent.http.server.listeners default host to 0.0.0.0
  • SEC-696: MDS - Adapt Swagger spec for Sphinx doc tool
  • CPKAFKA-4191: Make ClusterId json consistent with other components
  • MINOR: Use config prefix “confluent.metadata.” in MdsKafkaTokenIntegrationTest
  • CPKAFKA-4165: Add endpoint for schema registry URLs on default HTTP server
  • MINOR: Remove –confluent.metadata.rest.” prefix usage from MdsKafkaTokenIntegrationTest
  • SEC-655: MDS - Actually fill out / write the swagger spec
  • SEC-595: Test Token callback handlers with MDS over tls
  • CPKAFKA-4127: Changes to match broker config update for metadata apps
  • SEC-655 - MDS: Actually fill out / write the swagger spec
  • CPKAFKA-4130: Change MDS listeners default value to match ce-kafka
  • SEC-678: mds-in-a-box is not supposed to be part of CP 5.4 tarball
  • CPKAFKA-4111: Detaches the MetadataServer life-cycle from RBAC provider
  • SEC-651: Add waitForCondition check in AclMigrationClientTest
  • SEC-667: MDS - Remove unused endpoint - list groups for user
  • SEC-655: MDS - Actually fill out / write the swagger spec
  • MINOR: Define the JVM memory setting to be used by Maven
  • CPKAFKA-3985: Rename RBAC and ACL providers to CONFLUENT and ZK_ACL
  • MINOR: Put back test dependency on Audit logs
  • SEC-601: MDS : Remove 5.3 preview moved endpoints
  • SEC-655: MDS - Actually fill out / write the swagger spec
  • SEC-652: Optimize visibility lookup endpoint implementation in metada…
  • SEC-653: MDS : Suppress output of empty path elements in Scopes
  • HOTFIX: Remove duplicate methods from Stub classes
  • CPKAFKA-3853: Update ACL tests as per changes in CPKAFKA-3853
  • SEC-635: MDS : MDS in a Box
  • SEC-636: MDS - Remove Zookeeper dependency from build / artifacts
  • SEC-439: MDS Swagger : Make swagger available w/out extra jars
  • SEC 632: Excluded BC non-FIPS jars from packages
  • SEC-623: MDS Depend on ce-audit log code and test bootstrap issue
  • SEC-600: Improve error messaging for empty Scope body
  • SEC-575: - Rbac Api : Rolebinding Ability per Type
  • SEC-620: Added resource pattern validations
  • SEC-574: Rbac List Users and Groups
  • SEC-228: Add Scope ResourceType check
  • MINOR: Update tests to verfify SSL configuration in RestAuthorizer/MdsAclClient
  • SEC-229: Add validation for Role to ResourceTypes
  • SEC-227: Add basic validators for roleName, operation and resource type
  • SEC-615: Fix any timing issue involved in AclMigrationClientTest
  • SEC-227: Add basic validators for roleName, operation and resource type
  • CPKAFKA-3456: Add integration test for Acl Migration
  • SEC-355: MDS : Cleanup auth logic
  • SEC-437: Rbac Management: REST API to return all principal-managed role-bindings resources
  • SEC-576: MDS Define valid resourceTypes per clusterType
  • SEC-597: MDS : Move endpoints for 5.4
  • SEC-438: Rbac Management: REST API to return guidelines to create role bindings
  • CPKAFKA-3758: Move Acl API Request/Response classes to ce-kafka and add Integration tests
  • MMA-5319: changed logout path for clearing cookies
  • MINOR: Update tests to await Kafka/MDS bootup correctly
  • CPKAFKA-3677: Fix the tests as per changes in CPKAFKA-3677
  • MINOR: Comment out flaky test
  • SEC-435: Rbac Management: REST API to return all principal-managed clusters
  • MINOR: Update to new AclRule and Authorizer logic
  • MMA-3278: add support for cookie authentication to mds
  • SEC-436: migrate mds jenkins bot to mds-alerts
  • CPKAFKA-3457: Mark optional fields as not required
  • MINOR: Increase test timeout on more tests
  • SEC-333: MDS : Resource Create Specific Endpoint for the UI
  • SEC-367: Rbac Management : My Rolebindings Api
  • MINOR: Increase test timeout
  • MINOR: Fix timing flaky test by adding an Await
  • MINOR: Update SecurityMetadataAuthorizerTest to work with upstream
  • CPKAFKA-3457: Add APIs for Acl management
  • SEC-366: Configure compile and build environment for FIPS
  • SEC-356: MDS : Supress setting of the Token Public key for MDS
  • SEC-296: Ignore WS context, CPKAFKA-3316 optimize configuration for token auth (default)
  • SEC-233: MDS Test Infra : Find / build a better testing LDAP
  • SEC-300: MetadataService pom.xml unnecessarily overriding versions from common pom
  • Minor: Update Stub Server given change to token.auth.enabled
  • SEC-268: Increase default token lifetime to 6 hours
  • CONFLUENT: Rename SuperAdmin role to SystemAdmin
  • CONFLUENT: Update SuperUser to SuperAdmin, remove references to superuser

Secrets

  • CC-7331: Update name of secrets topic
  • CC-7305: close admin client in tests to reduce flakiness
  • CC-5587: Remove use of ZkUtils from secret registry

Community Features

Apache Kafka

  • PR-7904 - KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
  • PR-7839 - KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
  • PR-7795 - KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795)
  • PR-7765 - MINOR: Bump system test version from 2.2.1 to 2.2.2 (#7765)
  • PR-7789 - MINOR: Increase the timeout in one of Connect’s distributed system tests (#7789)
  • PR-7777 - MINOR: Convert last streams join test to TTD (#7777)
  • PR-7771 - KAFKA-9184: Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771)
  • PR-7763 - KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
  • PR-7752 - MINOR: Convert Stream-StreamJoin Integration Test to TTD (#7752)
  • PR-7723 - KAFKA-9190: Close connections with expired authentication sessions (#7723)
  • PR-7773 - KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
  • PR-7768 - KAFKA-9258 Check Connect Metrics non-null in task stop (#7768)
  • PR-7772 - KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
  • PR-7748 - KAFKA-9231: Streams Threads may die from recoverable errors with EOS enabled (#7748)
  • PR-7769 - KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
  • PR-7760 - KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
  • PR-7758 - KAFKA-9244: Update FK reference should unsubscribe old FK (#7758)
  • PR-7621 - KAFKA-9123 Test a large number of replicas (#7621)
  • PR-7446 - KAFKA-8981 Add rate limiting to NetworkDegradeSpec (#7446)
  • PR-7722 - MINOR: Updated StreamTableJoinIntegrationTest to use TTD (#7722)
  • PR-7532 - KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
  • PR-7729 - MINOR: Rat should ignore generated directories (#7729)
  • PR-7693 - MINOR: Fix Streams EOS system tests by adding clean-up of state dir (#7693)
  • PR-6031 - KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
  • PR-7691 - HOTFIX: safely clear all active state in onPartitionsLost (#7691)
  • PR-7707 - HOTFIX: Fix unit tests that failed when executed from IDE (#7707)
  • PR-7701 - KAFKA-9198: Complete purgatory operations on receiving StopReplica (#7701)
  • PR-7650 - MINOR: Fetch only from leader should be respected in purgatory (#7650)
  • PR-7704 - KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
  • PR-7695 - KAFKA-9196: Update high watermark metadata after segment roll (#7695)
  • PR-7629 - KAFKA-9072: Add Topology naming to the dev guide (#7629)
  • PR-7688 - KAFKA-9175: Update MirrorMaker 2 topic/partition metrics (#7688)
  • PR-7671 - MINOR: refactor replica last sent HW updates due to performance regression (#7671)
  • PR-7680 - HOTFIX: Fix compilation error in DelayedFetchTest (#7680)
  • PR-7678 - KAFKA-9171: Handle ReplicaNotAvailableException during DelayedFetch (#7678)
  • PR-7666 - MINOR: Change topic-exists log for CreateTopics from INFO to DEBUG (#7666)
  • PR-7673 - MINOR: Update Epoch field descrition in FetchRequest api doc (#7673)
  • PR-7662 - KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
  • PR-7613 - KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
  • PR-7659 - MINOR: Return null in key mapping of committed (#7659)
  • PR-7647 - KAFKA-9140: Also reset join future when generation was reset in order to re-join (#7647)
  • PR-7645 - HOTFIX: remove reference to unused Assignment error code (#7645)
  • PR-7638 - MINOR: Rework NewPartitionReassignment public API (#7638)
  • PR-7642 - HOTFIX: fix bug in VP test where it greps for the wrong log message (#7642)
  • PR-7631 - HOTFIX: Remove from restoringByPartition once restored (#7631)
  • PR-7620 - KAFKA-8972 (2.4 blocker): TaskManager state should always be updated after rebalance (#7620)
  • PR-7617 - KAFKA-8972 (2.4 blocker): bug fix for restoring task (#7617)
  • PR-7608 - KAFKA-8972 (2.4 blocker): clear all state for zombie task on TaskMigratedException (#7608)
  • PR-7441 - KAFKA-8972 (2.4 blocker): correctly release lost partitions during consumer.unsubscribe() (#7441)
  • PR-7597 - MINOR: improve logging of tasks on shutdown (#7597)
  • PR-7599 - KAFKA-9105: Add back truncateHead method to ProducerStateManager (#7599)
  • PR-7478 - KAFKA-8992: Redefine RemoveMembersFromGroup interface on AdminClient (#7478)
  • PR-7582 - KAFKA-9038: Allow creating partitions while a reassignment is in progress (#7582)
  • PR-7589 - KAFKA-8972: Need to flush state even on unclean close (#7589)
  • PR-7562 - KAFKA-9089: Reassignment should be resilient to unexpected errors (#7562)
  • PR-7522 - KAFKA-8729: Add upgrade docs for KIP-467 on augmented produce response (#7522)
  • PR-7592 - MINOR: Re-implement NewPartitionReassignment#of() (#7592)
  • PR-7535 - MINOR: Add documentation for foreign-key joins (KIP-213) (#7535)
  • PR-7579 - MINOR: Add toString to PartitionReassignment (#7579)
  • PR-7570 - MINOR: Rename brokers to replicas in the reassignment API (#7570)
  • PR-7574 - MINOR: Check against empty replicas in AlterPartitionReassignments (#7574)
  • PR-7550 - MINOR:Upgrade guide updates for KIP-479 (#7550)
  • PR-7547 - MINOR: KIP-307 upgrade guide docs (#7547)
  • PR-7557 - MINOR: don’t require key serde in join materialized (#7557)
  • PR-7564 - KAFKA-8943: Move SecurityProviderCreator to org.apache.kafka.common.security.auth package (#7564)
  • PR-7542 - KAFKA-9041: Flaky Test LogCleanerIntegrationTest#testIsThreadFailed (#7542)
  • PR-7527 - MINOR: Add metrics in operations doc for KIP-429 / KIP-467 (#7527)
  • PR-7553 - MINOR: fix typo in TestInputTopic.getTimestampAndAdvance (#7553)
  • PR-7421 - KAFKA-8962: Use least loaded node for AdminClient#describeTopics (#7421)
  • PR-7541 - KAFKA-9058: Lift queriable and materialized restrictions on FK Join (#7541)
  • PR-7500 - MINOR: Add ability to wait for all instances in an application to be RUNNING (#7500)
  • PR-7451 - MINOR: Augment log4j to add generation number in performAssign (#7451)
  • PR-7531 - KAFKA-9004: Prevent older clients from fetching from a follower (#7531)
  • PR-7544 - MINOR: Upgrade zk to 3.5.6 (#7544)
  • PR-7513 - MINOR: AbstractRequestResponse should be an interface (#7513)
  • PR-7511 - Kafka-8950: Fix KafkaConsumer Fetcher breaking on concurrent disconnect (#7511)
  • PR-7534 - MINOR: log reason for fatal error in locking state dir (#7534)
  • PR-7309 - KAFKA-8884: class cast exception improvement (#7309)
  • PR-7460 - KAFKA-8104: Consumer cannot rejoin to the group after rebalancing (#7460)
  • PR-7517 - KAFKA-9000: fix flaky FK join test by using TTD (#7517)
  • PR-7529 - KAFKA-8496: System test for KIP-429 upgrades and compatibility (#7529)
  • PR-7537 - KAFKA-9053: AssignmentInfo#encode hardcodes the LATEST_SUPPORTED_VERSION (#7537)
  • PR-7536 - MINOR: Improve FK Join docs and optimize null-fk case (#7536)
  • PR-7315 - KAFKA-8340, KAFKA-8819: Use PluginClassLoader while statically initializing plugins (#7315)
  • PR-6514 - KAFKA-7981: Add fetcher and log cleaner thread count metrics (#6514)
  • PR-7515 - MINOR: Update Kafka Streams upgrade docs for KIP-444, KIP-470, KIP-471, KIP-474, KIP-528 (#7515)
  • PR-7518 - KAFKA-9032: Bypass serdes for tombstones (#7518)
  • PR-7483 - KAFKA-8897: Warn about no guaranteed backwards compatibility in RocksDBConfigSetter (#7483)
  • PR-7488 - MINOR: Provide better messages when waiting for a condition in test (#7488)
  • PR-7392 - KAFKA-8945/KAFKA-8947: Fix bugs in Connect REST extension API (#7392)
  • PR-7491 - KAFKA-9014: Fix AssertionError when SourceTask.poll returns an empty list (#7491)
  • PR-7514 - KAFKA-9033: Use consumer/producer identity in generated clientId (#7514)
  • PR-7501 - KAFKA-9030: Document client-level (a.k.a. instance-level) metrics (#7501)
  • PR-7462 - MINOR: code and JavaDoc cleanup (#7462)
  • PR-7470 - KAFKA-8122: Fix Kafka Streams EOS integration test (#7470)
  • PR-7521 - MINOR: remove unused import in QueryableStateIntegrationTest (#7521)
  • PR-7519 - KAFKA-4422 / KAFKA-8700 / KAFKA-5566: Wait for state to transit to RUNNING upon start (#7519)
  • PR-7305 - KAFKA-8813: Refresh log config if it’s updated before initialization (#7305)
  • PR-7475 - KAFKA-8725: Improve LogCleanerManager#grabFilthiestLog error handling (#7475)
  • PR-7497 - MINOR: Increase spotBugs max heap to 2 GB (#7497)
  • PR-7437 - KAFKA-8671: NullPointerException occurs if topic associated with GlobalKTable changes (#7437)
  • PR-7463 - MINOR: unify calls to get committed offsets and metadata (#7463)
  • PR-7510 - MINOR: clarify wording around fault-tolerant state stores (#7510)
  • PR-7495 - KAFKA-9020: Streams sub-topologies should be sorted by sink -> source relationship (#7495)
  • PR-7503 - KAFKA-9029: Flaky Test CooperativeStickyAssignorTest.testReassignmentWithRand: bump to 4 (#7503)
  • PR-7497 - MINOR: Increase spotBugs max heap to 2 GB (#7497)
  • PR-7484 - MINOR: Improve assert in testCreateTopicsResponseMetadataAndConfig (#7484)
  • PR-7494 - HOTFIX: fix checkstyle in Streams system test (#7494)
  • PR-7494 - HOTFIX: fix checkstyle in Streams system test (#7494)
  • PR-7472 - KAFKA-8743: Flaky Test Repartition{WithMerge}OptimizingIntegrationTest (#7472)
  • PR-7472 - KAFKA-8743: Flaky Test Repartition{WithMerge}OptimizingIntegrationTest (#7472)
  • PR-7486 - MINOR: ListPartitionReassignmentsResponse should not be entirely failed when a topic-partition does not exist (#7486)
  • PR-7486 - MINOR: ListPartitionReassignmentsResponse should not be entirely failed when a topic-partition does not exist (#7486)
  • PR-7167 - KAFKA-8729, pt 3: Add broker-side logic to handle the case when there are record_errors and error_message (#7167)
  • PR-7167 - KAFKA-8729, pt 3: Add broker-side logic to handle the case when there are record_errors and error_message (#7167)
  • PR-7479 - MINOR: fix compatibility-breaking bug in RequestHeader (#7479)
  • PR-7479 - MINOR: fix compatibility-breaking bug in RequestHeader (#7479)
  • PR-7458 - MINOR: Change topic zNode’s new reassigning fields names to snake case in order to be consistent with other fields (#7458)
  • PR-7438 - MINOR: Fix stale comment in partition reassignment javadoc (#7438)
  • PR-7458 - MINOR: Change topic zNode’s new reassigning fields names to snake case in order to be consistent with other fields (#7458)
  • PR-7438 - MINOR: Fix stale comment in partition reassignment javadoc (#7438)
  • PR-7451 - MINOR: Augment log4j to add generation number in performAssign (#7451)
  • PR-7469 - MINOR: Just one put and flush to generation rocksDB File in RocksDBStoreTest (#7469)
  • PR-7452 - KAFKA-8991: Enable scalac optimizer (#7452)
  • PR-7406 - KAFKA-8954: Topic existence check is wrongly implemented in the DeleteOffset API (KIP-496) (#7406)
  • PR-7406 - KAFKA-8954: Topic existence check is wrongly implemented in the DeleteOffset API (KIP-496) (#7406)
  • PR-7464 - KAFKA-8932: Add tag for CreateTopicsResponse.TopicConfigErrorCode (KIP-525) (#7464)
  • PR-7464 - KAFKA-8932: Add tag for CreateTopicsResponse.TopicConfigErrorCode (KIP-525) (#7464)
  • PR-7393 - KAFKA-8944: Fixed KTable compiler warning. (#7393)
  • PR-7429 - KAFKA-8964: Rename tag client-id for thread-level metrics and below (#7429)
  • PR-7468 - MINOR: remove unused imports in Streams system tests (#7468)
  • PR-7388 - KAFKA-7190: Retain producer state until transactionalIdExpiration time passes (#7388)
  • PR-7388 - KAFKA-7190: Retain producer state until transactionalIdExpiration time passes (#7388)
  • PR-7449 - KAFKA-8983: AdminClient deleteRecords should not fail all partitions unnecessarily (#7449)
  • PR-7449 - KAFKA-8983: AdminClient deleteRecords should not fail all partitions unnecessarily (#7449)
  • PR-7461 - MINOR: Modified Exception handling for KIP-470 (#7461)
  • PR-7461 - MINOR: Modified Exception handling for KIP-470 (#7461)
  • PR-7105 - KAFKA-7245: Deprecate WindowStore#put(key, value) (#7105)
  • PR-7105 - KAFKA-7245: Deprecate WindowStore#put(key, value) (#7105)
  • PR-7459 - HOTFIX: Hide built-in metrics version (#7459)
  • PR-7453 - KAFKA-8985: Add flexible version support to inter-broker APIs (#7453)
  • PR-7386 - KAFKA-8179: Part 7, cooperative rebalancing in Streams (#7386)
  • PR-7386 - KAFKA-8179: Part 7, cooperative rebalancing in Streams (#7386)
  • PR-7453 - KAFKA-8985: Add flexible version support to inter-broker APIs (#7453)
  • PR-7455 - MINOR: Bump version to 2.5.0-SNAPSHOT (#7455)
  • PR-7454 - MINOR: Update documentation.html to refer to 2.4 (#7454)
  • PR-7378 - KAFKA-8233: TopologyTestDriver test input and output usability improvements (#7378)
  • PR-7325 - KAFKA-8885: The Kafka Protocol should Support Optional Tagged Fields (#7325)
  • PR-7383 - MINOR: Gracefully handle non-assigned case in fetcher metric (#7383)
  • PR-7445 - MINOR: Rework error messages in ConsumerGroupCommand (#7445)
  • PR-7416 - KAFKA-8934: Introduce instance-level metrics for streams applications (#7416)
  • PR-7447 - MINOR: just remove leader on trunk like we did on 2.3 (#7447)
  • PR-7442 - KAFKA-8974: Trim whitespaces in topic names in sink connector configs (#7442)
  • PR-5527 - KAFKA-3705 Added a foreignKeyJoin implementation for KTable. (#5527)
  • PR-7371 - KAFKA-6290: Support casting from logical types in cast transform (#7371)
  • PR-6914 - KAFKA-8523 Enabling InsertField transform to be used with tombstone events (#6914)
  • PR-7405 - MINOR: Make TopicDescription’s other constructor public (#7405)
  • PR-7432 - MINOR: Start correlation id at 0 in SaslClientAuthenticator (#7432)
  • PR-7285 - KAFKA-8558: Add StreamJoined config object to join (#7285)
  • PR-6843 - KAFKA-8447: New Metric to Measure Number of Tasks on a Connector (#6843)
  • PR-7310 - KAFKA-8804: Secure internal Connect REST endpoints (#7310)
  • PR-7430 - KAFKA-5609: Connect log4j should also log to a file by default (KIP-521) (#7430)
  • PR-7403 - KAFKA-7772: Dynamically Adjust Log Levels in Connect (#7403)
  • PR-7436 - HOTFIX: don’t throw if upgrading from very old versions (#7436)
  • PR-7423 - KAFKA-8649: Send latest commonly supported version in assignment (#7423)
  • PR-7431 - MINOR: Do not falsely log that partitions are being reassigned on controller startup (#7431)
  • PR-7407 - MINOR: Add provider name of SSLContext to debug log. (#7407)
  • PR-7417 - MINOR: Shutdown RockDB metrics recording trigger thread (#7417)
  • PR-7404 - KAFKA-8887: Use purgatory for ACL updates using async authorizers (#7404)
  • PR-7354 - KAFKA-8595: Support deserialization of JSON decimals encoded in NUMERIC (#7354)
  • PR-7418 - KAFKA-8807: Flaky GlobalStreamThread test (#7418)
  • PR-7377 - KAFKA-8896: Check group state before completing delayed heartbeat (#7377)
  • PR-7412 - MINOR: Mark RocksDBStoreTest as integration test (#7412)
  • PR-7150 - KAFKA-8729 pt 2: Add error_records and error_message to PartitionResponse (#7150)
  • PR-7376 - KAFKA-8927: Deprecate PartitionGrouper interface (#7376)
  • PR-7401 - KAFKA-8609: Add rebalance-latency-total (#7401)
  • PR-7419 - MINOR: Adjust logic of conditions to set number of partitions in step zero of assignment (#7419)
  • PR-7352 - KAFKA-8911: Using proper WindowSerdes constructors in their implicit definitions (#7352)
  • PR-7411 - MINOR: Update Jackson to 2.10.0 (#7411)
  • PR-7408 - MINOR: Add missing + in LogSegment.toString (#7408)
  • PR-7353 - KAFKA-8471: Replace control requests/responses with automated protocol (#7353)
  • PR-7353 - KAFKA-8471: Replace control requests/responses with automated protocol (#7353)
  • PR-7394 - MINOR: Don’t generate unnecessary strings for debug logging in FetchSessionHandler (#7394)
  • PR-7402 - MINOR:fixed typo and removed outdated varilable name (#7402)
  • PR-7397 - KAFKA-8934: Create version file during build for Streams (#7397)
  • PR-7382 - KAFKA-8319: Make KafkaStreamsTest a non-integration test class (#7382)
  • PR-7380 - KAFKA-8907: Return topic configs in CreateTopics response (KIP-525) (#7380)
  • PR-7379 - MINOR: Address review comments for KIP-504 authorizer changes (#7379)
  • PR-7372 - MINOR: add versioning to request and response headers (#7372)
  • PR-6362 - KAFKA-7273: Extend Connect Converter to support headers (#6362)
  • PR-7340 - MINOR: improve the Kafka RPC code generator (#7340)

MQTT Rest Proxy

  • PR-43 - MINOR: Upgrade netty version to 4.1.42

Rest Utils

  • PR-166 - MINOR: Log kafka exceptions for debugging
  • PR-160 - Enable support for multiple servlets on a single Jetty Server (#155)
  • PR-156 - Enable auto ssl keystore reload
  • PR-157 - Move shutdownLatch.countDown back to Application for better readability
  • PR-155 - Enable Rest Utils to provide multiple servlet contexts
  • PR-154 - Add error log with thrown exception
  • PR-151 - Update Jetty version to 9.4.20.v20190813 and Jersey version to 2.28
  • PR-149 - Include separate metrics for each HTTP error code
  • PR-147 - CC-5576: Use SslContextFactory.Server to disable server-side hostname verification
  • PR-144 - Suppress the Jetty server version header

Confluent REST Proxy Changelog

Version 5.4.0

  • PR-546 - CPKAFKA-2590: Bump ZK connection timeout to avoid test flakes
  • PR-580 - MINOR: Include zkclient dependency to kafka-rest-scala-consumer shaded jar
  • PR-587 - Add API to get beginning offset, end offset and offset for time
  • PR-601 - CLIENTS-1454: Propagate Schema Registry properties to KafkaAvroDecoder

Version 5.1.0

  • PR-511 - CPKAFKA-1876: Log cause of failure to produce messages in tests
  • PR-512 - Fix race condition when expiration thread iterates over partially-initialized consumer
  • PR-503 - Fix location of jars for dev testing
  • PR-481 - CPKAFKA-1681: Add long polling functionality
  • PR-508 - CPKAFKA-1868: Add more consumer tests
  • PR-507 - CPKAFKA-1871: Fix problem where ReadTaskSchedulerThread would loop too often
  • PR-504 - CPKAFKA-1865: Performance degrades drastically after running for extended periods
  • PR-495 - CPKAFKA-1794: Assign leader ID -1 on offline partitions in the topic describe API
  • PR-477 - Fix SimpleConsumerManager properly return requested offsets for compressed topic
  • PR-474 - MINOR: Fix checkstyle path
  • PR-466 - Change Consumer threading model to better handle rebalances
  • PR-459 - MINOR: Add utility createConsumerProperties() test method since it was removed from AK
  • PR-452 - Fix checkstyle to pass on 8.5 after upgrade from 6.19
  • PR-439 - Temporarily pin Kafka and Confluent versions
  • PR-438 - Don’t use deprecated RecordMetadata constructor

Version 5.0.0

  • PR-439 - Temporarily pin Kafka and Confluent versions
  • PR-438 - Don’t use deprecated RecordMetadata constructor
  • PR-434 - Fix usage of internal Jersey utilities that no longer exist
  • PR-428 - Use ExplicitGCInvokesConcurrent instead of disabling explicit GC

Version 3.3.0

  • PR-305 - Use max.block.ms instead of metadata.fetch.timeout.ms since the latter was deprecated and now removed in KAFKA-3763
  • Ability to add resources like filter through a custom extension and provide a custom context
  • PR-296 - Interceptor configuration options

Version 3.2.2

  • PR-303 - CLIENTS-304: Fix AvroConsumerRecord constructors and handling of topic for v1 consumer API.

Version 3.2.1

  • PR-285 - CLIENTS-281: Fix KafkaConsumerReadTask’s locking, handling of multiple polls, invalid handling of old ConsumerTimeout exceptions, and incorrect generic type parameters.
  • PR-277 - Documentation fixes

Version 3.2.0

  • PR-256 - Update ClusterTestHarness to use o.a.k.common.utils.Time.
  • PR-255 - clarified the key/value format limitations based on user questions
  • PR-264 - Fix build to work post KIP-103
  • PR-265 - Follow up for KIP-103 changes that fixes ProducerPool’s extraction of endpoints to use the security protocol in the URL instead of the listener name.
  • PR-268 - modified readme to use v2. also bumped installation up for usability
  • PR-271 - Updated quickstart and configuration docs for V2 and for security
  • PR-258 - REST Proxy security
  • PR-274 - added field ‘topic’ to tools/ConsumerPerformance.java for system test
  • PR-272 - added description of JVM parameters for SASL configuration

Version 3.1.2

No changes

Version 3.1.1

No changes

Version 3.1.0

  • PR-239 - Require bash since we use some bashisms and fix a copyright.
  • PR-235 - fix typo command line options -help check
  • PR-222 - fixing issue #91
  • PR-230 - Issue #229: Add blurb on Jetty jmx metrics
  • PR-214 - Small readme.md fix
  • PR-127 - Fix to ConsumerManager tests AND then handling of max bytes
  • PR-203 - Fix a typo that cause non-corresponding logger name
  • PR-202 - fix the implementation of topicExists

Version 3.0.0

  • PR-164 - 2.x merge to master
  • PR-166 - Bump version to 3.0.0-SNAPSHOT and Kafka dependency to 0.10.0.0-SNAPSHOT
  • PR-171 - Fix build to handle rack aware changes in Kafka.
  • PR-174 - Update CoreUtils usage to match kafka updates
  • PR-189 - Minor fixes for compatibility with newest 0.10.0 branch.
  • PR-192 - Minor fixes for compatibility with newest 0.10.0 branch.
  • PR-202 - fix the implementation of topicExists
  • PR-205 - Rearrange quickstart and use topics from earlier steps in requests for metadata so the example output will exactly match real output when starting from an empty cluster.

Version 2.0.1

  • PR-144 - Fix for Race condition when consuming while topic is being created (Issue #105)
  • PR-158 - Update Kafka version to 0.9.0.1-cp1

Version 2.0.0

  • PR-64 - Reduce integration test time.
  • PR-66 - Add support for SimpleConsumer-like access (Issue #26)
  • PR-67 - Handle conflicting IDs and separate IDs used in the REST Proxy and by Kafka’s consumer implementation.
  • PR-78 - Remove kafka from list of production directories to include in CLASSPATH.
  • PR-89 - JSON message support
  • PR-96 - Fixed log4j and daemon flag bugs in kafka-rest-run-class based on fix from schema-registry.
  • PR-99 - Require Java 7
  • PR-101 - rest-utils updates
  • PR-103 - Issue 94 rename main
  • PR-108 - Clarify partitioning behavior for produce requests
  • PR-117 - Update to Kafka 0.9.0.0-SNAPSHOT and make adjustments to work with updated ZkUtils.
  • PR-122 - Use x.y.z versioning scheme (i.e. 2.0.0-SNAPSHOT)
  • PR-123 - Updated args for JaasUtils.isZkSecurityEnabled()
  • PR-125 - Use Kafka compiled with Scala 2.11

Control Center Changelog

Version 5.4.0

  • Added UI for managing RBAC role assignments.
  • Added Replicator monitoring.
  • Added Auto-update UI feature.
  • Added Cluster Overview page.
  • Added Broker Metrics page.
  • Added ability to delete topics.
  • Added ability to upload and download connector configurations.
  • Added partition size to topic overview page.
  • Added time series data for offline topic partitions, under replicated partitions, and out-of-sync replicas.
  • Various enhancements and bug fixes to RBAC enforcement, Connect, KSQL, and License Manager.
  • Bug fixes for backend HTTP client.

Version 5.3.0

  • Redesigned the Control Center UI, including KSQL and Connect enhancements. New pages show cluster overview, topics pages, and consumers index.
  • RBAC enforcement.
  • Topic message browser enhancements, including jump to timestamp and download results in JSON format.
  • Alert enhancements that added Slack and PagerDuty webhook notifications, a new Consumer lead alert, pause and resume global alerts actions, and fast navigation to Alerts features from the Alerts bell icon.

Version 5.2.0

  • Added multiple Connect support.
  • Added multiple KSQL support.
  • Added broker configs editing.
  • Added Schema Registry editing.
  • Added license management UI.
  • Improved message browsing.
  • Improved KSQL UI.
  • Improved scalability.

Version 5.1.0

  • Browser support for Chrome, Firefox, and Safari. For more information, see Web Browsers.
  • A new, more readable light theme.
  • Many bug fixes and UX improvements.
  • Added configuration confluent.controlcenter.license.manager=_confluent-controlcenter-license-manager-5-1-0. (It was confluent.controlcenter..license.manager = _confluent-controlcenter-license-manager5-1-0 in first 5.1.x release, but it is fixed now.) No need for customers to override this configuration.

Version 5.0.0

  • Added KSQL interface to create/view streams, tables, and queries.
  • Added topic inspection to read messages from a topic.
  • Added consumer group message lag viewing and alerting.
  • Added Schema Registry viewing.
  • Added read only view of broker configurations.
  • Added feature flags for variety of user accessible features.

Version 4.1.0

  • Added ability to create and edit topics in Control Center.
  • Added topic filtering and ability to hide Apache Kafka®/Confluent Platform internal topics from the UI.
  • Added new alert to warn when Control Center stops receiving data from a Kafka cluster.
  • Added a restricted user type that prevents editing or creating of topics, alerts, or connectors.

Version 4.0.1

  • Made calls to Connect more resilient.
  • Fixed an issue that would prevent chart highlighting on System Health charts.
  • Fixed an issue that could cause Control Center to improperly warn about improper log dirs on brokers.
  • Fixed an issue that could cause Control Center to improperly alert about active controller count.

Version 4.0.0

  • Fixed an issue that could cause long request times if a cluster had brokers with noncontiguous IDs.
  • Additional logging during startup.
  • Fixed an issue that could cause Control Center to show blank graphs when data was present.
  • Set and enforce TimestampType for internal Control Center topics at startup.
  • Fixed an issue that caused us to link to an invalid consumer group on Streams Monitoring.
  • Fixed an issue where Broker Monitoring could check for data last update time on wrong cluster.
  • Allow overriding ConfluentMetricsReporter default producer settings.
  • Allow using a file for confluent.license config.
  • Docker image: allow passing license using CONTROL_CENTER_CONFLUENT_LICENSE environment variable.
  • Fixed an issue that resulted in Offline and Under Replicated Partitions data to be swapped.
  • Fixed an issue that could cause Control Center to shutdown if it lost connection to Kafka.
  • Fixed an issue that could cause Control Center to hang if it encountered an error creating topics on startup.

Version 3.3.1 (EOSL)

  • Allow text input in date picker.
  • Fixed an issue that could cause unsubmittable forms.
  • Show renamed cluster value rather than base clusterid on alerts page.
  • Added alert banner on Broker Monitoring to indicate a cluster may be down.
  • Scalability increase for Broker Monitoring to reduce number of messages sent to Kafka.
  • Fixed an issue where 100% Network Pool Usage was displayed instead of 0%.
  • Allow graceful shutdown of Control Center during startup.
  • Fixed a race condition that could happen if you start Control Center and your Kafka broker at the same time.

Version 3.3.0 (EOSL)

  • Sped up restore time.
  • Reduced the size of old data over long periods of time.
  • Upgraded to use new admin client (KIP-117).
  • Alert UI should render better alert information.
  • Triggers are createable directly though the UI on System Health.
  • Massive performance improvements to Stream Monitoring UI.
  • Fixed various time range display/selection bugs; persist time ranges though navigation.
  • Nonconfigurable form values being sent to Connect via UI should remain and be visible prior to submit.

Version 3.2.2 (EOSL)

  • Added shutdown script.
  • Force faster compaction and cleanup on changelog topics.
  • Several Streams improvements to reduce state-store restoration time after unclean shutdown.
  • Reduce the volume of data to be restored to state-stores on startup.
  • Fixed possible concurrency problem in setting RocksDB configuration options.
  • Fixed possible deadlock on exception after streams startup.

Version 3.2.1 (EOSL)

  • Enabled HTTP compression by default.
  • Fixed possible deadlock on streams exception.
  • Fixed fd leak when communicating with Kafka.
  • Update defaults for monitoring producers to be more resilient.
  • Prevent inadvertently creating misconfigured topics during startup during a failure.
  • Check Kafka version at startup.
  • Reduce minimum startup time.
  • More efficient duplicate checking to reduce write amplification.
  • Handle ‘hidden’ connect configs. This allows Control Center to manage Connectors that do not fully enumerate their configs.
  • Ensure users only see invalid license warning if their license is invalid.
  • Fixed topic display bug that would prevent topic details from showing.
  • Reduced logging in js.
  • Allow users to close missing data modal dialogs.
  • Better rendering of System Health metrics especially when some data is missing.
  • Better validation of triggers, actions, and connector configs.
  • Faster processing of trigger and action creation and update.
  • Corrected some misleading labels on System Health.
  • Better error handling around Connect communications.
  • Change default timespan in UI to 4 hours.
  • Check and warn about errors on Topic Management page.

Version 3.2.0 (EOSL)

  • Add capabilities to monitor brokers instrumented with Confluent Metrics Reporter.
  • Enable alerting on broker and topic metrics.
  • Add UI for viewing topic information.
  • Add status UI for tracking internal progress.
  • Fixed bug where we could inadvertently show license expired message.
  • Fixed bugs with alert trigger creation in UI.
  • Fixed issue with parsing certain error conditions from Connect.

Version 3.1.0 (EOSL)

  • Enable gathering stream monitoring data from multiple clusters.
  • Support alerting on stream monitoring conditions.
  • Ability to restrict UI access via user login.

Version 3.0.1 (EOSL)

  • Improved data fetching from the browser.
  • Handle authenticated/authorized Kafka clusters.
  • Bug fix for running in daemon mode.
  • More efficient topic creation at startup.
  • Reduced number of necessary topics/rocksdbs for aggregations.
  • Reduced total amount of stored data.
  • Fixed timing issues when receiving delayed data.
  • Configs updates to prefix Kafka/rest configs.
  • Added rate limit for logging to prevent logsize blowup.
  • Allow deleting connector in UI.

Release notes for Control Center UI auto-updates

Release notes for each interim version will be listed here by their release version ID as they become available. You can also view the release notes for an auto-update in the Updates tab.

Version 5.4.0

KSQL Changelog

Version 5.4.0

Detailed Changlog

  • PR-3980 - feat : Porting #3879, #3696 fixes for pull queries against secure clusters
  • PR-3952 - fix: fix issues with multi-statement requests failing to validate
  • PR-3819 - fix: /query rest endpoint should return valid JSON (BREAKING CHANGE)
  • PR-3820 - fix: pull queries available on /query rest & ws endpoint
  • PR-3778 - feat: add flag to disable pull queries (MINOR)
  • PR-3724 - feat: Added UDFs ENTRIES and GENERATE_SERIES
  • PR-3730 - fix: apply filter before flat mapping in logical planner
  • PR-3716 - feat: Implement describe and list functions for UDTFs
  • PR-3690 - feat: Implement schemaProvider for UDTFs
  • PR-3703 - fix: Remove duplicate ksql-common dependency in ksql-streams pom
  • PR-3695 - fix: quoted identifiers for source names
  • PR-3687 - feat: Implement user defined table functions
  • PR-3691 - fix: allow streams topic prefixed configs
  • PR-3683 - feat: Implement complex expressions for table functions
  • PR-3694 - fix: fix broken map coersion test case
  • PR-3685 - feat: Support multiple table functions in queries
  • PR-3639 - Don’t use rewrites to handle struct field access
  • PR-3589 - feat: Implement EXPLODE(ARRAY) for single table function in SELECT
  • PR-3588 - feat: support numeric json serde for decimals
  • PR-3613 - Combine rowkey rowtime steps in KS topology
  • PR-3592 - feat: Udf Invoker benchmark
  • PR-3612 - feat: expression support for insert values
  • PR-3609 - fix: Created new test for running topology generation manually from t…
  • PR-3605 - fix: do not allow inserts into tables with null key
  • PR-3599 - fix: be more lax on validating config
  • PR-3601 - fix: fix parsing issue with LIST property overrides
  • PR-3528 - fix: drop TopicDescriptionFactory class
  • PR-3501 - feat: add health check endpoint
  • PR-3531 - fix: support NULL return values from CASE statements
  • PR-3570 - feat: expose query status through EXPLAIN
  • PR-3540 - fix(3524): improve pull query error message (MINOR)
  • PR-3492 - feat: pass auth header to connect client for RBAC integration
  • PR-3550 - fix: Revert named stores in expected topologies, disable naming stores from StreamJoined, re-enable join tests.
  • PR-3529 - fix(3525): SET should only affect statements after it
  • PR-3516 - fix: standardize KSQL up-casting
  • PR-3467 - feat: change /metadata REST path to /v1/metadata
  • PR-3479 - Unique(ish) query Id per pull query
  • PR-3477 - feat: support quoted identifiers in column names
  • PR-3518 - feat: add request logging
  • PR-3499 - fix: fix Avro schema validation
  • PR-3490 - feat: add configurable MetricsReporter to MetricCollectors
  • PR-3466 - fix: changes required for compatibility with KIP-479
  • PR-3494 - feat: build materializations from the physical plan
  • PR-3486 - fix: make sure use of non-threadsafe UdfIndex is synchronized
  • PR-3488 - fix: Use the correct Immutable interface
  • PR-3464 - fix: improve print topic error message when topic does not exist
  • PR-3474 - fix: race condition in KsStateStore
  • PR-3471 - fix: Kafka Streams does not like negative time
  • PR-3470 - fix: fix NPE when printing records with empty value (MINOR)
  • PR-3473 - feat(static): switch partial datetime parser to use UTC by default
  • PR-3461 - Feat(static): forward auth header when proxying static query calls to another KSQL node
  • PR-3418 - feat: build ks app from an execution plan visitor
  • PR-3449 - fix: better error message if tz invalid in datetime string
  • PR-3439 - feat(static): support ROWKEY in the projection of static queries
  • PR-3438 - feat(static): allow windowed queries without bounds on rowtime
  • PR-3425 - feat: add KsqlUncaughtExceptionHandler and new KsqlRestConfig for enabling it
  • PR-3402 - fix: fix test for KIP-307 final PR
  • PR-3442 - fix(3441): stabilize the StaticQueryFunctionalTest
  • PR-3417 - feat: use coherent naming scheme for generated java code
  • PR-3435 - feat(static): support for partial datetimes on WindowStart bounds
  • PR-3437 - fix: address upstream change in KafkaAvroDeserializer (revert previous fix)
  • PR-3412 - fix: support UDAFs with different intermediate schema
  • PR-3432 - fix: do not allow WITH clause to be created with invalid WINDOW_SIZE
  • PR-3421 - fix: Remove dependencies on test-jars from ksql-functional-tests jar
  • PR-3430 - feat(static): fail on ROWTIME in projection
  • PR-3433 - fix: Rename Delimiter:parse(char c) to Delimiter.of(char c)
  • PR-3424 - fix: rename Delimiter.of(String) to Delimiter.parse(String)
  • PR-3427 - feat: Ignore local scratch QTT file called scratch.json from git
  • PR-3428 - feat(static): unordered table elements and meta columns serialization
  • PR-3423 - fix: typo in static query WHERE clause example
  • PR-3422 - feat(static): allow logical schema to have fields in any order
  • PR-3385 - feat: custom comparators for array, map and struct
  • PR-3411 - feat: remove equalsIgnoreCase from all Name classes (MINOR)
  • PR-3354 - feat: update query id generation to use command topic record offset
  • PR-3404 - feat: Implement ROUND() UDF
  • PR-3393 - feat: Implement user defined delimiter for value format
  • PR-3391 - feat: move aggregations to plan builder
  • PR-3407 - fix: error message when UDAF has STRUCT type with no schema
  • PR-3341 - feat(cli): add the feature to turn of WRAP for CLI output
  • PR-3397 - fix: better error message when users enter old style syntax for query
  • PR-3400 - Move statement re-write logic into engine
  • PR-3398 - fix(3356): struct rewritter missed EXPLAIN
  • PR-3386 - feat: back out Connect auto-import feature
  • PR-3361 - feat: move joins to plan builder
  • PR-3302 - feat: Add support for average aggregate function
  • PR-3360 - feat: move setup of the sink to plan builders
  • PR-3313 - feat: add REST /metadata path to display KSQL server information (replaces /info)
  • PR-3359 - feat: move groupBy into plan builders
  • PR-3369 - feat(static): static select support
  • PR-3362 - feat: move selectKey impl to plan builder
  • PR-3381 - Fix: Consistency between inserted rowkey and key values when inserting into a table with a key
  • PR-3368 - Avoid unnecessary repartition of stream on GROUP BY ROWKEY
  • PR-3372 - fix: address upstream change in KafkaAvroDeserializer
  • PR-3355 - Fix: NPE when coercing on INSERT + improve err msg + better coercion test coverage
  • PR-3337 - fix: improve error message on query or print statement on /ksql (MINOR)
  • PR-3345 - Better error message for missing join fields
  • PR-3346 - feat: move filters to plan builder
  • PR-3334 - feat: move toTable kstreams calls to plan builder
  • PR-3343 - feat: add offset to QueuedCommand and flag to Command
  • PR-3353 - feat: add logs/ to .gitignore (MINOR)
  • PR-3340 - feat(static): initial drop of static query functionality
  • PR-3351 - fix: switch AdminClient to be sandbox proxy
  • PR-3339 - fix:Case is now preserved for an unquoted topic name
  • PR-3328 - fix: KSQL should warn when forcing CSAS object names to upper cas…
  • PR-3326 - fix: band-aid around RestQueryTranslationTest
  • PR-3318 - fix: should be able to parse empty STRUCT schema (MINOR)
  • PR-3320 - feat(static): add forEach() to KsqlStruct (MINOR)
  • PR-3322 - feat(static): add custom jackson JSON serde for handling LogicalSchema
  • PR-3314 - fix: revert default listeners value and update docs
  • PR-3300 - feat(static): initial syntax for static queries

Version 5.3.2

Detailed Changlog

  • PR-3167 - feat: add KsqlRocksDBConfigSetter to bound memory and set num threads

Version 5.3.1

Detailed Changlog

  • PR-3225 - Fix quoted reserved identifier 5.3
  • PR-3197 - fix: validate CLI using REST / instead of REST /info
  • PR-3176 - fix: default timestamp extractor override is not working
  • PR-3111 - fix: add ksql-functional-tests to the ksql package
  • PR-3147 - fix: filter null entries before creating KafkaConfigStore
  • PR-3079 - feat: add config for enabling topic access validator
  • PR-3069 - feat(rest server): add warnings when listing fails with kafka error
  • PR-3104 - fix: COLLECT_LIST can now be applied to tables
  • PR-2997 - feat: add extension for custom metrics (5.3.x)
  • PR-2996 - feat: add config for custom metrics tags (5.3.x)

Version 5.3.0

KSQL 5.3.0 includes new features, including:

  • Drop the requirement that CREATE TABLE statements must have a KEY set in their WITH clause. This is now an optional optimization to avoid unnecessary repartition steps. See Github issue #2745 for more info.
  • Improved handling of KEY fields. The KEY field is an optional copy of the Kafka record’s key held within the record’s value. Users can supply the name of the field that holds the copy of the key within the WITH clause. The improved handling may eliminate unnecessary repartition steps in certain queries. Please note that preexisting persistent queries, e.g. those created via CREATE TABLE AS SELECT ... or CREATE STREAM AS SELECT ... or INSERT INTO ..., will continue to have the unnecessary repartition step. This is required to avoid the potential for data loss should this step be dropped. See Github issue #2636 for more info.
  • INSERT INTO ... VALUES is now supported, with standard SQL syntax to insert rows to existing KSQL streams/tables. To disable this functionality, set ksql.insert.into.values.enabled to false in the server properties.
  • CREATE STREAM and CREATE TABLE will now allow you to create the topic if it is missing. To do this, specify the PARTITIONS and optionally REPLICAS in the WITH clause.
  • CREATE STREAM AS SELECT ... and CREATE TABLE AS SELECT ... statements now use the source topic partitions and replica counts for the sink topic if the config properties are not set in the WITH clause. In case of JOIN, the left hand side topic is used. This deprecates ksql.sink.partitions and ksql.sink.replicas config properties.
  • A new config variable, ksql.internal.topic.replicas, was introduced to set the replica count for the internal topics created by KSQL Server. The internal topics include the command topic and the config topic.
  • A new KSQL testing tool was added. The tool is a command line utility that enables testing KSQL statements without requiring any infrastructure, like Apache Kafka and KSQL clusters.

KSQL 5.3.0 includes bug fixes, including:

  • The ROWTIME of the row generated when a JOIN encounters late data was previous the ROWTIME of the late event, where as now it is the max of ROWTIME of the rows involved in the join. This provides more deterministic join semantics.

  • Return values of UDF and UDAFs are now correctly marked as optional, where previously there was potential for non-optional fields, which would result in serialization issues in the presence of null values.

    This is a forward compatible change in Avro, i.e. after upgrading, KSQL will be able to read old values using the new schema. However, it is important to ensure downstream consumers of the data are using the updated schema before upgrading KSQL, as otherwise deserialization may fail. The updated schema is best obtained from running the query in another KSQL cluster, running version 5.3.

    See Github issue #2769 for more info.

  • Fixed issues with using AS keyword when aliasing sources. See #2732 for more info.

Detailed Changlog

  • PR-3061 - fix: Update version checker to use new dependencies
  • PR-3045 - fix: Create the streams state directory if it does not exist.
  • PR-3036 - fix: Reapply fix to only pass source topic input
  • PR-3027 - fix: SqlFormatter now handles windows
  • PR-2999 - Add explicit kafka core dependency for support metrics
  • PR-2948 - Using the correct props for the version checker in the headless mode.
  • PR-2937 - MINOR: Use function reference when assigning to java.func.Supplier
  • PR-2916 - Remove the hack to standarize expected topologies.
  • PR-2923 - Return correctly parametrized processing log type
  • PR-2919 - Mention ksql.streams.bootstrap.servers setting in scaling section
  • PR-2889 - Add support for checking external preconditions while still serving
  • PR-2898 - Minor: close websocket session on error
  • PR-2892 - Don’t accept empty output.
  • PR-2888 - Do not enable TopicAccessValidator if Kafka has unsupported authorizedOperations
  • PR-2893 - fix dead store in TestOptionsParserTest
  • PR-2881 - Testing tool split input
  • PR-2309 - Support -cp and -hotfix for packaging
  • PR-2882 - Backport insert into values bug fix
  • PR-2877 - Update support-metrics-common artifact version (#2874)
  • PR-2870 - Close ServiceContext when Session is closed
  • PR-2843 - Testing tool improvements/bug fixes
  • PR-2852 - Verify access authorization to Websockets endpoints
  • PR-2860 - Pipe serviceContext down to EngineContext (Fixes #2853)

Version 5.2.0

KSQL 5.2 includes new features, including:

  • Support for HTTPS.
  • Support for CASE expression: KSQL now supports CASE conditional expression in Searched form where KSQL evaluates each condition from left to right. It returns the result for the first condition that evaluates to true. If no condition evaluates to true, the result for the ELSE clause will be returned. If there is no ELSE clause, null is returned.
  • A new family of UDFs for improved handling of URIs (e.g. extracting information/decoding information), see UDF table for all URL functions
  • LIMIT keyword support for PRINT (#1316)
  • Support for read-after-write consistency: new commands don’t execute until previous commands have finished executing. This feature is enabled by default in the CLI (#2280) and can be implemented by the user for the REST API (Coordinate Multiple Requests).
  • A log of record processing events to help users debug their KSQL queries. The log can be configured to log to Kafka to be consumed as a KSQL stream. See KSQL processing log for more details.
  • Aggregation functionality has been extended. KSQL now supports:
    • GROUP BY more than just simple columns, including fields within structs, arithmetic results, functions, string concatenations and literals.
    • literals in the projection, (a.k.a the SELECT clause).
    • Multiple HAVING clauses, including the use of aggregate functions and literals.
  • Automatic compatibility management for queries in headless mode across versions. Starting with 5.2, KSQL will automatically take care of ensuring query compatibility when upgrading. This means you won’t need to worry about setting properties correctly during upgrade, as has been required for previous upgrades. Refer to the architecture documentation for details. Note that it is still up to the user to set properties correctly before upgrading to 5.2. The upgrade doc has details about the properties required to safely upgrade to 5.2.

KSQL 5.2 includes bug fixes, including:

  • Improved support for multi-line requests in interactive mode deployments. Prior to version 5.2 KSQL parsed the full request before attempting to execute any statements. Requests that contained later statements that were dependent the execution of prior statements may have failed. In version 5.2 and later, this is no longer an issue.
  • Improved support for non-interactive, “headless” mode deployments. Prior to version 5.2 KSQL parsed the full script before attempting to execute any statements. The full parse would often fail when later statements relied on the execution of earlier statements. In version 5.2 and later, this is no longer an issue.

KSQL 5.2 deprecates some features, including:

  • The use of the RUN SCRIPT statement via the REST API is now deprecated and will be removed in the next major release. (Github issue 2179). The feature circumnavigates certain correctness checks and is unnecessary, given the script content can be supplied in the main body of the request. If you are using the RUN SCRIPT functionality from the KSQL CLI you will not be affected, as this will continue to be supported. If you are using the RUN SCRIPT functionality directly against the REST API your requests will work with the 5.2 server, but will be rejected after the next major version release. Instead, include the contents of the script in the main body of your request.

Detailed Changlog

  • PR-2594 - Minor: clean up references to CURRENT_TOPOLOGY_VERSION
  • PR-2585 - Decrease Kafka Controller socket timeout in tests.
  • PR-2587 - 5.2 fix for UDFs not being passed their config.
  • PR-2584 - KSQL-2278: Ensure all SR props are passed where needed
  • PR-2537 - Fix bug in Avro schema cleanup during drop source
  • PR-2526 - Fix bug in Avro schema cleanup during cluster termination
  • PR-2532 - Adds a name to map schemas (#2504)
  • PR-2500 - SSL Support

Version 5.1.0

KSQL 5.1 includes new features, including:

  • WindowStart() and WindowEnd() UDFs
  • StringToDate() and DateToString() UDFs

Detailed Changlog

  • PR-2265 - MINOR: Fix bug encountered when restoring RUN SCRIPT
  • PR-2240 - Bring version checker improvements to 5.1.x
  • PR-2143 - Fix some bugs in recovery logic
  • PR-2150 - Currently we don’t support AS for aliasing stream/table.
  • PR-2149 - Using ksql topic name instead of Kafka topic name in topic map in metastore.
  • PR-2090 - Add WindowStart and WindowEnd UDFs (#1993)
  • PR-2075 - Disable optimizations for 5.1.x
  • PR-2051 - Preserve originals when merging configs
  • PR-2066 - Minor: Fix bug involving filters with NOT keyword.
  • PR-2056 - Added stringtodate and datetostring UDFs for 5.1.x
  • PR-2048 - Minor: Fix bug involving LIKE patterns without wildcards.
  • PR-2045 - List UDAFs for 5.1.x
  • PR-2023 - MINOR: Cause ‘ksql help’ and ‘ksql -help’ to behave the same as ‘ksql -h’ and ‘ksql –help’
  • PR-1979 - Metrics refactor + fix a couple issues
  • PR-2018 - Display stats timestamps in unambiguous format.
  • PR-1981 - ST-1153: Switch to use cp-base-new and bash-config to hide passwords by default
  • PR-1955 - Hide ssl configs and refactor KsqlResourceTest

Version 5.0.0

KSQL 5.0 includes a number of new features, including:

  • User-Defined Functions (UDF).
  • Support for nested data via the new STRUCT data type.
  • Support for writing to existing streams using the new INSERT INTO statement.
  • Stream-Stream Joins.
  • Table-Table Joins.
  • Table Aggregations.
  • A revamped REST API that will be kept compatible from this release onwards.
  • Confluent Platform Docker images for KSQL Server and CLI
  • Java 10 support.

Detailed Changlog

  • PR-1610 - Specify a namespace for avro schemas
  • PR-1588 - Add example avro schemas to the ksql packaging build
  • PR-1529 - Add a schema inference test
  • PR-1511 - Fix NPE when printing null values
  • PR-1547 - Fix the bug for null check in comparison expression code gen,
  • PR-1549 - Ensure that the list of interceptors is modifiable before mutating it
  • PR-1531 - Recommend the use of STRUCT instead of EXTRACTJSONFIELD
  • PR-1526 - Include monitoring interceptor config in streams configs
  • PR-1524 - Java 10 support
  • PR-1479 - Write configs to command topic and read back before building queries
  • PR-1512 - Data-gen now can create struct columns.
  • PR-1517 - Fix spacing in struct schema string
  • PR-1507 - Standardize on a single version for the avro-random-genator
  • PR-1481 - Consistent schema format between websocket endpoint and ksql endpoint
  • PR-1503 - Convert invalid avro field names before serializing
  • PR-1501 - sort udf names
  • PR-1500 - don’t bomb on non-existing plugin dir
  • PR-1495 - Pass schema registry client configs onto the schema registry client
  • PR-1482 - String masking functions
  • PR-1434 - Schema Translation Test + Nested Support in Query Validation Test
  • PR-1486 - list and describe updated to include aggregate functions
  • PR-1478 - Use ‘:’ for struct field reference.
  • PR-1467 - Support java based UDAFS
  • PR-1390 - Upgrade handling step 1: clean up KSQL configs
  • PR-1453 - Struct end to end
  • PR-1465 - support null literal in UDFs #1462
  • PR-1458 - add jar path to description of functions
  • PR-1457 - Simple Security manager for UDFS
  • PR-1449 - create module for UDF annotations to minimize dependencies
  • PR-1470 - Use a string for formatted time field in clickstream schema
  • PR-1448 - Add config for ext dir
  • PR-1463 - Handle rename of KafkaConsumerService to ConsumerService
  • PR-1455 - Rename SPAN to WITHIN everywhere
  • PR-1435 - Support List/Show Functions (UDFs) and Describe Function <name>
  • PR-1417 - Add new join types to KSQL
  • PR-1430 - Updated AST builder to handle dot notation.
  • PR-1442 - Simplified Security
  • PR-1445 - Prefixed output topics
  • PR-1426 - Throw exception when calling FunctionRegistry#getUdfFactory(name) if function doesn’t exist
  • PR-1429 - UDF tidy ups
  • PR-1423 - Collect metrics for UDFs
  • PR-1416 - Validate that function names are valid java identifiers
  • PR-1410 - Added query rewrite feature using AST.
  • PR-1366 - Create and load Java based UDFs from an ext directory
  • PR-1362 - Return schema description API entities as JSON objects
  • PR-1400 - Add struct support for avro by using Connect Converter API
  • PR-1406 - #1396 Fix regression issue with string concat using ‘+’ operator.
  • PR-1407 - Avoid NPE if not function exists that handles required types.
  • PR-1401 - Switch to optional schema fields since we can have null values.
  • PR-1409 - Issue #1111: Document schemaRegistryURL arg for ksql-datagen in usage…
  • PR-1383 - Using Connect data format in KSQL.
  • PR-1140 - Do schema inference before writing to output topic
  • PR-1370 - Wrap api query id strings in a class
  • PR-1368 - Using ExpressionTypeManager to detect the types for function look up.
  • PR-1379 - Disable server identification verification for SecureIntegrationTest
  • PR-1369 - Using the kafka topic name in delete topic message.
  • PR-1363 - Print an error if a message fails to produce with datagen
  • PR-1356 - New geo distance function
  • PR-1359 - handle array subsrcipts when generating function args
  • PR-1353 - Support functions with same name but different arguments
  • PR-1285 - Refactor WebSocket endpoints and add support for print topic
  • PR-1350 - Child first class loader for UDFs
  • PR-1337 - Fix the way LIMIT clauses are handled
  • PR-1321 - Make FunctionRegistry an interface and make MetaStore implement it
  • PR-1340 - Post fix udf instance names to ensure they are unique.
  • PR-1335 - Don’t ignore leading spaces when saving history
  • PR-1331 - Follow-on updates for KSQL_OPTS
  • PR-1329 - Fix run class to work with Java 10 and use ExplicitGCInvokesConcurrent
  • PR-1330 - Make KSQL_OPTS more prominent
  • PR-1301 - Minor: Admin client leak
  • PR-1315 - Consumed imports
  • PR-1277 - Ksql 1217 optionally delete kafka topic with drop statement
  • PR-1309 - Add crosslink from ‘starting KSQL server’ section to the headless mode instructions
  • PR-1114 - Struct Data Type(Part-1): New struct type, DDL statements and Describe
  • PR-1305 - MINOR: Add query validation test for stringtotimestamp with double single quote
  • PR-1303 - The DESCRIBE ACL on the __consumer_offsets topic is not required.
  • PR-1091 - Add parent reference to ast nodes
  • PR-1249 - Include function names in error message for unsupported table aggrega…
  • PR-1220 - Add external dependency on Avro Random Generator and remove its source code
  • PR-576 - Insert into implementation
  • PR-1256 - Better reporting of invalid serverAddress or connection issues in CLI
  • PR-1197 - KSQL-883: Add KafkaClientSupplier to KSQL API
  • PR-1278 - Add Preview Release information, update Latest News
  • PR-1219 - Improved error message for Stream/Table and Query relation correctness.
  • PR-1201 - Add versioning to the API (#1151)
  • PR-1210 - Fix some more rest api inconsistencies
  • PR-1136 - Table aggregations
  • PR-1163 - Minor: Fix lifecycle of AdminClient and KafkaTopicClientImpl instances.
  • PR-1208 - Add a Pull request template
  • PR-1050 - Escape discovered avro field name if it is a ksql lexer token literal #(1043)
  • PR-1198 - text is the new binary
  • PR-1147 - SourceDescription cleanup + listing with descriptions
  • PR-1179 - Shutdown cleaning should there be an error on start up.
  • PR-1174 - Fix handling of table tombstones
  • PR-1171 - Add back key-constraints
  • PR-838 - Rename ‘Kafka output topic’ to just ‘Kafka topic’ in describe ext (#838)
  • PR-1109 - Ksql 1054 better aggregation with complex expressions
  • PR-652 - Added referential integrity enforcement for streams/tables and queries
  • PR-756 - Add the ability to extract the record timestamp from a string field #646
  • PR-1130 - Return proper errors from the HTTP server endpoints
  • PR-944 - Ksql 660 schema registry clean up
  • PR-1126 - Don’t depend on the internal ‘PlainSaslServer’ class from kafka
  • PR-1103 - Return more stuff from the rest API
  • PR-1120 - update error messages when failing to parse an avro schema
  • PR-1108 - add websockets query endpoint

Schema Registry Changelog

Version 5.4.0

  • PR-1244 - DG-134: Fix NPE when schema.registry.ssl.protocol not specified
  • PR-1242 - DG-128: Allow DNS lookup policy to be configurable
  • PR-1235 - DG-114: Fix tombstone behavior that causes CPU utilization to go up
  • PR-1228 - DG-42: Upgrade to Avro 1.9.1; fixes #1122
  • PR-1225 - Remove javax.validation::validation-api dependency from Schema Registry client
  • PR-1026 - CC-3759 - Regenerate ID if in use during registration
  • PR-1207 - Add /v1/metadata/id endpoint to display Schema Registry cluster information
  • PR-1220 - Add optional query parameter to /schemas/ids/{id} to retrieve the current max ID of all schemas
  • PR-1214 - MINOR: remove type parameter on Avro deserializer
  • PR-1212 - CC-6428 - Use values to preserve defaults for SSL configuration
  • PR-1210 - MMA-5160 - Add defaultToGlobal flag for config API
  • PR-1144 - Add a configurable proxy to Schema Registry client
  • PR-1208 - Retain RestService SSL method API
  • PR-1167 - Mark API parameter for API Spec
  • PR-969 - Avro Console Consumer print.timestamp property
  • PR-957 - Ability to configure SSL truststore and keystore for Schema Registry clients
  • PR-1205 - Use constant time lookup to check subject exists
  • PR-1111 - Support reflection-based Avro serialization
  • PR-1196 - API to retrieve all subjects associated with a schema id
  • PR-1183 - MINOR: Change int to Integer for schemaRegistryPort
  • PR-1180 - MINOR: handle null in RecordNameStrategy; fixes #1165 (TopicRecordNameStrategy creates invalid names on null values)
  • PR-1179 - MINOR: add transitive levels to messages; fixes #1150 (Client reports misleading error when providing an unsupported compatibility level)
  • PR-965 - Invalid compatibility message update to include three transitive states
  • PR-1178 - Fix createParentPath issue where it was actually creating the child path
  • PR-1135 - #941. Avoid NPE on parsing Avro schemas by throwing InvalidSchemaException (which is handled as BadRequest)
  • PR-1172 - Add protected method for configuring forceOptional
  • PR-1038 - Use try-with-resources to avoid NPE in RestService
  • PR-1171 - MINOR: refactor configuration of RestService
  • PR-1170 - MINOR: close admin client in tests
  • PR-1006 - Fix issue “RecordNameStrategy throws SerializationException on null values”
  • PR-1149 - CC-5036: Use swagger-maven-plugin to generate API specifications
  • PR-1163 - CC-5283: Add temporary replacement for kafka.utils.ZkUtils
  • PR-1148 - Use the new AbstractCoordinator interface
  • PR-1140 - CC-4099: JsonErrorHandler for Schema Registry
  • PR-1088 - CC-4345: Add per-subject locking and lastOffset tracking
  • PR-1136 - Fix compilation error caused by signature change to TestUtils.createBrokerConfig
  • PR-1127 - MINOR: fix abstract coordinator usage

Version 5.3.0

  • PR-1009 - CC-3741: Remove use of IdentityHashMap
  • PR-1020 - CC-3803: Corrected support for handling defaults of logical schema types
  • PR-1018 - DOCS-319: Add reference to SR security plugin
  • PR-1064 - DEVX-768: 5.2 C3 feature focus: new Schema Registry capabilities
  • PR-1039 - DOCS-900: use compatibilityLevel for GET response examples
  • PR-1067 - DOCS-1437: Change sr-short to sr for proper substitution
  • PR-1073 - DOCS-1440: Remove CP install from SR lefthand TOC
  • PR-1065 - DOCS-1333: Add Schema Migration docs
  • PR-1055 - DOCS-1094: Schema Registry ACLs
  • PR-1060 - MINOR: Backport of: Fix checkstyle to pass on 8.5 after upgrade from 6.19 (#857)
  • PR-1074 - MINOR: Fix exception string formatting in Avro converter
  • PR-1022 - MINOR: improve err message during duplicate ID detection
  • PR-1021 - MINOR: add defensive check for tombstoning

Version 5.2.0

  • PR-1062 - CC-4091: Enable reregistration of schema with different ID
  • PR-1056 - CC-4062: Fix NPE caused after _schemas topic is compacted
  • PR-1054 - CC-4056: Pass id/version when secondary forwards register requests
  • PR-1045 - CC-4015 : Clean up tombstones from internal caches
  • PR-1043 - Support explicit names for non-string map schemas.
  • PR-1035 - CC-3956: Revert “Merge pull request #687 from sunbit-dev/canonical-schemas
  • PR-687 - Fix #28: Normalize schemas to canonical form
  • PR-1031 - CC-3930: Only send version/id if not null
  • PR-1023 - MINOR: swap deleted w/non-deleted before clearing; fix javadoc
  • PR-953 - CC-2669 Allow id and version to be passed to register; add modes
  • PR-992 - CC-3635 - Add config to specify location of static resources
  • PR-994 - Ability to pass additional topic configs to the schemas topic
  • PR-948 - Make CachedSchemaRegistryClient thread-safe
  • PR-930 - CC-3003 : Ability to inject custom headers to CachedSchemaRegistryClient
  • PR-939 - MINOR: ensure store is initialized for getAllKeys

Version 5.1.0

  • PR-933 - DOCS-989 - Fix legacy SR doc issues
  • PR-936 - DOCS-929 - Fix confusing anchor links
  • PR-938 - Duplicate Heading Update (#937)
  • PR-921 - Pass ClientDnsLookup to NetworkClient and parseAndValidateAddresses
  • PR-873 - Refactor SR to pull lookup datastructure to its own interface and Id Generation to its own interface
  • PR-888 - CC-2674: Point to the right checkstyle suppression file
  • PR-846 - CC-2367 : Tombstone deleted schemakey if the same schema is registered again in the subject
  • PR-853 - CC-2441 : Add Cyclic schema support
  • PR-861 - MINOR: Add Timer class. Change from Kafka (commit fc5f6b0e46ff81302b3e445fed0cdf454c942792)
  • PR-857 - Fix checkstyle to pass on 8.5 after upgrade from 6.19

Version 5.0.0

  • PR-837 - CC-2185: Deprecate schema.registry. prefixes on commonly modified configs and allow unprefixed versions
  • PR-834 - DOCS-381 - Rename quick start and improve navigation
  • PR-813 - Update maven-plugin-plugin to fix out of bounds error in build
  • PR-811 - Fix SchemaRegistryCoordinator for upstream changes
  • PR-810 - Remove KafkaJsonEncoder since Scala producers have been removed from Kafka
  • PR-809 - Remove KafkaAvroEncoder since the Scala producers have been removed from Kafka
  • PR-808 - Use java.util.Base64 instead of javax.xml.bind.DatatypeConverter
  • PR-807 - Fix usage of internal Jersey utilities that no longer exist
  • PR-806 - Inherit maven compiler plugin definition from common
  • PR-804 - Ability to plugin an implementation of BasicAuthCredentialProvider
  • PR-803 - CC-1958: Verify configs if createTopic fails because topic already exists
  • PR-792 - Improve GC options
  • PR-784 - Docs for Subject Name Strategy and Basic Auth
  • PR-774 - Revert “Remove ZooKeeper from title”
  • PR-773 - Remove ZooKeeper from title
  • PR-756 - CLIENTS-986 Forward Auth headers
  • PR-748 - confluent.maven.repo: use ‘https’
  • PR-731 - Issue #704 - Make Serde constructors public

Version 4.1.2

  • PR-840 - CC-2201: Handle NPE exception when tombstone messages are encountered
  • PR-839 - CC-2024 : Fix schema not found issue
  • PR-801 - CC-1902 : pass all configs to underlying Serializer/deserializer

Version 4.1.1

  • PR-792 - Improve GC options
  • PR-787 - Docs for Subject Name Strategy and Basic Auth

Version 4.1.0

  • PR-757 - Add WantedBy dependency to ensure startup order
  • PR-741 - CC-1511 : Fix for subject strategy when used with new SubjectNameStrategy.
  • PR-742 - Added delete to the client.
  • PR-738 - CLIENTS-896: Fix path to patch/series
  • PR-735 - Change to use updated Kafka internal API for JaasContext
  • PR-718 - Log properly when Exception raised on SchemaRegistryRestApplication
  • PR-723 - Merge packaging branches into master and add systemd init scripts
  • PR-545 - #544: Add null check for schema doc while adding the property “connect.enum.doc” for schema conversion of enum types
  • PR-680 - New (key|value).multi.type option for Avro serialization
  • PR-710 - Support Multiple SchemaRegistryResourceExtension
  • PR-702 - Set metrics tag for SR metrics
  • PR-694 - check kafkaTopicReader and producer not null before closing
  • PR-686 - issue#629: endpoint to return schema literal
  • PR-696 - Fix sasl inherit
  • PR-683 - Systemd
  • PR-681 - Add getAllVersions to SchemaRegistryClient
  • PR-673 - CLIENTS-729 : ability to inject BasicAuth credentials from producer/consumer SASL
  • PR-676 - CLIENTS-774 : ability to forward authorization header
  • PR-664 - CLIENTS-708 & CLIENTS-720 Fix enum type in fromConnectData
  • PR-571 - Support basic auth credentials embedded in the schema.registry.url
  • PR-657 - Add maxPollTimeout parameter to ConsumerNetworkClient constructor

Version 4.0.1

This release upgrades Jackson 2.5.4 to 2.9.4, and fixes handling of Avro enums.

  • PR-753 - Bump Jackson version
  • PR-692 - MINOR: Backport enum fixes to 4.0.x

Version 4.0.0

  • PR-686 <https://github.com/confluentinc/schema-registry/pull/686> - Added REST endpoint to return the unescaped Avro schema for a specific subject and version.

Upgrade Notes

In 4.0.0, initial creation or validation of the topic used to store schemas has been reimplemented to use native Kafka protocol requests instead of accessing ZooKeeper directly. This means that you are no longer required to have direct access to the ZooKeeper cluster backing your Kafka cluster. However, note that this also requires appropriate permissions to create topics (on first execution of Schema Registry) or describe topics and configs (on subsequent executions to validate the topic is configured correctly). If you have authentication and authorization enabled on your Kafka cluster, you must ensure your principal has the correct permissions before upgrading Schema Registry cluster. Your principal must have the following permissions:

Create Schemas Topic

Operation Resource Reason
Describe Topic: kafkastore.topic Check existence of topic
Create Cluster Create the schemas topic, set compaction policy

Validate Schemas Topic

Operation Resource Reason
Describe Topic: kafkastore.topic Check existence of topic
DescribeConfigs Topic: kafkastore.topic Validate correct compaction policy on topic

Version 3.3.0

  • Upgrade avro to 1.8.2
  • PR-561 - Use TO_AVRO_LOGICAL_CONVERTERS to convert default values that are Kafka connect logical data types to internal format which correspond to schema type. Logic copied from AvroData fromConnectData
  • PR-549 - Replace usage of deprecated ZkUtils.DefaultAcls()
  • Allow for some retries when validating that all nodes have the same primary
  • Relocate Avro serdes under a new avro package
  • Increment Magic Byte for SchemaKey and add compatibility tests
  • Add Delete Schema support
  • Added avro-serde for Kafka Streams. Pulled from the example project.
  • #506 - The AvroMessageFormatter passes byte[] to an Avro encoder, but Avro only likes ByteBuffer. So we need to ByteBuffer.wrap() instead.
  • Added optional kafkastore.group.id config to override the one automatically created by Schema Registry
  • PR-476 - Adapt to KAFKA-4636 changes: Per listener security settings overrides (KIP-103)

Version 3.2.1

  • PR-503 - CLIENTS-244: Update 3.2.0 changelog
  • PR-499 - making sure Schema Registry doesn’t start with uncompacted topic
  • PR-497 - CLIENTS-103: Fix ArrayIndexOutOfBoundsException in SchemaRegistryPerformance by counting the registration attempt even if it failed.
  • PR-493 - Fixes for CLIENTS-257
  • PR-494 - MINOR: Add compact schemas topic doc
  • PR-458 - CLIENTS-104: Add a few retries during startup to allow for slow metadata propagation after creating the _schemas topic.

Version 3.2.0

  • PR-428 - Maven Checkstyle
  • PR-425 - Logical Type support
  • PR-440 - Documentation changes to reflect pull request 415
  • PR-451 - Generalize schema incompatibility message.
  • PR-457 - Update ClusterTestHarness to use o.a.k.common.utils.Time.
  • PR-458 - CLIENTS-104: Add a few retries during startup to allow for slow metadata propagation after creating the _schemas topic.
  • PR-464 - Improve request URL building in client
  • PR-465 - Add topic to error string to make debugging easier.
  • PR-448 - Fixes the following avro issues (complex union, document preservation, output schema != input schema)
  • PR-468 - CC-443: AvroData’s caches should be synchronized for thread-safety
  • PR-473 - Fix build to work post KIP-103.
  • PR-474 - Fix broker endpoint extraction to correctly translate to the non-ListenerName version that clients use to initiate broker connections.
  • PR-472 - Handle primitive types when specific.avro.reader is true
  • PR-477 - Docchangefor3.2
  • PR-488 - Don’t re-invoke SchemaBuilder.version() and SchemaBuilder.name() if the value has already been set.
  • PR-494 - MINOR: Add compact schemas topic doc

Version 3.1.1

No changes

Version 3.1.0

Schema Registry

  • PR-429 - Moving licenses and notices to a new format, generated by an internal script.
  • PR-415 - Option to apply fully transitive schema compatibility checking
  • PR-412 - Require bash since we use some bashisms and fix a copyright.
  • PR-396 - ZooKeeper and Kafka SASL support.
  • PR-384 - Update the link to Google Java code Style
  • PR-372 - Update KafkaStore to use moved TopicExistsException class.
  • PR-373 - Added get all subjects.
  • PR-364 - Increase testing timeouts from 5000ms to 15000ms
  • PR-346 - configured log4j to write to log file

Serializers, Formatters, and Converters

  • PR-403 - CLIENTS-101: Document wire format for Avro.
  • PR-379 - Maven plugin
  • PR-355 - Adding support for Fixed data types
  • PR-352 - Fix schemas.cache.config can’t be overrided

Version 3.0.1

Schema Registry

  • PR-369 - Introducing a new config: kafkastore.bootstrap.servers
  • PR-371 - Fixing a bug where the listener port wasn’t used in ZooKeeper
  • PR-390 - Include cURL output in quickstart
  • PR-392 - Fix schemes used for listeners

Serializers, Formatters, and Converters

  • PR-363 - 3.0.x duplicate schema support
  • PR-365 - Fix API compatibility regression from #363
  • PR-378 - Support null namespace for Array of Records

Version 3.0.0

Schema Registry

  • PR-212 - change the documentation on port to have a high priority and list it higher up in the docs
  • PR-298 - Bump version to 3.0.0-SNAPSHOT and Kafka dependency to 0.10.0.0-SNAPSHOT
  • PR-300 - Using the new 0.9 Kafka consumer.
  • PR-302 - Fix build to handle rack aware changes in Kafka.
  • PR-305 - Update to match changed methods in CoreUtils
  • PR-317 - Change the ‘host.name’ importance to high
  • PR-319 - KafkaStore SSL support.
  • PR-320 - API reference uses ‘integer’ Avro type which isn’t supported. ‘int’ is supported.
  • PR-329 - https support.

Serializers, Formatters, and Converters

  • PR-264 - Add null checks to json serializer/deserializer
  • PR-274 - Add support for Avro projections in decoders
  • PR-275 - Fixed references to confluent common version
  • PR-276 - Unit tests and bugfix for NPE when using nested optional fields
  • PR-278 - Test cases for optional nested structs
  • PR-280 - Fix fromConnectData for optional complex types
  • PR-290 - Issue #284 Cannot set max.schemas.per.subject due to cast exception
  • PR-297 - Allows any CharSequence implementation to be considered a string
  • PR-318 - Minor cleanup
  • PR-323 - Fix #142 - handle parsing non-json responses in the RestService
  • PR-332 - Add status storage topic to Connect Avro sample config.

Version 2.0.1

Schema Registry

  • PR-286 - Update Kafka version to 0.9.0.1-cp1.

Version 2.0.0

Schema Registry

  • PR-141 - Incorrect path to log4j.properties file for simple zip file layout
  • PR-143 - schema-registry-start does not work with -daemon argument
  • PR-152 - Added point about Google code style to Readme.
  • PR-163 - Expose more information when registry fails to start
  • PR-165 - Add compatibility support for SchemaRegistryClient
  • PR-167 - Update the versionCache with a new schemaVersionMap for a subject
  • PR-169 - Use correct URL to update compatibility setting of a subject
  • PR-180 - GH-177: Remove unneeded content-type headers for GET ops in quickstart and README
  • PR-184 - Support multiple registry URLs in client
  • PR-186 - Rename LocalSchemaRegistryClient to make it clear it is only intended to be used as a mock in tests. Fixes #185.
  • PR-187 - Correct example response for POST /subjects/(string: subject)/versions
  • PR-188 - Address GH-168; enable unit testing of CachedSchemaRegistryClient
  • PR-195 - Fixed typo in Exception message
  • PR-196 - Require Java 7
  • PR-197 - Enable test code sharing
  • PR-198 - Update jersey, jackson and junit versions to match rest-utils
  • PR-202 - Correct minor docs error in example response
  • PR-203 - Issue 194 rename main
  • PR-207 - Issue #170: PUT /config/(string: subject) should return 4xx for unknown subjects
  • PR-210 - Issue #208: Several RestApiTest test cases don’t test proper exception behavior
  • PR-219 - Update Kafka version to 0.8.3-SNAPSHOT so we can start developing using upcoming 0.8.3 features.
  • PR-237 - Update uses of ZkUtils to match changes made in KAFKA-2639.
  • PR-240 - Fixes Typo in the docs. ‘actsas’ -> ‘acts as’
  • PR-242 - Fixed Schema Registry build against kafka trunk
  • PR-243 - Use x.y.z versioning scheme (i.e. 2.0.0-SNAPSHOT)
  • PR-245 - Fix mvn assembly setup
  • PR-252 - Use Kafka compiled with Scala 2.11
  • PR-257 - Updated classpath in schema-registry-run-class to reflect changes in pom.xml
  • PR-258 - CC-53: Add worker configs for Avro Kafka Connect that integrates with Schema Registry.

Serializers, Formatters, and Converters

  • PR-146 - Added decoder to return a specific Avro record from bytes.
  • PR-162 - Add implementation of new Deserializer interface.
  • PR-192 - Add new module for Kafka JSON serialization stuff
  • PR-193 - Add more JSON codec support
  • PR-200 - Switch serializer config classes to use AbstractConfig from confluent-common instead of from Kafka.
  • PR-222 - Add AvroConverter in new copycat-avro-converter jar to convert Copycat and Avro data.
  • PR-234 - Add AvroConverter support for Decimal, Date, Time, and Timestamp logical types.
  • PR-235 - Add caching of schema conversions in AvroData and AvroConverter.
  • PR-247 - Update for Copycat -> Kafka Connect renaming.
  • PR-251 - Add tests of conversion of null values from Kafka Connect and fix handling of null for int8 and int16 types.
  • PR-254 - Ensure AvroConverter passes through null values without adding schemas and that deserialized null values are converted to SchemaAndValue.NULL.
  • PR-255 - CC-44: Include version when deserializing Kafka Connect data.
  • PR-256 - Encode null values for schemaless array entries and map keys and values as an Anything record.