Confluent Platform Component Changelogs

This topic provides changelogs for the individual Confluent Platform components.

Version 6.1.1

Community Features

Common

  • PR-318 - ST-5085: Update resolver plugin
  • PR-314 - MMA-9930: Sync Google Guava version.

Kafka

  • PR-10311 - KAFKA-12462: Proceed with task revocation in case of thread in PENDING_SHUTDOWN (#10311)
  • PR-10118 - KAFKA-10192: Increase maximum time to wait for worker to start in some integration tests (#10118)
  • PR-529 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes
  • PR-525 - Enable downstream validation
  • PR-10260 - Remove ReassigningPartitions metric when shutting down ReplicaManager (#10260)
  • 14d7dbb5 - KAFKA-12400: Upgrade Jetty to fix CVE-2020-27223
  • 3b7cc02d - KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
  • PR-10205 - KAFKA-12323 Follow-up: Refactor the unit test a bit (#10205)
  • PR-10016 - KAFKA-10340: Proactively close producer when cancelling source tasks (#10016)
  • PR-9990 - KAFKA-12235: Fix a bug where describeConfigs does not work on 2+ config keys (#9990)
  • PR-10170 - KAFKA-12323: Set timestamp in record context when punctuate (#10170)
  • 3f005e38 - KAFKA-12324: Upgrade Jetty to fix CVE-2020-27218
  • PR-10152 - KAFKA-12339: Add retry to admin client’s listOffsets (#10152)
  • PR-10158 - KAFKA-12343: Handle exceptions better in TopicAdmin, including UnsupportedVersionException (#10158)
  • PR-10153 - KAFKA-12340: Fix potential resource leak in Kafka*BackingStore (#10153)
  • PR-9589 - KAFKA-10710; MM2 - Create herders only if source->target.enabled=true and heartbeats are disabled (#9589)
  • PR-10122 - KAFKA-12326: Corrected regresion in MirrorMaker 2 executable introduced with KAFKA-10021 (#10122)
  • PR-10098 - KAFKA-12321: The comparison function for UUID type should be equals rather than == (#10098)
  • PR-10104 - KAFKA-12272: Fix commit-interval metrics (#10104)
  • PR-9780 - KAFKA-10021: Changed Kafka backing stores to use shared admin client to get end offsets and create topics (#9780)
  • PR-9977 - KAFKA-12310: Update ZooKeeper to 3.5.9 (#9977)
  • PR-10060 - KAFKA-10716: Persist UUID in state directory for stable processId across restarts - 2.7 (#10060)
  • PR-9902 - KAFKA-12193: Re-resolve IPs after a client disconnects (#9902)
  • PR-10032 - KAFKA-12270: Handle race condition when Connect tasks attempt to create topics (#10032)
  • PR-10040 - KAFKA-12259: Use raw config to infer the connector type when returning a connector status response (#10040)
  • PR-9987 - KAFKA-10895: Handle invalid JAAS configs (follow up fix) (#9987)
  • PR-9319 - KAFKA-10413: Allow for even distribution of lost/new tasks when multiple Connect workers join at the same time (#9319)
  • PR-9525 - KAFKA-10658: ErrantRecordReporter.report always return completed futures (#9525)
  • PR-9983 - KAFKA-8930: MirrorMaker v2 documentation (#324) (#9983)
  • PR-9671 - KAFKA-10793: Move handling of FindCoordinatorFuture to fix race condition (#9671)
  • PR-9765 - KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations (#9765)
  • PR-9947 - KAFKA-12190: Fix setting of file permissions on non-POSIX filesystems (#9947)
  • PR-9832 - KAFKA-12152; Idempotent Producer does not reset the sequence number of partitions without in-flight batches (#9832)
  • PR-9904 - KAFKA-12211: Don’t change perm for base/state dir when no persistent store (#9904)
  • PR-9923 - KAFKA-12219: Add ‘synchronized’ keyword to InMemoryKeyValueStore#[reverseRange, reverseAll] (#9923)
  • PR-9673 - KAFKA-10289: Fix failed connect_distributed_test.py (ConnectDistributedTest.test_bounce) (#9673)
  • PR-9669 - KAFKA-10792: Prevent source task shutdown from blocking herder thread (#9669)
  • PR-9806 - KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806)
  • PR-9838 - KAFKA-12153: Update producer state before updating start/end offsets after truncation (#9838)
  • PR-9583 - KAFKA-10705: Make state stores not readable by others (#9583)
  • PR-9568 - KAFKA-10689: Fix windowed FKJ topology and put checks in assignor to avoid infinite loops (#9568)
  • PR-9782 - KAFKA-10815: EosTestDriver#verifyAllTransactionFinished should break loop if all partitions are verified (#9706) (#9782)

ksqlDB

  • PR-7032 - Fix: format cast arguments with passed context (6.0.x)
  • PR-6935 - Fix: ff-4240 upgrade HttpClient version

REST Proxy

  • f46271b4 - Fix import in test.
  • PR-799 - KREST-558: Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
  • 2ffd770b - KREST-558 Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
  • PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224
  • PR-229 - KC-1456: Update Jetty to 9.4.36.v20210114
  • PR-231 - UIF-298: Upgrade Jetty to 9.4.35.v20201120
  • PR-228 - MMA-9930: Use compatible version of Guava
  • PR-222 - UIF-300: Upgrade org.apache.httpcomponents.httpclient
  • 92e91bda - Remove extra Guava version during manual merge.
  • PR-223 - UIF-302: Upgrade Guava for 5.4.x and older.
  • PR-224 - Upgrade hibernate-validator to 6.1.7.Final

Schema Registry

  • PR-1799 - DGS-1391: Fix Avro Convertor for GenericData of union with null, string, enum
  • PR-1794 - Fix String.format in compatibility errorMessages
  • PR-1795 - DGS-1221: Ensure AvroSchema.hashCode is consistent with equals
  • PR-1791 - DGS-1364: Fix compatibility for JSON empty schema
  • PR-1737 - Backport #1733 to 5.4.x
  • PR-1784 - DGS-1221: Fix AvroSchema.equals
  • PR-1779 - DGS-772: Add API annotations for /mode
  • PR-1777 - Upgrade to everit 1.12.2
  • PR-1639 - Upgrade to wire 3.6.0
  • PR-1775 - DGS-104: Fix Avro compat check
  • PR-1774 - DGS-1333: Add config to remove Java props from Avro schemas
  • PR-1770 - DGS-1315: upgrade HibernateValidator
  • PR-1769 - DGS-1314: update JetBeanKotlin to 1.4.21
  • PR-1741 - DG-1162: Add verbose option to SR maven plugin test-compatibility goal

Commercial Features

Confluent Server

  • METRICS-2907: Make HttpExporterConfig proxy passwords Password objects
  • Merge: jmh-benchmark fixes after log start offset update refactoring
  • Update log start offset for MergedLog via callback from Log to avoid mistakes
  • CONFLUENT: Make Kafka dependencies for ce-metrics compileOnly
  • CONFLUENT: Backport KAFKA-12252 and KAFKA-12262 (#3268)
  • CONFLUENT: Revert “Limit the max topic name length to 209 (#425)”
  • Re-enable downstream validation check
  • MMA-9397: Filter out confluent.ssl configs when describing topic configs
  • KAFKA-12400: Upgrade Jetty to fix CVE-2020-27223
  • KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
  • KC-1522: Do not log parsed configs containing passwords in CruiseControlStartable
  • Use AWS CLI version 1.18.* for Python 3.4 compatibility
  • [METRICS-2820] Collect metrics on message conversion (#3189) (backport 6.1.x)
  • Backport METRICS-2819: Collect metrics about log cleaner (#3184)
  • Add authentication counters to default broker include regex
  • KAFKA-12324: Upgrade Jetty to fix CVE-2020-27218
  • KC-1457: Update HttpClient to 4.5.13 and add explicit HttpClient dependency
  • CONFLUENT: Update ZooKeeper node start timeout in system tests
  • KC-924: Add event processing metrics for ZK ACLs and MDS records
  • KC-922, KC-1238: Handle writer stop exceptions to ensure new writer is re-elected

Security

  • DGS-1381: Deserialize GLOBAL_SUBJECTS_READ to GLOBAL_READ for backward compatibility

Replicator

  • REPL-1410: Changed logging for WakeupException to DEBUG
  • REPL-1386: Adding clear logging statements for source and destination operations
  • REPL-1409: Added WARN logging if destination SR deemed not ready
  • REPL-1396: Fix CVE for HttpClient
  • SEC:1820: Fix CVE for codec
  • REPL-1396: Fix CVE for aws-java-sdk-ecr

Version 6.1.0

Community Features

Common

  • PR-309 - SEC-1789 Bump bouncycastle
  • PR-306 - Bump Junit version to 4.13.1
  • PR-305 - SEC-1474—Upgraded bcpkix-jdk15on to 1.66
  • PR-301 - Upgrade scala to 2.13.3 to align with Kafka

Kafka

  • PR-9678 - KAFKA-10798; Ensure response is delayed for failed SASL authentication with connection close delay
  • PR-9702 - CVE-2020-25649: bumping jackson to patched version 2.10.5.1
  • PR-9661 - KAFKA-10776: update the doc to add version attribute in RequstsPerSec metrics
  • PR-9729 - KAFKA-10839: improve consumer group coordinator unavailable message
  • PR-9747 - KAFKA-10852: AlterIsr should not be throttled
  • PR-9723 - Cherry-pick stream producer fix 10813
  • PR-9698 - KAFKA-10811: Correct the MirrorConnectorsIntegrationTest to correctly mask the exit procedures
  • PR-8965 - KAFKA-8147: Add changelog topic configuration to KTable suppress
  • PR-9669 - KAFKA-10792: Prevent source task shutdown from blocking herder thread
  • PR-9674 - KAFKA-10665: close all kafkaStreams before purgeLocalStreamsState
  • PR-9677 - KAFKA-10799 AlterIsr utilizes ReplicaManager ISR metrics
  • PR-9648 - KAFKA-10758: ProcessorTopology should only consider its own nodes when updating regex source topics
  • PR-9634 - KAFKA-10755: Should consider commit latency when computing next commit timestamp
  • PR-9633 - KAFKA-10706; Ensure leader epoch cache is cleaned after truncation to end offset
  • PR-9613 - Cherry-pick KAFKA-10687 to 2.7
  • PR-9515 - KAFKA-10651: read offsets directly from checkpoint for uninitialized tasks
  • PR-9534 - KAFKA-10664: Delete existing checkpoint when writing empty offsets
  • PR-9529 - Revert initial principal from 2.7
  • PR-9521 - KAFKA-10638: Fix QueryableStateIntegrationTest
  • PR-9506 - KAFKA-10647; Only serialize owned partitions when consumer protocol version >= 1
  • PR-9486 - KAFKA-9381: Fix publishing valid scaladoc for streams-scala
  • PR-9464 - KAFKA-10616: Always call prepare-commit before suspending for active tasks
  • PR-9479 - KAFKA-10631: Handle ProducerFencedException on offset commit
  • PR-9270 - KAFKA-10284: Group membership update due to static member rejoin should be persisted
  • PR-9406 - KAFKA-10520; Ensure transactional producers poll if leastLoadedNode not available with max.in.flight=1
  • PR-9237 - KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
  • PR-9338 - KAFKA-10515: Properly initialize nullable Serdes with default values
  • PR-9431 - KAFKA-10426: Deadlock on session key update.
  • PR-9455 - KAFKA-10599: Implement basic CLI tool for feature versioning system
  • PR-9429 - KAFKA-10572 mirror-maker config changes for KIP-629
  • PR-9448 - KAFKA-10605: KIP-478: Deprecate old PAPI registration methods
  • PR-9383 - KAFKA-10455: Ensure that probing rebalances always occur
  • PR-9343 - KAFKA-10332: Update MM2 refreshTopicPartitions() logic
  • PR-9425 - KAFKA-10600: Connect should not add error to connector validation values for properties not in connector’s ConfigDef
  • PR-9432 - KAFKA-10559: Not letting TimeoutException shutdown the app during internal topic validation
  • PR-9422 - KAFKA-10602: Make RetryWithToleranceOperator thread safe
  • PR-9407 - KAFKA-10611: Merge log error to avoid double error
  • PR-9438 - KAFKA-10613: Only set leader epoch when list-offset version >= 4
  • PR-8150 - KAFKA-9587: Producer configs are omitted in the documentation
  • PR-9353 - KAFKA-10521; Skip partition watch registration when AlterIsr is expected
  • PR-9403 - KAFKA-10573 Update connect transforms configs for KIP-629
  • PR-9415 - KAFKA-10494: eager handling of sending old values
  • PR-9396 - KAFKA-10437: Implement new PAPI support for test-utils
  • PR-8891 - KAFKA-10143: Improve test coverage for throttle changes during reassignment
  • PR-9375 - KAFKA-10574: Fix infinite loop in Values::parseString
  • PR-9408 - KAFKA-10598: Improve IQ name and type checks
  • PR-9020 - KAFKA-10271 Performance regression while fetching a key from a single partition
  • PR-9380 - KAFKA-7334: Suggest changing config for state.dir in case of FileNotF…
  • PR-9393 - KAFKA-10028: Minor fixes to describeFeatures and updateFeatures apis
  • PR-9139 - KAFKA-9929: Support backward iterator on SessionStore
  • PR-9388 - KAFKA-10562: Properly invoke new StateStoreContext init
  • PR-9373 - KAFKA-10564: only process non-empty task directories when internally cleaning obsolete state stores
  • PR-9385 - KAFKA-9274: fix incorrect default value for task.timeout.ms config
  • PR-9247 - KAFKA-10362: When resuming Streams active task with EOS, the checkpoint file is deleted
  • PR-9001 - KAFKA-10028: Implement write path for feature versioning system (KIP-584)
  • PR-9196 - KAFKA-10402: Upgrade system tests to python3
  • PR-9280 - KAFKA-10186: Abort transaction with pending data with TransactionAbortedException
  • PR-9321 - KAFKA-9929: fix: add missing default implementations
  • PR-8910 - KAFKA-10188: Prevent SinkTask::preCommit from being called after SinkTask::stop
  • PR-9345 - KAFKA-10338; Support PEM format for SSL key and trust stores (KIP-651)
  • PR-9348 - KAFKA-10527; Voters should not reinitialize as leader in same epoch
  • PR-9099 - KAFKA-6733: Printing additional ConsumerRecord fields in DefaultMessageFormatter
  • PR-9320 - KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale.
  • PR-9347 - KAFKA-10531: Check for negative values to Thread.sleep call
  • PR-9361 - KAFKA-10535: Split ProcessorContext into Processor/StateStore/Record Contexts
  • PR-8730 - KAFKA-10048: Possible data gap for a consumer after a failover when u…
  • PR-9306 - KAFKA-10477: Enabling the same behavior of NULL JsonNodeType to MISSI…
  • PR-9281 - KAFKA-10478: Allow duplicated ports in advertised.listeners
  • PR-9354 - KAFKA-10134 Follow-up: Set the re-join flag in heartbeat failure
  • PR-9356 - KAFKA-10556: NPE if sasl.mechanism is unrecognized
  • PR-9309 - KAFKA-10503: MockProducer doesn’t throw ClassCastException when no
  • PR-9102 - KAFKA-10326 Both serializer and deserializer should be able to see th…
  • PR-9333 - KAFKA-9274: Revert deprecation of retries for producer and admin clients
  • PR-9064 - KAFKA-10205: Documentation and handling of non deterministic Topologies
  • PR-9186 - KAFKA-10277: Allow null keys with non-null mappedKey in KStreamKGlobalTable join
  • PR-9121 - KAFKA-10351: add tests for IOExceptions for GlobalStateManagerImpl/OffsetCheckpoint
  • PR-9255 - KAFKA-6585: Consolidate duplicated logic on reset tools
  • PR-9266 - KAFKA-10469: Resolve logger levels hierarchically
  • PR-9284 - KAFKA-10479 Throw exception if users try to update configs of existen…
  • PR-9332 - KAFKA-10511; Ensure monotonic start epoch/offset updates in MockLog
  • PR-8973 - KAFKA-10218: Stop reading config topic in every subsequent tick if catchup fails once
  • PR-8181 - KAFKA-9584 Headers ConcurrentModificationException
  • PR-9317 - KAFKA-10509: Added throttle connection accept rate metric (KIP-612)
  • PR-9304 - KAFKA-10502:TimestampRouter may occur threadlocal leak
  • PR-9156 - KAFKA-10077: Filter downstream of state-store results in spurious tombstones
  • PR-9337 - KAFKA-10519; Add missing unit test for VotedState
  • PR-9334 - KAFKA-10516; Disable automatic retry of THROTTLING_QUOTA_EXCEEDED errors in the kafka-topics command (KIP-599)
  • PR-9100 - Add AlterISR RPC and use it for ISR modifications
  • PR-8295 - KAFKA-9627: Replace ListOffset request/response with automated protocol
  • PR-9083 - KAFKA-9450: Follow-up; Forbid process after closed
  • PR-9312 - KAFKA-10505: Fix parsing of generation log string.
  • PR-9323 - KAFKA-10514: Fix unit test for state directory cleanup
  • PR-9130 - KAFKA-10492; Core Kafka Raft Implementation (KIP-595)
  • PR-8892 - KAFKA-10068: verify assignment performance with large cluster
  • PR-9251 - KAFKA-10459: Document IQ APIs where order does not hold between stores
  • PR-9202 - KAFKA-10401: Fix the currentStateTimeStamp doesn’t get set correctly
  • PR-9223 - KAFKA-10438 Lazy initialization of record header to reduce memory usa…
  • PR-9290 - KAFKA-10487; Fetch response should return diverging epoch and end offset
  • PR-9275 - KAFKA-10435; Fetch protocol changes for KIP-595
  • PR-9286 - KAFKA-10292: Set min.insync.replicas to 1 of __consumer_offsets
  • PR-9274 - KAFKA-10131: Remove use_zk_connection flag
  • PR-9257 - KAFKA-10463 the necessary utilities in Dockerfile should include git
  • PR-9272 - KAFKA-10458; Updating controller quota does not work since Token Bucket
  • PR-9178 - KAFKA-8362: fix the old checkpoint won’t be removed after alter log dir
  • PR-9239 - Adding reverse iterator usage for sliding windows processing (extending KIP-450)
  • PR-9264 - KAFKA-5636: Add Sliding Windows documentation
  • PR-9231 - KAFKA-10447: Migrate tools module to JUnit 5 and mockito
  • PR-8834 - KAFKA-10134: Enable heartbeat during PrepareRebalance and Depend On State For Poll Timeout
  • PR-8657 - KAFKA-8334 Make sure the thread which tries to complete delayed reque…
  • PR-9157 - KAFKA-5636: Update for KIP-450 to handle early records
  • PR-9221 - KAFKA-10436: Implement KIP-478 Topology changes
  • PR-9182 - KAFKA-10403 Replace scala collection by java collection in generating…
  • PR-9219 - KAFKA-10432: LeaderEpochCache is incorrectly recovered for leader epoch 0
  • PR-9032 - KAFKA-10259: KIP-554 Broker-side SCRAM Config API
  • PR-9215 - KAFKA-10133: MM2 readme update on config
  • PR-9122 - KAFKA-10314: KafkaStorageException on reassignment when offline log d…
  • PR-9191 - KAFKA-10355: Throw error when source topic was deleted
  • PR-9232 - KAFKA-9924: Add remaining property-based RocksDB metrics as described in KIP-607
  • PR-9138 - KAFKA-9929: Support backward iterator on WindowStore
  • PR-9023 - KAFKA-10272: Add IBM i support to “stop” scripts
  • PR-8798 - KAFKA-10098: Remove unnecessary escaping in regular expression.
  • PR-9240 - KAFKA-10456: Wrong description in kafka-console-producer.sh help
  • PR-8955 - KAFKA-10020: Create a new version of a scala Serdes without name clash (KIP-616)
  • PR-9039 - KAFKA-5636: SlidingWindows (KIP-450)
  • PR-9177 - KAFKA-9924: Add RocksDB metric num-entries-active-mem-table
  • PR-9194 - KAFKA-10384: Separate converters from generated messages
  • PR-9094 - KAFKA-10054: KIP-613, add TRACE-level e2e latency metrics
  • PR-9112 - KAFKA-10312 Fix error code returned by getPartitionMetadata
  • PR-9131 - KAFKA-10367: Allow running the Streams demo app with a config file
  • PR-9136 - KAFKA-10211: Add DirectoryConfigProvider
  • PR-9137 - KAFKA-9929: Support reverse iterator on KeyValueStore
  • PR-9146 - KAFKA-10316 Updated Kafka Streams upgrade-guide.html
  • PR-9028 - KAFKA-10035: Safer conversion of consumer timeout parameters
  • PR-7222 - KAFKA-8806 Reduce calls to validateOffsetsIfNeeded
  • PR-9197 - Revert KAFKA-9309: Add the ability to translate Message to JSON
  • PR-9148 - KAFKA-10379: Implement the KIP-478 StreamBuilder#addGlobalStore()
  • PR-9189 - KAFKA-10407: Have KafkaLog4jAppender support linger.ms and batch.size
  • PR-9144 - KAFKA-9705: (part-1) add redirection fields in the request header
  • PR-9174 - KAFKA-10395: relax output topic check in TTD to work with dynamic routing
  • PR-9172 - KAFKA-10387: Fix inclusion of transformation configs when topic creation is enabled in Connect
  • PR-8768 - KAFKA-10023: Enforce broker-wide and per-listener connection creation…
  • PR-9183 - KAFKA-10404; Use higher poll timeout to avoid rebalance in testCoordinatorFailover
  • PR-9184 - KAFKA-8033; Wait for NoOffsetForPartitionException in testFetchInvalidOffset
  • PR-9181 - KAFKA-9516; Increase timeout in testNonBlockingProducer to make it more reliable
  • PR-9108 - KAFKA-9273: Extract testShouldAutoShutdownOnIncompleteMetadata from S…
  • PR-9098 - KAFKA-9924: Prepare RocksDB and metrics for RocksDB properties recording
  • PR-9163 - KAFKA-10386; Fix flexible version support for records type
  • PR-9170 - KAFKA-10391: Overwrite checkpoint in task corruption to remove corrupted partitions
  • PR-8549 - KAFKA-9911: Add new PRODUCER_FENCED error code
  • PR-9166 - KAFKA-10388; Fix struct conversion logic for tagged structures
  • PR-8964 - KAFKA-9450: Decouple flushing state from commiting
  • PR-9004 - KAFKA-10261: Introduce the KIP-478 apis with adapters
  • PR-9140 - KAFKA-10371; Partition reassignments can result in crashed ReplicaFetcherThreads
  • PR-9060 - KAFKA-9274: Remove retries from InternalTopicManager
  • PR-9120 - KAFKA-10316: Consider renaming getter method for Interactive Queries
  • PR-9114 - KAFKA-10162; Use Token Bucket algorithm for controller mutation quota (KIP-599, Part III)
  • PR-9047 - KAFKA-9274: Remove retries for global task
  • PR-9116 - KAFKA-10341: Add 2.6.0 to system tests and streams upgrade tests
  • PR-9092 - KAFKA-10163; Define controller_mutation_rate as a Double instead of a Long
  • PR-9008 - KAFKA-9629 Use generated protocol for Fetch API
  • PR-9097 - KAFKA-10319: Skip unknown offsets when computing sum of changelog offsets
  • PR-9007 - KAFKA-10120: Deprecate DescribeLogDirsResult.all() and .values()
  • PR-9095 - KAFKA-10321: fix infinite blocking for global stream thread startup
  • PR-9081 - KAFKA-10309: KafkaProducer’s sendOffsetsToTransaction should not block infinitively
  • PR-9012 - KAFKA-10270: A broker to controller channel manager
  • PR-9089 - KAFKA-10224: Update jersey license from CDDL to EPLv2
  • PR-9066 - KAFKA-10287: Skip unknown offsets when computing sum of changelog offsets
  • PR-9086 - FIX: Remove staticmethod tag to be able to use logger of instance
  • PR-9075 - KAFKA-10306: GlobalThread should fail on InvalidOffsetException
  • PR-9034 - KAFKA-10246 : AbstractProcessorContext topic() throws NPE
  • PR-9071 - KAFKA-10305: Print usage when parsing fails for ConsumerPerformance
  • PR-9061 - MINOR: removed incorrect deprecation annotations
  • PR-9065 - KAFKA-10301: Do not clear Partition#remoteReplicasMap during partition assignment updates
  • PR-9051 - KAFKA-10268: dynamic config like “–delete-config log.retention.ms” doesn’t work
  • PR-8200 - KAFKA-5876: IQ should throw different exceptions for different errors(part 1)
  • PR-9026 - KAFKA-10274; Consistent timeouts in transactions_test
  • PR-8968 - KAFKA-10164; Throttle Create Topic, Create Partition and Delete Topic Operations (KIP-599, Part II, Admin Changes)
  • PR-8933 - KAFKA-10163; Throttle Create Topic, Create Partition and Delete Topic Operations (KIP-599, Part I, Broker Changes)
  • PR-8864 - KAFKA-9274: Mark retries config as deprecated and add new task.timeout.ms config
  • PR-9044 - KAFKA-10279; Allow dynamic update of certificates with additional SubjectAltNames
  • PR-8935 - KAFKA-10189: reset event queue time histogram when queue is empty
  • PR-9043 - KAFKA-10295: Wait for connector recovery in test_bounce
  • PR-9040 - KAFKA-10286: Connect system tests should wait for workers to join group
  • PR-8979 - KAFKA-10223; Use NOT_LEADER_OR_FOLLOWER instead of non-retriable REPLICA_NOT_AVAILABLE for consumers
  • PR-8948 - KAFKA-10174: Prefer –bootstrap-server for configs command in ducker tests
  • PR-8239 - KAFKA-9666: Don’t increase transactional epoch when trying to fence if the log append fails
  • PR-9021 - KAFKA-10257 system test kafkatest.tests.core.security_rolling_upgrade…
  • PR-8928 - KAFKA-10192: Wait for REST API to become available before testing blocked connectors in integration tests
  • PR-9003 - KAFKA-10240: Stop throwing WakeupExceptions during sink task shutdown
  • PR-8672 - KAFKA-10002; Improve performances of StopReplicaRequest with large number of partitions to be deleted
  • PR-9013 - KAFKA-10044 Deprecate ConsumerConfig#addDeserializerToConfig and Prod…
  • PR-9009 - KAFKA-6453: Document how timestamps are computed for aggregations and joins
  • PR-8994 - KAFKA-10247: Correctly reset state when task is corrupted
  • PR-9010 - KAFKA-10262: Ensure that creating task directory is thread safe
  • PR-9005 - KAFKA-10263: Do not assign standby for revoking stateless tasks
  • PR-8996 - KAFKA-10249: don’t try to read un-checkpointed offsets of in-memory stores
  • PR-8995 - Restore stream-table duality description
  • PR-8902 - KAFKA-10179: Pass correct changelog topic to state serdes
  • PR-8974 - KAFKA-10225 Increase default zk session timeout for system tests
  • PR-8934 - KAFKA-10134: Use long poll if we do not have fetchable partitions
  • PR-8990 - KAFKA-10243; ConcurrentModificationException while processing connection setup timeouts
  • PR-8989 - KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups
  • PR-8579 - KAFKA-9930: Prevent ReplicaFetcherThread from throwing UnknownTopicOrPartitionException upon topic creation and deletion.
  • PR-8962 - KAFKA-10166: checkpoint recycled standbys and ignore empty rocksdb base directory
  • PR-8983 - KAFKA-8398: Prevent NPE in forceUnmap
  • PR-8479 - KAFKA-9769: Finish operations for leaderEpoch-updated partitions up to point ZK Exception
  • PR-8944 - KAFKA-10209: Fix connect_rest_test.py after the introduction of new connector configs
  • PR-8604 - KIP-597: MirrorMaker2 internal topics Formatters
  • PR-8938 - KAFKA-10173: Use SmokeTest for upgrade system tests
  • PR-8712 - KAFKA-10006: Don’t create internal topics when LeaderNotAvailableException
  • PR-7496 - KAFKA-9018: Throw clearer exceptions on serialisation errors
  • PR-8858 - KAFKA-10153: Error Reporting in Connect Documentation
  • PR-8929 - KAFKA-4996: Fix findbugs multithreaded correctness warnings for streams
  • PR-8927 - KAFKA-10200: Fix testability of PAPI with windowed stores
  • PR-8683 - KAFKA-9893: Configurable TCP connection timeout and improve the initial metadata fetch
  • PR-8925 - KAFKA-9974: Integration test shouldApplyUpdatesToStandbyStore; Make produce-sync flush
  • PR-8917 - KAFKA-10180: Fix security_config caching.
  • PR-8905 - KAFKA-10173: Fix suppress changelog binary schema compatibility
  • PR-8926 - KAFKA-10166: always write checkpoint before closing an (initialized) task
  • PR-7577 - KAFKA-9076: support consumer offset sync across clusters in MM 2.0
  • PR-8924 - KAFKA-10198: guard against recycling dirty state
  • PR-8900 - KAFKA-10169: swallow non-fatal KafkaException and don’t abort transaction during clean close
  • PR-8887 - KAFKA-10135: Extract Task#executeAndMaybeSwallow to be a general utility function into TaskManager…
  • PR-8845 - KAFKA-10126:Add a warning message for ConsumerPerformance
  • PR-8174 - KAFKA-9439: add KafkaProducer API unit tests
  • PR-8865 - KAFKA-10168: fix StreamsConfig parameter name variable
  • PR-8896 - KAFKA-10185: Restoration info logging
  • PR-8890 - KAFKA-9891: add integration tests for EOS and StandbyTask
  • PR-8898 - KAFKA-10138: Prefer –bootstrap-server for reassign_partitions command in ducktape tests
  • PR-8850 - KAFKA-10141: Add more detail to log segment delete messages
  • PR-8822 - KAFKA-10113; Specify fetch offsets correctly in LogTruncationException
  • PR-8876 - KAFKA-10167: use the admin client to read end-offset
  • PR-8841 - KAFKA-10123 Fix incorrect value for AWAIT_RESET#hasPosition
  • PR-8853 - KAFKA-10147 MockAdminClient#describeConfigs(Collection<ConfigResource…
  • PR-8882 - KAFKA-10165: Remove Percentiles from e2e metrics
  • PR-8856 - KAFKA-10150: task state transitions/management and committing cleanup
  • PR-8880 - KAFKA-10169: Error message when transit to Aborting / AbortableError / FatalError
  • PR-8872 - Fix log message for transition from standby to active
  • PR-8873 - Avoid WARN log message when re-init from checkpoint skipped
  • PR-8849 - KAFKA-10144: clean up corrupted standby tasks before attempting a commit
  • PR-8857 - KAFKA-10157: Fix broken tests due to InterruptedException from FinalizedFeatureChangeListener
  • PR-8312 - KAFKA-9432 automated protocol for DescribeConfigs
  • PR-8764 - KAFKA-10049: Fixed FKJ bug where wrapped serdes are set incorrectly when using default StreamsConfig serdes
  • PR-8680 - KAFKA-10027: Implement read path for feature versioning system (KIP-584)
  • PR-8787 - KAFKA-10085: correctly compute lag for optimized source changelogs
  • PR-8818 - KAFKA-10086: Integration test for ensuring warmups are effective
  • PR-8069 - KAFKA-9374: Make connector interactions asynchronous
  • PR-8829 - KAFKA-10115: Incorporate errors.tolerance with the Errant Record Reporter
  • PR-8828 - KAFKA-9216: Enforce that Connect’s internal topics use compact cleanup policy
  • PR-8455 - KAFKA-9845: Warn users about using config providers with plugin.path property
  • PR-8502 - KAFKA-9066: Retain metrics for failed tasks
  • PR-8847 - KAFKA-7833: Add missing test
  • PR-8453 - KAFKA-9841: Revoke duplicate connectors and tasks when zombie workers return with an outdated assignment
  • PR-8663 - KAFKA-9985: Sink connector may exhaust broker when writing in DLQ
  • PR-8833 - KAFKA-9441: remove prepareClose() to simplify task management
  • PR-8825 - KAFKA-7833: Add StateStore name conflict check in InternalTopologyBuilder
  • PR-8630 - KAFKA-9969: Exclude ConnectorClientConfigRequest from class loading isolation
  • PR-8685 - KAFKA-10014 Always try to close all channels in Selector#close
  • PR-8775 - KAFKA-10079: improve thread-level stickiness
  • PR-7384 - KAFKA-8938: Improve allocations during Struct validation in ConnectSchema
  • PR-8827 - KAFKA-9849: Fix issue with worker.unsync.backoff.ms creating zombie workers when incremental cooperative rebalancing is used
  • PR-8664 - KAFKA-9716: Clarify meaning of compression rate metrics
  • PR-8805 - KAFKA-9848: Avoid triggering scheduled rebalance delay when task assignment fails but Connect workers remain in the group
  • PR-8783 - KAFKA-10063 UnsupportedOperation when querying cleaner metrics after …
  • PR-8803 - KAFKA-10102: update ProcessorTopology instead of rebuilding it
  • PR-8676 - KAFKA-10005: Decouple RestoreListener from RestoreCallback
  • PR-8807 - KAFKA-10106: log time taken to handle LeaderAndIsr request
  • PR-8684 - KAFKA-10012 Reducing memory overhead associated with strings in Metri…
  • PR-8270 - KAFKA-9216: Enforce internal config topic settings for Connect workers during startup
  • PR-8820 - KAFKA-10097: Internalize checkpoint data
  • PR-8804 - KAFKA-9851: Revoking Connect tasks due to connectivity issues should also clear the running assignment
  • PR-8486 - KAFKA-9840: Skip End Offset validation when the leader epoch is not reliable
  • PR-8797 - Fixing KAFKA-10094
  • PR-8135 - KAFKA-9570: Define SSL configs in all worker config classes, not just distributed
  • PR-8814 - KAFKA-10111: Make SinkTaskContext.errantRecordReporter() a default method
  • PR-8811 - KAFKA-10110: Corrected potential NPE when null label value added to KafkaMetricsContext
  • PR-8776 - KAFKA-9441: Improve Kafka Streams task management
  • PR-8724 - KAFKA-10040; Make computing the PreferredReplicaImbalanceCount metric more efficient
  • PR-8759 - KAFKA-10066: TestOutputTopic should pass record headers into deserializers
  • PR-8755 - KAFKA-10069: Correctly remove user-defined “predicate” and “negate” configs from transformation properties
  • PR-8311 - KAFKA-9434: automated protocol for alterReplicaLogDirs
  • PR-8644 - KAFKA-9313: Set use_all_dns_ips as the new default for client.dns.lookup (KIP-602)
  • PR-8784 - KAFKA-9788: Use distinct names for transaction and group load time se…
  • PR-8793 - KAFKA-9514; The protocol generator generated useless condition when a field is made nullable and flexible version is used
  • PR-8792 - KAFKA-10089 The stale ssl engine factory is not closed after reconfigure
  • PR-8782 - KAFKA-10080; Fix race condition on txn completion which can cause duplicate appends
  • PR-8785 - KAFKA-10084: Fix EosTestDriver end offset
  • PR-8786 - KAFKA-10083: fix failed testReassignmentWithRandomSubscriptionsAndChanges tests
  • PR-8737 - KAFKA-9945: TopicCommand should support –if-exists and –if-not-exists when –bootstrap-server is used
  • PR-8695 - KAFKA-9320: Enable TLSv1.3 by default (KIP-573)
  • PR-8777 - KAFKA-10082: Fix the failed testMultiConsumerStickyAssignment
  • PR-8779 - [MINOR] Fixing spotbug fail - removing unused variable.
  • PR-8774 - KAFKA-10081: Remove an unused local variable to pass spotbugsMain check
  • PR-8706 - KAFKA-10030 allow fetching a key from a single partition
  • PR-8760 - Kafka-10064 Add documentation for KIP-571
  • PR-8668 - KAFKA-9987: optimize sticky assignment algorithm for same-subscription case
  • PR-7956 - KAFKA-9392; Clarify deleteAcls javadoc and add test for create/delete timing
  • PR-8769 - KAFKA-10074: Improve performance of matchingAcls
  • PR-8767 - KAFKA-10060 GroupMetadataManager should not log if there are no offse…
  • PR-8761 - Remove redundant code of KafkaProducer
  • PR-8723 - KIP-569-KAFKA-9494: DescribeConfigsResponse - include additional metadata information
  • PR-8739 - KAFKA-10056; Ensure consumer metadata contains new topics on subscription change
  • PR-8705 - KAFKA-10029; Don’t update completedReceives when channels are closed to avoid ConcurrentModificationException
  • PR-8238 - KAFKA-9130: KIP-518 Allow listing consumer groups per state
  • PR-8248 - KAFKA-9501: convert between active and standby without closing stores

ksqlDB

  • PR-6800 - fix: LDAP Authentication
  • PR-6732 - fix: (minor) don’t use deprecated jersey calls
  • PR-6702 - fix: use Java’s Base64 instead of jersey’s
  • PR-6668 - fix: fix error categorization on NPE from streams (#6655)
  • PR-6608 - fix: Check for index before removing value in undo of COLLECT_LIST (#…
  • PR-6558 - fix: add back configs for setting TLS protocols and cipher suites
  • PR-6551 - fix: avoid RUN SCRIPT to override CLI session variables/properties
  • PR-6550 - fix: support joins on key formats with different default serde features
  • PR-6541 - feat: CASE expressions can now handle 12+ conditions in docker + cloud cherry picking
  • PR-6537 - fix: ‘-e/-f’ CLI parameters are not handling session variables correctly
  • PR-6482 - fix: Internal Server Error for /healthcheck endpoint in RBAC-enabled
  • PR-6467 - fix: check for nested UnspportedVersionException during auth op check
  • PR-6400 - feat: Add support for ALTER STREAM|TABLE
  • PR-6437 - fix: Properly clean up state when executing a command fails
  • PR-6451 - fix: fixes performance regression in pull queries introduced by sessionConfig
  • PR-6448 - feat: support non-primitive types as keys
  • PR-6354 - feat: add aggregated rocksdb metrics
  • PR-6446 - fix: support unwrapped struct value inference
  • PR-6410 - fix: #6319 default port for CLI: set default port if not mentionned
  • PR-6411 - feat: enable support for JSON key format
  • PR-6393 - Join format mismatch
  • PR-6353 - feat: Add an endpoint for returning the query limit configuration
  • PR-6405 - feat: avoid spurious tombstones in table output
  • PR-6397 - fix: recovery hangs when using TERMINATE ALL
  • PR-6351 - refactor: split repartition node classes
  • PR-6366 - Topic info cache
  • PR-6370 - refactor: change QTT to store serialized data in mock Kafka
  • PR-6365 - Query schemas
  • PR-6346 - feat: add commandRunnerCheck to healthcheck detail
  • PR-6348 - fix: backup files are re-created on every restart
  • PR-6349 - feat: NONE format for key-less streams
  • PR-6267 - feat: new CLI parameter to execute a command and quit (without CLI interaction)
  • PR-6344 - feat: support for DELIMITED key format
  • PR-6292 - Json keys
  • PR-6149 - feat: Support Comparisons on complex types
  • PR-6148 - feat: Add metrics for pull query request/response size in bytes
  • PR-6295 - fix: JSON format to set correct scale of decimals
  • PR-6278 - fix: don’t use queryId of last terminate command after restore
  • PR-6253 - feat: add KSQL processing log message on uncaught streams exceptions
  • PR-6240 - feat: surface error to user when command topic deleted while server running
  • PR-6166 - bug: Improved error message for reserved keywords
  • PR-6238 - fix: delimited format should write decimals in a format it can read
  • PR-6164 - feat: CommandRunner enters degraded state when corruption detected in metastore
  • PR-6194 - feat: add KEY_FORMAT and FORMAT WITH clause properties
  • PR-6251 - revert: “fix: improve join predictability by increasing max task idle …
  • PR-6236 - fix: scale of ROUND() return value
  • PR-6245 - fix: format cast arguments with passed context
  • PR-6239 - fix: improve error handling of invalid Avro identifier
  • PR-6201 - feat: substantially improve avro deserialization performance
  • PR-6188 - fix: register correct unwrapped schema
  • PR-6173 - feat: Support IF NOT EXISTS on CREATE TYPE
  • PR-6182 - fix: fail on non-string MAP keys
  • PR-6036 - feat: Support for IF EXISTS on CREATE CONNECTOR
  • PR-6160 - fix: delete zombie consumer groups 🧟
  • PR-6172 - fix: missing topic classifier now uses MissingSourceTopicException
  • PR-6163 - fix: allow expressions in flat map
  • PR-6109 - feat: clarify key or value in (de)serialization processing log messages
  • PR-6118 - feat: Adds support for 0x, X’…’, x’…’ type hex strings in udf:encode
  • PR-6135 - fix: wrong value schema being registered for unwrapped primitives
  • PR-6096 - feat: support PARTITION BY NULL for creating keyless stream
  • PR-6039 - feat: add Ksql warning to KsqlResource response when CommandRunner degraded
  • PR-6066 - fix: Use a SandboxedPersistentQueryMetadata to not interact with KafkStreams
  • PR-6084 - feat: add serialization exceptions to processing logger
  • PR-6085 - fix: properly handle restore on CREATE OR REPLACE
  • PR-6067 - feat: Support IF EXISTS keyword on DROP CONNECTOR
  • PR-6040 - feat: add a serverStatus to ServerInfo and display the status in the CLI
  • PR-5983 - fix: Uses pull query metrics for all paths, not just /query
  • PR-6068 - fix: dont cleanup external resources on CREATE OR REPLACE
  • PR-6054 - Single value wrapping
  • PR-5998 - feat: Support subscript and nested functions in grouping queries
  • PR-6032 - feat: CommandRunner enters degraded states when it processes command with higher version than it supports
  • PR-6061 - feat: hard delete schemas for push queries
  • PR-6031 - feat: DistributingExecutor fails DDL statement if CommandRunner DEGRADED
  • PR-6012 - feat: move command topic deserialization to CommandRunner and introduce DEGRADED CommandRunnerStatus
  • PR-6048 - fix: remove unnecessary parser token (MINOR)
  • PR-6005 - fix: allow implicit cast of numbers literals to decimals on insert/select
  • PR-6033 - fix: protobuf format does not support unwrapping
  • PR-5944 - feat(client): support describe source in Java client
  • PR-6018 - fix: set restarted query as healthy during a time threshold
  • PR-5984 - refactor: Call the correct pull/push query validator
  • PR-5982 - feat: support UDAFs with and without init Args with same param type
  • PR-5979 - fix: replace kafka-streams deprecated code
  • PR-5960 - fix: NPE when udf metrics enabled
  • PR-5969 - feat: add config for suppress buffer
  • PR-5962 - feat: Support [IF EXISTS] on DROP TYPE command
  • PR-5476 - feat: Add consumer offsets to DESCRIBE EXTENDED
  • PR-5877 - feat: New ksql.properties.overrides.denylist to deny clients configs overrides
  • PR-5884 - feat: add suppress functionality
  • PR-5807 - feat: add service to restart failed persistent queries
  • PR-5594 - fix: improve join predictability by increasing max task idle
  • PR-5883 - fix: close query on invalid use of HTTP/2 with /query endpoint
  • PR-5874 - fix: change default grace period to zero
  • PR-5879 - fix: create the metastore backups directory if it does not exist
  • PR-5849 - feat: Enable datagen to set the message timestamp
  • PR-5779 - feat: add suppress node
  • PR-5843 - feat: new array_remove udf
  • PR-5831 - feat: Replay command topic to local file to backup KSQL Metastore
  • PR-5835 - fix: configure topic retention based on retention clause for windowed tables
  • PR-5823 - fix: always use the changelog subject in table state stores
  • PR-5816 - feat(client): return query ID when persistent query started via Java client
  • PR-5814 - feat: expose query ID in CommandStatusEntity (MINOR)
  • PR-5775 - feat(client): support DDL/DML statements in Java client
  • PR-5813 - feat: organize UDFs by category
  • PR-5818 - fix: Allow backported ALPN to be used and warn if no OpenSSL
  • PR-5769 - fix: ensure null values cast to varchar/string remain null
  • PR-5743 - fix: windowed tables now have cleanup policy compact+delete
  • PR-5781 - fix: circumvent KAFKA-10179 by forcing changelog topics for tables
  • PR-5744 - fix: Make sure UDTF describe shows actual function description
  • PR-5739 - feat: add syntax for suppression
  • PR-5766 - feat: support CREATE OR REPLACE w/ config guard but w/o restrictions
  • PR-5742 - fix: Reuse KsqlClient instance for inter node requests
  • PR-5763 - fix: ksqlDB should not truncate decimals
  • PR-5750 - fix: show overridden props in CLI
  • PR-5679 - refactor: Convert remaining Kudfs
  • PR-5730 - fix: make sure old query stream doesn’t block on close
  • PR-5641 - feat(client): support streaming inserts in Java client
  • PR-5682 - feat(client): support list queries in Java client
  • PR-5680 - feat: support WINDOWEND in WHERE of pull queries
  • PR-5671 - feat(client): support admin operations in Java client
  • PR-5672 - fix: simplify pull query error message
  • PR-5644 - fix: support GROUP BY with no source columns used
  • PR-5645 - fix: ensure only deserializable cmds are written to command topic
  • PR-5482 - feat: Adds SSL mutual auth support to intra-cluster requests
  • PR-5474 - feat: implements ARRAY_JOIN as requested in (#5028)
  • PR-5563 - feat: new split_to_map udf
  • PR-5559 - feat: add CHR UDF
  • PR-5597 - fix: fail AVRO/Protobuf/JSON Schema statements if SR is missing
  • PR-5573 - Stream with no key should not fail if key has data

REST Proxy

  • PR-773 - KREST-287: Syncronize resource pools/managers lazy instantiation.
  • PR-756 - KREST-26: Add a blocklist config to allow disabling individual endpoints.
  • PR-761 - Add toBuilder method to Cluster
  • PR-755 - Update JUnit dependency to JUnit 5.
  • PR-216 - MMA-7520: add CsrfTokenProtectionFilter
  • PR-209 - (backport) SEC-592: Customize Jetty request queue and threadpool size (#172)
  • PR-215 - ff-3928: bump version of Guava in parent of schema-registry.
  • PR-210 - ST-4542: Upgrade jetty to address CVE-2020-27216
  • PR-205 - CIAM-261: Add Jetty ThreadPool Metrics
  • PR-194 - Fix build failure

Schema Registry

  • PR-1747 - DG-1189 Add latest.compatibility.strict flag
  • PR-1746 - DG-931: throw ConfigException in Avro/Json/Protobuf Converter
  • PR-1724 - DG-1124 NPE in ProtobufMessageFormatter
  • PR-1722 - DG-1122 Treat reordering Protobuf msgs as incompatible
  • PR-1711 - add testUsingSubjectNameStrategy()
  • PR-1709 - QEC-2994: fix bug in getting subject name for value
  • PR-1700 - DG-293: Improve exception messages in JsonSchemaConverter and ProtobufConverter
  • PR-1699 - DG-1040 Register ids for soft deletes
  • PR-1688 - DG-1007 Preserve refs when converting schema to envelope
  • PR-1687 - DG-924 Disable default validation by default
  • PR-1686 - MMA-8608 Handle duplicate configs after stripping prefix
  • PR-1684 - MMA-8608: parse configs before calling restService.configure()
  • PR-1678 - DG-990 Allow SpecificAvroSerde to work with array/map
  • PR-1676 - DG-972 Fix name collisions with field and message
  • PR-1675 - DG-962 Handle const schemas in JSON Schema converter
  • PR-1668 - Removed property ${confluent.maven.repo}
  • PR-1658 - DG-951 Handle duplicate message names during deserialization
  • PR-1654 - Adding an NPE check for permanent delete
  • PR-1648 - DG-935 Treat removing Protobuf message types as incompatible
  • PR-1642 - Avro Console producer should support subject name strategy option
  • PR-1647 - DG-933 Resend old value when handling invalid key-value
  • PR-1638 - DG-924 Enable Avro default validation for new schemas
  • PR-1634 - DG-921 Fix Avro specific reader for unions
  • PR-1625 - DG-410 Fix doc/defaults in Avro converter
  • PR-1614 - Add failing compatibility reason in rest response
  • PR-1621 - DG-848 Enable mode mutability by default, fixes #1610
  • PR-1591 - Add offset checkpoints for persistent stores
  • PR-1571 - Add setTenant
  • PR-1559 - Add API to get subjects by predicate
  • PR-1544 - DG-616 Permanent Delete support in SR client
  • PR-1533 - DG-534 Enhance compat endpoint to validate schema
  • PR-1532 - DG-540 Add batch get schema API
  • PR-1527 - NetworkClient fix for upstream Kafka changes
  • PR-1519 - #1515 - Cache Avro DatumReader and DatumWriter instances to increase performance
  • PR-1498 - update README to include protobuf, json schema
  • PR-1491 - Revert “Add node information to SchemaRegistryProtocol (#1485)”

Commercial Features

Confluent Server

  • KSTORAGE-1348 Flush epoch cache before reinitialization in Log.renameDir
  • SEC-1805: Reduce failed MDS request send log threshold from ERROR to Debug
  • Fix test failure in test_kibosh_service (CPKAFKA-6180)
  • SEC-1802 Check for topic creation
  • SEC-1802 Don’t block TopicManager creation on describeTopics
  • SEC-1770 NonBlockingKafkaExporter should notice that topics exist
  • KSTORAGE-1178: Moved missing tier state file recovery code under assignTopicId (#2702)
  • KSQL-5843: prevent log spam from MDS RestClient on client close
  • Add back specific log message for unclean recovery test (#2938)
  • SEC-1789 Bump Bouncy Castle dep to resolve CVE-2020-28052
  • AUDIT-249– Added mime type “application/protobuf” supporting to BinaryDeserializer
  • CPKAFKA-6413: test_offset_migration_single_group_offset_not_exists_on_dest
  • CPKAFKA-6375: Decode python3 bytes object in tier fence restore test
  • CPKAFKA-6374: Handle None in tier roundtrip system test
  • CPKAFKA-6370: Use python3 range (versus xrange) to have BrokerStorageFailureTest work with python3
  • CPKAFKA-6371: Use python3 sys.maxsize in tiered storage system tests
  • KC-1275: Update KafkaReader to handle entries from multiple partitions
  • CPKAFKA-6373: Fix deprecated has_key python2 method
  • CPKAFKA-6376: Use range over xrange to fix TierMetadataCorruptionTest to work with python3
  • KSTORAGE-1290: Bump google storage client dependency to 1.113.4
  • CPKAFKA-6372: Fix bad assertion in cluster link test
  • CPKAFKA-3011: fix timing issue with onConfigChange and ConfluentLeaderAndIsrRequest
  • Backport of CPKAFKA-6268
  • CNK-1431: Add confluent.ansible.managed property
  • KC-1228 cherry pick for 6.1.x
  • CPKAFKA-6018: Fix cluster link test timing issue with metrics and replica status
  • KSTORAGE-1260: tiered segments do not correctly block retention for local segments (#2831)
  • CNK-1325: Add a metric for initiated executions.
  • CNK-819: Don’t self-heal if a rebalance doesn’t improve detection goals.
  • Port fix for KSTORAGE-1230 to 6.1.x
  • CPKAFKA-6126: Update commons codec (#2785)
  • CNK-1210: Introduce kafka-add-brokers CLI
  • CPKAFKA-6233: Update lastUpdateTimeMs before updating the current broker addition state
  • CNK-1396: Register BrokerAddCount metric in CDBE
  • CNK-617: Temporary pull out CloudNetworkFault + deps
  • CNK-1349: Switch broker removal state from PLAN_EXECUTION_INITIATED to PLAN_COMPUTATION_INITIATED on failover
  • KC-1226: Enable confluent.cluster.link.io.max.bytes.per.second to be configured at cluster-level
  • KC-1225: Fix NPE on broker restart with paused links, fix blocked state
  • CNK-1228: Update SBK add_broker system tests to utilize kafka-add-brokers for checking operation completion
  • KC-1027 backport into 6.1.x
  • Backport Update system test to use –comparator.json (#2743)
  • CNK-1332: Handle TopicExistsException when creating SBK topics
  • KC-1203: Stop mirroring if destination partition is under-min-isr, metrics changes
  • KC-1195: Ensure connections are closed if principal cannot be built
  • backport METRICS-2459: update default include list for hosted monitoring
  • backport METRICS-2396: Add extra checks to ensure that we do not leak EventExporter objects
  • backport METRICS-2110: document Telemetry Reporter packaging (#2710)
  • backport METRICS-1805: remove duplicate resource type label (#2744, #2755)
  • Add unit tests to disable tier storage at broker level
  • KSTORAGE-1180 Update tier enable flag at FileTierPartitionState unconditionally
  • KC-1183: Fix exception in writer metric for non-MDS brokers
  • KC-1025 automatic observer promotion/demotion in MRC (#2649) cherry-pick
  • CNK-1304: Update remove broker validation to use strict collections
  • CNK-1331: Fix typo in SBK Persistence Store initialization
  • CNK-1209: Describe AddBroker functionality and integration tests
  • KC-1132: Add cluster link request metrics for source cluster
  • CPKAFKA-6030: Fix broker removal system test
  • CNK-1290: Update the sub task status field in BrokerAdditionDescription and BrokerRemovalDescription and CNK-1288 - add description() method to DescribeBrokerRemovalsResult for feature parity
  • CNK-1233: Adds support for logical cluster type link-healthcheck.
  • CNK-1208: Return a high level balancer operation status in describe broker removals
  • KSTORAGE-1075: Add debug logging when PendingFetch completes
  • [METRICS-2356] patch version for thread leak fix
  • Increase test tolerance in non deterministic test
  • CC-12141: Confluent’s WorkerConfigDecorator in Connect should handle connector validate responses with extra properties
  • KSTORAGE-1004: Restore tool should be able to work standalone without comparator tool
  • KC-1149: Ensure stopped mirrors are not treated as read-only partitions
  • CNK-1012: Add broker state machine
  • CNKAF-1236: Add startTimeMs and lastUpdateTimeMs to the describe broker removal RPC
  • KC-1148: Fixing metric conflict issues where link can’t be recreated due to metric conflict issues
  • Add config option to override schema validator interceptor class
  • SEC-1477 Add trace logging on LDAP search success
  • CCS -> CE sync upstream 28 sept 2020
  • CONFLUENT: Add config for the tests that rely on MockAuditLogProvider.
  • KSTORAGE-1176: chmod -R ag+w in ccloud startup script increases startup time
  • CPKAFKA-4479: Make operationally important Kafka configs dynamic (fetch.max.bytes)
  • KC-907: Update to cc-base 7.4.0 and remove prometheus and async profiler downloads
  • CNK-1126: Request quota auto-tuning sorts by an unstable value
  • CNK-374: Return task history for only configured retention period
  • ce-metrics: add space to reconcileConfigs() log message
  • KC-375: Include cluster-link-id in requests from destination brokers to source
  • METRICS-2358 : only initialize event logger when enabled, properly close it, and add better support for reconfiguration
  • CNKAF-1134: Close AdminClient, Producers and Consumers immediately
  • KC-1120: Ensure that RBAC metadata write failures are reset after writer elections
  • CNKAF-856: DescribeBrokerAdditions API - RPC and Admin API skeleton
  • KC-938: Add config for dedicated-cluster topics for cluster-linking soak setup.
  • KC-955: Adds describeMirrors API.
  • CNKAF-1068: Add validation for broker removal for rf equal to alive brokers
  • CNKAF-1235: Revise state machine for broker removal
  • CIAM-349 CloudClusterAdmin can see Schema Registry
  • KSTORAGE-1100: Offset cache can’t be disabled although we have the facility to do so
  • KSTORAGE-986: Check for invalid topic partitions within the recovery tool
  • CNK-1206: Update dependencies for SBK serializers
  • CNK-42: Only enable Databalancer Count metric if balancer is enabled
  • CNKAF-1109: Use AdminClient for Leader Elections, handle exceptions
  • KSTORAGE-1160: avoid unnecessary maxCompactionDelay calculations when now < max.compaction.lag.ms
  • KC-899: Monitor auto-topic creation and compaction
  • Propagate CC release branch to ce-kafka-images
  • CNKAF-1001: Retry plan computation in Broker Removal
  • KSTORAGE-1150: Allow customers to change log.cleaner.max.compaction.lag.ms cluster default
  • CIAM-371 Add Delete operation to Cluster for CloudClusterAdmin
  • CNKAF-1174: improve fetch session cache performance
  • KC-1017: Extends replica status to include mirror information.
  • KC-44: Maintain topics per tenant in metadata cache
  • KC-1021: Add RequestConvertToJson tests
  • KSTORAGE-797: allow recovery path materializer to use secured listeners
  • CC-11832: add connector name to metrics context when calling Kafka Cl…
  • CNKAF-1062: Add validation for broker removal leading to under min isr
  • CNKAF-1224: Log all uncaught SBK exception and errors
  • CNKAF-920: Raise invalid broker removal exception with specific error message
  • KC-1019: ScramUserCredentials matches missing in RequestConvertToJson
  • [CNKAF-1195] Avoid stale cluster metadata for AnomalyDetection plans.
  • CIAM-321 Add Audit Resource Type for OrgAdmin role
  • Configure ListenerConnectionQuota on broker startup
  • Suspending rbac system tests till we fix hang issue
  • local start offset check is done outside committed offset check block.
  • KSTORAGE-785: allow CCloud users to set max.compaction.lag.ms
  • CNKAF-915 Improve log level in executor
  • Durability Audit fixes
  • KSTORAGE-1013: add broker internal log dir storage probe
  • CNKAF-1075: Improve Confluent SASL secrets caching
  • KC-190: add metric to count responding to disconnected clients
  • cherry-pick from AK - Fix that support, resolve of “kafka.apache.org ” to 3 IP addresses (#9294)
  • Reverted changes in TestUtils which added additional parameter for enabling durability audit.
  • Fixed unit test in AuditManager
  • KSTORAGE-1023: Durability audit follow up
  • CNKAF-765; Update the TenantQuotaCallback to support Controller Mutation Quota
  • CPKAFKA-6013: Skip tier state presence validation for deleted partitions
  • KSTORAGE-972: Modify TopicIdPartition.toString() implementation to use ‘:’ and ‘-‘ as delimiters
  • KSTORAGE-885: add lagging and realtime consumer to retention ms/bytes system test
  • KC-883: Update soak-cluster configuration for source and destination.
  • Migrated AuditJob away from KafkaScheduler
  • KSTORAGE-973: Check for invalid topic partitions within the fencing tool
  • KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline
  • KSTORAGE-725: Exclude segments where the high offset is lower than the lowerBoundOffset for tiering.
  • KSTORAGE-1074: retry should log at INFO level
  • KSTORAGE-1082: tier replication causes incorrect HWM after restore (#2507)
  • KSTORAGE-1075: Add more debug info for consumption tracking
  • feat: Add load metrics for logical cluster metadata
  • KSTORAGE-1086: Better validations to check if tiering is disabled
  • CNKAF-1138: Emit JSONs with auto-generated schema
  • CNKAF-546: Reinitialize expired partition detail sensors
  • KSTORAGE-909: Follow-up PR to durability audit PR
  • KSTORAGE-1054: add another Azure auth method via config file
  • KC–603: Fix NPE in KafkaBasedLog during broker shutdown if license topic creation fails
  • KC-902: Fix cluster link authorization, add end-to-end authorization tests
  • KSTORAGE-970: Fix help screen in TierTopic tooling
  • KSTORAGE-1084: use file name convention for tier_unclean_leader_election_test.py
  • CPKAFKA-5114: Add group size metrics
  • Sync upstream august 18 2020
  • KSTORAGE-1082: tier replication causes incorrect HWM after restore
  • CNKAF-1169: Correctly handle single racks and no racks when creating topics in the MT interceptor
  • CPKAFKA-5114: Add group size metrics
  • Update mk-include
  • KSTORAGE-887: Integrate TierFetcherBenchmark into compatibility checker
  • KSTORAGE-887: Make TierFetcherBenchmark configurable for Pure Storage testing
  • CNKAF-1087: Fix rack aware topic placement for cloud use case
  • KC-854: Allow socket buffer size configurable for high latency cluster links on Cloud
  • KC-801: Reassignment succeeds with empty constraints
  • [CNKAF-1144] Confluent.operator.managed property (#2484)
  • [Backport]METRICS-2089 Adding the correct NOTICE / COPYRIGHT for the confluent-telemetry package
  • KSTORAGE-1042: Harden compatibility framework #1
  • METRICS-2089 Adding the correct NOTICE / COPYRIGHT for the confluent-telemetry package
  • KSTORAGE-1067: Dump fetch context when tiered storage runs into an unexpected error
  • [cnkaf-1144] Confluent.operator.managed property
  • Durability Audit - Audit Manager, Materialization and Reporting and Event generation
  • CNKAF-902: Add metrics to track SSL handshake failures and avg/max time
  • switch to using DefaultAzureCredentials for Azure authentication
  • CNKAF-108: Improve log message for dynamic certificate update
  • KC-166: Add metric for controller loading time
  • KSTORAGE-974: Print tiertopic partition and timestamp for fencing tool
  • KSTORAGE-859: MergedLog.createUploadableSegment deleted segment check
  • CNKAF-758:Add cpu load metric
  • KSTORAGE-1040: Better checks to prevent tiering from being disabled dynamically
  • METRICS-2159: use a loading cache to avoid expensive pattern matching calls
  • METRICS-2160: optimize local exporter regex
  • METRICS-2159: use a loading cache to avoid expensive pattern matching calls
  • KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale.
  • avoid calls to gettimeofday on every metric key in telemetry
  • KC-102 Add cc-zookeeper docker build
  • CPKAFKA-5882. Add copyDependantLibs dependency to ce-sbk and ce-broker-plugins module
  • [CNKAF-1112 WORKAROUND] Do not allow enabling of strict quotas.
  • feat: Cache auth failures
  • SEC-1016: Changes to NonBlockingKafkaExporter
  • CONFLUENT: Upgrade netty-tcnative to 2.0.34.Final
  • CONFLUENT: Change visibility of ConfluentProvider.createRbacAuthorizer() to public
  • KC-845: Ensure LogConfig includes broker defaults to pass schema config validation
  • CPKAFKA-5873 Don’t load Netty classes when using default SSL engine
  • CONFLUENT: Remove shading from ce-broker-plugins and ce-sbk
  • [KSTORAGE-829] Storage Fault Injection: Integrate Kibosh with Ducktape
  • CPKAFKA-5851: Align joda-time version with common
  • KSTORAGE-1000: TierDeletedPartitionsCoordinator Handling of Emigration During Read
  • CNKAF-863: Only showed broker removal status for given broker.
  • KC-613: Use kafka-rest-lib instead of kafka-rest on classpath.
  • KC-613: Use kafka-rest-lib instead of kafka-rest on classpath.
  • Update compatibility checker README.md
  • SEC-1373 EmbeddedKafkaCluster sets Audit Log replication factor to 1
  • CPKAFKA-5849: Don’t log password configs in request log filter
  • CONFLUENT: Remove unused jacksonDatabind version definition
  • CPKAFKA-5795: Change guava dependency to 28.1-jre
  • CPKAFKA-5796: Remove ZK dependency from ce-licensing
  • CPKAFKA-5835: Fix regression in cluster link compatibility test
  • Add secrets to default RBAC policies
  • KC-810: Add cluster link destination tenant metrics suitable for billing
  • METRICS-2108 Telemetry-client dependency version update from 1.174.0 to 1.192.0 to update default base URL for TelemetryHttpClient
  • KC-770: Support backpressure for cluster link when disk usage is high
  • METRICS-2108 Telemetry-client dependency version update from 1.174.0 to 1.192.0 to update default base URL for TelemetryHttpClient
  • KSTORAGE-996 Fix logic to check bytes fetched from local log
  • [CNKAF-1089][RM-32] Remove SBK dependencies on grpc-stub. (#2408)
  • KSTORAGE-991 Implementing Additional Config Validations
  • KSTORAGE-334, KSTORAGE-893, KSTORAGE-894: DeleteRecords, retention time and bytes system tests
  • backport METRICS-2098 exclude zstd/lz4 from telemetry-clients to avoid conflicting versions
  • KSTORAGE-994 Improved Configuration Documentation
  • [CNKAF-1089][RM-32] Remove SBK dependencies on grpc-stub.
  • Redo SEC-866
  • CNKAF-655: SBK Soak cluster workload
  • METRICS-2090: Remove duplicate labels on metrics
  • METRICS-2097: Don’t capture ‘total_size` and ‘tier_size` metrics
  • METRICS-2098 exclude zstd / lz4 from telemetry-clients to avoid conflicting versions
  • KC-637 cherry pick to 6.0.x
  • KC-637 Offset migration system tests.
  • KSTORAGE-957: Only set SSEAwsKeyManagementParams if a customer encryption key is supplied and sseAlgorithm is aws:kms. (#2353)
  • KSTORAGE-886 [MINOR] Move scripts from bin to compatibility checker directory
  • SEC-866-6.0.x again
  • KSTORAGE-886: Implementing basic tiering load test utils
  • CONFLUENT: Remove the soak clients status reporter cronjob
  • KSTORAGE-957: Only set SSEAwsKeyManagementParams if a customer encryption key is supplied and sseAlgorithm is aws:kms.
  • KSTORAGE-506: Prevent accidental change to critical configurations for tier topic
  • CIAM-227: Add requestor principal to AuthWriter interface methods
  • METRICS-2031 Prune the hardcoded Kafka broker default include list to avoid collecting high cardinality metrics.
  • METRICS-2048 Sanitize ksql service ids from their metric name to make them more practical to use. Also ignore legacy metrics.
  • [CNKAF-1038] SBK Rebalance Cluster Test: add rolling bounces (#2326)
  • METRICS-2031 Prune the hardcoded Kafka broker default include list to avoid collecting high cardinality metrics.
  • CNKAF-1058; Ensure that Control Request Batching does not send LeaderAndIsrRequest with yet unknown broker(s)
  • CNKAF-896: Avoid expensive calls to MetadataCache#getNonExistingTopics
  • KSTORAGE-947: batch tier segment deletion delays to improve deletion SLA (#2309)
  • METRICS-2048 Sanitize ksql service ids from their metric name to make them more practical to use. Also ignore legacy metrics.
  • KC-238: add max rebalance time gauge
  • CNKAF-1028:Passed SNI host name to OAuthBearerValidatorCallbackHandler
  • CNKAF-1039: Update error for insufficient metrics to compute plan
  • backport METRICS-2010: create a new ‘releaseTarGz’ task for ce-metrics (#2282)
  • METRICS-2010: create a new ‘releaseTarGz’ task for ce-metrics
  • CPKAFKA-2864: Update base docker image for new tooling
  • CONFLUENT: Allow both AK and Confluent class for system test process
  • CPKAFKA-5762: Fix topic deletion in system tests
  • CNKAF-1039: Update error for insufficient metrics to compute plan
  • KC-671: SBK system test with CL
  • METRICS-1988: add confluent-telemetry directory to kafka classpath
  • CONFLUENT: Bump cc-trogdor chart docker image to cc-trogdor:v0.207.0-5.3.0-ce-SNAPSHOT
  • KAFKA-10371; Partition reassignments can result in crashed ReplicaFetcherThreads
  • [Cherry 2367] [SEC-1345] Read HTTP Reason phrase when connection closed
  • CC-4539: Use plugin.path mechanism to load rest servlet initializers in Connect
  • Revert “SEC-866 - Migrate audit log “other” category into “management” category. (#2330)”
  • [SEC-1345] Read HTTP Reason phrase when connection closed
  • backport METRICS-2037 remove non-relocated dependencies from confluent-metrics fat jar (#2332)
  • [CLIENTS-1633] Schedule MDS url refresher asynchronously
  • Increase root FS to 20GB
  • METRICS-1763, METRICS-1767 Configure telemetry metrics whitelist for Connect and MQTT Proxy Proactive Support (#2241)
  • SEC-866 - Migrate audit log “other” category into “management” category.
  • METRICS-1763,METRICS-1767 Configure telemetry metrics include list for Connect and MQTT Proxy Proactive Support
  • SEC-866 - Migrate audit log “other” category into “management” category.
  • KC-391: Source topic deletion system test for CL
  • METRICS-2037 remove non-relocated dependencies from confluent-metrics fat jar
  • KSTORAGE-947: batch tier segment deletion delays to improve deletion SLA
  • METRICS-2056 Update api.secret datatype from String to password in event’s HttpExporterConfig.
  • KC-727: Prevent source topic config mirroring from violating alter config policies
  • [CLIENTS-1633] Authorizer RestClient should fail updating active node…
  • KAFKA-10282; Remove Log metrics immediately when deleting log (#2342)
  • [CNKAF-1038] SBK Rebalance Cluster Test: add rolling bounces
  • CNKAF-1036: Fix issue where dynamic throttle applies for the current rebalance only - cherry pick to 6.0
  • Rebalance Cluster System Test
  • KAFKA-10282: Remove Log metrics immediately when deleting log
  • CNKAF-1008: Add Broker/Remove Broker proposal logging
  • CNKAF-1036: Fix issue where dynamic throttle applies for the current rebalance only
  • KC-633: Auth failures system test for Cluster Linking
  • Add CloudNetworkFault to Trogdor
  • [CHERRY-2308] Move kafka-http-server project into clients project
  • METRICS-2007 Updating Telemetry Reporter to exclude exporter configs or disabled exporters with no type set.
  • Move kafka-http-server project into clients project
  • KC-638: System tests with tier storage enabled on destination
  • KC-754: Cluster link compatibility system tests
  • KC-631 Good path cluster linking test using SASL_SSL
  • backport METRICS-1992: Fix schema registry cluster id resource label (#2285)
  • SBK: System test for cluster rebalancing
  • METRICS-1760 Configuring additional Kafka Broker include list metrics for Proactive Support.
  • KC-749: initial single-topic setup for cluster-linking soak clusters
  • KC-755: Fix NoSuchElementException warnings for consumer groups with no new offsets
  • CONFLUENT: Fix broken cluster link system tests in 6.0.x
  • CNKAF-481: Oauth validation of SNI HostName
  • METRICS-1950 Adding integration test to verify proxy configurations are propagated correctly.
  • CNKAF-481: Passed down SNI hostname and enable parsing in transport layer.
  • CNKAF-1008: Log full proposal dump for add/remove
  • METRICS-1992: Follow up comment based on review
  • CNKAF-1029: Ensure Executor state is reset after unexpected exceptions
  • KC-735: Improve cluster link addition.
  • KSTORAGE 806: Tiering Functionality Tests
  • KC-726: Cluster link config support for CCloud.
  • CNKAF-1035: Roll cluster while a new broker is getting added
  • clean up auto aborting input stream
  • KSTORAGE-888: Add tier fetches metrics for tiered storage
  • METRICS-1998: add rbac metrics to default metrics list (#2269)
  • METRICS-1992: Fix schema registry cluster id resource label
  • backport METRICS-1993: add connect.cluster.id label (#2264)
  • Updated default topic replicas factor to 3 for audit log.
  • KC-741: Don’t include passwords in cluster link request logs
  • KC-736: Allow deletion of cluster links if mirrored topics are marked for deletion
  • KSTORAGE-896: fix tier deletion manager system test (#2256)
  • CNKAF-822: Cherry pick add broker system test to 6.0.x
  • KC-725: Backport of KC-725 to 6.0.x
  • METRICS-1993: add connect.cluster.id to telemetry resource labels
  • CNKAF-822: Adds an SBK AddBroker system test
  • METRICS-1998: add rbac metrics to default metrics list
  • KC-725: Fixed issue where metrics were not cleaned up when cluster links were deleted
  • KSTORAGE-823: Log.maybeIncrementLogStartOffset should not flush leader epoch cache
  • backport METRICS-1986 Fix to give higher precedence to “confluent.telemetry.metrics.collector.whitelist” over default provider whitelist. (#2266)
  • backport METRICS-1985 Add missing metrics to the _local exporter for SBK feature to work as expected.
  • CONFLUENT: Make RbacClusters.java to work with Writer/AuthWriter interfaces
  • METRICS-1985 Add missing metrics to the _local exporter for SBK feature to work as expected.
  • KC-728: Return INVALID_CONFIG error for invalid link configs, log unexpected errors
  • METRICS-1986 Fix to give higher precedence to “confluent.telemetry.metrics.collector.whitelist” over default provider whitelist.
  • KSTORAGE-920: TierTopicConsumer primary consumer long poll throttles catchup (#2253)
  • KSTORAGE-864 Fetch requests to return NotLeaderForPartition during ULE recovery
  • CNKAF-819: Test SBK broker removal with roll
  • METRICS-1762 Configure telemetry metrics whitelist for KSQL Proactive Support
  • KSTORAGE-896: fix tier deletion manager system test
  • KSTORAGE-920: TierTopicConsumer primary consumer long poll throttles catchup
  • CIAM-12: update mk-include
  • METRICS-1762 Configure telemetry metrics whitelist for KSQL Proactive Support
  • Backporting fix for METRICS-1764 and METRICS-1766
  • update kafkatest template for balancer replication factor (for < 3-node clusters)
  • CNKAF-1015: Fix issue in SBK startup by moving CruiseControl instantiation after the startup components
  • KSTORAGE-876: Adding Tiered Storage TotalSizeAggregate Metric
  • KC-638: Cluster linking system test with tiered storage enabled in source cluster
  • KSTORAGE-890: Initial commit with the AuditJob infrastructure
  • CONFLUENT: Fix reassign partitions system tests
  • METRICS-1764 Configure telemetry metrics whitelist for Schema Registry Proactive Support
  • METRICS-1766 Configure telemetry metrics whitelist for Rest proxy Proactive Support
  • CNKAF-825: Introduce a system test for replica placement constraints
  • CNKAF-1014: Rename usages of SBK in logs to DataBalancer
  • CNKAF-613: Introduce a broker removal system test
  • METRICS-1894:Telemetry Reporter is missing labels configured in prope…
  • METRICS-1817: add include regex default value to Provider interface
  • METRICS-1938: upgrade jose4j version (#2219)–backport
  • METRICS-1929: use advertised listeners/host-port when deriving local producer config
  • METRICS-1938: upgrade jose4j version
  • [CNKAF-963] Switch AddBroker plan computation to wait METADATA_TTL.
  • CNKAF-954: Removes GoalOptimizer proposal caching
  • KSTORAGE-892 Re add couple of lines of ULE code that got deleted duri…
  • Backporting config/events Telemetry Reporter changes along with all shading issues.
  • Added one more toJson method in protoserde for webhook alert
  • (backport) MMA-8432: fix wrong package name
  • MMA-8432: fix wrong package name
  • KC-428: Always log authorize results to file based on log4j settings
  • KAFKA-10279; Allow dynamic update of certificates with additional SubjectAltNames
  • CNKAF-1004: Don’t describe removed broker configs
  • METRICS-1894:Telemetry Reporter is missing labels configured in properties file
  • KC-714: SslTransportLayer write should invoke SSLEngine.wrap once with all the buffers
  • CNKAF-1004: Don’t describe removed broker configs
  • CNKAF-762: Abstract CruiseControl startup and config translation logic into own package
  • METRICS-1895: Update telemetry client version
  • KC-705: Retry metadata topic consumer.position() until authorizer startup times out
  • METRICS-1918: bump gson from 2.8.5 -> 2.8.6
  • METRICS-1895: upgrade telemetry client version
  • (backport) prevent ‘io.confluent.controlcenter’ from being renamed by shadow plugin
  • KSTORAGE-526 Recover from corrupt or missing TierState file during bootup
  • KSTORAGE-822: Expose Tiered Storage configs for GA in 6.0
  • KSTORAGE-822: Expose Tiered Storage configs for GA in 6.0
  • (Backport) include UnderMinIsrPartitionCount in the default broker metrics list
  • Add UnderMinIsrPartitionCount to default broker metrics list
  • CNKAF-481: Parse SNI in sslTransportLayer
  • CPKAFKA-2852: CCloud metadata listener thread should not die on unexpected exception
  • METRICS-1875 fix shading issues with cloudevent proto generated classes
  • SEC-1276 Allow empty resourceNames in CRNs
  • MINOR: Improve AuthenticationException logging
  • SEC-384: Use the correct log level for debug output
  • KC-634: Cluster link system tests with broker failures
  • CIAM-110 Associate RBAC Roles with organizations
  • METRICS-1875: Shade generated ce-events protobuf
  • prevent ‘io.confluent.controlcenter’ from being renamed by shadow plugin
  • CNKAF-985: Bump leader distribution goal priority
  • KSTORAGE-500 Remove TierTopicAppender.becomeArchiver from the interface and inline the implementation
  • KC-614: Check that topic name is not empty before tenant prefixing
  • CNKAF-964: Shade opencensus protobufs into SBK jar (#2155)
  • CONFLUENT: Update log statements in alterBrokerConfigs/alterTopicConfigs methods
  • SBK: cherry pick javadoc updates for removeBroker
  • METRICS-1878: Shade telemetry client classes
  • Add KafkaRest Metrics Provider
  • KSTORAGE-845 Check if tier topic exists already before calling AdminZkClient#createTopic
  • CNKAF-952: Update javadoc for removeBroker ConfluentAdmin API with InvalidBrokerRemovalException
  • CNKAF-851: log goals skipped during detection in one line
  • Add KafkaRest Metrics Provider
  • CNKAF-964: Shade opencensus protobufs into SBK jar
  • KSTORAGE-806 Modifying Correctness Framework to accept .properties
  • KC-630: Cluster link system test support with good path mirror test
  • CONFLUENT: Prevent log start offset overrun during retention
  • METRICS-1862 Remove config events proto defition from ce-kafka and use the one checked in into Schroedinger instead.
  • KSTORAGE-861: fatal NPE in TierTopicConsumer thread caused by initialization order (#2133)
  • Back port [METRICS-1815] Connect client metrics missing some resource level tag…
  • KSTORAGE-806 Patch to add Confluent License Information
  • KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups (#8989)
  • CPKAFKA-5456: Shade telemetry-api classes
  • KSTORAGE-806: Implementing a CLI for compatibility correctness tests & building object store correctness tests
  • CNKAF-936: Remove dot-handling logic in metrics processor (#2121)
  • KSTORAGE-861: fatal NPE in TierTopicConsumer thread caused by initialization order
  • METRICS-1833 Special handling of KSQL metrics in telemetry reporter that are getting mangled.
  • [METRICS-1815] Connect client metrics missing some resource level tags
  • METRICS-1833 Special handling of KSQL metrics in telemetry reporter that are getting mangled.
  • CNKAF-936: Remove dot-handling logic in metrics processor
  • Azure block blob implementation for tiered storage
  • Deprecate KafkaHttpServlet in favour of rest-utils Application. (#2135)
  • CLIENTS-1599: Deprecate KafkaHttpServlet in favour of rest-utils Application.
  • KC-532: Support observers in replica assignment tool
  • CONFLUENT: Remove unused val and stale documentation in ReplicaManager
  • [METRICS-1844, METRICS-1839, METRICS-1756,METRICS-1853] - Reconcile configs in Telemetry Reporter before creating a ConfluentTelemetryConfig object. Use password object for api.secret in ConfluentTelemetryConfig.java.
  • [CNKAF-885, CNKAF-946, CNKAF-947] Set SBK Default Throttle
  • [METRICS-1844, METRICS-1839, METRICS-1756, METRICS-1853] - Reconcile configs in Telemetry Reporter before creating a ConfluentTelemetryConfig object. Use password object for api.secret in ConfluentTelemetryConfig.java.
  • fix: Make EmbeddedKafka threadsafe
  • METRICS-1635: Generate and send config events
  • METRICS-1801: align telemetry domain metric names
  • METRICS-1835 Shade confluent-serializers dependencies in confluent-metrics (#2111)
  • [CNKAF-885, CNKAF-946, CNKAF-947] Set SBK Default Throttle, handle absent brokers when setting throttle.
  • METRICS-1835 Shade ce-serializer dependencies in ce-metrics
  • CNKAF-481: Verify Cluster Id in FileBasedPlainSaslAuthenticator
  • KSTORAGE-708: Durability Topic Partition management
  • METRICS-1733 replace ‘whitelist’ with ‘include’ for telemetry and metrics reporter (#2096)
  • METRICS-1810 use SBK topic replicas config if present (#2109)
  • METRICS-1733 replace ‘whitelist’ with ‘include’ for telemetry and metrics reporter
  • CNKAF-481: Allow embedding of logical cluster name in advertised listener config
  • METRICS-1810 use SBK topic replicas config if present
  • fix: fixes exception in server shutdown due to metrics registration
  • METRICS-1809: add telemetry settings to default server.properties file
  • KC-681: Make num.partitions a dynamic config
  • CNKAF-849: Sanitize log messages when checking for startup conditions
  • METRICS-1819: support using port 0 for telemetry local exporter
  • [CNKAF-927] Fix typo in server.properties. (#2090)
  • [CNKAF-924, CNKAF-926] Fix Databalancer Topic RF config, remove Teleme…
  • [CNKAF-927] Fix typo in server.properties.
  • CONFLUENT: Added check for CL metrics
  • CNKAF-918: Relax GoalUtils.eligibleBrokers criteria to allow leadership movements off observers.
  • CONFLUENT: Added check for CL metrics for 6.0.x
  • CIAM-109 Add option to fetch role bindings for multiple scopes
  • [CNKAF924, CNKAF-926] Fix Databalancer Topic RF config, remove Telemetry RF dependency
  • 6.0.x
  • CPKAFKA-4955: Fix failing testTopicDeletion
  • CIAM-109 Add ALL DescribeAccess and AlterAccess for resource specific grants
  • cherry-pick PR 2047, 2095 from 6.0.x
  • KC-655 Offset migration configs with racially neutral terms.
  • Removed two testing files accidentally added.
  • SEC-1130-Inject ConfluentAdmin from ce-kafka
  • backport METRICS-1820: use password object for telemetry api secret
  • CNKAF-893: Update SBK to work with secure Zookeeper
  • CIAM-96 Added cloud-cluster scope and describe access for higher levels.
  • [METRICS-1820] use password object for telemetry api secret
  • SEC-1034: use confluent-log4j instead of log4j
  • METRICS-1816, METRICS-1705 local telemetry exporter metrics whitelist configuration for SBK (#2031)
  • CONFLUENT: deleting extra topic config change in cluster linking test
  • KC-505: Integration tests for CL metrics
  • CNKAF-909: Add direct dependency on opencensus for SBK (#2057)
  • KAFKA-10149: Allow auto preferred leader election when partitions are…
  • CNKAF-918: Relax GoalUtils.eligibleBrokers criteria to allow leadership movements off observers.
  • CNKAF-909: Add direct dependency on opencensus for SBK
  • Add RackAwareGoal to goal list
  • Add RackAwareGoal to goal list
  • CNKAF-884: Remove unnecessary metrics (#2059)
  • Migrate MDS to run on KafkaHttpServer implementation
  • KAFKA-10149: Allow auto preferred leader election when partitions are…
  • Fix ce-http-server import control violation for SslContextFactory
  • KC-505: integration tests for CL metrics for 6.0.x
  • [METRICS-1816] [METRICS-1705] Fixing some metric names in default whitelist and configured local telemetry exporter metrics whitelist configuration for SBK
  • [CNKAF-867] Handle empty proposals for addBroker
  • METRICS-1717 use shared domain for jvm level metrics (#1949)
  • Migrate MDS to KafkaHttpServlet implementation
  • CNKAF-854: Disabling SBK should persist the broker removal cancellation status
  • KC-138: Implements cluster link pausing.
  • [CNKAF-873] Don’t require network capacity configs. (#1996)
  • CNKAF-884: Remove unnecessary metrics
  • SEC-1183 - Treat empty authority in CRN pattern as a wildcard
  • CNKAF-882: Add proposal logging for add/remove
  • CNKAF-837: Add broker should be able to add removed brokers.
  • METRICS-1717 use shared domain for system level metrics
  • KC-393: Integration tests for mirroring compacted topics
  • Add inter broker listener endpoint information to HTTP server binder.
  • Add inter broker listener endpoint information to HTTP server binder.
  • CNKAF-773: Do not throw an exception when the broker is not present in the cluster model
  • CNKAF-847: Complete removal callback when there are no proposals to execute
  • KSTORAGE-802: Allow specifying AWS credentials through configuration file
  • [METRICS-1552][METRICS-1719] remove telemetry broker assumptions and use metrics context for broker check
  • [CNKAF-848] Set preferred user SBK config in server.properties. (#1999)
  • [CNKAF-873] Don’t require network capacity configs.
  • [CNKAF-719] Add option to disable dynamic throttle setting when stati…
  • KSTORAGE-819: Force roll segments only for partitions eligible for tiering
  • KSTORAGE-799: reduce Log.lock contention in ArchiveTask segment selection logic
  • KSTORAGE-774: Fix active segment roll logic when roll.min.bytes setting is enabled
  • KSTORAGE-766: Ensure delete segment throttler does not block active segment rolls
  • KSTORAGE-769: Fix to remove the costly toSet conversions in disk throttler
  • KSTORAGE-758: Include ERROR status in TierPartitionStatus.isOpen()
  • KSTORAGE-595 NPE in TierDeletedPartitionsCoordinator after tier topic is reassigned / deleted from broker
  • [CNKAF-848] Set preferred user SBK config in server.properties.
  • KSTORAGE-754: KafkaMetrics Whitelist Creation and new KafkaMetricsWrapper util (#1906)
  • KSTORAGE-754: KafkaMetrics Whitelist Creation and new KafkaMetricsWrapper util
  • Revert “KC-138: Implements cluster link pausing. (#1994)”
  • KC-138: Implements cluster link pausing.
  • KC-611: Fix mirror topic create processing incorrect config.
  • KSTORAGE-547: state recovery injection tooling
  • [METRICS-1235] implement exporter level whitelist
  • CONFLUENT: Fix Confluent provider start future completion, add unit tests
  • [CNKAF-881] SBK: Don’t treat all new brokers as empty. (#2005)
  • SBK/CruiseControl: low utilization fix
  • CNAKF-833, CNKAF-842: Fix missing request rate metric and enable tele…
  • KC-521: Add per listener authentication audit log enable config
  • CNAKF-833, CNKAF-842: Fix missing request rate metric and enable telemetry reporter sampler by default
  • KSTORAGE-547: state recovery injection tooling
  • CIAM-94 Decouple scope path keys and Role scope type
  • RackAwareGoal should allow for uneven replica distribution based on t…
  • [CNKAF-881] SBK: Don’t treat all new brokers as empty.
  • [CNKAF-719] Add option to disable dynamic throttle setting when stati…
  • KC-365: Add Jmh benchmarks for Authentication Audit Events
  • CNKAF-788: Introduce proper API exceptions in the broker removal API
  • KSTORAGE-706: Durability events and database.
  • METRICS-1708: add telemetry version header for forward-compatibility (#1883)
  • Allowing user to disable tiered storage safely - Phase 1
  • CONFLUENT: Use undeprecated ciphers for TLSv1.3 cipher change test
  • Port from CC: Fix low utilization threshold computation (#92)
  • CNKAF-791: Change data format of failedBrokers in ZK
  • CNKAF-807: Remove zkclient from gradle dependencies.
  • CNKAF-575: Remove IOtec:ZkClient dependency.
  • [CNKAF-776] Switch SBK Maven publish group to “io.confluent”
  • [CNKAF-776] Make SBK publish to “io.confluent”
  • RackAwareGoal should allow for uneven replica distribution based on t…
  • [METRICS-1609] dynamically add telemetry reporter at runtime
  • CONFLUENT: Move clusterId set on AuditLogProvider Instance to AuditLogProviderFactory
  • KC-389: Retry segment deletion in cluster link test to fix timing issue
  • KC-389: Cluster link integration tests with tier storage enabled
  • CPKAFKA-5239 - Fix AddPartitionsToTxn audit event category
  • CPKAFKA-5207: Complete authorizer start future only when sufficient nodes are ready
  • CNKAF-843: Fix remove broker validation check
  • Persist the broker removal state and handle failover
  • CNKAF-773: Do not throw exception when there are no replicas to move on broker removal
  • CNKAF-428: Integration test for replica placement self-healing
  • Add retries to telemetry sampler integration test (#1922)
  • [CNKAF-803] Add a metric to track the number of pending add-broker requests
  • KC-72: Add topic listing for cluster links.
  • KC-519: Support Authentication events in audit log routing
  • CPKAFKA-5207: Fail fast during broker start up if multiple brokers use same MDS URL
  • KC-536 bug fix to make offsets/acls configs dynamic (#1893)
  • [CNKAF-803] Add a metric to track the number of pending add-broker requests.
  • CNKAF-428: Update ReplicaPlacementGoal to handle replication factor change
  • KC-545: Allow configuring ssl cipher suites at the cluster level
  • fix telemetry reporter reconfiguration
  • KSTORAGE-623: added disk throttling metrics for producer and follower throttle rates
  • KC-536 bug fix to make offsets/acls configs dynamic
  • [KSTORAGE-610] Add metric for tier materialization lag
  • update ksql and streams provider
  • CNKAF-649: Handle broker restart during broker removal
  • KC-390: Cluster link authorization tests
  • KC-366: Add authentication audit logging support to Selector and authentication implementations
  • KC-594: use BufferSupplier GroupMetadataManager.scheduleLoadGroupAndOffsets
  • METRICS-1708: add telemetry version header for forward-compatibility
  • backport [METRICS-1564] telemetry - derive interbroker client configs (#1816)
  • ksql and streams provider use configs for building labels
  • (Backport) make commit-id optional, update whitelist to include C3 metrics
  • make commit-id optional, update whitelist to include C3 metrics
  • SEC-1083 - AuditAdmin role’s scopeType = Cluster
  • SEC-1083 - AuditAdmin role’s scopeType = Cluster
  • KC-544: Alter configs from external listener should only update said listener configs
  • CNKAF-428: Update ReplicaPlacementGoal to handle replication factor change
  • KC-392: Integration tests for cluster link quotas on source and destination
  • KC-582: Add multi-tenant support for alterMirrors.
  • KSTORAGE-705-Define per broker durability metrics and its management
  • CNKAF-795: Ensure SBK clients have bootstrap.servers configured
  • [METRICS-1564] telemetry - derive interbroker client configs
  • CIAM-71 Handle RBAC access policies at different scopeLevels
  • fix checkstyle error
  • [KAFKA-8528] Expose Trogdor-specific JMX metrics for Tasks and Agents
  • Backport PR 1897 to 6.0.x
  • update ksql and streams provider
  • SEC-1083 - Add AuditAdmin role
  • CNKAF-794: Properly wrap broker removal exceptions in listener and improve remove broker test
  • Add schemas topic label to SchemaRegistryProvider

Security

  • Fix auth for SchemasResource
  • Fix SR security test classpath
  • Implement asychronous schema fetching in LruSchemaRegistryClient
  • SEC-1478 Update from JUnit4 to JUnit5
  • Implements FF-873 in a generic way by supporting wildcards *
  • Check authentication.roles in KSQL bearer token requests
  • Include Kafka dependencies to KSQL packaging
  • DG-540 Add batch get schema API
  • fix: Make rest-utils compile scope as ksqlDB no longer pulls this in

MQTT Proxy

  • Fix telemetry reporter test on 6.0.x

Metadata Service

  • SEC-1565: fix unit test based on SEC-1565 on 5.4.x
  • SEC-1565: Incorrect implementation of equals in ManagedRoleBindings
  • SEC-1566: Switch to https for packages.confluent.io
  • SEC-1507 cluster information migration script should validate properties file
  • [MMA-9298]: bump up the version for nice name feature
  • [SEC-1506]: cluster-information-migration-script - allow user to specify curl options
  • [SEC-1514]: cluster-information-migration-script - suppress progress but show errors
  • [SEC-1499]: Add –help option to cluster registry import/export script
  • CIAM-399: decouple rbac-db from rbac-api-server
  • SEC-1494: Update to junit5
  • CIAM-338 - EnvAdmin can see more that it should in cli
  • [SEC-1471]: Backport Friendly cluster name from master to 6.0.x
  • CIAM-371 Add Delete operation to Cluster for CloudClusterAdmin
  • CIAM-338 - Test MyRoleBindings endpoints
  • CIAM-325 Add ON_ERROR_STOP inside rbac_schema_postgres.sql
  • CIAM-318 Define cc-rbac Postgres users in SQL
  • SEC-1452: refactor out cluster name validation to its own validator
  • CIAM-121 Environment variable is named DBMIGRATE_SHARE_DIR
  • SEC-1419: fix script to exit with 0 after successful execution
  • CIAM-121 Support DB Migrations
  • CIAM-313 give USAGE privilege to RBAC DB users
  • Add implementation for newly added “describe and alter SCRAM user” API
  • SEC-1402 Cluster Registry Unicode Cluster Names
  • CIAM-257: Add db healthcheck skip option for cpd
  • CIAM-269 Terraform to create schema and DB roles
  • CIAM-295 - Verify MDS behavior with CCloud setup and ANON super.user
  • Restore Cluster Registry import script in master
  • CIAM-241 - Port v2 tests to demonstrate calls to DB
  • MMA-7359 - Space in Principal Breaks Control Center Authorization
  • CIAM-248 - Parity Suite Infrastructure in MDS
  • CIAM-269 Terraform for CC RBAC RDS
  • CIAM-261: Add metrics for all rest end points
  • CIAM-262 Use connection pool for RBAC DB
  • CIAM-222: Config support for db
  • CIAM-134 Add test to demonstrate /lookup/rolebindings/principal
  • CIAM-245: Add DbAuthCache/DbAuthWriter metrics
  • SEC-1382: validate groups in “super.users”
  • CIAM-236 Add test for removal of env describe when cluster admin removed
  • CIAM-221 Implement list rolebindings queries
  • CIAM-102: Add caching to DBAuthCache findRule method
  • SEC-1372 - MDS : Address V1 Visibility Testing Gap found by C3
  • CIAM-227: (follow-up) Add DbAuthStoreConfig and minor cleanups to DbAuthWriter/RbacOrmService
  • CIAM-237 rename resource_id to id
  • CIAM-227: Implement resource RoleBinding methods and add validations to DBAuthWriter
  • CIAM-223: Support add, delete, lookup rolebindings from DB
  • CIAM-82 Cloud RBAC Schema
  • CIAM-232 Add endpoint to delete all role bindings for a principal
  • CIAM-82 Data generator
  • CIAM-119 - Rbac DB Stubs
  • CIAM-119 - DB Auth Maven Module and Stubs
  • CIAM-162 Provide /lookup/rolebindings endpoint
  • CIAM-86: Add perf tests for RBAC control plane authorizations
  • CIAM-110 Fix yaml indentation
  • CIAM-140 Add tests for /principals endpoints
  • CIAM-110 Associate RBAC Roles with Namespaces
  • CIAM-59: Readme update
  • CIAM-59: Update mds-in-a-box to support v2alpha1 rbac
  • SEC-1089 Fix v2 yaml MdsScope
  • CIAM-145 Update sdk yaml to use lookup role endpoint, make bindWithRe…
  • Fix aquirePort typo in test helper function
  • CIAM-138 Test /roles endpoint
  • SEC-1226 - Updated audit log api code after ce-audit added a new category event authentication
  • CIAM-132 Don’t allow empty principal name
  • SEC-1229 - MINOR: Disable getUpdatedAuditRouteConfigTest and dependents because …
  • SEC-1205 Gracefully handle roles request for nonexistent role
  • CIAM-59: Update MdsWritesFilter pathMatcher for v2alpha1
  • CIAM-109 fixed checkstyle
  • CIAM-109 add hierarchy traversal for managed resource endpoint
  • Have ce-kafka inject the (internal) ConfluentAdmin directly, instead …
  • Cherry pick 297: Move MDS to KafkaHttpServlet
  • CIAM-96 fix unit tests based on changes in cloud cluster admin role
  • CIAM-126 Make /roles/{roleName} and /roleNames v2 endpoints available to mds sdk
  • CIAM-125 Update Roles object
  • SEC-1127 - Fixed a defect that AuditLogConfigSpec didn’t get updated
  • CIAM-59: Disable token endpoints if no user store configured
  • CIAM-100 Allow scopes with SR cluster but no Kafka cluster
  • CIAM-27 Lookup scope includes contained scope
  • CIAM-94 Upstream interface changes
  • CIAM-59: Config to skip userstore setup/basicauth fallback
  • CIAM-79 Update MDS for changed interfaces in CIAM-71
  • removed required kafka-cluster in Scope

Replicator

  • REPL-1310 fix for monitor thread failures in start stop start cycle
  • REP-1300: Null out licenseManager on connector stop to avoid hang on restart
  • REPL-1303: Setting deadline correctly when trying to resume partitions
  • FF-3928:Exclude test dependency javassisst from curator-test to be assembled
  • FF-3928: Bump the version of Guava
  • REPL-1263: Supporting source url configs for active active SR
  • REPL-1204: Enable retries on cluster id fetch for Replicator task start
  • REP-1193: Removed error logging for failed executable connector submission
  • REP-1056: Handle empty assignments
  • REP-1046: Improve logging on conversion error
  • REP-917: Executable should not log an errors when attempting to start connectors on followers
  • REP-762: Add a READ ACLs check in verifier tool for destination topics
  • REP-763: Improve WARN log to explain if the error is on source or dest location