Confluent Platform Component Changelogs

This topic provides changelogs for the individual Confluent Platform components.

Version 6.2.15

Released April 2024

Community Features

Common

  • 4fb11814 - Remove auto-generated CODEOWNERS. It’s inaccurate.
  • 52b0f806 - Remove CODEOWNERS file
  • 7a96a215 - Add service.yml
  • c4b849a5 - update jose4j to address
  • 4c8d3021 - update jetty to 9.4.54
  • PR-581 - fix: no implicit activation of docker-arm profile
  • PR-563 - added a new docker-arm profile to build docker containers in arm64

Kafka

  • PR-1079 - [2.8] Update jetty to 9.4.54.v20240208

ksqlDB

  • PR-10238 - Update apache.io.version
  • PR-10237 - Update commons-compress
  • 2f84873b - Change groupID to a more secure name
  • f969d5c0 - Enable pinned nanoversions.
  • 37e06168 - Fix the parent version in commons.
  • 5467ea20 - set java version to 8
  • 0a1cf171 - Increase the execution time limit.
  • db81cd09 - Fixed the extra args in service.yml file.

REST Proxy

  • PR-1235 - Deflake KafkaConsumerManagerTest.testBackoffMsControlsPollCalls test
  • PR-1236 - disable flakey test

Schema Registry

  • PR-3012 - Minor: Apply service bot changes to earlier branches
  • abbab267 - Ensure package change for Protobuf is incompatible
  • 3bd1cdd5 - update common-compress
  • 177af0d0 - Add Protobuf compat check for moving field into existing oneOf
  • bbe261f4 - exclude okio
  • PR-2943 - Update Bouncycastle dependencies to address

Version 6.2.14

Released December 2023

Community Features

Common

  • e99d8b1a - exclude logback-core and logback-classic in line with the changes in kafka-rest
  • eb200670 - update zookeeper component to match version in ce-kafka
  • 0c034e86 - re-add dependencymanagement sections for old bouncycastle to satisfy downstream validation
  • 33734c5f - add additional version definition to ease the update of the downstream repos
  • 8e71caf2 - update bouncycastle

Kafka

  • PR-12103 - KAFKA-13270 Backport Jute buffer default to 6.2.x (#12103)
  • PR-12034 - CONFLUENT: Bump ZK version to 3.8.3 for 6.2.x (#12034)

REST Proxy

  • PR-1230 - KREST-12372 Upgrade dependency to JDK 1.8 with version 1.77
  • PR-1221 - KREST-12630 Exclude logback from zookeeper dependency in tests
  • PR-1219 - KREST-12630 Add missing dependency due to zookeeper upgrade
  • PR-1212 - KREST-12277 semaphore migration to release branches
  • PR-440 - KREST-11812 upgrade jaxb-api dependency
  • ee8dd479 - KREST-12364: update bcpix
  • PR-427 - KRES-12288 migration semaphore release branches

Schema Registry

  • 98a1a4de - update swagger core
  • PR-2888 - Apply service bot changes to version branches
  • bafbad8c - DGS-9343 Allow removing enum in JSON Schema to be backward compatible
  • PR-2874 - DGS-9048: Semaphore Migration of Version Branches
  • 7f7a7ab1 - add metrics-core to fix the build
  • PR-2816 - DGS-8960 Fix handling of diamond structure in Protobuf converter

Version 6.2.13

Released October 2023

Community Features

Common

  • 35a1415d - APPSEC-3039: Update jetty and netty
  • 81295738 - update snappy to 1.10.5
  • aea31af8 - APPSEC-2796: add guava to dependencyManagement

Kafka

  • PR-974 - KSECURITY-2013: Upgraded Jetty to 9.4.53.v20231009 and Netty to 4.1.100.Final
  • PR-962 - KAFKA-15498: bump snappy-java version to 1.1.10.5 (#14434)
  • PR-14494 - HOTIFX: fix Kafka versions for system tests (#14494)

Version 6.2.12

Released September 2023

Community Features

Common

Kafka

ksqlDB

  • d3f0e149 - fix: fix compile error introduced by merge conflict resolution

REST Proxy

Schema Registry

  • 3f0e1198 - DGS-7927: update io.grpc_grpc-protobuf
  • PR-2695 - do not reinstantiate objectmapper as it’s expensive (#2686)

Version 6.2.11

Released June 2023

Community Features

Common

  • PR-530 - Backport log redactor to CP patches.
  • 22a9da57 - APPSEC-2551: Update Prometheus JMX Exporter
  • 5ad03d12 - APPSEC-2551: Update Jackson
  • PR-514 - Update snakeyaml
  • b4334b94 - pin jose4j version to match ce-kafka
  • 95ed5124 - include snappy and jetty in the dependency management to match versions with ce-kafka, cleanup jackson imports

Kafka

  • 41791013 - KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires
  • PR-910 - KSECURITY-1627 KSECURITY-1556: update snappy and jetty

ksqlDB

  • PR-9810 - Bump minimist and mkdirp
  • 4ce2d5f3 - fix: Replace regex in CommandParser with a split by space
  • 08853cd3 - Fix log4j-extensions

REST Proxy

  • ab17aa29 - Fix the overzealous conflict resolution
  • 5be3406e - KREST-9942: Remove SnakeYaml from dependency management
  • a54dde7e - CNC-366: Update Jetty

Schema Registry

  • e0626f64 - Update the test to make it clear to customer
  • dc6f2096 - update the doc
  • 33659feb - Update schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java
  • 3ffd8fc7 - Update schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java
  • 1ca2a39d - Update the codes with checking string for http connect timeout
  • ffba37fe - * changed the config names
  • 4c383ee4 - Apply suggestions from code review
  • 7155f0fb - * checkstyle error
  • 7e2dec9c - * removing unused import
  • 9ab1b0ec - * adding defaults
  • 01c6d7a7 - * Made SR client http connect and read timeout configurable.
  • 89aa28be - * Made SR client http connect and read timeout configurable.
  • PR-2642 - DGS-7367 Propagate error messages even if not JSON
  • ed4ec914 - DGS-6698: Remove SnakeYaml from the dependencyManagement
  • PR-2589 - DGS-6613 Ignore invalid Avro defaults in Avro Converter
  • PR-2588 - DGS-6603 Fix default conversions in JsonSchemaConverter
  • PR-2693 - DGS-7746 Fix perf regression due to DGS-6331
  • PR-2684 - DGS-7675 Fix NPE when print.schema.ids=true
  • 1bf961b7 - Update json-schema

Commercial Features

Confluent Server

  • add logredactor dependency for 6.2.x
  • CC-18990 : Backport log redactor connect 6.2.x
  • KSECURITY-1568: Upgrading jose4j
  • KSECURITY-1556: updated jetty version
  • KAFKA-14887: FinalizedFeatureChangeListener should not shut down when
  • KSECURITY-1466: Updated json-smart, nimbus-jose-jwt
  • KSECURITY-1459: updated golang.org/x/net mod.
  • KSECURITY-1455: updated golang.org/x/text mod.
  • KSECURITY-1453: Updated version of golang.org/x/sys
  • KSECURITY-1627: Updated snappy-java
  • KSECURITY-1618: update guava version

Security

  • CIAM-2998 - Log redactor backport for confluent-security-plugins

Secret Registry

  • unpin netty, as rest-utils brings updated version

Control Center

  • [MMA-13078] revisit session caching in basic auth
  • [MMA-13089] Handle large message JSON on topic inspection
  • [MMA-13042] Fixing log statement for websocket time out
  • [MMA-13216] update jose4j

MQTT Proxy

  • CC-19108 | Rename isChannelClosed to isAuthenticated.
  • CC-19108 | Use an atomic reference to avoid publishing message when the channel is closed or marked for closure.

Metadata Service

  • Update pom.xml
  • CIAM-2986 - Older MDS builds are not running all the tests they should be
  • update comment, clean up whitespaces
  • another attempt with mina-core
  • pin mina core in cli, scope mina-core as test in main pom
  • pin mina-core in testingLdap only
  • SEC-xyz: Update mina-core

Replicator

  • disable builds of connect-replicator version 5.5.x
  • remove aws-maven plugin to unblock the downstream validation for common: RCCA-10412, APPSEC-2551

Version 6.2.10

Released March 2023

Community Features

Common

  • PR-512 - Add snakeyaml to dependency management
  • 7bcaab9e - Fix typo in with method calls
  • 3868e27b - Fix typo
  • 783c5721 - Return the original “version_range” if it isn’t an actual Maven version range
  • PR-502 - Fix: override maven version plugin’s default versionrange
  • 9350d231 - Update resolver-maven-plugin to 0.6.0
  • PR-497 - Upgrade Netty to 4.1.86.Final
  • PR-496 - Bump to 7.4.4

Kafka

  • PR-13245 - KAFKA-14704; Follower should truncate before incrementing high watermark (#13245)
  • PR-875 - KC-2332: Upgrade netty to 4.1.86
  • PR-866 - revert: CONFLUENT: Skip publishing for kafka-streams-upgrade-system-tests & CONFLUENT: Fix filter for not publishing streams upgrade test artifacts
  • PR-857 - CONFLUENT: Fix filter for not publishing streams upgrade test artifacts
  • PR-853 - CONFLUENT: Skip publishing for kafka-streams-upgrade-system-tests
  • PR-860 - KAFKA-9648: Add configuration to adjust listen backlog size for Accep
  • PR-846 - Added support for skipping upload task for older gradle versions

ksqlDB

  • aefe6338 - Fix log4j-extensions
  • PR-9821 - Fix: migrations tool without basic auth works again
  • PR-9770 - Upgrade netty to 4.1.86 #2
  • PR-9717 - Fix: change password-based configs to Type.PASSWORD

REST Proxy

  • PR-1108 - Set kafka-schema-registry version in dependency mamanagement
  • 7a7f5dcd - Remove unnecessary ${project.version} tag
  • PR-1102 - Fix transitive dependency of kafka-schema-registry
  • PR-1101 - Fix dependency org.yaml_snakeyaml, upgrade to fixed

Schema Registry

  • PR-2575 - DGS-6331 Handle javaType for oneOfs during JSON deserialization
  • PR-2568 - DGS-6306 Allow multiple oneofs in Protobuf converter
  • PR-2540 - DGS-6075 Add explicit definition of snakeyaml.
  • PR-2519 - DGS-6014 cherry pick Maven plugin enhancement to 5.5.x
  • PR-2514 - DGS-5950: Allow optional map key in Avro

Commercial Features

Confluent Server

  • KMETA-478 Enable zk audit logs by default
  • Fix version comparison in kafkatests (#8922)
  • KSECURITY-897: Make expiration timeout for ZK ACL change notification path configurable
  • KC-2328: Update yaml.v2 to 2.2.4
  • KSECURITY-953: Update azure-identity to 1.7.3
  • KSECURITY-981: Add FeatureZNode zk node path to secure root paths
  • KC-2333: Update io.netty libraries to 4.1.86.Final
  • CNKAF-103; KIP-496 does not work in multi-tenant clusters
  • HOTFIX: increasing file read buffer size for SSL transfer
  • KSTORAGE-82: Tiered storage topic deletion support
  • CPKAFKA-3847 Reuse AdminClient in event appender
  • added support for skipping upload task for older gradle versions
  • MINOR: avoid unnecessary generation of PartitionState objects in ReplicaFetcher
  • KAFKA-9038: [WIP] Allow creating partitions for topics partitions not in reassignment
  • CONFLUENT: Use single audit log provider for MDS and its hosting broker
  • CONFLUENT: Sync from confluentinc/kafka (18 October 2019)
  • KSTORAGE-252: update partition state endOffset on deleteInitiate

Security

  • Remove logging statement
  • SEC-5350: Update netty-codec-http to 4.1.86.Final

Secret Registry

  • Remove unused variables
  • Update RestService.java

Control Center

  • RCCA-9557 - fix broker flapping issue
  • Change to make this compatible with 6.2.x
  • KC-2261 timeout websocket connection via jwt token
  • Update CODEOWNWERS to match latest definition from master
  • Refactoring- made code more functional and improved readability.
  • Updated NotFoundException message in scenario where topicName provided returns 0 no consumergroupdata.
  • Updated code based on PR feedback.
  • Updated consumerGroupId API to also support topicName queryParam.
  • Improving test cases with scenarios of single consumergroup covering multiple topics. 2. updated consumerGroupId to not support topicName queryParam, it is not required.
  • Adding unit test for CachedConsumerOffsetsResource
  • Adding support for topicName in query param to return only consumer groups for that topic
  • Reading password using getPassword instead of getString, this fixes the casting exception
  • Fixing failing test case, due to merge issue
  • Renaming Health_check to health check
  • Adding config based functionality to force http1 for armeria health checks
  • MINOR: Update repo to use mvn-wrapper to speed up builds

MQTT Proxy

  • Remove the pin of netty version to use the (current) version defined in common

Metadata Service

  • Clean testng
  • [Fix build] Drop down to testNG 7.5 - which is the last version to support jdk8
  • Clean up surefire
  • CIAM-2615 - Bump TestNG to 7.7.0
  • Fix NotNull import
  • CIAM-2570: Backport CIAM-2183 fix to CP in active 6.1.x
  • CIAM-2570: Backport CIAM-2183 fix to CP in active 6.0.x
  • Increase buffer size for Proxy Servlet

Replicator

  • Testing a build fix

Version 6.2.9

Released February 2023

Commercial Features

Security

Version 6.2.8

Released December 2022

Community Features

Common

  • PR-492 - APIF-3112: Upgrade jmx_prometheus_javaagent to 0.17.2.
  • f5764727 - Merge conflict resolution.
  • PR-489 - APIF-3117: Upgrade scala to 2.13.10.
  • PR-485 - APIF-2957: Upgrade Jackson and Protobuf versions.

Kafka

  • PR-845 - CONFLUENT: Skip publishing for projects with no scala suffix when the scala version is not the default
  • PR-842 - DP-9030: Use the new withGradleFile closure
  • PR-834 - KSECURITY-792: Upgrade from Scala 2.13.8 to 2.13.10
  • PR-820 - KSECURITY-788: Upgrade jackson-databind version to 2.13.4.2

ksqlDB

  • PR-9708 - APIF-3117: Unpin scala version.
  • PR-9697 - APIF-3122: Unpin Protobuf version.

REST Proxy

  • PR-1072 - KREST-7107 Shouldn’t provide overly detailed info or stack traces in responses -

Schema Registry

  • PR-2440 - Update the Snakeyaml dependency
  • 8e5ce168 - Unpin Protobuf version.
  • PR-2414 - DGS-5253 Handle cycles when resolving references
  • PR-2405 - DGS-4754 Handle empty record default at field level
  • PR-2399 - DGS-5084 Ignore compat check in IMPORT mode
  • PR-2395 - DGS-4971 Handle map types with enhanced.protobuf.schema.support

Commercial Features

Confluent Server

  • CONFLUENT: Skip publishing for projects with no scala suffix when the scala version is not the default
  • DP-9030 - Switch to codeartifact repo
  • Increase vagrant aws timeout for system test
  • BouncyCastle Issue fix for Connect service [6.2.x]
  • KSECURITY-792: Upgrade from Scala 2.13.8 to 2.13.10
  • KSECURITY-788: Upgrade jackson-databind version to 2.13.4.2
  • KSECURITY-832: Upgrade bc fips drivers
  • CIAM-2424 - Name RestClient threads for stackdump identification (#7887)
  • KSECURITY-794: Upgrade to protobuf version 3.19.6
  • CONFLUENT: Remove “principal.builder.class” config from Broker Client configs
  • Add stripped down transaction tool which allows aborting hanging transactions
  • CIAM-2304: Add SecurityMetadata:Describe to SecurityAdmin
  • RCCA-8564: log a warning if LDAP login fails becuase of network issues

Security

  • Exclude bcpkix-jdk15on as compile dep and add as test dependency
  • Pin bc-nonfips dep as a test dependency
  • Pin bouncycastle fips dependency in confluent-security-plugins
  • Removed bcfips from common
  • excluded bcfips jars
  • AUTHN-1087: Upgrade vertx to 3.9.14
  • AUTHN-1087: Upgrade vertx to 3.9.14
  • AUTHN-1087: Upgrade vertx to 3.9.14
  • AUTHN-1325: Update bc-fips version to 1.0.2.3
  • fixed checkstyle error
  • Requesting permission only for Subject Read operation instead of any of the subject operation in case of GET /schemas

Secret Registry

  • Exclude Non-FIPS dep and use bc-fips dependency

Control Center

  • MMA-9023: memory bounding for streams applications [5.4.x, master]
  • Revert “MMA-12206 upgrade io.netty:netty-codec-http”

MQTT Proxy

  • Fixing the tests and config validator, to incorporate new changes in validation of MqttDecoder brought in by netty upgrade
  • Upgrading netty

Metadata Service

  • CIAM-2392 Bump PSQL Version
  • Remove spaces
  • Add back newline
  • Implement writeTransactionMarkers in DelegateConfluentAdmin
  • CIAM-2304: Add unit tests for SecurityAdmin

Replicator

  • REPL-2055-: Log WakeupException at debug level
  • RCCA-7678: Reverse proxy header check added

Version 6.2.7

Released August 2022

Community Features

Common

  • PR-472 - Upgrade Netty to 4.1.79.Final
  • PR-468 - Fix dependency in disk-usage-agent [6.2 and 7.0]
  • PR-464 - Migrate from confluent-log4j to reload4j [5.4.x – 7.0.x]
  • PR-467 - Upgrade JUNIT in 5.4.x only

Kafka

  • 62cef718 - Upgrade Jetty to 9.4.48.v20220622
  • PR-765 - migrating log4j12 to reload4j, slf4j-log4j12 to slf4j-reload4j.jar (2.8)
  • 967bc9cf - upgrading netty to v4.1.79.Final (2.8) #758

ksqlDB

  • PR-9337 - bugfix: remove log4j from the classpath (#9334)

REST Proxy

Schema Registry

  • PR-2351 - DGS-4358 Fix NPE in Protobuf converter for null map value
  • 0577dffb - Add comments
  • 042d3e3c - DGS-4303: Support different stat types in SchemaRegistryMetric
  • PR-2342 - APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
  • PR-2339 - APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.

Commercial Features

Confluent Server

  • KGLOBAL-1918: Source brokers only create active-link-count metrics wh
  • KAFKA-14107: Upgrade Jetty version.
  • Migrating log4j12 to reload4j, slf4j-log4j12 to slf4j-reload4j.jar (6.2.x)
  • Avoid sending partial updates during LDAP group manager startup
  • Fix exception handling in RBAC writer coordinator
  • Upgrading aws-java-sdk-s3 to v1.12.268 (6.2.x) #6974
  • Upgrading netty version to 4.1.79.Final (backport 6.2.x)
  • Upgrading google-cloud-storage to v2.6.1 (6.2.x) #6990
  • Update bouncy castle to 1.70 (back port 6.2.x)

Security

  • add auth token provider to ksql security extensions
  • revert pom
  • add auth token provider to ksql security extensions
  • AUTHN-1085: Fix NPE in debug log
  • APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.

Secret Registry

  • Migrate from log4j to reload4j for connect-plugin
  • CC-17641,CC-17644 Bump jetty version
  • APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.

Control Center

  • MMA-12201 remove slf4j from transitive deps
  • MMA-12206 upgrade io.netty:netty-codec-http
  • MMA-12201 Exclude slf4j in transitive dependency
  • MMA-12201: Use reload4j version properties from common
  • MMA-8781: Respect SR Config and handle nulls in CachingSchemaRegistryClientSupplier
  • MMA-12201 Replace log4j with reload-4j
  • MMA-12127: Change type from String to Password for CONTROL_CENTER_EMAIL_PASSWORD

MQTT Proxy

  • CC-17620: remove dependency on slf4j-log4j12 and confluent-log4j

Metadata Service

  • CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS
  • fixes 6.0.x build from 5.5.x merge
  • CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS

Replicator

  • Migrate confluent-log4j to reload4j.

Version 6.2.6

Released July 2022

Community Features

ksqlDB

  • PR-9260 - fix: Excludes Guava from Guava-retrying in order to manage Guava depe
  • 617df451 - merge conflicts
  • 4ef7a1df - merge conflicts

REST Proxy

  • PR-329 - APIF-2705: Update jersey version to 2.36.

Schema Registry

  • 3636b48c - Fix merge issue
  • af27c4c0 - Fix merge issue
  • 31183d32 - Fix merge issue
  • ce22bb34 - Fix merge issue
  • a930a7b6 - Fix merge issue
  • 6c78adf4 - Fix merge issue
  • d46bebb5 - Fix merge issue
  • PR-2331 - DGS-4172 Bound size of Avro datumReader/Writer caches
  • PR-2328 - DGS-4162 Handle refs at different levels in Protobuf converter
  • PR-2327 - DGS-1648 Allow Protobuf msg fullname to be passed for console producer
  • PR-2321 - DGS-4016 Fix handling of UInt32Value in Protobuf converter
  • PR-2318 - DGS-3974 Fix StackOverflow with recursive JSON schema in converter

Commercial Features

Control Center

  • MMA-11938 Make connect healthcheck path configurable
  • Deprecate CONTROL_CENTER_STREAMS_RETRIES_CONFIG and remove default usage

Metadata Service

  • CIAM-2141 - Remove usages of MiniKdcWithLdapService

Version 6.2.5

Released June 2022

Community Features

Common

  • PR-452 - KSECURITY-277: Upgrading gson
  • PR-456 - DGS-3674 Upgrade gson to 2.8.6
  • PR-446 - Upgrade jackson related packages
  • PR-445 - Unify protocol buffer version.

Kafka

  • PR-704 - KSECURITY-220: Update jackson packages to 2.13.2
  • PR-11946 - KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (#11946)
  • PR-11891 - KAFKA-13727; Preserve txn markers after partial segment cleaning (#11891)
  • PR-11872 - KAFKA-12879: Remove extra sleep (#11872)
  • PR-11871 - KAFKA-12879: Addendum to reduce flakiness of tests (#11871)
  • PR-11797 - KAFKA-12879: Revert changes from KAFKA-12339 and instead add retry capability to KafkaBasedLog (#11797)

REST Proxy

  • PR-1016 - KREST-5096 Fix NPE in Topics Resource when no key, or no value specified
  • PR-942 - Fix(#341): Remove deadlock when rebalancing by multiple consumer instances
  • PR-991 - KREST-4825 Huge timeout in consume test
  • PR-1026 - Backport consumer deadlock fix to 5.4.x

Schema Registry

  • PR-2290 - DGS-3674 Upgrade to Protobuf 3.19.4
  • PR-2285 - Merge Conflict Resolution (from 5.4.x to 5.5.x)
  • PR-2283 - DGS-3659 Handle null default in JSON Schema converter
  • a5c4a3b7 - Override dependency of kotlin-scripting-compiler-embeddable to Kotlin version
  • bumping kotlin componenets version to 1.6 for kafka-connect-protobuf-converter
  • 96471721 - Upgrade kotlin to 1.6.0
  • PR-2259 - Fix upgrade to Jackson 2.13
  • PR-2257 - DGS-3548 Optimize cache for JSON Schema converter
  • PR-2253 - DGS-3535: Remove enforcement of building with JDK8
  • PR-2241 - DGS-3484 Allow NaN when parsing JSON
  • PR-2227 - DGS-3417 Fix default value conversions in JsonSchemaConverter
  • Upgrading snakeyaml
  • 165e0f5f - Fix merge issue
  • PR-2215 - DGS-3250 Return 422 instead of 500 if schema lookup is invalid
  • PR-2214 - DGS-3248 Return 422 instead of 500 if subject is READONLY during DELETE
  • PR-2201 - DGS-3066 Perf optimizations for AvroSchema
  • PR-2200 - DGS-3133: Compare canonical string instead of jsonNode when comparing JsonSchema

Commercial Features

Confluent Server

  • Upgrade Gson
  • KSTORAGE-1606: FileTierPartitionIterator incorrectly handles mixed size prefixed entries (#3893)
  • use correct cp urls in dockerfile - 6.2.x
  • KSECURITY-221: Update jackson packages to 2.13.2 version
  • KSTORAGE-82: Tiered storage topic deletion support
  • KSECURITY-198: Update protobuf-java packages to 3.19.4
  • Backport of KMETA-8 to 6.2.x
  • KAFKALESS-1032: Use wall clock to prune task history
  • KENGINE-139: do not synchronize when reporting fetch session cache metrics #5831
  • build: kafka system tests should support larger EBS volumes for newer instances (#6371) (#6406)

Control Center

  • Bump maven-shade-plugin to 3.3.0
  • Fix build break
  • Override transitive dependency for analytics library of kotlin-stdlib-common
  • Add exclusion to override dependency kotlin-stdlib-jdk8
  • LOG_DIR to match rest of confluent components

Metadata Service

  • CIAM-2020 - Fix null string in MDSLoggingFilter
  • CIAM-2019 - Fix 6.x MDS builds from Jackson update
  • MINOR SEC-3087 SEC-3028 Avoid NPE in integration test
  • RCCA-6743: Test that we can allow rest client to disable hostname verification
  • This release version includes a fix to prevent stack trace exposure from getting surfaced in HTTP responses for malformed requests.

Version 6.2.4

Released April 2022

Commercial Features

Metadata Service

  • MDS-38 MDS-39 - Make MDSLoggingFilter case-insensitive

Version 6.2.3

Released March 2022

Community Features

Common

  • PR-428 - Upgrade Netty
  • 605b9b9d - use 3.17.0 protobuf for version after 5.5.x
  • PR-422 - MMA-11603: Add centralized protobuf version from 5.3.x onwards
  • PR-421 - Update confluent-log4j version
  • PR-413 - Remove redundant Netty Version definition.
  • PR-412 - CVE-2021-43797: Upgrade Netty
  • PR-411 - Upgrade Log4j2 to 2.17.1
  • PR-403 - Upgrade Log4j2

Kafka

  • PR-11742 - KAFKA-13636: Fix for the group coordinator issue where the offsets are deleted for unstable groups (#11742)
  • PR-654 - KC-1933; Upgrade confluent-log4j to 1.2.17-cp2.2
  • PR-11607 - KAFKA-13544: fix FinalizedFeatureChangeListener deadlock (#11607)
  • PR-11552 - KAFKA-13488: Producer fails to recover if topic gets deleted midway (#11552)
  • 13bc00df - KAFKA-12293: Remove JCenter from buildscript and delete buildscript.gradle

ksqlDB

REST Proxy

  • PR-984 - KREST-4608 port changes to 6.2.x onwards
  • PR-983 - KREST-4605 Merge the master changes to back levels too
  • PR-982 - KREST-4577 Port changes to 6.2.x onwards
  • PR-963 - KREST-4113 Don’t delete all ACLs if an empty DELETE request is received.
  • PR-969 - KREST-3836 update maven to pull from https
  • 5514b21c - MMA-11468: Add option to enable nosniff protection
  • 19657b29 - Remove Redundant Netty Definition
  • PR-305 - Remove Redundant Netty Definition
  • PR-302 - APIF-2027: Upgrade jetty-server to 9.4.44.v20210927.
  • PR-301 - APIF-2005: Change the Maven repository protocol: http → https

Schema Registry

  • PR-2187 - DGS-3066 Handle unions in AvroSchema.metaEquals
  • PR-2186 - DGS-3061 Only cache top-level schemas in AvroData
  • PR-2175 - DGS-2990: Backport PR 2174 to 5.2.x
  • PR-2174 - DGS-2990 Removing message value from exception to avoid logging custo
  • PR-2164 - DGS-2937 Use default scale of 0 in Avro converter
  • PR-2151 - DGS-2875: Return correct default value for connect union
  • PR-2148 - DGS-2830 Check for addition/removal of required Protobuf fields
  • PR-2147 - DGS-2826 Add Avro UUID conversion; minor optimization
  • PR-2140 - DGS-2814 Ensure id generation is thread-safe in converters
  • PR-2128 - MINOR: Update 6.1.x OpenAPI Spec
  • PR-2126 - DGS-2719: add explicit dependency for kotlin-scripting-jvm
  • PR-2125 - DGS-2727 Handle multiple refs to same Protobuf enum
  • PR-2124 - DGS-2722 Use identity comparison for DatumReader/Writer caches
  • PR-2122 - DGS-2707 Ensure Protobuf reserved range uses exclusive end
  • PR-2108 - DGS-2646 Support name method on ProtobufSchema with only enums
  • PR-2102 - DGS-2621 Support raw primitives in the Protobuf converter
  • PR-2107 - DGS-2636 Collect properties for allOf in JSON Schema converter
  • PR-2100 - Backport PR2039 to 5.5.x branch
  • PR-2103 - DGS-2626 Handle combined schema with title and/or object

Commercial Features

Confluent Server

  • Ignore failing quota_test (#5651)
  • Cherry-pick “Disabled bubbling errors from JsonSerializer and Deserializers for connect that print the message in stack trace (#5558)” onto 6.2.x
  • Cherrypick to 6.2.x “KSTORAGE-1915: Check if usage of S3 API’s GetBucketLocation can be eliminated”
  • [CLI-1430] Add GCM functionality to DecryptionEngine (#5563)
  • Upgrade TCNative
  • Bump Protobuf Dependency Version
  • KC-1933; Upgrade confluent-log4j to 1.2.17-cp2.2 (6.2.x)
  • EventLogger interface
  • KAFKA-13461: Don’t re-initialize ZK client session after auth failure if connection still alive
  • KAFKA-9648: Add configuration to adjust listen backlog size for Accep
  • KAFKA-12293: Remove JCenter from buildscript and delete buildscript.gradle

Security

  • Added fix for empty bulkAuthorize requests
  • DG-1249, DG-1257, DG-1353, DG-1423, DG-1462, DG-1491, DGS-2950: Fix testGetLastOffsetWithExistingMessages
  • APPSEC-827:Upgrade bc-fips
  • DGS-2730: Use fail-open logic in license validation

Secret Registry

Control Center

  • MMA-11603: Update protobuf version
  • MMA-11121: Fix compatibility issues for partition size metric
  • MMA-11478: enable CORS check on websocket when confluent.controlcenter.rest.access.control.allow.origin property is enabled
  • MMA-11451: fix @JsonProperty usage to yield expected output
  • (MINOR): update armeria version and exclude netty artifacts
  • (MINOR): fix merge
  • MMA-11468: Add option to enable nosniff protection. Header is now set to nosniff by default.
  • MMA-10986: fix health check resource for connect
  • MMA-10571: add SR basic auth to armeria healthcheck
  • (MINOR): fix armeria test
  • MMA-9126: Fix broken RocksDB stats endpoint
  • MMA-9292: unit test for ClusterDataExporter extraction of security protocol
  • MMA-9292: control-center-export script reads security.protocol from prop file
  • MMA-10114: Change default values for command topic configuration
  • MMA-10668: reduce the broker id retention to 30ms
  • MMA-9479: Remove JDK dependency from control center stop script
  • DGS-2570: call /scope instead of /permissions to get SR scope in RbacServiceVisibilityFilter
  • Added support for bigint format for numbers coming from ksql JSON

MQTT Proxy

  • Fixing error in repeated manual merge.
  • Fixing error in repeated manual merge.
  • [CC-16389] Changing the maven URL in pom.xml from http to https
  • Revert “[CC-16389] Changing the maven URL in pom.xml from http to https”
  • [CC-16389] Changing the maven URL in pom.xml from http to https

Metadata Service

  • MDS-33 - Add AES GCM enable variable
  • CIAM-1761 - Update postgres db driver to 42.3.3 for cve issue
  • CIAM-1737 Bump PSQL Version
  • SEC-238 MDS Test : Silent Proxy of Writes

Version 6.2.2

Released November 2021

Community Features

Common

  • PR-383 - Upgrade Netty to 4.1.68.Final

Kafka

  • PR-11439 - KAFKA-13406: skip assignment validation for built-in cooperativeStickyAssignor (#11439)
  • PR-10952 - KAFKA-12257; Consumer mishandles topics deleted and recreated with the same name (#10952)
  • a73b47a9 - upgrade netty version to 4.1.68.final
  • PR-11283 - KAFKA-13249: Always update changelog offsets before writing the checkpoint file (#11283)
  • PR-11299 - KAFKA-13258/KAFKA-13259: Fix error response generation (#11299)
  • PR-11308 - KAFKA-13277; Fix size calculation for tagged string fields in message generator (#11308)
  • PR-11174 - KAFKA-9747: Creating connect reconfiguration URL safely (#11174)
  • PR-11073 - KAFKA-13092: Perf regression in LISR requests (#11073)
  • PR-11271 - KAFKA-13236: TopologyTestDriver should not crash for EOS-beta config (#11271)
  • PR-11266 - KAFKA-13233 Log zkVersion in more places (#11266)
  • PR-11264 - KAFKA-13231; TransactionalMessageCopier.start_node should wait until the process if fully started (#11264)
  • PR-11231 - KAFKA-13214; Consumer should not reset state after retriable error in rebalance (#11231)
  • PR-11068 - KAFKA-13081: detect and handle doubly assigned partitions in cooperative assignor (#11068)

ksqlDB

REST Proxy

  • PR-924 - KREST-1936 Remove mask filtering of producer configs.
  • PR-913 - KREST-1965 Disable ProduceActionIntegrationTest on 6.2.x.
  • PR-910 - Log setup/teardown of integration tests
  • PR-292 - Upgrade Netty Version
  • PR-290 - Disabled concurrent jobs
  • PR-289 - Update jakarta.el-api
  • 6fa4e875 - MMA-11014: Wrap correct exception
  • PR-286 - MMA-11014: wrap SSLException

Schema Registry

  • PR-2086 - Backport Netty version Override change.
  • PR-2079 - DGS-2507 Use fullName if enhanced.protobuf.schema.support=true
  • PR-2075 - DGS-2491 Ensure ProtobufSchema.name returns qualified name
  • PR-2070 - DGS-2401 Add converter flag to allow old doc/default behavior
  • PR-2039 - Fixes #1930: Support Jakarta EE 9 by eliminating dependencies on jersey ey-common and jakarta.ws.rs in schema-registry-client. (#1933)
  • PR-2036 - Fix default value for array of records
  • PR-2068 - Demostrate wrong cache hit for union type reader schema
  • PR-2061 - DGS-2417 Change moving Protobuf messages as compatible
  • PR-2058 - DGS-2393 Fix special chars in options for oneof/enum
  • PR-2052 - DGS-2315: Only tombstone previous schemas during hard deletion
  • PR-2043 - DGS-2259 Omit defaults for fields with type Message
  • PR-2026 - DGS-2078 Fix optional Avro schema generation
  • PR-2024 - DGS-2076 Fix NPE when cross referencing schema w/o name
  • PR-2023 - DGS-2073 Fix Protobuf converter for multiple map refs

Commercial Features

Confluent Server

  • KSTORAGE-1528: Remove flakey tests for TierObjectStoreFactory
  • CONFLUENT: Upgrade netty to 4.1.68.Final and disable io.netty.handler.ssl.openssl.useTasks (#4606)
  • KSTORAGE-1834: followerRestorePoint is the Last Tiered Segment with offsets before localLogStartOffset
  • SEC-2731 - Improve Ldap handling for stale connections (#4497)
  • CC-13747 CIAM-1263 FF-4419: Limit max retries in MDS URL Refresh loop
  • CIAM-1253 Fix Logging of various timeouts in ce-rest-auth RestClient
  • MINOR: remove azure backend for tiered storage system tests in 6.2.x
  • SEC-1537 RestClient Retry Logic Fixed (#4395)
  • KC-1878 Debug Logging for LDAP Context creation

Control Center

  • (MINOR): MMA-11236: clean and reorganize checkstyle suppressions
  • PG-389: short-lived http connection for non-idempotent reqs and configurable mds client
  • MMA-10607: fix testBadClientDoesNotStallRegistration by awaiting method invocation before verify
  • KEXP-114 Fixed broker details page to show correct values under Production and Consumption latency
  • DGS-2570: Call /scope instead of /permissions to get Schema Registry scope in RbacServiceVisibilityFilter (related to DGS-2570 under Schema Registry). This fixes an issue where Control Center previously had to make a Get Permission call instead of using a specialized endpoint. When there were a large number of Subject or RBAC rules, the Get Permission call would timeout, causing the Schema tab to disappear from Control Center.

Metadata Service

  • SEC-2805 - Manually tested fix for MDS forwarding logic to handle unicode and request with query params
  • Update commons-codec

Replicator

  • Upgrade jakarta.el:jakarta.el-api
  • REPL-1843: Track uncommited records given to Connect framework.

Schema Registry

  • DGS-2570: Expose Scope for SchemaRegistrySecurityResourceExtension; provides an endpoint on Schema Registry for Confluent Control Center to filter the visible Schema Registry clusters to show based on the principal’s permission, which is needed when RBAC is enabled on Schema Registry (related to DGS-2570 under Control Center). This fixes an issue where Control Center previously had to make a Get Permission call instead of using a specialized endpoint. When there were a large number of Subject or RBAC rules, the Get Permission call would timeout, causing the Schema tab to disappear from Control Center.

Confluent Platform Docker Images

  • Confluent Platform Docker Images will no longer ship with Yum/Apt configuration that allowed users to update running Confluent Platform containers to the next Confluent Platform release. Confluent recommends upgrading Confluent Platform service components running in containers by replacing the running container with the next Confluent Platform released Docker image and using Docker volumes to persist data between upgrades.
  • Confluent Platform Docker images formerly shipped with world writable data and configuration directories (chmod a+w). Those permissions are now removed, and adjustments made to make those directories writable from the image’s appuser USER.
  • Confluent Platform Docker images are now built with yum --nodocs ... to prevent installing Red Hat Documentation. This slims down the images slightly. Confluent documentation from Confluent packages are still installed.

Version 6.2.1

Released September 2021

Community Features

Common

  • PR-357 - log4j migration to confluent repackaged version
  • PR-358 - Upgrade Zookeeper to 3.5.9.
  • PR-361 - Upgrade bcprov-jdk15on to 1.68
  • PR-365 - Upgrade log4j2 to 2.13.2
  • PR-309 - Upgrade bouncycastle to 1.68

Kafka

  • PR-11221 - KAFKA-13207: Skip truncation on fetch response with diverging epoch if partition removed from fetcher (#11221)
  • 0867e916 - KAFKA-13209: Upgrade jetty-server
  • PR-10906 - KAFKA-12835: Topic IDs can mismatch on brokers (after interbroker protocol version update) (#10754) (#10906)
  • PR-10895 - KAFKA-12890; Consumer group stuck in CompletingRebalance (2.8) (#10895)
  • PR-10915 - KAFKA-13041: Enable connecting VS Code remote debugger (#10915)
  • PR-11136 - KAFKA-13141; Skip follower fetch offset update in leader if diverging epoch is present (#11136)
  • PR-11098 - KAFKA-13099; Transactional expiration should account for max batch size (#11098)
  • PR-11113 - KAFKA-13128: wait for all keys to be fully processed in #shouldQueryStoresAfterAddingAndRemovingStreamThread (#11113)
  • PR-10919 - KAFKA-12985: Upgrade jetty to 9.4.42 (#10919)
  • PR-10921 - KAFKA-13096: Ensure queryable store providers is up to date after adding stream thread (#10921)
  • PR-8844 - KAFKA-9887 fix failed task or connector count on startup failure (#8844)
  • PR-10877 - KAFKA-12925: adding presfixScan operation for missed implementations (#10877)
  • 75ae1a8c - KAFKA-13037: “Thread state is already PENDING_SHUTDOWN” log spam
  • PR-9441 - KAFKA-10614: Ensure group state (un)load is executed in the right order (#9441)
  • PR-10986 - KAFKA-12983: reset needsJoinPrepare flag before rejoining the group (#10986)
  • PR-9541 - KAFKA-10675: Add schema name to ConnectSchema.validateValue() error message (#9541)
  • PR-10940 - KAFKA-13007; KafkaAdminClient getListOffsetsCalls reuse cluster snapshot (#10940)
  • PR-10930 - KAFKA-12996; Return OFFSET_OUT_OF_RANGE for fetchOffset < startOffset even for diverging epochs (#10930)
  • PR-10894 - KAFKA-12951: restore must terminate for tx global topic (#10894)
  • PR-10923 - KAFKA-12976; Remove UNSUPPORTED_VERSION error from delete topics call (#10923)
  • PR-10922 - KAFKA-12790: Remove SslTransportLayerTest.testUnsupportedTlsVersion (#10922)
  • 74d2f8ab - Update cherry-pick to fit 2.8 testing style for scala
  • PR-10190 - KAFKA-12336 Custom stream naming does not work while calling stream[K… (#10190)
  • PR-10879 - KAFKA-12991; Fix unsafe access to AbstractCoordinator.state (#10879)
  • PR-10878 - KAFKA-12898; Owned partitions in the subscription must be sorted (#10878)
  • PR-10834 - KAFKA-12904: Corrected the timeout for config validation REST API resource (#10834)
  • PR-10880 - KAFKA-12870; Flush in progress not cleared after transaction completion (#10880)
  • PR-10885 - KAFKA-12701: NPE in MetadataRequest when using topic IDs (#10885)
  • PR-10882 - KAFKA-12948: Remove node from ClusterConnectionStates.connectingNodes when node is removed (#10882)
  • PR-10846 - KAFKA-12914: StreamSourceNode should return null topic name for pattern subscription (#10846)
  • 675bd1ca - KAFKA-12820: Upgrade maven-artifact dependency
  • PR-10633 - KAFKA-12751: Reset AlterIsr in-flight state for duplicate update requests (#10633)
  • PR-10059 - KAFKA-8562; SaslChannelBuilder - Avoid (reverse) DNS lookup while building SslTransportLayer (#10059)
  • PR-10643 - KAFKA-12747: Fix flakiness in shouldReturnUUIDsWithStringPrefix (#10643)
  • PR-10636 - KAFKA-12752: Bump Jersey deps to 2.34 (#10636)
  • PR-10014 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes (#10014)
  • PR-10016 - KAFKA-10340: Proactively close producer when cancelling source tasks (#10016)
  • PR-10611 - KAFKA-12730; Avoid duplicate logout if Kerberos login fails (#10611)
  • PR-10586 - KAFKA-12702: Fix NPE in networkListeners from BrokerServer in 2.8 (#10586)

ksqlDB

  • PR-8051 - chore: commons-codec override
  • PR-7639 - fix: Sync Vertx version

REST Proxy

  • PR-885 - KREST-1577 Replace a manually set Avro version with the Avro version variable.
  • PR-870 - Use a more lenient test for testBackoffMsControlsPollCalls.
  • 0c83d02d - SEC-1034: log4j migration to confluent repackaged version
  • 19154f1d - SEC-1034: log4j migration to confluent repackaged version
  • 8a9c055f - build: backport formatting checkstyle issues from #873
  • PR-858 - KREST-728: Add OpenAPI spec for Produce V3.
  • PR-857 - KREST-1263 Exclude a vulnerable jackson-mapper-asl dependency from avro 1.8.2 in 5.3.x.
  • PR-280 - KREST-578: Add configurable DosFilter support. (#238)
  • 81d7a448 - build: Fix checkstyle
  • PR-275 - Fix rest utils checkstyle and log4j updates
  • PR-270 - Upgrade Jetty to 9.4.43.v20210629
  • PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224

Schema Registry

  • PR-2016 - DGS-2026 Allow generating JSON schemas with additionalProperties=true
  • PR-2014 - DGS-2020 Allow optional to be overridden in SchemaWrapper
  • PR-2012 - DGS-2018 Ensure names in Avro union are scrubbed
  • PR-1995 - Corrected Typo in ProtobufConverter and JsonSchemaConverter
  • PR-1990 - DGS-1879 changed timeout parameter
  • PR-1989 - Upgrade commons-compress to 1.21 (5.4.x and later)
  • PR-1986 - DGS-1915: Upgrade commons-compress version
  • PR-1977 - DGS-1909 Fix AVRO_FIELD_DEFAULT_FLAG_PROP in AvroData
  • PR-1972 - DGS-1902 Use fully qualified names for Protobuf Map types
  • PR-1969 - SEC-2492 exclude guava version with CVE
  • PR-1968 - DGS-1880 Allow additionalProperties to be set in converter
  • PR-1967 - Logging schema size when register
  • PR-1965 - Remove use of WrappedParsedSchema in mock client
  • PR-1956 - DGS-1853 Fix handling of slashes with refs in maven plugin
  • PR-1954 - DGS-1850 Prevent duplicate imports in Protobuf converter
  • PR-1947 - DGS-1829 Handle duplicate refs during Protobuf Schema compat check
  • PR-1943 - DGS-1826 Fix NPE when struct default has null
  • PR-1945 - DGS-1825 Fix DataException when scrubbing invalid names (Avro)
  • PR-1942 - DGS-1825 Fix DataException when scrubbing invalid names (Protobuf)
  • PR-1939 - DGS-1820 Pass ObjectMapper to JSON Schema generator
  • PR-1937 - DGS-1820 JSON Schema validation should take a JsonNode
  • PR-1931 - DGS-1794 Don’t return read-only byte buffers from Protobuf converter
  • PR-1928 - DGS-1758: Allow AvroData take JSON null as default
  • PR-1923 - DGS-1737: MockSchemaRegistryClient doesn’t generate new schema id with protobuf
  • PR-1920 - DGS-404: Improving error messages - adding 40408 and 40409 for subject level compatibility and mode error
  • PR-1918 - small fix in deleteSchemaVersion for MockSchemaRegistryClient
  • PR-1917 - DGS-1734: MockSchemaRegistryClient need to perform a correct deleteSchemaVersion
  • PR-1907 - DGS-1658: Preserve field names when rendering Protobuf objects as JSON
  • PR-1901 - DGS-1726 Only convert wrappers to primitive if flag is set
  • PR-1902 - DGS-1727 Don’t throw exception when comparing array item
  • PR-1897 - DGS-1716 Fix logging
  • PR-1896 - DGS-1704: override commons-compress
  • PR-1895 - DGS-1704: Upgrade jackson-mapper-asl
  • PR-1323 - API update for Jetty 9.4.26.v20200117

Commercial Features

Confluent Server

  • KC-1800: Upgrade Guava to 30.0-android
  • KC-1869; IllegalStateException in ControlMetadataBatch due to topicId mismatch
  • KAFKA-13209: Upgrade jetty-server
  • AUDIT-670 - Moved DESCRIBE_USER_SCRAM_CREDENTIALS from MANAGEMENT to DESCRIBE
  • KC-1789: Asynchronously flush partition metadata file
  • DGS-1903: (backport to 6.2.x) Add config option for missing schemas TTL (#4118)
  • Upgrade AzureIdentity
  • KC-1810: Upgrade commons.io in ce-kafka
  • KAFKA-13037: “Thread state is already PENDING_SHUTDOWN” log spam
  • backport KC-1266 to 6.2.1 (wait until 6.2.0 is released)
  • KAFKALESS-523: Topic placement constraints not used in CreatePartitions
  • KC-1784: Use URI instead of URL for RBAC writer comparison to avoid hostname resolution issues
  • CONFLUENT: Complete version regex check in system tests (#580)
  • KC-1625: DynamicBrokerConfig must not log sensitive config keys
  • KC-1752 Remove public facing String redact API which is not currently used
  • KC-1752 Fix potential alter/update config related issue
  • CPKAFKA-3262: TierTopicManager sees incorrect __confluent-tier-state partition count on startup
  • KAFKA-12820: Upgrade maven-artifact dependency

Security

  • Override commons-codec
  • fix formatting issues with newer checkstyle versions
  • fix: Checkstyle issues in AuthorizationFilter.java
  • SEC-1034: log4j migration to confluent repackaged version
  • DGS-1903: Add missing schema cache to LruSchemaRegistryClient
  • SEC-2533 upgrade commons-io to 2.11.0
  • SEC-2493 Remove bouncy-castle explicit dependency
  • DGS-1691: Perform license checks in background thread
  • KQE-229: Update vertx-core to 3.9.7
  • Pins Ksql to a jetty-jaas 9.4.28.v20200408 (#365) (#366)

Secret Registry

  • CC-15204: Upgrade bcprov-jdk15on to 1.68

Control Center

  • MMA-7623: Hide stack trace in HTML output for failed responses
  • MMA-10895: Add support for prefixed ‘auth.bearer.roles.claim’ property
  • DGS-1654: Upgrade Protobuf to v3.17.0
  • MMA-10673: Introduces a new SSL-specific SMTP port configuration for email alerts

MQTT Proxy

  • CCMSG-1101: Upgrade Netty to 4.1.63.Final
  • CC-14445: Remove compile dependency on Kafka broker

Metadata Service

  • SEC-2440 - MDS - Fix MdsScope Validation and messages and add tests
  • Fix lookupPrincipalsWithRoleOnResource to authorize on valid ResourcePattern using DESCRIBE_ACCESS operation
  • SEC-1034: log4j migration to confluent repackaged version
  • Fixed issue where users with ResourceOwner role, could not see rolebindings of other users on resources they own

Version 6.2.0

Released June 2021

Community Features

Common

  • PR-342 - Fix jacksondatabind and jackson-cbor version
  • PR-343 - Change zookeeper version to 3.4.14
  • PR-340 - Fix zookeeper, jacksondatabind and jackson-cbor version
  • PR-339 - update spotbugs and jacoco for improved jdk support
  • PR-337 - DGS-1551: upgrade jackson-bom to 2.9.10.20210106
  • PR-334 - CC-14690 : Define com.fasterxml.jackson.dataformat:jackson-dataformat-cbor.
  • PR-332 - Update dependency-check to 6.1.6
  • PR-330 - ST-5293: Explicit load of netty dependencies
  • PR-325 - ST-4106: Uplift metrics related jars to CP Images
  • PR-329 - ST-5293: Override version of netty used by ZK to address CVE
  • PR-326 - KREST-1066 Upgrade Guava to the latest version to patch CVE-2020-8908.
  • PR-322 - ST-4106: Uplift metrics related jars to CP Images
  • PR-320 - Build: Sync Upstream Kafka scala and zookeeper versions
  • PR-318 - ST-5085:Update resolver plugin
  • PR-314 - MMA-9930: Sync Google Guava version.
  • PR-310 - update Spotbugs and Jacoco plugins for recent JDK support
  • PR-307 - Upgrade to Avro 1.10.1

Kafka

  • PR-564 - APPSEC-60: Upgrade org.apache.maven:apache-maven
  • PR-554 - fix: CVE-2021-28168 Upgrade jersey to 2.34
  • PR-10545 - KAFKA-12672: Added config for raft testing server (#10545)
  • PR-10536 - KAFKA-12667: Fix incorrect error log on StateDirectory close (#10536)
  • PR-10526 - KAFKA-12655 Update Jetty: 9.4.38.v20210224 → 9.4.39.v20210325 (#10526)
  • PR-10531 - KAFKA-12658: Include kafka-shell jar and dependencies in release tar (#10531)
  • PR-10301 - KAFKA-9831: increase max.poll.interval.ms to avoid unexpected rebalance (#10301)
  • PR-10481 - KAFKA-12619; Raft leader should expose hw only after committing LeaderChange (#10481)
  • PR-10474 - KAFKA-12602: Fix LICENSE file (#10474)
  • PR-9992 - KAFKA-10759 Add ARM build stage (#9992)
  • PR-10396 - KAFKA-12474: Handle failure to write new session keys gracefully (#10396)
  • PR-10452 - KAFKA-12593: Fix Apache License headers (#10452)
  • 6703cc20 - KAFKA-12583: Upgrade netty to 4.1.62.Final
  • PR-10404 - KAFKA-12557: Fix hanging KafkaAdminClientTest (#10404)
  • PR-10407 - KAFKA-12523: handle TaskCorruption and TimeoutException during handleCorruption and handleRevocation (#10407)
  • PR-10387 - KAFKA-12537: fix application shutdown corner case with only one thread (#10387)
  • PR-10397 - KAFKA-12508: Disable KIP-557 (#10397)
  • PR-10392 - KAFKA-12435: Fix javadoc errors (#10392)
  • PR-10374 - (Cherry-pick) KAFKA-9274: handle TimeoutException on task reset (#10000) (#10374)
  • PR-10360 - KAFKA-12508: Emit records with same value and same timestamp (#10360)
  • PR-10355 - KAFKA-12500: fix memory leak in thread cache (#10355)
  • PR-10356 - KAFKA-12503: inform threads to resize their cache instead of doing so for them (#10356)
  • PR-10227 - KAFKA-12382: add a README for KIP-500 (#10227)
  • PR-10322 - KAFKA-12455: Fix OffsetValidationTest.test_broker_rolling_bounce failure with Raft (#10322)
  • PR-10320 - revert stream logging level back to ERROR (#10320)
  • PR-10285 - KAFKA-12442: Upgrade ZSTD JNI from 1.4.8-4 to 1.4.9-1 (#10285)
  • PR-10311 - KAFKA-12462: proceed with task revocation in case of thread in PENDING_SHUTDOWN (#10311)
  • PR-10118 - KAFKA-10192: Increase max time to wait for worker to start in some integration tests (#10118)
  • PR-528 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes
  • PR-10252 - KAFKA-12403; Ensure local state deleted on RemoveTopicRecord received (#10252)
  • PR-10240 - KAFKA-12381: remove live broker checks for forwarding topic creation (#10240)
  • PR-10253 - KAFKA-12376: Apply atomic append to the log (#10253)
  • PR-10263 - KAFKA-12393: Document multi-tenancy considerations (#334) (#10263)
  • 8ed0c228 - KAFKA-12400: Upgrade jetty to fix CVE-2020-27223
  • PR-10215 - KAFKA-12375: don’t reuse thread.id until a thread has fully shut down (#10215)
  • PR-10217 - KAFKA-12254: Ensure MM2 creates topics with source topic configs (#10217)
  • PR-10223 - KAFKA-12394; Return TOPIC_AUTHORIZATION_FAILED in delete topic response if no describe permission (#10223)
  • PR-9717 - KAFKA-10766: Unit test cases for RocksDBRangeIterator (#9717)
  • PR-10052 - KAFKA-12289: Adding test cases for prefix scan in InMemoryKeyValueStore (#10052)
  • PR-10205 - KAFKA-12323 Follow-up: Refactor the unit test a bit (#10205)
  • PR-10199 - KAFKA-12374: Add missing config sasl.mechanism.controller.protocol (#10199)
  • PR-8812 - KAFKA-10101: Fix edge cases in Log.recoverLog and LogManager.loadLogs (#8812)
  • PR-10213 - KAFKA-12375: fix concurrency issue in application shutdown (#10213)
  • PR-9990 - KAFKA-12235: Fix a bug where describeConfigs does not work on 2+ config keys (#9990)
  • PR-10194 - KAFKA-12365; Disable APIs not supported by KIP-500 broker/controller (#10194)
  • PR-10200 - KAFKA-12367; Ensure partition epoch is propagated to Partition state (#10200)
  • PR-10170 - KAFKA-12323: Set timestamp in record context when punctuate (#10170)
  • PR-10191 - KAFKA-12341: Ensure consistent versions for javassist (#10191)
  • PR-10174 - KAFKA-12357: Do not inline methods from the scala package by default (#10174)
  • PR-10024 - KAFKA-12273 InterBrokerSendThread#pollOnce throws FatalExitError even… (#10024)
  • 9e452950 - KAFKA-12324: Upgrade jetty to fix CVE-2020-27218
  • PR-10152 - KAFKA-12339: Add retry to admin client’s listOffsets (#10152)
  • PR-10153 - KAFKA-12340: Fix potential resource leak in Kafka*BackingStore (#10153)
  • PR-10070 - KAFKA-12276: Add the quorum controller code (#10070)
  • 276630e2 - KAFKA-12334: Add the KIP-500 metadata shell
  • PR-10129 - KAFKA-10817; Add clusterId validation to raft Fetch handling (#10129)
  • PR-10158 - KAFKA-12343: Handle exceptions better in TopicAdmin, including UnsupportedVersionException (#10158)
  • PR-10072 - KAFKA-9274: Throw TaskCorruptedException instead of TimeoutException when TX commit times out (#10072)
  • PR-10143 - KAFKA-12332; Error partitions from topics with invalid IDs in LISR requests (#10143)
  • PR-10063 - KAFKA-12258; Add support for splitting appending records (#10063)
  • PR-10091 - KAFKA-9524: increase retention time for window and grace periods longer than one day (#10091)
  • PR-10138 - KAFKA-12331: Use LEO for the base offset of LeaderChangeMessage batch (#10138)
  • PR-10666 - KAFKA-12278; Ensure exposed api versions are consistent within listener (#10666)
  • PR-10133 - Fixed README and added clearer error message. (#10133)
  • PR-10102 - KAFKA-12272: Fix commit-interval metrics (#10102)
  • PR-10123 - KAFKA-12327: Remove MethodHandle usage in CompressionType (#10123)
  • PR-10122 - KAFKA-12326: Corrected regresion in MirrorMaker 2 executable introduced with KAFKA-10021 (#10122)
  • PR-9780 - KAFKA-10021: Changed Kafka backing stores to use shared admin client to get end offsets and create topics (#9780)
  • PR-10098 - KAFKA-12321 the comparison function for uuid type should be ‘equals’ rather than ‘==’ (#10098)
  • PR-9986 - JUnit extensions for integration tests (#9986)
  • PR-9865 - Fill in the 2.8 release note for Authorizer (#9865)
  • PR-9579 - KAFKA-9751: Forward CreateTopicsRequest for FindCoordinator/Metadata when topic creation is needed (#9579)
  • PR-10000 - KAFKA-9274: handle TimeoutException on task reset (#10000)
  • PR-10049 - Refactor the MetadataCache into two implementations (#10049)
  • PR-9997 - KAFKA-9274: Add timeout handling for StreamPartitioner (#9997)
  • PR-10053 - KAFKA-10834: Remove redundant type casts in Connect (#10053)
  • PR-9726 - KAFKA-10833: Expose task configurations in Connect REST API (KIP-661) (#9726)
  • PR-9107 - KAFKA-5488: Add type-safe split() operator (#9107)
  • PR-9867 - KAFKA-10835: Replace Runnable and Callable overrides with lambdas in Connect (#9867)
  • PR-9902 - KAFKA-12193: Re-resolve IPs after a client disconnects (#9902)
  • PR-9978 - KAFKA-10716: persist UUID in state directory for stable processId across restarts (#9978)
  • PR-10032 - KAFKA-12270: Handle race condition when Connect tasks attempt to create topics (#10032)
  • PR-9998 - KAFKA-12250; Add metadata record serde for KIP-631 (#9998)
  • PR-10040 - KAFKA-12259: Use raw config to infer the connector type when returning a connector status response (#10040)
  • PR-10018 - KAFKA-12271 Immutable container classes to support new MetadataCache (#10018)
  • PR-9987 - KAFKA-10895: Gracefully handle invalid JAAS configs (follow up fix) (#9987)
  • PR-9508 - KAFKA-10648: Add Prefix Scan support to State Stores (#9508)
  • PR-9996 - KAFKA-12249: Add client-side Decommission Broker RPC (KIP-500) (#9996)
  • PR-9319 - KAFKA-10413: Allow for even distribution of lost/new tasks when multiple Connect workers join at the same time (#9319)
  • PR-9901 - KAFKA-12209: Add the timeline data structures for the KIP-631 controller (#9901)
  • PR-10007 - KAFKA-10700 - Support mutual TLS authentication for SASL_SSL listeners (KIP-684) (#10007)
  • PR-9989 - KAFKA-12246: Remove redundant suppression in KafkaAdminClient (#9989)
  • PR-9994 - KAFKA-12248; Add BrokerHeartbeat/BrokerRegistration RPCs for KIP-500 (#9994)
  • PR-9253 - KAFKA-10366 & KAFKA-9649: Implement KIP-659 to allow TimeWindowedDeserializer and TimeWindowedSerde to handle window size (#9253)
  • PR-9739 - KAFKA-10636; Bypass log validation and offset assignment for writes from the raft leader (#9739)
  • PR-9967 - KAFKA-12236; New meta.properties logic for KIP-500 (#9967)
  • PR-9979 - KAFKA-12238; Implement DescribeProducers API from KIP-664 (#9979)
  • PR-9816 - KAFKA-10761; Kafka Raft update log start offset (#9816)
  • PR-9984 - KAFKA-12247: add timeout and static group rebalance to remove thread (#9984)
  • PR-9684 - KAFKA-10764: Add support for returning topic IDs on create, supplying topic IDs for delete (#9684)
  • PR-9420 - KAFKA-10604: Fix Streams default state.dir (#9420)
  • PR-9985 - KAFKA-12237; Support lazy initialization of quorum voter addresses (#9985)
  • PR-9589 - KAFKA-10710; MM2 - Create herders only if source->target.enabled=true and heartbeats are disabled (#9589)
  • PR-9738 - KAFKA-8744: Update Scala API to give names to processors (#9738)
  • PR-9525 - KAFKA-10658 ErrantRecordReporter.report always return completed futur… (#9525)
  • PR-9840 - KAFKA-10867: Improved task idling (#9840)
  • PR-9983 - KAFKA-8930: MirrorMaker v2 documentation (#324) (#9983)
  • PR-9970 - KAFKA-12233: Align the length passed to FileChannel by FileRecords.writeTo (#9970)
  • PR-9671 - KAFKA-10793: move handling of FindCoordinatorFuture to fix race condition (#9671)
  • PR-9720 - KAFKA-10555: Improve client state machine (#9720)
  • PR-9765 - KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations (#9765)
  • PR-9819 - KAFKA-10694; Implement zero copy for FetchSnapshot (#9819)
  • PR-9942 - KAFKA-12229: Restore original class loader in integration tests using EmbeddedConnectCluster during shutdown (#9942)
  • PR-9947 - KAFKA-12190: Fix setting of file permissions on non-POSIX filesystems (#9947)
  • PR-9848 - KAFKA-10357: Extract setup of repartition topics from Streams partition assignor (#9848)
  • PR-9945 - KAFKA-12212; Bump Metadata API version to remove ClusterAuthorizedOperations fields (KIP-700) (#9945)
  • PR-9940 - KAFKA-12185: fix ConcurrentModificationException in newly added Tasks container class (#9940)
  • PR-9832 - KAFKA-12152; Idempotent Producer does not reset the sequence number of partitions without in-flight batches (#9832)
  • PR-9926 - KAFKA-12175 Migrate generator module to junit5 (#9926)
  • PR-9814 - KAFKA-10869: Gate topic IDs behind IBP 2.8 (KIP-516) (#9814)
  • PR-9904 - KAFKA-12211: don’t change perm for base/state dir when no persistent store (#9904)
  • PR-9905 - KAFKA-12210; AdminClient should use DescribeCluster API when available (KIP-700) (#9905)
  • PR-9877 - KAFKA-8460: produce records with current timestamp (#9877)
  • PR-9923 - KAFKA-12219: Add ‘synchronized’ keyword to InMemoryKeyValueStore#[reverseRange, reverseAll] (#9923)
  • PR-9733 - KAFKA-10017: fix uncaucht-exception handling in EosBetaUpgradeIntegrationTest (#9733)
  • PR-9903 - KAFKA-12204; Implement DescribeCluster API in the broker (KIP-700) (#9903)
  • PR-9887 - KAFKA-12195 Fix synchronization issue happening in KafkaStreams (#9887)
  • PR-9888 - KAFKA-12194: use stateListener to catch each state change (#9888)
  • PR-9743 - KAFKA-10841: Extract conversion from LogReadResult to FetchPartitionData (#9743)
  • PR-9924 - KAFKA-12221 remove PowerMock from connect-json module and connect-transforms module (#9924)
  • PR-9855 - KAFKA-7341 Migrate core module to JUnit 5 (#9855)
  • PR-9909 - KAFKA-12196: Migrate connect:api module to JUnit 5 (#9909)
  • PR-9917 - KAFKA-12200: Migrate connect:file module to JUnit 5 (#9917)
  • PR-9894 - KAFKA-12202 Migrate connect:mirror module to JUnit 5 (#9894)
  • PR-9907 - KAFKA-12197: Migrate connect:transforms module to JUnit 5 (#9907)
  • PR-9600 - KAFKA-10674: Controller API version bond with forwardable APIs (#9600)
  • PR-9914 - KAFKA-12214: Generated code does not include UUID or struct fields in its toString output (#9914)
  • PR-9871 - KAFKA-12161; Support raft observers with optional id (#9871)
  • PR-9900 - KAFKA-12208: Rename AdminManager to ZkAdminManager (#9900)
  • PR-9890 - KAFKA-12198: Migrate connect:json module to JUnit 5 (#9890)
  • PR-9896 - KAFKA-12206: o.a.k.common.Uuid should implement Comparable (#9896)
  • PR-9899 - KAFKA-12191 SslTransportTls12Tls13Test can replace ‘assumeTrue’ by (junit 5) conditional test (#9899)
  • PR-9898 - KAFKA-12189 ShellTest can replace ‘assumeTrue’ by (junit 5) conditional test (#9898)
  • PR-9889 - KAFKA-12203 Migrate connect:mirror-client module to JUnit 5 (#9889)
  • PR-9892 - KAFKA-12201: Migrate connect:basic-auth-extensio module to JUnit 5 (#9892)
  • PR-9876 - KAFKA-12183: Add the KIP-631 metadata record definitions (#9876)
  • PR-9224 - KAFKA-10304: Refactor MM2 integration tests (#9224)
  • PR-9856 - KAFKA-12171: Migrate streams:test-utils module to JUnit 5 (#9856)
  • PR-9874 - KAFKA-7340: Migrate clients module to JUnit 5 (#9874)
  • PR-9749 - Only schedule AlterIsr thread when we have an ISR change (#9749)
  • PR-9857 - KAFKA-12172 Migrate streams:examples module to JUnit 5 (#9857)
  • PR-9850 - KAFKA-12168; Move envelope request parsing out of SocketServer (#9850)
  • 1e4d3352 - KAFKA-12180: Implement the KIP-631 message generator changes
  • PR-9806 - KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806)
  • PR-9695 - KAFKA-10500: Add KafkaStreams#removeStreamThread (#9695)
  • PR-9842 - KAFKA-12156: Document single threaded response handling in Admin client (#9842)
  • PR-9776 - KAFKA-10878 Check failed message in ProtocolSerializationTest (#9776)
  • PR-9785 - KAFKA-10887 Migrate log4j-appender module to JUnit 5 (#9785)
  • PR-9838 - KAFKA-12153; Update producer state before updating start/end offsets after truncation (#9838)
  • PR-9846 - KAFKA-12165: Include org.apache.kafka.common.quota in javadoc (#9846)
  • PR-9807 - KAFKA-10779; Reassignment tool sets throttles incorrectly when overriding a reassignment (#9807)
  • PR-9800 - KAFKA-9274: Fix commit-TimeoutException handling for EOS (#9800)
  • PR-9676 - KAFKA-10778; Fence appends after write failure (#9676)
  • PR-9802 - KAFKA-10894; Ensure PartitionInfo replicas are not null in client quota callback (#9802)
  • PR-9792 - KAFKA-10870; Handle REBALANCE_IN_PROGRESS error in JoinGroup (#9792)
  • PR-9797 - KAFKA-10893 Increase target_messages_per_sec of ReplicaScaleTest to reduce the run time (#9797)
  • PR-9811 - rename produce to produces (#9811)
  • PR-9761 - KAFKA-10768 Make ByteBufferInputStream.read(byte[], int, int) to follow the contract (#9761)
  • PR-9553 - KAFKA-10427: Fetch snapshot API (#9553)
  • PR-9606 - KAFKA-10722: Improve JavaDoc for KGroupedStream.aggregate and other similar methods (#9606)
  • PR-9706 - KAFKA-10815 EosTestDriver#verifyAllTransactionFinished should break loop if all partitions are verified (#9706)
  • PR-9737 - KAFKA-10828; Replacing endorsing with acknowledging for voters (#9737)
  • PR-9732 - KAFKA-10842; Use InterBrokerSendThread for raft’s outbound network channel (#9732)
  • PR-9713 - KAFKA-10825 ZooKeeper ISR manager (#9713)
  • PR-9626 - KAFKA-10545: Create topic IDs and propagate to brokers (#9626)
  • PR-9622 - KAFKA-10547; Add TopicId in MetadataResponse (#9622)
  • PR-9736 - Add ConfigurableProducerSpec to Trogdor for improved E2E latency tracking. (#9736)
  • PR-9485 - KAKFA-10619: Idempotent producer will get authorized once it has a WRITE access to at least one topic (KIP-679) (#9485)
  • PR-9735 - KAFKA-10846: Grow buffer in FileSourceStreamTask only when needed (#9735)
  • PR-9760 - KAFKA-10850: Use primitive type to replace deprecated ‘new Integer’ from BrokerToControllerRequestThreadTest (#9760)
  • PR-9689 - KAFKA-10740; Replace OffsetsForLeaderEpochRequest.PartitionData with automated protocol (#9689)
  • PR-9708 - KAFKA-9126: KIP-689: StreamJoined changelog configuration (#9708)
  • PR-9678 - KAFKA-10798; Ensure response is delayed for failed SASL authentication with connection close delay (#9678)
  • PR-9552 - KAFKA-10656: Log the feature flags received by the client (#9552)
  • PR-9661 - KAFKA-10776: Add version attribute in RequestsPerSec metrics documentation (#9661)
  • PR-9729 - KAFKA-10839; improve consumer group coordinator unavailable message (#9729)
  • PR-9753 - KAFKA-10855; Fix non-local return in KafkaApis.handle (#9753)
  • PR-9727 - KAFKA-10417: Update Cogrouped processor to work with suppress() and joins (#9727)
  • PR-4090 - KAFKA-6084: Propagate JSON parsing errors in ReassignPartitionsCommand (#4090)
  • PR-9526 - KAFKA-10525: Emit JSONs with new auto-generated schema (KIP-673) (#9526)
  • PR-9747 - KAFKA-10852: AlterIsr should not be throttled (#9747)
  • PR-9718 - KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers (#9718)
  • PR-9697 - KAFKA-10810: Replace stream threads (#9697)
  • PR-9639 - KAFKA-10677; Complete fetches in purgatory immediately after resigning (#9639)
  • PR-9723 - throw corresponding invalid producer epoch (#9700) (#9723)
  • PR-9685 - KAFKA-10748: Add IP connection rate throttling metric (KIP-612) (#9685)
  • PR-9628 - KAFKA-10747: Extend DescribeClientQuotas and AlterClientQuotas APIs to support IP connection rate quota (KIP-612) (#9628)
  • PR-9700 - throw corresponding invalid producer epoch (#9700)
  • PR-9621 - KAFKA-9892; Producer state snapshot should be forced to disk (#9621)
  • PR-9673 - KAFKA-10289; Fix failed connect_distributed_test.py (ConnectDistributedTest.test_bounce) (#9673)
  • PR-9716 - KAFKA-10826; Ensure raft io thread respects linger timeout (#9716)
  • PR-9435 - KAFKA-10606: Disable auto topic creation for fetch-all-topic-metadata request (#9435)
  • PR-9539 - KAFKA-10634; Adding LeaderId to voters list in LeaderChangeMessage along with granting voters (#9539)
  • PR-9564 - KAFKA-10667: add timeout for forwarding requests (#9564)
  • PR-9635 - KAFKA-10756; Add missing unit test for UnattachedState (#9635)
  • PR-7409 - KAFKA-10818: Skip conversion to Struct when serializing generated requests/responses (#7409)
  • PR-9512 - KAFKA-10394: Add classes to read and write snapshot for KIP-630 (#9512)
  • PR-9698 - KAFKA-10811: Correct the MirrorConnectorsIntegrationTest to correctly mask the exit procedures (#9698)
  • PR-8965 - KAFKA-8147: Update upgrade notes for KIP-446 (#8965)
  • PR-9615 - KAFKA-10500: Allow people to add new StreamThread at runtime (#9615)
  • PR-9660 - KAFKA-10629: TopologyTestDriver should not require a Properties argument (#9660)
  • PR-9674 - KAFKA-10665: close all kafkaStreams before purgeLocalStreamsState (#9674)
  • PR-9614 - KAFKA-10500: Add failed-stream-threads metric for adding + removing stream threads (#9614)
  • PR-9682 - KAFKA-10803: Fix improper removal of bad dynamic config (#9682)
  • PR-9677 - KAFKA-10799 AlterIsr utilizes ReplicaManager ISR metrics (#9677)
  • PR-9630 - KAFKA-10739; Replace EpochEndOffset with automated protocol (#9630)
  • PR-9326 - KAFKA-10460: ReplicaListValidator format checking is incomplete (#9326)
  • PR-9382 - KAFKA-10554; Perform follower truncation based on diverging epochs in Fetch response (#9382)
  • PR-9675 - KAFKA-10794 Replica leader election is too slow in the case of too many partitions (#9675)
  • PR-8826 - KAFKA-10090 Misleading warnings: The configuration was supplied but i… (#8826)
  • PR-9423 - KAFKA-9263 The new hw is added to incorrect log when ReplicaAlterLogDirsThread is replacing log (fix PlaintextAdminIntegrationTest.testAlterReplicaLogDirs) (#9423)
  • PR-9601 - KAFKA-10729; Bump remaining RPC’s to use tagged fields. (#9601)
  • PR-9609 - KAFKA-6687: restrict DSL to allow only Streams from the same source topics (#9609)
  • PR-9659 - KAFKA-10770: Remove duplicate defination of Metrics#getTags (#9659)
  • PR-9607 - KAFKA-10722: Described the types of the used state stores (#9607)
  • PR-9632 - KAFKA-10702; Skip bookkeeping of empty transactions (#9632)
  • PR-9611 - KAFKA-10736 Convert transaction coordinator metadata schemas to use g… (#9611)
  • PR-9597 - KAFKA-10720: Document prohibition on header mutation by SMTs (#9597)
  • PR-9644 - KAFKA-10565: Only print console producer prompt with a tty (#9644)
  • PR-9648 - KAFKA-10758: ProcessorTopology should only consider its own nodes when updating regex source topics (#9648)
  • PR-9634 - KAFKA-10755: Should consider commit latency when computing next commit timestamp (#9634)
  • PR-9605 - KAFKA-10727; Handle Kerberos error during re-login as transient failure in clients (#9605)
  • PR-9633 - KAFKA-10706; Ensure leader epoch cache is cleaned after truncation to end offset (#9633)
  • PR-9204 - KAFKA-6181 Examining log messages with {{–deep-iteration}} should show superset of fields (#9204)
  • PR-9636 - KAFKA-10757: Fix compilation failure in StreamThreadTest (#9636)
  • PR-9433 - KAFKA-10607: Consistent behaviour for response errorCounts() (#9433)
  • PR-9596 - KAFKA-10723: Fix LogManager shutdown error handling (#9596)
  • PR-7498 - KAFKA-9023: Log request destination when the Producer gets disconnected (#7498)
  • PR-9386 - KAFKA-10024: Add dynamic configuration and enforce quota for per-IP connection rate limits (KIP-612, part 2) (#9386)
  • PR-9623 - KAFKA-10692: Add delegation.token.secret.key, deprecate …master.key (#9623)
  • PR-9598 - KAFKA-10701 First line of detailed stats from consumer-perf-test.sh incorrect (#9598)
  • PR-9547 - KAFKA-9630; Replace OffsetsForLeaderEpoch request/response with automated protocol (#9547)
  • PR-9507 - KAFKA-10628: remove all the unnecessary parameters from the tests which are using TopologyTestDriver (#9507)
  • PR-9570 - KAFKA-9274: Handle TimeoutException on commit (#9570)
  • PR-9572 - KAFKA-10500: Allow resizing of StreamThread state store caches (#9572)
  • PR-9473 - KAFKA-10545: Create topic IDs in ZooKeeper and Controller (#9473)
  • PR-9401 - KAFKA-9628; Replace Produce request/response with automated protocol (#9401)
  • PR-9566 - KAFKA-10618: Rename UUID to Uuid and make it more efficient (#9566)
  • PR-9487 - KAFKA-9331: Add a streams specific uncaught exception handler (#9487)
  • PR-9318 - KAFKA-10497 Convert group coordinator metadata schemas to use generat… (#9318)
  • PR-9569 - KAFKA-10687: make ProduceRespone only returns INVALID_PRODUCER_EPOCH (#9569)
  • PR-9568 - KAFKA-10689: fix windowed FKJ topology and put checks in assignor to avoid infinite loops (#9568)
  • PR-9576 - KAFKA-10685: strictly parsing the date/time format (#9576)
  • PR-9594 - Improve JavaDoc (#9594)
  • PR-9563 - KAFKA-10684; Avoid additional envelope copies during network transmission (#9563)
  • PR-9582 - KAFKA-6687: rewrite topology to allow reading the same topic multiple times in the DSL (#9582)
  • PR-9583 - KAFKA-10705: Make state stores not readable by others (#9583)
  • PR-9588 - Add system test coverage for group coordinator migration (#9588)
  • PR-9580 - KAFKA-10350: add forwarding manager implementation with metrics (#9580)
  • PR-9573 - KAFKA-10693: Close quota managers created in tests (#9573)
  • PR-9499 - KAFKA-10470: Zstd upgrade and buffering (#9499)
  • PR-9558 - migrate remaining RPCs (#9558)
  • PR-9266 - KAFKA-10469: Respect logging hierarchy (KIP-676) (#9266)
  • PR-9531 - KAFKA-10661; Add new resigned state for graceful shutdown/initialization (#9531)
  • PR-9555 - KAFKA-10673: Cache inter broker listener name used in connection quotas (#9555)
  • PR-9561 - KAFKA-10624: For FeatureZNodeStatus, use sealed trait instead of Enumeration (#9561)
  • PR-9543 - KAFKA-10500: Makes the Stream thread list resizable (#9543)
  • PR-9103 - KAFKA-10181: Use Envelope RPC to do redirection for (Incremental)AlterConfig, AlterClientQuota and CreateTopics (#9103)
  • PR-7751 - KAFKA-7987: Reinitialize ZookeeperClient after auth failures (#7751)
  • PR-9000 - KAFKA-10036: Improve handling and documentation of Suppliers (#9000)
  • PR-9482 - KAFKA-10632; Raft client should push all committed data to state machines (#9482)
  • PR-9364 - KAFKA-10471 Mark broker crash during log loading as unclean shutdown (#9364)
  • edc2c4fd - KAFKA-10669: Make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field default to 1
  • 236d7dc8 - KAFKA-10669: Make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field default to 1
  • PR-9515 - KAFKA-10651: read offsets directly from checkpoint for uninitialized tasks (#9515)
  • PR-9534 - KAFKA-10664: Delete existing checkpoint when writing empty offsets (#9534)
  • PR-9521 - KAFKA-10638: Fix QueryableStateIntegrationTest (#9521)
  • PR-9503 - KAFKA-10644; Fix VotedToUnattached test error (#9503)
  • PR-9418 - KAFKA-10601; Add support for append linger to Raft implementation (#9418)
  • PR-9506 - KAFKA-10647; Only serialize owned partitions when consumer protocol version >= 1 (#9506)
  • PR-9486 - KAFKA-9381: Fix publishing valid scaladoc for streams-scala (#9486)
  • PR-9464 - KAFKA-10616: Always call prepare-commit before suspending for active tasks (#9464)
  • PR-8794 - KAFKA-10092: Remove unnecessary contructor and exception in NioEchoServer (#8794)
  • 35eea09a - KAFKA-10592: Fix vagrant for a system tests with python3
  • c8c1baf4 - KAFKA-10592: Fix vagrant for a system tests with python3
  • PR-9479 - Handle ProducerFencedException on offset commit (#9479)
  • PR-9270 - KAFKA-10284: Group membership update due to static member rejoin should be persisted (#9270)
  • PR-9406 - KAFKA-10520; Ensure transactional producers poll if leastLoadedNode not available with max.in.flight=1 (#9406)
  • PR-9454 - KAFKA-10618: Add UUID class, use in protocols (part of KIP-516) (#9454)
  • PR-9237 - KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered (#9237)
  • PR-9338 - KAFKA-10515: Properly initialize nullable Serdes with default values (#9338)
  • PR-9431 - KAFKA-10426: Deadlock in DistributedHerder during session key update. (#9431)
  • PR-9455 - KAFKA-10599: Implement basic CLI tool for feature versioning system (#9409) (#9455)
  • PR-9429 - KAFKA-10572 mirror-maker config changes for KIP-629 (#9429)
  • PR-9448 - KAFKA-10605: Deprecate old PAPI registration methods (#9448)
  • PR-9383 - KAFKA-10455: Ensure that probing rebalances always occur (#9383)
  • PR-9368 - KAFKA-9274: Add timeout handling for state restore and StandbyTasks (#9368)
  • PR-9409 - KAFKA-10599: Implement basic CLI tool for feature versioning system (#9409)
  • PR-9343 - KAFKA-10332: Update MM2 refreshTopicPartitions() logic (#9343)
  • PR-9397 - KAFKA-10583: Add documentation on the thread-safety of KafkaAdminClient (#9397)
  • PR-9425 - KAFKA-10600: Connect should not add error to connector validation values for properties not in connector’s ConfigDef (#9425)
  • PR-9432 - KAFKA-10559: Not letting TimeoutException shutdown the app during internal topic validation (#9432)
  • PR-9422 - KAFKA-10602: Make RetryWithToleranceOperator thread safe (#9422)
  • PR-9407 - KAFKA-10611: Merge log error to avoid double error (#9407)
  • PR-9149 - KAFKA-10340: Improve trace logging under connector based topic creation (#9149)
  • PR-9438 - KAFKA-10613: Only set leader epoch when list-offset version >= 4 (#9438)
  • PR-8150 - KAFKA-9587: Add omitted configs in KafkaProducer javadoc (#8150)
  • PR-9353 - KAFKA-10521; Skip partition watch registration when AlterIsr is expected (#9353)
  • PR-9403 - KAFKA-10573 Update connect transforms configs for KIP-629 (#9403)
  • ff1e2271 - backport KAFKA-10571
  • e6b543a4 - KAFKA-10570; Rename JMXReporter configs for KIP-629
  • f46d4f4f - KAFKA-10570; Rename JMXReporter configs for KIP-629
  • PR-9415 - KAFKA-10494: Eager handling of sending old values (#9415)
  • PR-9396 - KAFKA-10437: Implement new PAPI support for test-utils (#9396)
  • PR-9352 - KAFKA-10533; KafkaRaftClient should flush log after appends (#9352)
  • PR-8891 - KAFKA-10143; Improve test coverage for throttle changes during reassignment (#8891)
  • PR-9375 - KAFKA-10574: Fix infinite loop in Values::parseString (#9375)
  • PR-9408 - KAFKA-10598: Improve IQ name and type checks (#9408)
  • PR-9399 - KAFKA-10584:IndexSearchType should use sealed trait instead of Enumeration (#9399)
  • PR-7929 - KAFKA-9393: DeleteRecords may cause extreme lock contention for large partition directories (#7929)
  • 4ab72780 - KAFKA-10571; Replace blackout with backoff for KIP-629
  • PR-9020 - KAFKA-10271: Performance regression while fetching a key from a single partition (#9020)
  • PR-9380 - KAFKA-7334: Suggest changing config for state.dir in case of FileNotFoundException (#9380)
  • PR-9393 - KAFKA-10028: Minor fixes to describeFeatures and updateFeatures apis (#9393)
  • PR-9139 - KAFKA-9929: Support backward iterator on SessionStore (#9139)
  • 4d3036bb - Updating trunk versions after cutting branch for 2.7

ksqlDB

  • PR-7545 - test: fix QTT tests after upstream change (#7306)
  • PR-7501 - fix: updated Guava version
  • PR-7534 - Use Java’s Base64 instead of Jersey’s.
  • PR-7477 - fix: multi-column keys are broken in some scenarios when rearranged
  • PR-7494 - chore: Updates to Vert.x 3.9.7 and Netty 4.1.60.Final
  • PR-7388 - fix(migrations tool): allow non-capitalized set/unset commands to be used
  • PR-7412 - fix: ignore streams threads in KSQL uncaught exception handler
  • PR-7376 - fix: source key format compatibility with non primitive types (#7337)
  • PR-7342 - fix: append additional fields to row values in KsqlMaterialization.java to fix pull query filtering
  • PR-7333 - fix: upgrade io.netty:netty-codec-http2 to 4.1.62.Final.
  • PR-7307 - feat: add maven wrapper
  • PR-7305 - fix: only list known connect worker properties when listing server props (5.5.x)
  • PR-7212 - fix: Ensures BaseSubscriber.makeRequest is called on context in PollableSubscriber
  • PR-7097 - feat: CLI should fail for unsupported server version
  • PR-7192 - fix: ksql.service.id should not be usable as a query parameter
  • PR-6944 - fix: Make pull query metrics apply only to pull and not also push
  • PR-7180 - fix: check for null inputs for various timestamp functions
  • PR-7188 - feat: Adds max concurrent pull queries to limit effects of table scans
  • PR-7190 - feat(migrations): enable set/unset statements for migration tool
  • PR-7193 - fix: Pull Queries: Avoids inefficient KsqlConfig copy with overrides
  • PR-7161 - feat(migrations): enable insert values and create connector for migrations tool
  • PR-7181 - fix: fix the cache max bytes buffering check
  • PR-7109 - feat: limit the number of active push queries everywhere using “ksql.max.push.queries” config
  • PR-7178 - fix: incorporate lambdas into CoercionUtil and change LambdaVariable from Literal to Expression
  • PR-7162 - fix: prevent IOB when printing topics with a key/value with an empty string
  • PR-7152 - feat: Adds Lambda functionality to the interpreter
  • PR-7155 - feat: Allows lots of table scans cases where keys cannot easily be extracted
  • PR-7148 - feat: add lambda feature flag
  • PR-7150 - refactor: decouple context from ExpressionTypeManager, SqlToJavaVisitor, and CodeGenRunner
  • PR-7159 - fix: return null on null input for timestamp math functions
  • PR-7153 - feat(migrations): implement clean command
  • PR-6901 - feat: Add timestamp arithmetic functionality
  • PR-7145 - feat(migrations): implement info command
  • PR-7085 - feat: Introduce materialization support for tables using CTAS to make tables queryable
  • PR-7137 - feat(migrations): add ability to apply specific migration version
  • PR-7103 - refactor: pull lambda type to sql lambda check out of UdfIndex and GenericUtils
  • PR-7006 - feat: Adds an expression interpreter to be used for pull queries
  • PR-7133 - feat(migrations): implement create command
  • PR-7134 - fix: script used in kci-170 is now exporting an array instead of string
  • PR-7118 - feat: added script to export antlr tokens
  • PR-7099 - feat(migrations): apply command
  • PR-7086 - fix: Table Scan: Filters out StreamsMetadata for wrong subtopology
  • PR-7078 - fix: quick fix for appender
  • PR-7056 - feat: implement correct logic for nested lambdas and more complex lambda expressions
  • PR-7061 - feat: classify authorization exception as user error
  • PR-7093 - fix: get unique function names in correct order
  • PR-7075 - feat: add filter lambda udf
  • PR-6994 - feat: introduce transform and reduce invocation function for lambdas
  • PR-7087 - feat(migrations): implement validate command
  • PR-6985 - fix: get all available restore commands even on poll timeout
  • PR-6966 - feat: add lambda type checking without adding lambda sql type
  • PR-7030 - feat(client): add getInfo method to java client
  • PR-7032 - fix: format cast arguments with passed context (6.0.x)
  • PR-7011 - refactor: use SqlArgument wrapper to look up functions
  • PR-6827 - feat: add describe option to list streams/tables
  • PR-6988 - feat: new and initialize commands for migration tool
  • PR-6989 - feat: adding TypeContext for Lambdas
  • PR-6995 - update changelog for 0.15
  • PR-6954 - fix: make format_timestamp and parse_timestamp default to utc
  • PR-6891 - feat: Adds ability to bypass cache for pull queries
  • PR-6927 - feat: Support timestamp protobuf serde
  • PR-6987 - build: move alerts to #ksqldb-warn
  • PR-6981 - build: fix build after AK bump (#6979)
  • PR-6939 - feat: Allows pull queries with generic WHERE clauses
  • PR-6955 - feat: add functions as UDF input types
  • PR-6979 - build: fix build after AK bump
  • PR-6935 - fix: ff-4240 upgrade httpclient version
  • PR-6959 - Revert “fix: throw error message on create source with no value colum…ns (#6680)”
  • PR-6946 - fix: npe when getting topic configs
  • PR-6930 - fix: do not log data on JSON deserialization errors (MINOR)
  • PR-6933 - fix: use internal format with multi-col support when needed
  • PR-6936 - fix: Adds groupMetadata as a faked method of SandboxedConsumer
  • PR-6868 - feat: add lambda syntax to grammar
  • PR-6874 - feat: Rewrites pull query WHERE clause to be in DNF and allow more expressions
  • PR-6908 - fix: Select star table scans need keys in the intermediate row
  • PR-6905 - fix: close transient queries when network splits
  • PR-6909 - fix: dont use the streamthread to construct the log-metric-and-continue logger
  • PR-6813 - feat: Make pull queries streamed asynchronously
  • PR-6895 - fix: Fixes auth for forwarded requests for pull queries
  • PR-6889 - build: Fix up build process & remove virtual packages from ksqldb
  • PR-6852 - feat: add functions to support use of timestamp data type
  • PR-6884 - feat: enable decimal for Protobuf
  • PR-6876 - fix: fix how the buffer limit check evaluates streams config
  • PR-6872 - feat: display precision and scale when describing DECIMAL columns
  • PR-6763 - fix: client/server SSL settings fail when ‘ssl.key.password’ is set
  • PR-6846 - fix: Make Select * avoid code gen for projections
  • PR-6845 - feat: add standard deviation udf
  • PR-6851 - feat: make UDAFs configurable and remove limit on COLLECT_LIST/SET
  • PR-6821 - fix: use the right name for fallback subject for transient queries
  • PR-6814 - feat: Move WHERE clause handling for pull queries into logical plan
  • PR-6726 - feat: Adds table scans to pull queries
  • PR-6806 - feat: timestamp support - casting, comparisons and serde
  • PR-6812 - refactor: Tune Package Relationship
  • PR-6823 - refactor: wrap Vert.x SSL configurations inside a VertxSslOptionsFactory class
  • PR-6797 - feat: bound the size of the shared buffer cache used by queries
  • PR-6816 - chore: bump command version to 3 for 0.15
  • PR-6803 - feat: support PARTITION BY on multiple expressions
  • PR-6786 - feat: ungate support for multi-column GROUP BY
  • PR-6684 - feat: Move code from physical to logical plan for Projection
  • PR-6714 - fix: Removes orphaned topics from transient queries
  • PR-6753 - fix: include ‘ksql.streams.topic.*’ prefix properties on LIST PROPERTIES output
  • PR-6727 - fix: catch stack overflow error when parsing/preparing statements
  • PR-6073 - fix: CREATE IF NOT EXISTS does not work at all
  • PR-6722 - feat: expose support for array and struct keys
  • PR-6701 - fix: Makes response codes rate limited as well as prints a message when it is hit
  • PR-6665 - fix: don’t create threads per request
  • PR-6708 - feat: support table joins on key format mismatch
  • PR-6709 - fix: change locate() error message for a more user-friendly message
  • PR-6694 - feat: expose AVRO and JSON_SR as key formats
  • PR-6692 - feat: support PROTOBUF keys
  • PR-6693 - fix: Refactors rate limiter within LoggingHandler to be singleton
  • PR-6680 - fix: throw error message on create source with no value columns
  • PR-6615 - feat: Adds logging for every request to ksqlDB
  • PR-6667 - refactor: swap Struct key to GenericKey
  • PR-6663 - fix: Pull query thread leak
  • PR-6655 - fix: fix error categorization on NPE from streams
  • PR-6653 - refactor: remove unused validation code
  • PR-6641 - feat: add partitions to PRINT TOPIC output
  • PR-6647 - fix: propagate null-valued records in repartition
  • PR-6375 - feat: Implement a physical plan builder and physical plan for pull queries
  • PR-6574 - feat: add CLI parameter to define variables
  • PR-6572 - fix: allow reserved keywords on variables names
  • PR-6573 - fix: cannot reference variables in DEFINE statement
  • PR-6475 - feat: support IN operator for non-pull queries
  • PR-6594 - refactor: re-use CreateSourceFactory to create streams/tables for CREATE_AS statements
  • PR-6603 - fix: Check for index before removing value in undo of COLLECT_LIST
  • PR-6524 - feat: update ksql restore command to skip incompatible commands if flag set
  • PR-6593 - refactor: remove duplicate decimal code (MINOR)
  • PR-6587 - feat: Extend CAST to ARRAY, MAP and STRUCT
  • PR-6553 - feat: add WITH clause and QUERY_ID property to INSERT INTO statements
  • PR-6143 - feat: terminate persistent query on DROP command
  • PR-6544 - feat: support multi-column key declarations
  • PR-6545 - feat: add constraint to deny drop sources referenced by other CREATE_AS sources
  • PR-6462 - feat: cli to show tombstones in transient query output
  • PR-6548 - fix: Bypass window store cache when doing windowed pull queries
  • PR-6551 - fix: avoid RUN SCRIPT to override CLI session variables/properties
  • PR-6550 - fix: support joins on key formats with different default serde features
  • PR-6535 - feat: CASE expressions can now handle 12+ conditions in docker + cloud
  • PR-6537 - fix: ‘-e/-f’ CLI parameters are not handling session variables correctly
  • PR-6532 - fix: clean up leaked admin client threads when issuing join query to query-stream endpoint
  • PR-6504 - feat: support variable substitution in SQL statements
  • PR-6508 - fix: NPE in PARTITON BY on null value
  • PR-6485 - refactor: add engine-common module + move SqlValueCoercer
  • PR-6474 - feat: new syntax to interact with session variables (define/undefine/show variables)
  • PR-6440 - feat: new CLI parameter (-f,–file) to execute commands from a file
  • PR-6409 - feat: Add support for IN clause to pull queries
  • PR-6361 - feat: new command to restore ksqlDB command topic backups

REST Proxy

  • PR-853 - Fix error code for missing schema in Produce V2 and V3.
  • PR-850 - KREST-1187: Propagate client.sasl.jaas.config to Schema Registry.
  • PR-849 - QEC-5809: Make SchemaRegistryClient as request scoped
  • PR-846 - KREST-481 Fix a problem with configuring monitoring interceptors
  • 8f7008ac - fix: resolve ProduceController early/outside of lambda
  • PR-831 - KREST-865 Remove usage of the deprecated SimpleAclAuthorizer.
  • PR-830 - KREST-826, KREST-837 Reduce coupling to kafka-clients
  • PR-827 - Add a SSL-everywhere* test environment for integration tests.
  • PR-822 - KREST-770 Annotate the v3 Produce API
  • PR-815 - KREST-729 Set the endpoint blocking filter’s priority to be higher than all the predefined filter priorities.
  • PR-797 - Produce V3 API
  • PR-806 - KREST-638 Some small deployment improvements.
  • PR-804 - Return Bad Request on JSON parsing errors.
  • PR-799 - KREST-558 Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
  • 2ffd770b - KREST-558 Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
  • PR-794 - Allow setting the request contentType in testProduceToAuthorizationError
  • PR-793 - Add schema-registry to test environments.
  • PR-792 - KREST-456 Add API for creating a topic with custom partition-to-replicas assignments
  • PR-787 - Multiplex all produce requests into a single Producer.
  • PR-780 - KREST-360: Start brokers concurrently in ClusterTestHarness
  • PR-776 - KREST-297: Correctly propage schema.registry.url config to clients.
  • PR-251 - bump guava version to 30.1.1-jre
  • PR-248 - ST-5293: Explicit load of netty dependencies
  • PR-249 - DGS-1621 Upgrade Jetty to 9.4.40
  • PR-246 - fix: CVE-2021-28168
  • PR-245 - Bump netty-version 4.1.63
  • PR-244 - Upgrade Jetty to 9.4.39.v20210325
  • PR-243 - DGS-1546 Map JsonMappingException to 400 response
  • PR-242 - Upgrade Jetty to 9.4.39.v20210325
  • PR-239 - KREST-860 Fix deprecated usages of org.apache.kafka.common.Metric.value()
  • PR-237 - Define Netty version to be shared to downstreams.
  • PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224
  • PR-229 - KC-1456: Update Jetty to 9.4.36.v20210114
  • PR-231 - UIF-298: Upgrade Jetty to 9.4.35.v20201120
  • PR-228 - MMA-9930: Use compatible version of Guava
  • PR-222 - UIF-300: Upgrade org.apache.httpcomponents.httpclient
  • PR-223 - UIF-302: Upgrade guava for 5.4.x and older.
  • PR-224 - upgrade hibernate-validator to 6.1.7.Final
  • PR-218 - DG:1183 Address http status 422 which is not supported in Response.status in rest-util’s ExceptionMapper

Schema Registry

  • PR-41 - SEC-2105: Update guava to version 30.1.1-jre
  • PR-1852 - DGS-1595 Ignore invalid Avro defaults in Avro converter
  • PR-1847 - DGS-1570 Handle optional integer/number with format in JSON Schema
  • PR-1845 - DGS-1557: Upgrade io.netty_netty-codec
  • PR-1846 - DGS-1569 Ensure recursive types are not built prematurely
  • PR-1844 - DGS-1562 Don’t require refs be declared with root schema
  • PR-1839 - DGS-1550 Add cache for transformed schemas
  • PR-1833 - DGS-1534 Use raw Avro schema to check doc equality
  • PR-1821 - DGS-1469 Add server-side schema cache
  • PR-1816 - DGS-1391: fix Avro Convertor for GenericData of Enum Union with names
  • PR-1815 - DGS-1451 Add Jackson annotations for object evolution
  • PR-1813 - Fix NPE when checking compatibility
  • PR-1812 - Add more JSON Schema tests
  • PR-1808 - DGS-1403 Allow register different schema types if compatibility level
  • PR-1807 - UIF-302: Add Guava declaration in dependency.
  • PR-1806 - DGS-1402 Flag to set non-required fields to optional in JSON Schema
  • PR-1805 - DGS-1400 Escape Special Character in Protobuf Option
  • PR-1804 - DGS-1392 Allow removing false from closed content model in JSON Schema
  • PR-1802 - DGS-1398 Allow adding empty schema to open content model for JSON Schema
  • PR-1801 - DGS-1396 Fix optionality of ref schemas in JSON Schema converter
  • PR-1799 - DGS-1391: fix Avro Convertor for GenericData of union with null, string, enum
  • PR-1794 - Fix String.format in compatibility errorMessages
  • PR-1795 - DGS-1221 Ensure AvroSchema.hashCode is consistent with equals
  • PR-1791 - DGS-1364 Fix compatibility for JSON empty schema
  • PR-1737 - Backport #1733 to 5.4.x
  • PR-1784 - DGS-1221 Fix AvroSchema.equals
  • PR-1781 - DGS-785 Add logical type converter to Kafka avro serializer/deserializer (#17
  • PR-1779 - DGS-772 Add API annotations for /mode
  • PR-1777 - Upgrade to everit 1.12.2
  • PR-1639 - Upgrade to wire 3.6.0
  • PR-1775 - DGS-104 Fix Avro compat check
  • PR-1774 - DGS-1333 Add config to remove Java props from Avro schemas
  • PR-1771 - Allow handlers to override checkpoints
  • PR-1770 - DGS-1315: upgrade HibernateValidator
  • PR-1769 - DGS-1314: update JetBeanKotlin to 1.4.21
  • PR-1741 - DG-1162: Add verbose option to SR maven plugin test-compatibility goal
  • PR-1768 - DGS-1285 Minor API cleanup
  • PR-1767 - DG-615 Include dependencies for logical types if missing
  • PR-1761 - DGS-1251 Only rollback invalid updates on leader
  • PR-1756 - Fix spotbugs version
  • PR-1752 - DG-1210 Add flag to enable primitive wrappers for Protobuf converter
  • PR-1749 - DG-615 Allow native schemas to be used w/o refs
  • PR-1743 - DG-615 Add precision to Decimal logical type
  • PR-1742 - DG-716 Upgrade to Avro 1.10.1
  • PR-1738 - Increase timeout to 3 hrs
  • PR-1734 - DG-1135: Add support for nullable fields in ReflectionAvroSerializer
  • PR-1733 - Add kafkastore.topic.skip.validation configurations option
  • PR-1727 - Add getSchemas functionality to the SchemaRegistryClient.
  • PR-1731 - DG-615 Add logical types to Protobuf converter
  • PR-1713 - Enable coordinator test
  • PR-1712 - Temporarily disable coordinator test
  • PR-1707 - Re-enable SASL test
  • PR-1705 - Fix protobuf provider compilation
  • PR-1703 - Temporarily ignore sasl tests
  • PR-1694 - Skip plugin execution by configuration/CLI
  • PR-1632 - Add Mode.READONLY_OVERRIDE and global mode

Commercial Features

Confluent Server

  • KC-1740: Upgrading to IBP 2.8 and writing to the ZNode can reassign topic IDs
  • SEC-298 - Log errors when MDS can not start
  • APPSEC-90: Upgrade Azure dependencies
  • KC-1610: Upgrade dataformat-cbor version (#3589)
  • KC-1624: fix perf regression due to removal of BufferSupplier (#3555)
  • KC-1625: DynamicBrokerConfig must not log sensitive config values
  • KC-1614: Update BrokerFailureDetector to use executeRemoval
  • CNK-1695: Log cause of unfixable hard goals in goal violation self-healing
  • ST-5293: Upgrading version of netty due to CVE
  • KGLOBAL-414: Reset AlterIsr in-flight state for duplicate update requests
  • CONFLUENT: Change visibility of LicenseConfig constants to public
  • KSTORAGE-1511: Remove the delay of 1ms when scheduling flush in Log.roll()
  • [KSTORAGE-1474] set unclean.leader.election.enable to false for tier state topic (#3429)
  • KSTORAGE-1545: Partition deletion does not stop leadership of tier tasks
  • CNK-617: Temporary pull out CloudNetworkFault + deps
  • Fix SemaphoreCI integration
  • KAFKALESS-400: Handle non-PLAINTEXT setups for SBC commands (#3488)
  • KAFKALESS-409: Handle License topic MinISR of 2 in SBC tests
  • Fix TaskManagerTest: correct merge error
  • CONFLUENT: Fix storage test failures
  • CONFLUENT: Make sure to filter out confluent configuration
  • Start licesne validator after unfencing KRaft broker
  • CONFLUENT: Fix all of the tests in LogManagerTest
  • Handle node.id in metrics context and null auditLogProvider for KRaft
  • KAFKALESS-387: Respect broker concurrency limit even for “dataless” m
  • Fixed BrokerEpochIntegrationTest, ensure compatability for error, and minor style fix
  • Add license validator and disable unsupported features for KRaft
  • Merge: Fix KafkaApisTest failures
  • Fix PartitionTest and MergedLogTest
  • Revert changes to ce-kafka handling of topic ID and partition metadata file, allow all versions to write partition metadata file, and handle deletion more cleanly
  • Fixed ReplicaManager test with new understanding of how tiered storage ids work with new topic IDs
  • Create forward and api version managers after metacache
  • KC-1580 Remove clearPending method from AlterIsrManager
  • Fix PartitionTest
  • CONFLUENT: Manually apply upstream revert
  • topic ID changes for kstorage-1413
  • CONFLUENT: Add listeners to all of the CE requests
  • add new create_zookeeper_if_necessary method
  • KSTORAGE-1494: Prevent deletion of empty compacted segments
  • Added back removed line so system tests can run
  • removed AK topic ID methods in system test files
  • KAFKALESS-342: Retry topic description failures in the Executor
  • KAFKALESS-368: Update balancer state to DISABLED/ABORTED after cruiseControl is shutdown
  • KSTORAGE-1471 Fix fallback bootstrap server (#3405)
  • KAFKALESS-348: Fix previous status null case returned from EvenClusterLoadStateRecord
  • SBC-309: Update EvenClusterLoad status in persistenceStore and return from it
  • SBC-306: Retry on InvalidReplicaAssignmentException
  • KAFKA-12583: Upgrade netty to 4.1.62.Final
  • MMA-10222: start ConfluentLicenseValidator after Kafka started
  • Increase timeout for EvenClusterLoadStatusTest
  • AUDIT-173: Hide client IP address for Confluent Cloud
  • Increase timeout for AddBrokerTest
  • Use the correct util method to wait for a topic to be created
  • Use method instead of variable for auto-topic create meters
  • SEC-2008 Fix intermittent failure of testDefaultConfigurationAndReconfigure
  • Fix upstream merge caused auto-topic creation related issue
  • KSTORAGE-1466: retry of putSegment may cause violation of transactions guarantees
  • use waitUntilBrokerMetadataIsPropagated in TierTopicPartitionOfflineTest Stream test fix post merge from upstream.
  • DGS-908: Schema validation metrics api integration
  • CL-99: Added link status metrics
  • CL-107: Adding more logging for CL
  • Allow for Cluster Link types in ZK repository test
  • Don’t shutdown the broker in the middle of RequestQuotaTest
  • Fix shutdown to use MetadataCache instead of ZK
  • Allow ClusterLink configs in ZkConfigRepository
  • MMA-9336 topic creation race may prevent metrics collection on startup
  • SBC-324: Setup mock databalancer for integration tests
  • KSTORAGE-1464: tiered fetch of transaction index should throw on cancellation
  • SBC-334: Reduce number of describeTopics calls in SBC Executor
  • Upgrade ce-events to cloudevent SDK v2
  • Catch topic creation race during telemetry startup
  • SBC-309: Write EvenClusterLoad status to ApiPersistenceStore and update start state
  • Fix mocks for ClusterLinkAdminManager in KafkaApisTest
  • Reuse the same createKafkaMetricsContext
  • METRICS-2925: Use Keyed class to filter on MetricKey and emit SerializedMetric
  • Stub out support for tags in MetadataBroker
  • Include full CP describeConfigs() method in ConfigHelper
  • CL-293: Active link count metrics that can be used for billing for source clusters
  • pull up metrics, eliminate metric dups due to pullup
  • Undo pullup adminManager,kafkaController,_zkClient,configRepository
  • Core unit tests all passing
  • merge: upstream Feb06 fixed build issues
  • SBC-54: ConcurrentModificationException on SBC shutdown
  • METRICS-2934: Do not log configuration values on failed Provider validation
  • Cherry-pick KAFKA-12268: Implement task idling semantics via currentLag API
  • METRICS-2924: Update telemetry-client to v1.593.0 and use Suppliers for HttpExporter updates
  • KC-1480; Enable admin api logging by default
  • CNK-1649; Improve how ReplicaFetcherThread apply the follower quota
  • KAFKA-9189: Use MetadataCache instead of ZK during controlled shutdown to avoid hang
  • AUDIT-359: Added config multitenant.oauth.superuser.disable
  • Changed to JUnit5 format
  • merge: DESCRIBE_PRODUCERS api made hidden from exposed multitenant APIs
  • CL-292: Revert change to RequestChannel from KC-130 that was wrongly updating the CreateClusterLinksRequest
  • merge: Disable testReaderDoesNotWaitForNewWriterBeforeCompletingFuture, till we merge 38fee5f from CCS
  • SBC-333: Do not re-execute reassignments in the Executor
  • merge: DESCRIBE_PRODUCERS exposed under multitenant API.
  • merge: Port DECSRIBE_PRODUCER and CREATE/DELETE_TOPIC API changes for testIsApiAllowedHandlesAllApiVersions
  • merge: Fix testCleanupSnapshots to work with ce-kafka and also removed unused imports.
  • CL-100: Integration tests for link-count.
  • SBC-179: Refactor Executor 3-step process (inter-broker, intra-broker, leadership movements) (part 1/2)
  • CL-130: Refactored CL Admin APIs to use admin objects
  • CL-118: Adding failure metrics for periodic tasks
  • KAFKA-12330; FetchSessionCache may cause starvation for partitions when FetchResponse is full
  • CNK-1578: Fix TLS handshake time metric
  • merge: Fixed issue in merge conflicts and ported MessageTest not to use toStruct and fromStruct methods.
  • Set entity type for new topic name field in DeleteTopicsRequest. Removed some invalid uses of data.topicNames()
  • DGS-1397: Schema interceptor override default
  • CL-13: Syncing offsets for only active mirrors
  • merge: DESCRIBE_PRODUCERS support in AuditLogCategoryResultRouter and fixed merge conflict issue in TopicWithId
  • SBC-310: SBC system test support for AnyUnevenLoad CLI
  • Cherry-pick KAFKA-10413 for Connect
  • CL-100: Add link mode tag to link-count metric.
  • DGS-909: Disallow cloud user from enabling schema validation while feature flag disabled
  • DGS-1289: Move schema validation interceptors to Log class
  • merge: Remove KafkaConfig#usesTopicId
  • merge: Revert to returning Node in DescribeClusterResponse#nodes.
  • merge: Add test for cluster link in MultiTenantRequestContextTest#testDescribeClusterResponse
  • merge: tweak dependency resolution for bcpkix-jdk15on. io.kubernetes:client-java brings an older version than clients.
  • merge: Add unit tests for DescribeCluster API in MultiTenantRequestContextTest
  • merge: add DESCRIBE_CLUSTER in ClusterLinkApis; add entityType for ClusterId in DescribeClusterResponse.
  • METRICS-2907: Make HttpExporterConfig proxy passwords Password objects
  • merge: migrate JUnit in TierAbortedTxnReaderTest
  • SBC-316: Throw BalancerLoadError for Balancer requests
  • CNK-1684: Add per-IP throttle metrics
  • CPKAFKA-6514: Extend the broker failure test timeout from 30s to 300s
  • KC-1223: expose CurrentLeaderEpoch as part of the ReplicaStatus API
  • PG-100: make LicenseManagers check internal configurable
  • KC-1479: Improve error message when creating topic with invalid clust
  • CPKAFKA-2389; Remove dependency on core from ce-licensing
  • SBC-317: Ensure RackAwareGoal allows replica swaps for the same partition
  • KC-1470: Increase per-partition fetch size for cluster link to work better with tiered storage
  • SBC-307: Fix issue where replica changes are recorded incorrectly and can result endless rebalances
  • KAFKA-12252 and KAFKA-12262: Fix key rotation when leadership changes
  • KSTORAGE-1423: Fix offsetMap bug in TierMetadataDebugger
  • CONFLUENT: Revert “Limit the max topic name length to 209 (#425)”
  • KC-1523: Fix inconsistencies in cluster link metrics with source connections
  • re enable downstream validation check
  • CNK-1326: FileBasedPlainSaslAuthenticator that loads from topic
  • merge: fix remaining issues in PartitionTest
  • CIAM-526 Comments on error catching behavior sending audit logs auth
  • Fix - Clear unclean leader state + AlterIsr
  • Improve MetricLoader logging
  • Add “Create” operation to Billing resource type
  • CONFLUENT: Update log message in PlainSaslServer
  • SBC-311: Gracefully handle failure to load KafkaDataBalanceManager
  • KAFKA-12400: Upgrade jetty to fix CVE-2020-27223
  • KSTORAGE-1397: Additional Logging in TierTopicConsumer Registration
  • KC-179: log request on disconnected client
  • METRICS-2868: add kafka exporter for events
  • KC-1451: Change Topic ID logic to support downgrades in ce-kafka
  • KC-1337: Simplify AlterMirrors protocol using operation codes instead of nested arrays
  • KC-1467: Delete unbalanced workload to reduce total workload in the Kafka soak cluster
  • KC-1522: Do not log parsed configs containing passwords in CruiseControlStartable
  • SBC-292: Implement kafka-rebalance-cluster –describe
  • SBC-304: Return even cluster load balance information from Kafka Api
  • KC-1517: Initialize cluster link quota when brokers start up
  • SBC-303: Backend implementation of Even Cluster Load Status api
  • KC-1461: Changing test_tiered_topic_mirror to use hotset_bytes for tier check
  • merge: remove ReplicaManager FIXME comment, track via KSTORAGE-1413
  • KC-788: Source connection origination for cluster links
  • Use AWS CLI version 1.18.* for Python 3.4 compatibility
  • merge: KafkaApis#handleDeleteAcls valides the response when the response is constructed now. Before, the validation was done when the response was serialized. This broke an assumption that we makde in MultiTenantRequestContextTest#testDeleteAclsResponse. This test verifies all the version. This fails for version 0 because it does not support PREFIX filters which is used by the interceptor. The interceptor always use version >= 1 so this commit updates the test to reflect this and thus avoid the issue.
  • merge: jmh-benchmark fixes after log start offset update refactoring
  • Remove -Xlint:unused and unused imports
  • Update log start offset for MergedLog via callback from Log to avoid mistakes
  • KSTORAGE-1404: Add fetcher processing time metrics
  • SBC-305: Don’t throw if rebalance plan computation fixes some goals at the expense of others.
  • Fix KafkaMetadataLog
  • PG-79: inject tracking identifiers through LicenseTrackingInfoHolder
  • Clean up AclAuthorizerTest
  • Fix ProducerStateManagerTest.testReloadFromTieredSnapshot
  • Fix OffsetsForLeaderEpochRequestTest#testOffsetsForLeaderEpochWhenPartitionUnderRecovery
  • KafkaApisTest#testInvalidEnvelopeRequestAsPrimary was removed in 474ad1eebd7d80f5cfa2e5f783e6fd6aafcf388b.
  • Address couple of FIXMEs related to topic ids
  • Fix compilation issue with Scala 2.12
  • Refactor ControlMetadataBatch and ControllerChannelManager
  • Changing ControlMetadataBatchResult
  • Add authentication counters to default broker include regex
  • KSTORAGE-1368: Metrics for object store fetch latencies
  • [KSTORAGE-1377] New metric for stray partition
  • Add topic id to ControlMetadataBatches.
  • KC-788: Source connection origination for cluster links (Part 1)
  • KAFKA-12324: Upgrade jetty to fix CVE-2020-27218
  • Set CE Kafka Master to next CP 7.0 Version
  • CONFLUENT: Add trace logging in SslTransportLayer when flush doesn’t complete
  • KAFKA-12334: Add the KIP-500 metadata shell
  • [METRICS-2820] Collect metrics on message conversion
  • METRICS-2819:Collect metrics about log cleaner
  • SBC-291: Add CLI for balancer status
  • KSTORAGE-1398: revert log level for request samples to info level
  • KSTORAGE-1390: Ensure async topic deletion is idempotent
  • [PG-66]: add licenseId and platformId in License Class
  • SBC: Add create and last update time to state machine
  • Fix AbstractHerderTest failures and merge conflicts
  • Fixup Partition#sendAlterIsrRequest
  • SBC: Make broker id available to KafkaCruiseControl
  • CONFLUENT: Support KIP-679 authorizeByResourceType in ConfluentServerAuthorizer
  • Forward cloud schema validation enablement config
  • CIAM-457: Add a healhcheck method to AuthCache
  • CONFLUENT: Update Zookeeper node start timeout in system tests
  • SBC-300: Catch NPE caused by BalancerStatusTracker in KDBM
  • SBC-295: Remove telemetry metrics enable check
  • Reduce CNK-819 plan computation cost
  • KC-1387: Fix promote then failover test
  • Remove unused imports, update buffer supplier for envelope request
  • CNK-1568: fix FetchRequestBenchmark BufferUnderflowException
  • Comment out imports in AbstractHerderTest to make progress on build
  • Fix checkstyle, some test failures, cluster link update
  • KC-1478: Fix timing issue in metrics check in MultiTenantClusterLinkTest.testMultiTenantClusterLink
  • [CNKAF-1033] Handle rapid exit of Anomaly healing.
  • SBC-283: Get rid of Zookeeper references in SBC
  • SBC-285: Remove diff computation from GoalViolationDetector
  • CONFLUENT: Add logger entry for mds request logger
  • CNK-1568: Fix MergedLogTierBenchmark IllegalArgumentException
  • CC-1053 (bandaid): Prevent offset commit flush failures from shadowing original exceptions for source tasks
  • KC-1460: Add support for cluster link configuration on source clusters
  • KC-1388: Add pending-stoped-only option to filter for describe mirrors
  • CNK-1548: Add option to not construct producer in KafkaBasedLog
  • SBC-272: Use Persistence store to save failed broker list in BrokerFailureDetector
  • RCCA-1881.4: Add OauthBearerLoginCallbackHandler empty config test
  • Revert “CNK-1548: Add option to not construct producer in KafkaBasedL
  • KAFKA-12193: Re-resolve IPs after a client disconnects (#9902)
  • CNK-1568: fix StateWriteBenchmark IllegalStateException
  • CNK-1564: Reduce StandardMetadataRequestBenchmark matrix
  • SBC-186: Update even cluster load protocol
  • Remove superfluous publicKeyPath check from OAuthBearerServerLoginCallbackHandler
  • SBC-175: Don’t rebalance within some time from cluster membership changes.
  • CNK-1564: Reduce BaseRecordBatchBenchmark matrix
  • SBC-165: Backend implementation for describe balancer status api
  • KSTORAGE-1073: retrieving /proc/diskstats metrics
  • KSTORAGE-1218: Replace the go-based storage k8s liveness probe
  • MMA-9397: Filter out confluent.ssl configs when describing topic configs
  • KSTORAGE-390: Return an objectId for the uploaded segment prior to the leaders local log start offset for use as a restore point
  • KSTORAGE-1372:replace the sleep with mock time in TierObjectStoreMetricsEnabledTest
  • SBC-285: Improve performance of goal computation by reducing calls to get diff of original and optimized cluster model
  • CNK-502: Add a system test exercising rack-aware topic creation with the multi tenant interceptor
  • CNK-1457: Add JMH Benchmark for SBK Plan Computation
  • MINOR: fix MultiTenantRequestContextBenchmark ConfigException
  • CPKAFKA-6334: Add better logging to SBC integration tests
  • CNK-1383: Adds a prop to configure max throttle time for connection rate throttling
  • Fix WorkerWithTopicCreationTest
  • MINOR: Fixing auth failures system tests
  • Get branch building - Add FIXMEs for ControllerChanneManagerTests - Cleanup FIXMEs for ServerShutdownTest - Cleanup imports
  • CNK-1564: reduce matrix for FetchSessionCacheBenchmark
  • KC-1454: Disable truncation-on-fetch for cluster link fetchers
    • Handle envelope requests early in KafkaApis - Fix unused imports
  • MINOR: fix MultiTenantMetadataRequestBenchmark ConfigException
  • KSTORAGE-1335: Modify logging level of durability lapse error messages
  • merge: make KafkaMetadataLog use AbstactLog
  • merge: start on rework of MultiTenantRequestContextTest for topic IDs
  • merge: ClusterLinkFetcherThead make compatible with regular fetcher
  • merge: attempt to add topic ID support to ControlMetadataBatch
  • merge: move many fields from PartitionTest to AbstractPartitionTest as in AK, comment out some tests for rework
  • merge: ocmment out broken test OffsetsForLeaderEpochRequestTest.testOffsetsForLeaderEpochWhenPartitionUnderRecovery
  • merge: comment out broken test ProducerStateManagerTest.testReloadFromTieredSnapshot for now
  • merge: minor test fixes
  • merge: switch to assertThrows in KafkaConfigTest
  • merge: add AK topicId interop with MergedLog
  • SBC-189: Add even cluster load adminApi
  • CNK-1564: fix ZSTD magic 1 jmh error
  • Add isTerminal() method to BalancerOperationState
  • Remove Zk dependency from BrokerFailureDetector
  • CNK-1542: Adds static config for configuring default IP rate limits
  • CONFLUENT: Make cluster link test metrics methods shareable, use different broker ids
  • KC-1314: Added In-Sync Mirror Reporting to MirrorCommand
  • KC-786: Test for replication with cluster link source failure
  • KC-1155: Support for reversing connections using Kafka protocol
  • KSTORAGE-1358 Make testArchiveAndTierFetchFenceAndRecovery align with system tests
  • CNK-1548: Add option to not construct producer in KafkaBasedLog
  • CNK-1464; Make Broker Tags Dynamic
  • Revert “KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806) (#3076)”
  • RCCA-2440: Update MDS HTTP listener config handling
  • KSTORAGE-1154 Metrics for object store get/put/delete latencies
  • KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806)
  • RCCA-1881.2: JwtAuthenticator usability improvements
  • CNK-1557: CreateTopicPolicy#maxPartitionsPerTenant is not dynamically updated
  • KC-1263 DR-LITE system tests
  • KSTORAGE-1339: Ensure segment is force rolled when it is the only untiered segment
  • CPKAFKA-6476: fix the system test error caused by #3013
  • MINOR: Remove unreleased versions
  • KSTORAGE-1353: Add an aggregate metric for amount of data in compacted topics
  • Add note explaining absence of clusterId tag in ACL apis
  • Remove overload parseResponse from NetworkClient
  • Allow non-request/response types to be constructed without MessageContext
  • Force MessageContext to be provided by caller
  • KSTORAGE-1348 Flush epoch cache before reinitialization in Log.rename
  • CNK-1530: Use regular rebalance tolerances for GoalViolation dry runs.
  • KSTORAGE-1348 Flush epoch cache before reinitialization in Log.renameDir
  • CONFLUENT: Upgrade to cc-base:v8.2.0-jdk-15
  • SEC-1805: Reduce failed MDS request send log threshold from ERROR to Debug
  • Fix shutdown ordering bug in TierTasks
  • KC-1324 Adding metric to track active incremental fetch session eviction rate
  • KC-785: Test for cluster link using SSL PEM key and trust stores
  • Fix test failure in test_kibosh_service (CPKAFKA-6180)
  • SEC-1802 Check for topic creation
  • SEC-1802 Don’t block TopicManager creation on describeTopics
  • SEC-1770 NonBlockingKafkaExporter should notice that topics exist
  • Flush epoch cache before renaming log dir
  • KAFKA-12193: Re-resolve IPs after a client disconnects
  • KSTORAGE-1178: Moved missing tier state file recovery code under assignTopicId (#2702)
  • Fix unused import in DescribeEvenClusterLoadStatusRequest
  • Enable remote JWT validation key loading
  • CNK-1442: Add even cluster load status protocol
  • Add note about ReclaimableMemoryRecords in SendBuilder
  • KSTORAGE-977 Fix TierMetadataComparator to clean working dir before a run
  • KAFKA-12180: Implement the KIP-631 message generator changes
  • Coalesce sends when records type class is MemoryRecords
  • KSQL-5843: prevent log spam from MDS RestClient on client close
  • KSQL-5843: prevent log spam from MDS RestClient on client close
  • CNK-1456: Split FileBasedPlainSaslAuthenticator into PlainSaslAuthenticator and FileBasedPlainSaslAuthenticator
  • Avoid extracting the MemoryRecords buffer to coalesce sends in favor of enqueuing the DefaultRecordsSend in SendBuilder.
  • KC-859 add idle-connection-close metric to common selector class
  • KSTORAGE-1171: Validating local epoch chain with start offset
  • KSTORAGE-1338: Eagerly close FTPS listeners when performing state replacement.
  • Add test case which fails when exposed api versions are bumped
  • Address deprecation warnings in produce request/response
  • Ensure error messages have entity type set
  • Forwarded requests should be built through interceptor
  • CPKAFKA-5921: Kafka’s java command excees ps command limit
  • Add back specific log message for unclean recovery test (#2938)
  • SEC-1789 Bump Bouncy Castle dep to resolve CVE-2020-28052
  • METRICS-2629: Wrap KafkaMetric.metricValue() call inside a try/catch block to avoid failure of entire collector task
  • Cloud schema validation subject name strategy
  • Always truncate to log end offset before starting cluster link fetcher
  • Fix AbstractFetcherThreadTest
  • AUDIT-249– Added mime type “application/protobuf” supporting to BinaryDeserializer
  • Fix failing tests in ClusterLinkFetcherThreadTest
  • Revert “METRICS-2109: Return false if METRICS_INCLUDE_LIST is empty (
  • Move EntityType into protocol package
  • Remove no longer needed transformation classes
  • Rename generalOperationStatus field to generalBalancerStatus
  • Use enum for entity type
  • Fix a few test failures
  • Fix 2.12 compilation error
  • CNK-1443: Add balancer status admin client api
  • Fix some of the failing unit tests
  • Fix checkstyle issues
  • Add a few more context parse methods
  • Fix lingering compilation issues
  • Remove unneeded test case
  • Add more context parse methods
  • Use generated code for tenant transformations
  • SEC-1781 - Improve MDS builds by making managed kafka cluster optional
  • KC-1336; Add MT transformation logic to ReplicaStatus
  • CONFLUENT: Add operations used in tests to verifyPermissions() method
  • CIAM-571 - Fix high thread usage in MDS test suite
  • Reverted wrong info msg in Shutdownable thread class(#2995)
  • Fix mistaken change in ZkData.encode
  • Reverted wrong info msg in Shutdownable thread class(#2995)
  • Remove duplicate sourceSets in core module
  • Ensure AlterIsr and UpdateFeatures are not exposed to users
  • CPKAFKA-6149: fix flaky cluster link system test
  • CPKAFKA-6413: test_offset_migration_single_group_offset_not_exists_on_dest
  • Cleans up merge of AK topic IDs. Topic IDs used before this merge are now known as legacy topic IDs and will be used when IBP is less than 2.8. They will still only be used for tiered topics. When IBP is 2.8 or greater, AK topic IDs (modern topic IDs) will be used for all topics. ZK contains optional fields for both legacy and modern topic IDs. At most one of these fields can be defined at one time. Upon upgrading to 2.8 or above, any legacy topic ID will be converted to a modern one and the legacy topic ID will be removed from ZK.
  • KSTORAGE-1317 - Nit Durability Audit producer needs to shutdown after 1 sec wait during close call.
  • CPKAFKA-6403: fix cluster link test_offset_migration
  • CNK-1460: Remove zkAdminClient references from ReplicationThrottleHelper
  • Try to get core compiling again
  • KC-1317: Fix incrementalAlterConfig for list property with default null
  • CNK-482: Handle optional max_partitions field in logicalclusters secret
  • KC-1320: Fix tenant prefix for cluster link metrics
  • CNK-1316; Update TenantPartitionAssignor to support cells
  • KC-923: Improve start up time for RBAC providers by reducing status records and delaying writer startup
  • MINOR: MergedLog. deleteOldSegments: Flip condition for readability
  • KSTORAGE-1311: Implement customizable user agent prefix for S3 client
  • CNK-1441: Add protocol for balancer status
  • CNK-1435: Fix RandomClusterTest to work with SBK’s goals
  • CNK-1344: Close all resources if KafkaCruiseControl cannot be created successfully
  • CNK-1405: Remove the PreferredLeaderElectionGoal
  • CPKAFKA-6375: Decode python3 bytes object in tier fence restore test
  • CPKAFKA-6374: Handle None in tier roundtrip system test
  • CPKAFKA-6370: Use python3 range (versus xrange) to have BrokerStorageFailureTest work with python3
  • CPKAFKA-6371: Use python3 sys.maxsize in tiered storage system tests
  • KC-1275: Update KafkaReader to handle entries from multiple partitions
  • CNK-1277; Broker Tags (Config, UpdateMetadataRequest, MetadataCache)
  • CNK-1162: Handle multiple replica movements by accounting for their l
  • CPKAFKA-6373: Fix deprecated has_key python2 method
  • CPKAFKA-6376: Use range over xrange to fix TierMetadataCorruptionTest to work with python3
  • KSTORAGE-1295: Durability audits should setup topics with specific topic configs
  • KSTORAGE-1290: Bump google storage client dependency to 1.113.4
  • KC-1214: Propagate link-level pause state to partitions.
  • Add back in specific log message for unclean recovery test
  • CPKAFKA-6372: Fix bad assertion in cluster link test
  • REFACTOR: Infrastructure changes from CNK-1162
  • CNK-215: Delay closing connections on request parsing failure
  • KSTORAGE-1220: Prevent log truncation when logEndOffset is less than firstUntieredOffset
  • KSTORAGE-684: Simplify locking inside FileTierPartitionState by killing stateResourceReadWriteLock
  • CPKAFKA-3011: fix timing issue with onConfigChange and ConfluentLeaderAndIsrRequest
  • make the TelemetryPushModeIntegrationTest more easily extendable
  • Fix partition tests for new IsrChangeListener arg
  • Backport of CPKAFKA-6268
  • Add getter for Group and Type fields in YammerMetric interface
  • KAFKA-10792 preemptive patch
  • CPKAFKA-6268 fix for failing ClusterLinking topic deletion test
  • CNK-836: Basic Cloud Configured System test
  • CNK-1431: Add confluent.ansible.managed property
  • KC-1228 cherry pick for 6.1.x
  • KC-1206: Add partition availability metric to controller
  • fix indentation issue
  • fix indentation issue
  • CC-12309: Connect should not add error to connector validation values for properties not in connector’s ConfigDef
  • KC-922, KC-1238: Handle writer stop exceptions to ensure new writer is re-elected
  • KSTORAGE-1255: Reduce default interval to detect deleted partitions
  • KC-1228 AlterIsr needs additional fields in CP
  • KC-955: Updates topic mirror stopping to use task.
  • CPKAFKA-6018: Fix cluster link test timing issue with metrics and replica status
  • KSTORAGE-1273: Avoid duplicate lapse reports in a single durability run and update message formatter
  • KSTORAGE-1274: Close FileTierPartitionState segments iterator just once inside AuditJob
  • CPKAFKA-6336: update properties list for config test.
  • CPKAFKA-6311: Don’t attempt to read metrics from dead brokers.
  • CC-12309: Connect should not add error to connector validation values for properties not in connector’s ConfigDef
  • KSTORAGE-1260: tiered segments do not correctly block retention for local segments (#2831)
  • Apply fix for KAFKA-10574 to master
  • CNK-1325: Add a metric for initiated executions.
  • CNK-819: Don’t self-heal if a rebalance doesn’t improve detection goals.
  • Port fix for KSTORAGE-1230 to 6.1.x
  • CPKAFKA-6126: Update commons codec (#2785)
  • KSTORAGE-1172 Prevent durability audit to do start offset validation with overlapping previous segment
  • CPKAFKA-6126: Update commons codec
  • KC-1154: Adding metric for unreachability
  • KC-955: Implements task to gracefully stop a topic’s mirror.
  • CNK-1210: Introduce kafka-add-brokers CLI
  • KSTORAGE-1260: tiered segments do not correctly block retention for local segments
  • CNK-266; Tenant Quotas need better logging
  • CPKAFKA-6233: Update lastUpdateTimeMs before updating the current broker addition state
  • KSTORAGE-835: producer state is loaded twice in MergedLog/Log init
  • CNK-1396: Register BrokerAddCount metric in CDBE
  • kstorage-1236: dump durability db tool
  • KSTORAGE-1131: print TierObjectStore paths in DumpTierPartitionState
  • KSTORAGE-1047: Calculate and Log Expected Shrinkage Ratio
  • Use commons-lang v3 in ConfluentDataBalanceEngineTest.java
  • CNK-1389: Report request queue size percentile metrics in JMX
  • KC-942: Adds tenant tags to cluster link metrics + refactoring.
  • make OwnedPartitions ignorable in ConsumerProtocolSubscription.json
  • CNK-1220; Limit the maximum number of (new) partitions per create topics/partitions request
  • KC-924: Add event processing metrics for ZK ACLs and MDS records
  • CNK-617: Temporary pull out CloudNetworkFault + deps
  • CNK-165: Add number of tenants metric
  • KC-1214: Add support for pausing a mirror topic’s replication.
  • CNK-1349: Switch broker removal state from PLAN_EXECUTION_INITIATED to PLAN_COMPUTATION_INITIATED on failover
  • CNK-1362; Token Bucket metric spikes to maximum when a tenant is removed
  • KSTORAGE-1241: Make TierS3EndpointOverrideProp and TierS3ForcePathStyleAccessProp non-internal
  • KSTORAGE-1241: Allow user to configure S3 path-access-style
  • KC-1226: Enable confluent.cluster.link.io.max.bytes.per.second to be configured at cluster-level
  • KSTORAGE-1178: Moved missing tier state file recovery code under assignTopicId
  • KSTORAGE-1237: tools to debug durability events
  • KC-1225: Fix NPE on broker restart with paused links, fix blocked state
  • METRICS-2109: Return false if METRICS_INCLUDE_LIST is empty
  • CNK-1228: Update SBK add_broker system tests to utilize kafka-add-brokers for checking operation completion
  • KSTORAGE-1230: Avoid creating new futures to wait on TierTask futures.
  • CNK-1371: Shutdown quota managers in BrokerConfigHandlerTest
  • CNK-1359: LogicalClusterMetadata excessive logging of inactive clusters
  • KC-1027 backport into 6.1.x
  • KC-1027 MRC observer promotion system tests
  • KC-955: Improve MirrorCommand to contain per-partition mirror information.
  • [Minor] fetch previous offset less then startoffset to compare against start offset validation.
  • CNK-1332: Handle TopicExistsException when creating SBK topics
  • KC-1203: Stop mirroring if destination partition is under-min-isr, metrics changes
  • KC-1195: Ensure connections are closed if principal cannot be built
  • KSTORAGE-1217 Update durability metrics every 30mins
  • backport METRICS-2459: update default include list for hosted monitoring
  • Update quickstart to point to 2.7.0 relase version
  • update version in quickstart to current
  • METRICS-2459: update default include list for hosted monitoring
  • DG-965: Allow enabling of key/value schema validation topic configs in CCloud
  • backport METRICS-2396: Add extra checks to ensure that we do not leak EventExporter objects
  • backport METRICS-2110: document Telemetry Reporter packaging (#2710)
  • backport METRICS-1805: remove duplicate resource type label (#2744, #2755)
  • Add unit tests to disable tier storage at broker level
  • KAFKA-10669: Make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field default to 1
  • KAFKA-10669: Make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field default to 1
  • KSTORAGE-1180 Update tier enable flag at FileTierPartitionState unconditionally
  • KC-1183: Fix exception in writer metric for non-MDS brokers
  • METRICS-1805: followup commit to address comments
  • KC-1025 automatic observer promotion/demotion in MRC (#2649) cherry-pick
  • METRICS-1805: remove duplicate resource type label
  • CNK-1304: Update remove broker validation to use strict collections
  • CNK-1331: Fix typo in SBK Persistence Store initialization
  • Always compare Instant values inside TelemetryPushModeIntegrationTest
  • Fix downstream partition unit tests
  • METRICS-2396: Add extra checks to ensure that we do not leak EventExporter objects
  • CNK-1209: Describe AddBroker functionality and integration tests
  • make OwnedPartitions ignorable in ConsumerProtocolSubscription.json
  • KC-1025 automatic observer promotion/demotion in MRC
  • METRICS-2110: document Telemetry Reporter packaging
  • Reconcile downstream PartitionTest tests with KAFKA-8836
  • KAFKA-10602: Make RetryWithToleranceOperator thread safe
  • MINOR: Change log level on initialization of file based authenticator
  • KC-1132: Add cluster link request metrics for source cluster
  • CNKAF-1132: Record & unrecord total-throttle-time metric
  • Support for path prefixed SNI hostnames (for V4 networking support)
  • KAFKA-10592: Fix vagrant for a system tests with python3
  • KAFKA-10592: Fix vagrant for a system tests with python3
  • METRICS-2132: Support push-based mode of Telemetry Reporter
  • CPKAFKA-6030: Fix broker removal system test
  • CNK-1290: Update the sub task status field in BrokerAdditionDescription and BrokerRemovalDescription and CNK-1288 - add description() method to DescribeBrokerRemovalsResult for feature parity
  • KSTORAGE-1173: Segment set for avoiding duplicate alerts in a session
  • CNK-1233: Adds support for logical cluster type link-healthcheck.
  • KC-1166: Fix the name of the _second_ large-throughput many-partition topic
  • Revert “CPKAFKA-4479: Make operationally important Kafka configs dynamic (fetch.max.bytes)”
  • backport KAFKA-10571
  • KAFKA-10570; Rename JMXReporter configs for KIP-629
  • KAFKA-10570; Rename JMXReporter configs for KIP-629
  • KAFKA-10571; Replace blackout with backoff for KIP-629
  • Updating trunk versions after cutting branch for 2.7

Security

  • Pins Ksql to a jetty-jaas 9.4.28.v20200408 (#365)
  • KREST-1187: Propagate client.sasl.jaas.config to Schema Registry.
  • KQE-229: Update vertx-core to 3.9.7
  • DGS-1463: Modify config types for sensitive SR validator plugin configs
  • SEC-2023: TestUtils method was renamed upstream (#352)
  • DGS-1381: Deserialize GLOBAL_SUBJECTS_READ to GLOBAL_READ for backward compatibility
  • Bugfix ksql 5447 correct status code on unauthorized
  • RCCA-1881: Use new JwtAuthenticator instead of JwtConsumer
  • Fix InvalidSslUserProducerTest
  • Reenable failing test InvalidSslUserProducerTest
  • DG-161: Add schema validator metrics
  • Fix auth for SchemasResource

Secret Registry

  • SEC-2105: Update guava to version 30.1.1-jre

MQTT Proxy

  • CC-14445: Remove compile dependency on Kafka broker
  • SEC-2022: Upgrade Netty to 4.1.62.Final.

Metadata Service

  • SEC-2191: Properly handle unicode characters when rewriting urls
  • CIAM-426 Test Rolebindings are escaped
  • CIAM-659 - Add implementations for new interface methods added upstream
  • PG-80: add /trackingId endpoint to retrieve tracking identifiers
  • CIAM-457: Add healthcheck endpoint to metadata service API
  • Directly pass through JwtPrincipal rather than unpacking it
  • Replace TokenUtils with PublicKeyJwks
  • SEC-1800 MDS : Ability to run individual test from cli
  • CIAM-475 - DB : Deploy Encryption at Rest
  • SEC-1790 - Make crappy ldap names testing clearer
  • SEC-1782 - Improve MDS build by updating maven failsafe
  • SEC-1785 Use Dynamic Proxy for DelegateConfluentAdmin
  • SEC-1781 - Improve MDS builds by not running secondary cluster
  • CIAM-571 - Fix high thread usage in MDS test suite
  • CIAM-281: Migrate ControlPlaneDbProviderTest test to use RbacOrmDbService
  • CIAM-290 - Doc updates from doing one-off db shapshot restore in devel
  • CIAM-515 Upgrade TF Version
  • SEC-1707: Add static listener ports for MdsInABox
  • CIAM-505 Use snapshots When Recreating Postgres RBAC
  • CIAM-259 - Cloud Rbac needs a real DNS name
  • CIAM-246: Add Perf tests for DB auth store
  • CIAM-499 - Doc Request Logging changes
  • SEC-1577: Test stability for dynamicUserGroupMembershipWorks
  • Add an additional CIDR to the allowed list
  • Pin the terraform version to 0.11.13
  • CIAM-297: Use Jersey LoggingFeature to enable request/response logging
  • Check in schema.sql and fix urlencode for stage password
  • CIAM-305 db migrate scripts to seed the migration cli as CCloudRoleBi

Replicator

  • REPL-1410: Changed logging for WakeupException to DEBUG
  • REPL-1386: Adding clear logging statements for src and dest operations
  • REPL-1409: Added WARN logging if destination SR deemed not ready
  • REPL-1396: Fix CVE for HttpClient
  • SEC:1820: Upgrade codec version
  • REPL-1396: Fix CVE for aws-java-sdk-ecr

Confluent Platform Documentation

  • DOCS-9480: Update ksqlDB limitations for pull queries
  • DOCS-9461 Clean up the API docs landing page
  • Fix cp-javadoc link macro
  • DOCS-9459 Show correct syntax for kafka-mirrors –promote, –failover, add note, add pause/resume
  • DOCS-9459 CP Cluster Linking add docs for promote, failover, deprecate –mirror-action stop
  • DOCS-9441 confluent-security package is needed on 6.0.x versions to enable license manager
  • KCI-409: Table for ksqlDB RBAC role mapping
  • DOCS-9363 - Cleanup unused link macros and text-string replace
  • DOCS-7396 - Fix nav for Clients
  • DOCS-9279: Add link to Kafka broker configurations
  • Change link back to docs version of cloud
  • DOCS-8943: Add TransactionalId role for ksqlDB EOS
  • DOCS-9197: Add links to new ksqlDB courses
  • DOCS-9087: Add setCompressionType to example code for RocksDB
  • Update CP quickstart pages for Learn Kafka project
  • DOCS-9203 - Add links to Confluent Developer
  • DOCS-9219 MRC replica pages description of leader-is-observer (DOCS)
  • DOCS-8451 No Platinum support for CFK
  • DOCS-8846 - Fix typo
  • DOCS-8966: Explain that the ConsumerLag metric is not client lag
  • DOCS-8681: Mention transaction timeout change in Streams upgrade topic
  • Fix readme syntax
  • DOCS-8538 Added CONNECT_CONFLUENT_LICENSE property for Replicator
  • DOCS-6852 K8S CPU units
  • DOCS-9088: Remove co-partitioning note in Streams foreign-key join table
  • DOCS-9100: Replace uses of SerDe with Serde (and KQE-289)
  • DOCS-3175 - mentioning both confluent packages
  • ANSI-881 Added the known issue about Replicator RBAC
  • DOCS-9016: Fix typo in SerDes topic
  • fixing issue with headers causing duplication
  • DOCS-7895: Fix typo in Streams DSL API topic
  • DOCS-8826 Azure is not supported for tiered storage
  • DOCS-8459 -adding text/links between triggers for consumer groups and monitoring
  • PG-86 Adding links to blog posts
  • numerous updates to headers, adding redirects, and more
  • DOCS-8851, DOCS-8850, DOCS-8852, DOCS-8820, DOCS-8822 fix build error on Cluster Linking, use updated link target
  • DOCS-8552, DGS-1725 Add docs for schema references UI on C3 for 6.2
  • DOCS-8739: Add descriptions for Health+ free and paid pricing tiers
  • DOCS-7462 - Update what is CP for new narrative
  • Simplify and update the platform README
  • DOCS-8762 Cluster Linking mirror topics updates (CP)
  • DOCS-8286 Cluster Linking 6.2: Add Mirror Topics concepts (CP)
  • DOCS-8698 - fix typo in cluster-linking/docker-quickstart.rst
  • Fixed the link to CFK telemetry config
  • adding more info to disconnects per second
  • DOCS-8386 - Changing wording on buffer
  • DOCS-8704 - removing duplicated list
  • DOCS-8650 More detail about Ansible and CFK with Cluster Linking
  • DOCS-7974 Fixed broken anchors to ACL
  • DOCS-8092: Link to the new CP 6.2 overview video
  • DOCS-8654 - Fix link to log compaction
  • DOCS-8612 - Add video for 6.2.0 release notes
  • KCI-446: Update docs for eos-beta to eos-v2 renaming
  • DOCS-8144: Port sliding windows content from old docs repo
  • Updates to the Ansible compat table; removed the dup item in the RN
  • DOCS-8637: Update caption text for Health+ Monitoring dashboard screenshot
  • DOCS-8282: Fix HTML encoding in Streams config example
  • Refactor Proactive Support content for Health+ (DOCS-7448)
  • DOCS-8570 - Add C3 known issue
  • DOCS-8583 - Add link to blog and podcast for 6.2.00
  • DOCS-8594 Updated the release date of CP 6.2
  • Generate Changelog for 6.2.0
  • DOCS-8539 - Add Avro deprecation to release notes
  • DOCS-7842: Update quick start signup workflows and screenshot
  • DOCS-7897 - 6.2.0 release notes
  • DOCS-7682 DOCS-7681 Version interoperability for CP 6.2.0
  • PG-86, PG-139, PG-56 Document new c3 features for 6.2.0
  • DOCS-7683: Update Upgrade Guide for CP 6.2
  • Bump AK version for CP 6.2
  • DOCS-8351 Upgrading from 5.3.1 to 6.1.x is supported
  • DOCS-8388: Update ksqlDB metrics list for CP 6.2
  • DOCS-8421: Update AK monitoring topic with autogenerated metrics for CP 6.2
  • DOCS-7899: Update autogenerated AK config parameters for CP 6.2.x
  • DOCS-8088: Update queries in ksqlDB section of COS Quick Start for 6.2 (also DOCS-8089)
  • DOCS-8248 Removed references to Operator 1.x in CP 6.2 docs
  • PG-140 6.2 change - Remove deprecated config properties
  • DOCS-7446 SBC 6.2, Update card on brokers overview in C3
  • DOCS-8084: Update Persistent Queries section in ksqlDB C3 UI reference
  • DOCS-8086: Update queries in ksqlDB section of CE Quick Start (local)
  • DOCS-8083: Refresh screenshot for Integrate ksqlDB with C3 topic
  • DOCS-7449 SBC 6.2, Add Balancer Status, Any Uneven Load docs
  • DOCS-8091: Refresh workflow in the Write Streaming Queries with CLI tutorial
  • DOCS-8090: Refresh screenshots in the Write Streaming Queries in C3 tutorial for 6.2
  • DOCS-8139: Update the CP ksqlDB CLI help text for 6.2
  • DOCS-8138: Update the CP ksqlDB CLI splash screen for 6.2
  • DOCS-7945: Update syntax for ksqlDB DESCRIBE EXTENDED statement
  • DOCS-7935: Update Kafka local services start command in Secrets tutorial
  • DOCS-7900: Update the ksqlDB supported versions table for 0.17.0/6.2.x
  • DOCS-7444: Update the CP Supported Versions table for ksqlDB 0.17.0
  • DOCS-7442: Bump ksqlDB version to 0.17.0
  • DOCS-7840: Update CREATE TABLE statement with key format
  • KSQL-6023: Add note to ksqlDB upgrade guide about deprecated configs
  • DOCS-7324: Add new split method to 6.2.0 Streams Upgrade Guide
  • KSE-130: Port Streams DSL API docs for new KIP-418 split method from AK to CP
  • DOCS-7268, DOCS-7259, DOCS-7260 Add SR subject-level DELETE API for /config, /mode
  • MINOR: Better master branch detection when running in CI systems