Confluent Platform Component Changelogs¶
This topic provides changelogs for the individual Confluent Platform components.
Version 6.1.15¶
Released December 2023
Community Features¶
Common¶
- e99d8b1a - exclude logback-core and logback-classic in line with the changes in kafka-rest
- eb200670 - update zookeeper component to match version in ce-kafka
- 0c034e86 - re-add dependencymanagement sections for old bouncycastle to satisfy downstream validation
- 33734c5f - add additional version definition to ease the update of the downstream repos
- 8e71caf2 - update bouncycastle
Kafka¶
REST Proxy¶
- PR-1221 - KREST-12630 Exclude logback from zookeeper dependency in tests
- PR-1219 - KREST-12630 Add missing dependency due to zookeeper upgrade
- PR-1212 - KREST-12277 semaphore migration to release branches
- PR-440 - KREST-11812 upgrade jaxb-api dependency
- ee8dd479 - KREST-12364: update bcpix
- PR-427 - KRES-12288 migration semaphore release branches
Schema Registry¶
- 98a1a4de - update swagger core
- PR-2888 - Apply service bot changes to version branches
- bafbad8c - DGS-9343 Allow removing enum in JSON Schema to be backward compatible
- PR-2874 - DGS-9048: Semaphore Migration of Version Branches
- 7f7a7ab1 - add metrics-core to fix the build
- PR-2816 - DGS-8960 Fix handling of diamond structure in Protobuf converter
Version 6.1.13¶
Released September 2023
Version 6.1.12¶
Released June 2023
Community Features¶
Common¶
Kafka¶
REST Proxy¶
Schema Registry¶
- e0626f64 - Update the test to make it clear to customer
- dc6f2096 - update the doc
- 33659feb - Update schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java
- 3ffd8fc7 - Update schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java
- 1ca2a39d - Update the codes with checking string for http connect timeout
- ffba37fe - * changed the config names
- 4c383ee4 - Apply suggestions from code review
- 7155f0fb - * checkstyle error
- 7e2dec9c - * removing unused import
- 9ab1b0ec - * adding defaults
- 01c6d7a7 - * Made SR client http connect and read timeout configurable.
- 89aa28be - * Made SR client http connect and read timeout configurable.
- PR-2642 - DGS-7367 Propagate error messages even if not JSON
- ed4ec914 - DGS-6698: Remove SnakeYaml from the dependencyManagement
- PR-2589 - DGS-6613 Ignore invalid Avro defaults in Avro Converter
- PR-2588 - DGS-6603 Fix default conversions in JsonSchemaConverter
- PR-2693 - DGS-7746 Fix perf regression due to DGS-6331
- PR-2684 - DGS-7675 Fix NPE when print.schema.ids=true
- 1bf961b7 - Update json-schema
Commercial Features¶
Confluent Server¶
- Add log redactor 6.1.x new
- CC-18990 : Backport log redactor connect for 6.1.x
- KSECURITY-1568: Upgrading jose4j.
- KSECURITY-1556: updated jetty version
- KAFKA-14887: FinalizedFeatureChangeListener should not shut down when
- KSECURITY-1466: Updated json-smart and nimbus-jose-jwt (#9482)
- KSECURITY-1459: updated golang.org/x/net mod.
- KSECURITY-1455: updated golang.org/x/text mod.
- KSECURITY-1453: Updated version of golang.org/x/sys
- update grgit to resolve recent build failures in packaging branch 6.1.x
- KSECURITY-1640: Update hibernate-validator
- KSECURITY-1627: Updated snappy-java
- KSECURITY-1618: update guava
Security¶
- CIAM-2998 - Log redactor backport for confluent-security-plugins
Secret Registry¶
- unpin netty, as rest-utils brings updated version
Control Center¶
- [MMA-13078] revisit session caching in basic auth
- [MMA-13089] Handle large message JSON on topic inspection
- [MMA-13042] Fixing log statement for websocket time out
- [MMA-13216] update jose4j
MQTT Proxy¶
- CC-19108 | Rename isChannelClosed to isAuthenticated.
- CC-19108 | Use an atomic reference to avoid publishing message when the channel is closed or marked for closure.
Metadata Service¶
- Update pom.xml
- CIAM-2986 - Older MDS builds are not running all the tests they should be
- update comment, clean up whitespaces
- another attempt with mina-core
- pin mina core in cli, scope mina-core as test in main pom
- pin mina-core in testingLdap only
- SEC-xyz: Update mina-core
Replicator¶
- disable builds of connect-replicator version 5.5.x
- remove aws-maven plugin to unblock the downstream validation for common: RCCA-10412, APPSEC-2551
Version 6.1.11¶
Released March 2023
Community Features¶
Common¶
- PR-512 - Add snakeyaml to dependency management
- 7bcaab9e - Fix typo in with method calls
- 3868e27b - Fix typo
- 783c5721 - Return the original “version_range” if it isn’t an actual Maven version range
- PR-502 - Fix: override maven version plugin’s default versionrange
- 9350d231 - Update resolver-maven-plugin to 0.6.0
- PR-497 - Upgrade Netty to 4.1.86.Final
- PR-496 - Bump to 7.4.4
Kafka¶
REST Proxy¶
Schema Registry¶
- PR-2575 - DGS-6331 Handle javaType for oneOfs during JSON deserialization
- PR-2568 - DGS-6306 Allow multiple oneofs in Protobuf converter
- PR-2540 - DGS-6075 Add explicit definition of snakeyaml.
- PR-2519 - DGS-6014 cherry pick Maven plugin enhancement to 5.5.x
- PR-2514 - DGS-5950: Allow optional map key in Avro
Commercial Features¶
Confluent Server¶
- KMETA-478 Enable zk audit logs by default
- KC-2328: Update yaml.v2 to 2.2.4
- KSECURITY-953: Update azure-identity to 1.7.3
- KSECURITY-981: Add FeatureZNode zk node path to secure root paths
- KC-2333: Update io.netty libraries to 4.1.86.Final
- [MMA-5596]: Add ‘ObserverReplicasCount’ to monitored metrics list
- CPKAFKA-3552: Move license validator out of authorizer and make broker wide
- CONFLUENT: Add unit tests to check observers tagged field
- Added support for skipping upload task for older gradle versions
- MINOR: avoid unnecessary generation of PartitionState objects in ReplicaFetcher
- KAFKA-9038: [WIP] Allow creating partitions for topics partitions not in reassignment
- CONFLUENT: Use single audit log provider for MDS and its hosting broker
- CPKAFKA-3721: Update topic describe to use metadata for observers
- CONFLUENT: Disable scalac optimizer
Security¶
- SEC-5350: Update netty-codec-http to 4.1.86.Final
Secret Registry¶
- Remove unused variables
- Update RestService.java
Control Center¶
- KC-2261 timeout websocket connection via jwt token
- Update CODEOWNWERS to match latest definition from master
- Refactoring- made code more functional and improved readability.
- Updated NotFoundException message in scenario where topicName provided returns 0 no consumergroupdata.
- Updated code based on PR feedback.
- Updated consumerGroupId API to also support topicName queryParam.
- Improving test cases with scenarios of single consumergroup covering multiple topics. Updated consumerGroupId to not support topicName queryParam, it is not required.
- Adding unit test for CachedConsumerOffsetsResource
- Adding support for topicName in query param to return only consumer groups for that topic
- Reading password using getPassword instead of getString, this fixes the casting exception
- Fixing failing test case, due to merge issue
- Renaming Health_check to health check
- Adding config based functionality to force http1 for armeria health checks
- MINOR: Update repo to use mvn-wrapper to speed up builds
MQTT Proxy¶
- Remove the pin of netty version to use the (current) version defined in common
Metadata Service¶
- Clean testng
- CIAM-2615 - Bump TestNG to testNG 7.5
- Increase buffer size for Proxy Servlet
Replicator¶
- Testing a build fix
Version 6.1.9¶
Released December 2022
Community Features¶
Common¶
Kafka¶
Schema Registry¶
- PR-2440 - Update the Snakeyaml dependency
- 8e5ce168 - Unpin Protobuf version.
- PR-2414 - DGS-5253 Handle cycles when resolving references
- PR-2405 - DGS-4754 Handle empty record default at field level
- PR-2399 - DGS-5084 Ignore compat check in IMPORT mode
- PR-2395 - DGS-4971 Handle map types with enhanced.protobuf.schema.support
Commercial Features¶
Confluent Server¶
- CONFLUENT: Skip publishing for projects with no scala suffix when the scala version is not the default
- DP-9030 - Switch to codeartifact repo
- KENGINE-283 Topic ID Mismatch Issue - Exception added to disallow Alt
- Increase vagrant aws timeout for system test
- BouncyCastle Issue fix for connect service [6.1.x]
- KSECURITY-792: Upgrade from Scala 2.13.8 to 2.13.10
- KSECURITY-788: Upgrade jackson-databind version to 2.13.4.2
- KSECURITY-832: Upgrade bc fips drivers
- CIAM-2424 - Name RestClient threads for stackdump identification (#7887)
- KSECURITY-794: Upgrade to protobuf version 3.19.6
- CONFLUENT: Remove “principal.builder.class” config from Broker Client configs
- Add stripped down transaction tool which allows aborting hanging transactions
- CIAM-2304: Add SecurityMetadata:Describe to SecurityAdmin
- RCCA-8564: log a warning if LDAP login fails becuase of network issues
Security¶
- Exclude bcpkix-jdk15on as compile dep and add as test dependency
- Pin bc-nonfips dep as a test dependency
- Pin bouncycastle fips dependency in confluent-security-plugins
- Removed bcfips from common
- excluded bcfips jars
- AUTHN-1087: Upgrade vertx to 3.9.14
- AUTHN-1087: Upgrade vertx to 3.9.14
- AUTHN-1325: Update bc-fips version to 1.0.2.3
- fixed checkstyle error
- Requesting permission only for Subject Read operation instead of any of the subject operation in case of GET /schemas
Secret Registry¶
- Exclude Non-FIPS dep and use bc-fips dependency
Control Center¶
- MMA-9023: memory bounding for streams applications [5.4.x, master]
- Revert “MMA-12206 upgrade io.netty:netty-codec-http”
MQTT Proxy¶
- Fixing the tests and config validator, to incorporate new changes in validation of MqttDecoder brought in by netty upgrade
- Upgrading netty
Metadata Service¶
- Remove spaces
- Add back newline
- Implement writeTransactionMarkers in DelegateConfluentAdmin
- CIAM-2304: Add unit tests for SecurityAdmin
Replicator¶
- REPL-2055-: Log WakeupException at debug level
- RCCA-7678: Reverse proxy header check added
Version 6.1.8¶
Released August 2022
Community Features¶
Kafka¶
REST Proxy¶
- PR-1049 - APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
- PR-1047 - : Migrate from confluent-log4j to slf4j-reload4j.
- PR-350 - Cherry-pick https://github.com/confluentinc/rest-utils/pull/349 to 5.4.x
- 3d1250af - APIF-2739: Upgrade Jetty to 9.4.48.v20220622.
- PR-320 - APIF-2714: Switch from confluent-log4j to reload4j (5.4.x)
Schema Registry¶
- PR-2351 - DGS-4358 Fix NPE in Protobuf converter for null map value
- 0577dffb - Add comments
- 042d3e3c - DGS-4303: Support different stat types in SchemaRegistryMetric
- PR-2342 - APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
- PR-2339 - APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Commercial Features¶
Confluent Server¶
- [METRICS-3888][METRICS-4448] update to telemetry-client without shaded Netty for 6.1.x
- KAFKA-14107: Upgrade Jetty version.
- migrating log4j12 to reload4j, slf4j-log4j12 to slf4j-reload4j.jar (6.1.x)
- [METRICS-4448][METRICS-3338] upgrade telemetry to pick up netty not shaded change for 6.1.x
- Fix exception handling in RBAC writer coordinator (#7
- Avoid sending partial updates during LDAP group manager startup
- Upgrading aws-java-sdk-s3 to v1.12.268 (6.1.x) #6975
- Upgrading google-cloud-storage to v2.6.1 (6.1.x) #6991
- Upgrading netty version to 4.1.79.Final (backport 6.1.x)
- KAFKA-13474: Allow reconfiguration of SSL certs for broker to control
Security¶
- add auth token provider to ksql security extensions
- revert pom
- add auth token provider to ksql security extensions
- AUTHN-1085: Fix NPE in debug log
- APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Secret Registry¶
- Migrate from log4j to reload4j for connect-plugin
- CC-17641,CC-17644 Bump jetty version
- APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Control Center¶
- MMA-12201 remove slf4j from transitive deps
- MMA-12206 upgrade io.netty:netty-codec-http
- MMA-12201 Exclude slf4j in transitive dependency
- MMA-12201: Use reload4j version properties from common
- MMA-8781: Respect SR Config and handle nulls in CachingSchemaRegistryClientSupplier
- MMA-12201 Replace log4j with reload-4j
- MMA-12127: Change type from String to Password for CONTROL_CENTER_EMAIL_PASSWORD
MQTT Proxy¶
- CC-17620: remove dependency on slf4j-log4j12 and confluent-log4j
Metadata Service¶
- CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS
- fixes 6.0.x build from 5.5.x merge
- CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS
Version 6.1.7¶
Released July 2022
Community Features¶
Schema Registry¶
- af27c4c0 - Fix merge issue
- 31183d32 - Fix merge issue
- ce22bb34 - Fix merge issue
- a930a7b6 - Fix merge issue
- 6c78adf4 - Fix merge issue
- d46bebb5 - Fix merge issue
- PR-2331 - DGS-4172 Bound size of Avro datumReader/Writer caches
- PR-2328 - DGS-4162 Handle refs at different levels in Protobuf converter
- PR-2327 - DGS-1648 Allow Protobuf msg fullname to be passed for console producer
- PR-2318 - DGS-3974 Fix StackOverflow with recursive JSON schema in converter
Version 6.1.6¶
Released June 2022
Community Features¶
Common¶
Kafka¶
- PR-705 - KSECURITY-220: Update jackson packages 2.13.2
- PR-11946 - KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (#11946)
- PR-11872 - KAFKA-12879: Remove extra sleep (#11872)
- PR-11871 - KAFKA-12879: Addendum to reduce flakiness of tests (#11871)
- PR-11797 - KAFKA-12879: Revert changes from KAFKA-12339 and instead add retry capability to KafkaBasedLog (#11797)
REST Proxy¶
Schema Registry¶
- PR-2290 - DGS-3674 Upgrade to Protobuf 3.19.4
- PR-2285 - Merge Conflict Resolution (from 5.4.x to 5.5.x)
- PR-2283 - DGS-3659 Handle null default in JSON Schema converter
- a5c4a3b7 - Override dependency of kotlin-scripting-compiler-embeddable to Kotlin version
- bd9cb11e bumping kotlin components version to 1.6 for kafka-connect-protobuf-converter
- 96471721 - Upgrade kotlin to 1.6.0
- PR-2259 - Fix upgrade to Jackson 2.13
- PR-2257 - DGS-3548 Optimize cache for JSON Schema converter
- PR-2253 - DGS-3535: Remove enforcement of building with JDK8
- PR-2241 - DGS-3484 Allow NaN when parsing JSON
- PR-2227 - DGS-3417 Fix default value conversions in JsonSchemaConverter
- 7b222add Fix swagger
- 165e0f5f - Fix merge issue
- PR-2215 - DGS-3250 Return 422 instead of 500 if schema lookup is invalid
- PR-2214 - DGS-3248 Return 422 instead of 500 if subject is READONLY during DELETE
- PR-2201 - DGS-3066 Perf optimizations for AvroSchema
- PR-2200 - DGS-3133: Compare canonical string instead of jsonNode when comparing JsonSchema
- This release version includes a fix to prevent stack trace exposure from getting surfaced in HTTP responses for malformed requests.
Commercial Features¶
Confluent Server¶
- Upgrade Gson
- use correct cp urls in dockerfile - 6.1.x
- KSECURITY-221: Update jackson packages to 2.13.2 version
- CONFLUENT: Sync from confluentinc/kafka (30 August 2019)
- KSECURITY-198: Update protobuf-java packages to 3.19.4
- Backport of KMETA-8 to 6.1.x
- KAFKA-6502: Update consumed offsets on corrupted records. (#11683)
Control Center¶
- Bump maven-shade-plugin to 3.3.0
- Fix build break
- LOG_DIR to match rest of confluent components
Metadata Service¶
- CIAM-2019 - Fix 6.x MDS builds from Jackson update
- MINOR SEC-3087 SEC-3028 Avoid NPE in integration test
- This release version includes a fix to prevent stack trace exposure from getting surfaced in HTTP responses for malformed requests.
Replicator¶
Version 6.1.5¶
Released March 2022
Community Features¶
Common¶
Kafka¶
ksqlDB¶
REST Proxy¶
- PR-963 - KREST-4113 Don’t delete all ACLs if an empty DELETE request is received.
- PR-969 - KREST-3836 update maven to pull from https
- 5514b21c - MMA-11468: Add option to enable nosniff protection
- PR-305 - Remove Redundant Netty Definition
- PR-302 - APIF-2027: Upgrade jetty-server to 9.4.44.v20210927.
- PR-301 - APIF-2005: Change the Maven repository protocol: http → https
Schema Registry¶
- PR-2187 - DGS-3066 Handle unions in AvroSchema.metaEquals
- PR-2186 - DGS-3061 Only cache top-level schemas in AvroData
- PR-2175 - DGS-2990: Backport PR 2174 to 5.2.x
- PR-2174 - DGS-2990 Removing message value from exception to avoid logging custo
- PR-2164 - DGS-2937 Use default scale of 0 in Avro converter
- PR-2151 - DGS-2875: Return correct default value for connect union
- PR-2148 - DGS-2830 Check for addition/removal of required Protobuf fields
- PR-2140 - DGS-2814 Ensure id generation is thread-safe in converters
- PR-2128 - MINOR: Update 6.1.x OpenAPI Spec
- PR-2126 - DGS-2719: add explicit dependency for kotlin-scripting-jvm
- PR-2125 - DGS-2727 Handle multiple refs to same Protobuf enum
- PR-2124 - DGS-2722 Use identity comparison for DatumReader/Writer caches
- PR-2122 - DGS-2707 Ensure Protobuf reserved range uses exclusive end
- PR-2108 - DGS-2646 Support name method on ProtobufSchema with only enums
- PR-2107 - DGS-2636 Collect properties for allOf in JSON Schema converter
- PR-2100 - Backport PR2039 to 5.5.x branch
- PR-2103 - DGS-2626 Handle combined schema with title and/or object
- PR-2089 - DGS-2541 Fallback to simple name comp in Protobuf compat
Commercial Features¶
Confluent Server¶
- Cherry-pick “Disbaled bubbling errors from JsonSerializer and Deserializers for connect that print the message in stack trace (#5558)” onto 6.1.x
- [CLI-1430] Add GCM functionality to DecryptionEngine (#5563)
- Upgrade TCNative
- APPSEC-860: Upgrade Telemtry to adopt new Netty version.
- Bump Protobuf Dependency Version
- [DO NOT MERGE] Cache+upgrade+optimize metadata many partitions
- KC-1933; Upgrade confluent-log4j to 1.2.17-cp2.2 (6.1.x)
- KAFKA-9648: Add configuration to adjust listen backlog size for Accep
Security¶
- Added fix for empty bulkAuthorize requests
- DG-1249, DG-1257, DG-1353, DG-1423, DG-1462, DG-1491, DGS-2950: Fix testGetLastOffsetWithExistingMessages
- APPSEC-827:Upgrade bc-fips
- DGS-2730: Use fail-open logic in license validation
- DGS-2570: add ScopeResource for SchemaRegistrySecurityResourceExtension
Secret Registry¶
Control Center¶
- MMA-11603: Update protobuf version
- MMA-11121: Fix compatibility issues for partition size metric
- MMA-11478: enable cors check on websocket
- MMA-11451: fix @JsonProperty usage to yield expected output
- (MINOR): update armeria version and exclude netty artifacts
- (MINOR): fix merge
- MMA-11468: Add option to enable nosniff protection
- MMA-10986: fix health check resource for connect
- MMA-10571: add SR basic auth to armeria healthcheck
- (MINOR): fix armeria test
- MMA-9126: Fix broken RocksDB stats endpoint
- MMA-9292: unit test for ClusterDataExporter extraction of security protocol
- MMA-9292: control-center-export script reads security.protocol from prop file
- MMA-10114: Change default values for command topic configuration
- MMA-9479: Remove JDK dependency from control center stop script
- DGS-2570: call /scope instead of /permissions to get SR scope in RbacServiceVisibilityFilter
- Added support for bigint format for numbers coming from ksql JSON
MQTT Proxy¶
- Fixing error in repeated manual merge.
- [CC-16389] Changing the maven URL in pom.xml from http to https
- Revert “[CC-16389] Changing the maven URL in pom.xml from http to https”
- [CC-16389] Changing the maven URL in pom.xml from http to https
Metadata Service¶
- MDS-33 - Add AES GCM enable variable
- CIAM-1761 - Update postgres db driver to 42.3.3 for cve issue
- CIAM-1737 Bump PSQL Version
- SEC-238 MDS Test : Silent Proxy of Writes
- SEC-2805 - Manually tested fix for MDS forwarding logic to handle unicode and request with query params
Version 6.1.4¶
Released November 2021
Community Features¶
Kafka¶
- 16a28cdb - upgrade netty version to 4.1.68.final
- PR-11068 - KAFKA-13081: detect and handle doubly assigned partitions in cooperative assignor (#11068)
- PR-10986 - KAFKA-12983: reset needsJoinPrepare flag before rejoining the group (#10986)
- PR-11308 - KAFKA-13277; Fix size calculation for tagged string fields in message generator (#11308)
- PR-11279 - KAFKA-13236: TopologyTestDriver should not crash for EOS-beta config (#11279)
- PR-11266 - KAFKA-13233 Log zkVersion in more places (#11266)
- PR-11231 - KAFKA-13214; Consumer should not reset state after retriable error in rebalance (#11231)
REST Proxy¶
Schema Registry¶
- PR-2079 - DGS-2507 Use fullName if enhanced.protobuf.schema.support=true
- PR-2075 - DGS-2491 Ensure ProtobufSchema.name returns qualified name
- PR-2070 - DGS-2401 Add converter flag to allow old doc/default behavior
- PR-2036 - Fix default value for array of records
- PR-2068 - Demostrate wrong cache hit for union type reader schema
- PR-2061 - DGS-2417 Change moving Protobuf messages as compatible
- PR-2058 - DGS-2393 Fix special chars in options for oneof/enum
- PR-2052 - DGS-2315: Only tombstone previous schemas during hard deletion
- PR-2043 - DGS-2259 Omit defaults for fields with type Message
- PR-2026 - DGS-2078 Fix optional Avro schema generation
- PR-2024 - DGS-2076 Fix NPE when cross referencing schema w/o name
- PR-2023 - DGS-2073 Fix Protobuf converter for multiple map refs
Commercial Features¶
Confluent Server¶
- KSTORAGE-1528: Remove flakey tests for TierObjectStoreFactory
- CONFLUENT: Upgrade netty to 4.1.68.Final and disable io.netty.handler.ssl.openssl.useTasks (#4606)
- upgrade netty version to 4.1.68.final
- Backport KSTORAGE-1369 to 6.1.x
- SEC-2731 - Improve Ldap handling for stale connections (#4497)
- CC-13747 CIAM-1263 FF-4419: Limit max retries in MDS URL Refresh loop
- CIAM-1253 Fix Logging of various timeouts in ce-rest-auth RestClient
- SEC-1537 RestClient Retry Logic Fixed (#4395)
- KC-1878 Debug Logging for LDAP Context creation
Control Center¶
- (MINOR): MMA-11236: clean and reorganize checkstyle suppressions
- PG-389: short-lived http connection for non-idempotent reqs and configurable mds client
- MMA-10607: fix testBadClientDoesNotStallRegistration by awaiting method invocation before verify
Metadata Service¶
- Update commons-codec
Replicator¶
- Upgrade jakarta.el:jakarta.el-api
- REPL-1843: Track uncommited records given to Connect framework.
Confluent Platform Docker Images¶
- Confluent Platform Docker Images will no longer ship with Yum/Apt configuration that allowed users to update running Confluent Platform containers to the next Confluent Platform release. Confluent recommends upgrading Confluent Platform service components running in containers by replacing the running container with the next Confluent Platform released Docker image and using Docker volumes to persist data between upgrades.
- Confluent Platform Docker images formerly shipped with world writable data and configuration directories (
chmod a+w). Those permissions are now removed, and adjustments made to make those directories writable from the image’sappuserUSER. - Confluent Platform Docker images are now built with
yum --nodocs ...to prevent installing Red Hat Documentation. This slims down the images slightly. Confluent documentation from Confluent packages are still installed.
Version 6.1.3¶
Released September 2021
Community Features¶
Common¶
Kafka¶
- 7b5db94c - KAFKA-13209: Upgrade jetty-server
- PR-10915 - KAFKA-13041: Enable connecting VS Code remote debugger (#10915)
- PR-11136 - KAFKA-13141; Skip follower fetch offset update in leader if diverging epoch is present (#11136)
- PR-11098 - KAFKA-13099; Transactional expiration should account for max batch size (#11098)
- PR-10919 - KAFKA-12985: Upgrade jetty to 9.4.42 (#10919)
- PR-8844 - KAFKA-9887 fix failed task or connector count on startup failure (#8844)
- PR-9541 - KAFKA-10675: Add schema name to ConnectSchema.validateValue() error message (#9541)
- PR-10930 - KAFKA-12996; Return OFFSET_OUT_OF_RANGE for fetchOffset < startOffset even for diverging epochs (#10930)
- PR-10894 - KAFKA-12951: restore must terminate for tx global topic (#10894)
- PR-10922 - KAFKA-12790: Remove SslTransportLayerTest.testUnsupportedTlsVersion (#10922)
- PR-10834 - KAFKA-12904: Corrected the timeout for config validation REST API resource (#10834)
- PR-10014 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes (#10014)
- PR-10882 - KAFKA-12948: Remove node from ClusterConnectionStates.connectingNodes when node is removed (#10882)
REST Proxy¶
- PR-886 - Re-enable nano-version builds
- PR-885 - KREST-1577 Replace a manually set Avro version with the Avro version variable.
- PR-870 - Use a more lenient test for testBackoffMsControlsPollCalls.
- 0c83d02d - SEC-1034: log4j migration to confluent repackaged version
- d0b719bf - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- 19154f1d - SEC-1034: log4j migration to confluent repackaged version
- 8a9c055f - build: backport formatting checkstyle issues from #873
- PR-280 - KREST-578: Add configurable DosFilter support. (#238)
- PR-275 - Fix rest utils checkstyle and log4j updates
- PR-270 - Upgrade Jetty to 9.4.43.v20210629
Schema Registry¶
- PR-2016 - DGS-2026 Allow generating JSON schemas with additionalProperties=true
- PR-2014 - DGS-2020 Allow optional to be overridden in SchemaWrapper
- PR-2012 - DGS-2018 Ensure names in Avro union are scrubbed
- PR-1995 - Corrected Typo in ProtobufConverter and JsonSchemaConverter
- PR-1990 - DGS-1879 changed timeout parameter
- PR-1989 - Upgrade commons-compress to 1.21 (5.4.x and later)
- PR-1993 - Fixing checkstyle complaints (5.5.x and later)
- PR-1986 - DGS-1915: Upgrade commons-compress version
- PR-1984 - Add confluent-log4j as dependency
- PR-1977 - DGS-1909 Fix AVRO_FIELD_DEFAULT_FLAG_PROP in AvroData
- PR-1972 - DGS-1902 Use fully qualified names for Protobuf Map types
- PR-1969 - SEC-2492 exclude guava version with CVE
- PR-1968 - DGS-1880 Allow additionalProperties to be set in converter
- PR-1967 - Logging schema size when register
- a0fa5210 - Remove use of WrappedParsedSchema
- PR-1965 - Remove use of WrappedParsedSchema in mock client
- PR-1956 - DGS-1853 Fix handling of slashes with refs in maven plugin
- PR-1954 - DGS-1850 Prevent duplicate imports in Protobuf converter
- PR-1947 - DGS-1829 Handle duplicate refs during Protobuf Schema compat check
- PR-1943 - DGS-1826 Fix NPE when struct default has null
- PR-1945 - DGS-1825 Fix DataException when scrubbing invalid names (Avro)
- PR-1942 - DGS-1825 Fix DataException when scrubbing invalid names (Protobuf)
- PR-1939 - DGS-1820 Pass ObjectMapper to JSON Schema generator
- PR-1937 - DGS-1820 JSON Schema validation should take a JsonNode
- PR-1931 - DGS-1794 Don’t return read-only byte buffers from Protobuf converter
- PR-1928 - DGS-1758: Allow AvroData take JSON null as default
- PR-1923 - DGS-1737: MockSchemaRegistryClient doesn’t generate new schema id with protobuf
- PR-1920 - DGS-404: Improving error messages - adding 40408 and 40409 for subject level compatibility and mode error
- PR-1918 - small fix in deleteSchemaVersion for MockSchemaRegistryClient
- PR-1917 - DGS-1734: MockSchemaRegistryClient need to perform a correct deleteSchemaVersion
Commercial Features¶
Confluent Server¶
- Change Guava from andriod to jre
- remove azure backend for tiered storage system tests in 6.1+
- KC-1800: Upgrade Guava to 30.0-android
- KAFKA-13209: Upgrade jetty-server
- AUDIT-670 - Moved DESCRIBE_USER_SCRAM_CREDENTIALS from MANAGEMENT to DESCRIBE
- DGS-1903: (backport to 6.1.x) Add config option for missing schemas TTL (#4118)
- Upgrade AzureIdentity
- KC-1809 Upgrade transitive hibernate-validator dependency
- KC-1810: Upgrade commons.io in ce-kafka
- Backport KC-1580 to 6.1.x
- KAFKALESS-523: Topic placement constraints not used in CreatePartitions
- KAFKALESS-521: Don’t join empty throttled replicas with new throttled replicas
- KC-1784: Use URI instead of URL for RBAC writer comparison to avoid hostname resolution issues
- KC-1317: Fix incrementalAlterConfig for list property with default null
- KSTORAGE-1626: Fix logging during topic creation
- Hide secrets when logging during exception in DynamicBrokerConfig
- KC-1625: Fix logging in DynamicBrokerConfig
- KC-1752 Remove public facing String redact API which is not currently used
- KC-1752 Fix potential alter/update config related logging
Security¶
- APPSEC-287: Override commons-codec
- fix formatting issues with newer checkstyle versions
- fix: Checkstyle issues in AuthorizationFilter.java
- SEC-1034: log4j migration to confluent repackaged version
- DGS-1903: Add missing schema cache to LruSchemaRegistryClient
- SEC-2533 upgrade commons-io to 2.11.0
- SEC-2493 Remove bouncy-castle explicit dependency
Secret Registry¶
- CC-15204: Upgrade bcprov-jdk15on to 1.68
Control Center¶
- MMA-7623: Hide stack trace in HTML output for failed responses
- MMA-10895: Add support for prefixed ‘auth.bearer.roles.claim’ property
- Upgrade Apache io-common to v2.7
- DGS-1654: Upgrade Protobuf to v3.17.0
MQTT Proxy¶
- CCMSG-1251: log4j migration to confluent repackaged version
Metadata Service¶
- SEC-2440 - MDS - Fix MdsScope Validation and messages and add tests
- Add try catch to first authorize on SECURITY_METADATA
- Fix lookupPrincipalsWithRoleOnResource to authorize on valid ResourcePattern using DESCRIBE_ACCESS operation
- SEC-1034: log4j migration to confluent repackaged version
- Fixed issue where users with ResourceOwner role, could not see rolebindings of other users on resources they own
Replicator¶
- No Changes
Version 6.1.2¶
Released June 2021
Community Features¶
Common¶
- PR-339 - update spotbugs and jacoco for improved jdk support
- PR-337 - DGS-1551: upgrade jackson-bom to 2.9.10.20210106
- PR-334 - CC-14690 : Define com.fasterxml.jackson.dataformat:jackson-dataformat-cbor.
- PR-332 - Update dependency-check to 6.1.6
- PR-326 - KREST-1066 Upgrade Guava to the latest version to patch CVE-2020-8908.
- PR-344 - APPSEC-110: Exclude Netty in zookeeper 3.4.14
- PR-342 - Fix jacksondatabind and jackson-cbor version
- PR-343 - Change zookeeper version to 3.4.14
- PR-340 - Fix zookeeper, jacksondatabind and jackson-cbor version
Kafka¶
- 1a355d42 - KC-1752 Remove public facing String redact API which is not currently used
- PR-9735 - KAFKA-10846: Grow buffer in FileSourceStreamTask only when needed (#9735)
- PR-575 - KC-1747: Upgrade Jetty, commons-codec, http-client
- f8ffc037 - KAFKA-12820: Upgrade maven-artifact dependency to resolve CVE-2021-26291
- PR-564 - APPSEC-60: Upgrade org.apache.maven:apache-maven
- PR-10633 - KAFKA-12751: Reset AlterIsr in-flight state for duplicate update requests (#10633)
- PR-10059 - KAFKA-8562; SaslChannelBuilder - Avoid (reverse) DNS lookup while building SslTransportLayer (#10059)
- PR-10636 - KAFKA-12752: Bump Jersey deps to 2.34 due to CVE-2021-28168 (#10636)
- PR-10611 - KAFKA-12730; Avoid duplicate logout if Kerberos login fails (#10611)
- PR-9605 - KAFKA-10727; Handle Kerberos error during re-login as transient failure in clients (#9605)
- PR-10536 - KAFKA-12667: Fix incorrect error log on StateDirectory close (#10536)
- PR-10526 - KAFKA-12655 Update Jetty: 9.4.38.v20210224 → 9.4.39.v20210325 (#10526)
- PR-10301 - KAFKA-9831: increase max.poll.interval.ms to avoid unexpected rebalance (#10301)
- PR-10474 - KAFKA-12602: Fix LICENSE file (#10474)
- PR-539 - SEC-1883: Upgrade Netty version to 4.1.62.Final
- PR-10396 - KAFKA-12474: Handle failure to write new session keys gracefully (#10396)
- PR-10452 - KAFKA-12593: Fix Apache License headers (#10452)
- PR-10397 - KAFKA-12508: Disable KIP-557 (#10397)
- PR-10360 - KAFKA-12508: Emit records with same value and same timestamp (#10360)
ksqlDB¶
REST Proxy¶
- PR-832 - KREST-46: Removed property ${confluent.maven.repo}
- PR-857 - KREST-1263 Exclude a vulnerable jackson-mapper-asl dependency from avro 1.8.2 in 5.3.x.
- d7a631a4 - KREST-1029 Remove a new occurrence of SLF4J versioning in kafka-rest.
- 40e94513 - KREST-1029 Remove SLF4J versioning from kafka-rest.
- PR-831 - KREST-865 Remove usage of the deprecated SimpleAclAuthorizer.
- PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224
- 66aa0c15 - KREST-1030 Upgrade hibernate-validator to fix CVE-2020-10693.
- 6b8327ac - KREST-1030 Upgrade Jersey to fix a number of CVEs.
- 2e5ff421 - KREST-1030 Upgrade Jetty to fix a number of CVEs.
- bfb547cc - KREST-1030 Upgrade hibernate-validator to fix CVE-2020-10693.
- PR-251 - bump guava version to 30.1.1-jre
- PR-249 - DGS-1621 Upgrade Jetty to 9.4.40
- PR-246 - fix: CVE-2021-28168
- PR-244 - Upgrade Jetty to 9.4.39.v20210325
- PR-243 - DGS-1546 Map JsonMappingException to 400 response
- PR-242 - Upgrade Jetty to 9.4.39.v20210325
- PR-239 - KREST-860 Fix deprecated usages of org.apache.kafka.common.Metric.value()
- PR-237 - Define Netty version to be shared to downstreams.
Schema Registry¶
- PR-1907 - DGS-1658 Preserve field names when rendering Protobuf objects as JSON
- PR-1902 - DGS-1727 Don’t throw exception when comparing array item
- PR-1897 - DGS-1716 Remove logging of configs
- PR-1896 - DGS-1704 override commons-compress
- PR-1895 - DGS-1704 Upgrade jackson-mapper-asl
- PR-1890 - DGS-1698 Handle recursive JSON schemas in Converter
- PR-1889 - DGS-49 Fix StackOverFlowException with cyclic JSON Schema
- PR-1885 - DGS-1677 Normalize Protobuf schemas during validation
- PR-1879 - DGS-1551 Upgrade hibernate-validator
- PR-1881 - DGS-469: Reuse MetricsContext object in KafkaGroupLeaderElector
- PR-1877 - DGS-1664 Allow URL encoded subject names in maven plugin
- PR-1873 - DGS-1657 Add converter flag to enable scrubbing invalid names
- PR-1871 - DGS-1660 Ensure schemas are sorted by version
- PR-1870 - DGS-1659 Be more lenient with non-string enums
- PR-1863 - DGS-1631 Ensure enums are emitted after message types
- PR-1852 - DGS-1595 Ignore invalid Avro defaults in Avro converter
- PR-1847 - DGS-1570 Handle optional integer/number with format in JSON Schema
- PR-1846 - DGS-1569 Ensure recursive types are not built prematurely
- PR-1844 - DGS-1562 Don’t require refs be declared with root schema
- PR-1839 - DGS-1550 Add cache for transformed schemas
- PR-1833 - DGS-1534 Use raw Avro schema to check doc equality
- PR-1821 - DGS-1469 Add server-side schema cache
- PR-1816 - DGS-1391 fix Avro Convertor for GenericData of Enum Union with names
- PR-1815 - DGS-1451 Add Jackson annotations for object evolution
- PR-1813 - Fix NPE when checking compatibility
- PR-1808 - DGS-1403 Allow register different schema types if compatibility level
- PR-1807 - UIF-302 Add Guava declaration in dependency.
- PR-1806 - DGS-1402 Flag to set non-required fields to optional in JSON Schema
- PR-1805 - DGS-1400 Escape Special Character in Protobuf Option
- PR-1804 - DGS-1392 Allow removing false from closed content model in JSON Schema
- PR-1802 - DGS-1398 Allow adding empty schema to open content model for JSON Schema
- PR-1801 - DGS-1396 Fix optionality of ref schemas in JSON Schema converter
- PR-1323 - API update for Jetty 9.4.26.v20200117
Commercial Features¶
Confluent Server¶
- Correct syntax for password based schema registry config
- Fix secret type for Schema registry client
- METRICS-3247:Upgrade telemtry-client version
- CPKAFKA-3268: Remove Apache LICENSE, add COPYRIGHT and NOTICE to all ce-kafka jars
- KC-1742: Sync Jetty, Jersey, Netty, etc versions from 6.2.x
- Change azure dependency to compileOnly for 6.2 only.
- Add an exclusion for simple-xml on the azure identity dependency
- KAFKA-12820: Upgrade maven-artifact dependency to resolve CVE-2021-26291
- Fix Azure Version
- SEC-298 - Log errors when MDS can not start
- APPSEC-90: Upgrade Azure dependencies
- KC-1610: Upgrade dataformat-cbor version (#3589)
- CNK-1695: Log cause of unfixable hard goals in goal violation self-healing. Cherry-pick to 6.1
- CONFLUENT: Change visibility of LicenseConfig constants to public
- KSTORAGE-1464: tiered fetch of transaction index should throw on cancellation (#3364)
- Added missing version in downgrade_test.py
- KSTORAGE-1466: retry of putSegment may cause violation of transactions guarantees (#3376)
- Added classpath for mds jars and on jars mds depends.
- METRICS-2934: Do not log configuration values on failed Provider validation
- DGS-1397: Schema interceptor override default (#3324)
Security¶
- DGS-1691: Perform license checks in background thread
- KQE-229: Update vertx-core to 3.9.7
- Pins Ksql to a jetty-jaas 9.4.28.v20200408 (#365) (#366)
- DGS-1463: Modify config types for sensitive SR validator plugin configs
Secret Registry¶
- SEC-2105: Update guava to version 30.1.1-jre
MQTT Proxy¶
- CCMSG-1101: Upgrade Netty to 4.1.63.Final
- CC-14445: Remove compile dependency on Kafka broker
- SEC-2022: Upgrade Netty to 4.1.62.Final
Metadata Service¶
- SEC-2191: Properly handle unicode characters when rewriting urls
- Fix spelling of dataProvider
- CIAM-426 Test Rolebindings are escaped
Version 6.1.1¶
Released May 2021
Community Features¶
Kafka¶
- PR-10311 - KAFKA-12462: Proceed with task revocation in case of thread in PENDING_SHUTDOWN (#10311)
- PR-10118 - KAFKA-10192: Increase maximum time to wait for worker to start in some integration tests (#10118)
- PR-529 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes
- PR-525 - Enable downstream validation
- PR-10260 - Remove ReassigningPartitions metric when shutting down ReplicaManager (#10260)
- 14d7dbb5 - KAFKA-12400: Upgrade Jetty to fix CVE-2020-27223
- 3b7cc02d - KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
- PR-10205 - KAFKA-12323 Follow-up: Refactor the unit test a bit (#10205)
- PR-10016 - KAFKA-10340: Proactively close producer when cancelling source tasks (#10016)
- PR-9990 - KAFKA-12235: Fix a bug where describeConfigs does not work on 2+ config keys (#9990)
- PR-10170 - KAFKA-12323: Set timestamp in record context when punctuate (#10170)
- 3f005e38 - KAFKA-12324: Upgrade Jetty to fix CVE-2020-27218
- PR-10152 - KAFKA-12339: Add retry to admin client’s listOffsets (#10152)
- PR-10158 - KAFKA-12343: Handle exceptions better in TopicAdmin, including UnsupportedVersionException (#10158)
- PR-10153 - KAFKA-12340: Fix potential resource leak in Kafka*BackingStore (#10153)
- PR-9589 - KAFKA-10710; MM2 - Create herders only if source->target.enabled=true and heartbeats are disabled (#9589)
- PR-10122 - KAFKA-12326: Corrected regresion in MirrorMaker 2 executable introduced with KAFKA-10021 (#10122)
- PR-10098 - KAFKA-12321: The comparison function for UUID type should be
equalsrather than==(#10098) - PR-10104 - KAFKA-12272: Fix commit-interval metrics (#10104)
- PR-9780 - KAFKA-10021: Changed Kafka backing stores to use shared admin client to get end offsets and create topics (#9780)
- PR-9977 - KAFKA-12310: Update ZooKeeper to 3.5.9 (#9977)
- PR-10060 - KAFKA-10716: Persist UUID in state directory for stable processId across restarts - 2.7 (#10060)
- PR-9902 - KAFKA-12193: Re-resolve IPs after a client disconnects (#9902)
- PR-10032 - KAFKA-12270: Handle race condition when Connect tasks attempt to create topics (#10032)
- PR-10040 - KAFKA-12259: Use raw config to infer the connector type when returning a connector status response (#10040)
- PR-9987 - KAFKA-10895: Handle invalid JAAS configs (follow up fix) (#9987)
- PR-9319 - KAFKA-10413: Allow for even distribution of lost/new tasks when multiple Connect workers join at the same time (#9319)
- PR-9525 - KAFKA-10658: ErrantRecordReporter.report always return completed futures (#9525)
- PR-9983 - KAFKA-8930: MirrorMaker v2 documentation (#324) (#9983)
- PR-9671 - KAFKA-10793: Move handling of FindCoordinatorFuture to fix race condition (#9671)
- PR-9765 - KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations (#9765)
- PR-9947 - KAFKA-12190: Fix setting of file permissions on non-POSIX filesystems (#9947)
- PR-9832 - KAFKA-12152; Idempotent Producer does not reset the sequence number of partitions without in-flight batches (#9832)
- PR-9904 - KAFKA-12211: Don’t change perm for base/state dir when no persistent store (#9904)
- PR-9923 - KAFKA-12219: Add ‘synchronized’ keyword to InMemoryKeyValueStore#[reverseRange, reverseAll] (#9923)
- PR-9673 - KAFKA-10289: Fix failed connect_distributed_test.py (ConnectDistributedTest.test_bounce) (#9673)
- PR-9669 - KAFKA-10792: Prevent source task shutdown from blocking herder thread (#9669)
- PR-9806 - KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806)
- PR-9838 - KAFKA-12153: Update producer state before updating start/end offsets after truncation (#9838)
- PR-9583 - KAFKA-10705: Make state stores not readable by others (#9583)
- PR-9568 - KAFKA-10689: Fix windowed FKJ topology and put checks in assignor to avoid infinite loops (#9568)
- PR-9782 - KAFKA-10815: EosTestDriver#verifyAllTransactionFinished should break loop if all partitions are verified (#9706) (#9782)
ksqlDB¶
REST Proxy¶
- f46271b4 - Fix import in test.
- PR-799 - KREST-558: Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
- 2ffd770b - KREST-558 Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
- PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224
- PR-229 - KC-1456: Update Jetty to 9.4.36.v20210114
- PR-231 - UIF-298: Upgrade Jetty to 9.4.35.v20201120
- PR-228 - MMA-9930: Use compatible version of Guava
- PR-222 - UIF-300: Upgrade org.apache.httpcomponents.httpclient
- 92e91bda - Remove extra Guava version during manual merge.
- PR-223 - UIF-302: Upgrade Guava for 5.4.x and older.
- PR-224 - Upgrade hibernate-validator to 6.1.7.Final
Schema Registry¶
- PR-1799 - DGS-1391: Fix Avro Convertor for GenericData of union with null, string, enum
- PR-1794 - Fix String.format in compatibility errorMessages
- PR-1795 - DGS-1221: Ensure AvroSchema.hashCode is consistent with equals
- PR-1791 - DGS-1364: Fix compatibility for JSON empty schema
- PR-1737 - Backport #1733 to 5.4.x
- PR-1784 - DGS-1221: Fix AvroSchema.equals
- PR-1779 - DGS-772: Add API annotations for /mode
- PR-1777 - Upgrade to everit 1.12.2
- PR-1639 - Upgrade to wire 3.6.0
- PR-1775 - DGS-104: Fix Avro compat check
- PR-1774 - DGS-1333: Add config to remove Java props from Avro schemas
- PR-1770 - DGS-1315: upgrade HibernateValidator
- PR-1769 - DGS-1314: update JetBeanKotlin to 1.4.21
- PR-1741 - DG-1162: Add verbose option to SR maven plugin test-compatibility goal
Commercial Features¶
Confluent Server¶
- METRICS-2907: Make HttpExporterConfig proxy passwords Password objects
- Merge: jmh-benchmark fixes after log start offset update refactoring
- Update log start offset for MergedLog via callback from Log to avoid mistakes
- CONFLUENT: Make Kafka dependencies for ce-metrics compileOnly
- CONFLUENT: Backport KAFKA-12252 and KAFKA-12262 (#3268)
- CONFLUENT: Revert “Limit the max topic name length to 209 (#425)”
- Re-enable downstream validation check
- MMA-9397: Filter out
confluent.sslconfigs when describing topic configs - KAFKA-12400: Upgrade Jetty to fix CVE-2020-27223
- KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
- KC-1522: Do not log parsed configs containing passwords in CruiseControlStartable
- Use AWS CLI version 1.18.* for Python 3.4 compatibility
- [METRICS-2820] Collect metrics on message conversion (#3189) (backport 6.1.x)
- Backport METRICS-2819: Collect metrics about log cleaner (#3184)
- Add authentication counters to default broker include regex
- KAFKA-12324: Upgrade Jetty to fix CVE-2020-27218
- KC-1457: Update HttpClient to 4.5.13 and add explicit HttpClient dependency
- CONFLUENT: Update ZooKeeper node start timeout in system tests
- KC-924: Add event processing metrics for ZK ACLs and MDS records
- KC-922, KC-1238: Handle writer stop exceptions to ensure new writer is re-elected
Security¶
- DGS-1381: Deserialize GLOBAL_SUBJECTS_READ to GLOBAL_READ for backward compatibility
Replicator¶
- REPL-1410: Changed logging for WakeupException to DEBUG
- REPL-1386: Adding clear logging statements for source and destination operations
- REPL-1409: Added WARN logging if destination SR deemed not ready
- REPL-1396: Fix CVE for HttpClient
- SEC:1820: Fix CVE for codec
- REPL-1396: Fix CVE for aws-java-sdk-ecr
Version 6.1.0¶
Released February 2021
Community Features¶
Common¶
Kafka¶
- PR-9678 - KAFKA-10798; Ensure response is delayed for failed SASL authentication with connection close delay
- PR-9702 - CVE-2020-25649: bumping jackson to patched version 2.10.5.1
- PR-9661 - KAFKA-10776: update the doc to add version attribute in RequstsPerSec metrics
- PR-9729 - KAFKA-10839: improve consumer group coordinator unavailable message
- PR-9747 - KAFKA-10852: AlterIsr should not be throttled
- PR-9723 - Cherry-pick stream producer fix 10813
- PR-9698 - KAFKA-10811: Correct the MirrorConnectorsIntegrationTest to correctly mask the exit procedures
- PR-8965 - KAFKA-8147: Add changelog topic configuration to KTable suppress
- PR-9669 - KAFKA-10792: Prevent source task shutdown from blocking herder thread
- PR-9674 - KAFKA-10665: close all kafkaStreams before purgeLocalStreamsState
- PR-9677 - KAFKA-10799 AlterIsr utilizes ReplicaManager ISR metrics
- PR-9648 - KAFKA-10758: ProcessorTopology should only consider its own nodes when updating regex source topics
- PR-9634 - KAFKA-10755: Should consider commit latency when computing next commit timestamp
- PR-9633 - KAFKA-10706; Ensure leader epoch cache is cleaned after truncation to end offset
- PR-9613 - Cherry-pick KAFKA-10687 to 2.7
- PR-9515 - KAFKA-10651: read offsets directly from checkpoint for uninitialized tasks
- PR-9534 - KAFKA-10664: Delete existing checkpoint when writing empty offsets
- PR-9529 - Revert initial principal from 2.7
- PR-9521 - KAFKA-10638: Fix QueryableStateIntegrationTest
- PR-9506 - KAFKA-10647; Only serialize owned partitions when consumer protocol version >= 1
- PR-9486 - KAFKA-9381: Fix publishing valid scaladoc for streams-scala
- PR-9464 - KAFKA-10616: Always call prepare-commit before suspending for active tasks
- PR-9479 - KAFKA-10631: Handle ProducerFencedException on offset commit
- PR-9270 - KAFKA-10284: Group membership update due to static member rejoin should be persisted
- PR-9406 - KAFKA-10520; Ensure transactional producers poll if leastLoadedNode not available with max.in.flight=1
- PR-9237 - KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
- PR-9338 - KAFKA-10515: Properly initialize nullable Serdes with default values
- PR-9431 - KAFKA-10426: Deadlock on session key update.
- PR-9455 - KAFKA-10599: Implement basic CLI tool for feature versioning system
- PR-9429 - KAFKA-10572 mirror-maker config changes for KIP-629
- PR-9448 - KAFKA-10605: KIP-478: Deprecate old PAPI registration methods
- PR-9383 - KAFKA-10455: Ensure that probing rebalances always occur
- PR-9343 - KAFKA-10332: Update MM2 refreshTopicPartitions() logic
- PR-9425 - KAFKA-10600: Connect should not add error to connector validation values for properties not in connector’s ConfigDef
- PR-9432 - KAFKA-10559: Not letting TimeoutException shutdown the app during internal topic validation
- PR-9422 - KAFKA-10602: Make RetryWithToleranceOperator thread safe
- PR-9407 - KAFKA-10611: Merge log error to avoid double error
- PR-9438 - KAFKA-10613: Only set leader epoch when list-offset version >= 4
- PR-8150 - KAFKA-9587: Producer configs are omitted in the documentation
- PR-9353 - KAFKA-10521; Skip partition watch registration when AlterIsr is expected
- PR-9403 - KAFKA-10573 Update connect transforms configs for KIP-629
- PR-9415 - KAFKA-10494: eager handling of sending old values
- PR-9396 - KAFKA-10437: Implement new PAPI support for test-utils
- PR-8891 - KAFKA-10143: Improve test coverage for throttle changes during reassignment
- PR-9375 - KAFKA-10574: Fix infinite loop in Values::parseString
- PR-9408 - KAFKA-10598: Improve IQ name and type checks
- PR-9020 - KAFKA-10271 Performance regression while fetching a key from a single partition
- PR-9380 - KAFKA-7334: Suggest changing config for state.dir in case of FileNotF…
- PR-9393 - KAFKA-10028: Minor fixes to describeFeatures and updateFeatures apis
- PR-9139 - KAFKA-9929: Support backward iterator on SessionStore
- PR-9388 - KAFKA-10562: Properly invoke new StateStoreContext init
- PR-9373 - KAFKA-10564: only process non-empty task directories when internally cleaning obsolete state stores
- PR-9385 - KAFKA-9274: fix incorrect default value for task.timeout.ms config
- PR-9247 - KAFKA-10362: When resuming Streams active task with EOS, the checkpoint file is deleted
- PR-9001 - KAFKA-10028: Implement write path for feature versioning system (KIP-584)
- PR-9196 - KAFKA-10402: Upgrade system tests to python3
- PR-9280 - KAFKA-10186: Abort transaction with pending data with TransactionAbortedException
- PR-9321 - KAFKA-9929: fix: add missing default implementations
- PR-8910 - KAFKA-10188: Prevent SinkTask::preCommit from being called after SinkTask::stop
- PR-9345 - KAFKA-10338; Support PEM format for SSL key and trust stores (KIP-651)
- PR-9348 - KAFKA-10527; Voters should not reinitialize as leader in same epoch
- PR-9099 - KAFKA-6733: Printing additional ConsumerRecord fields in DefaultMessageFormatter
- PR-9320 - KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale.
- PR-9347 - KAFKA-10531: Check for negative values to Thread.sleep call
- PR-9361 - KAFKA-10535: Split ProcessorContext into Processor/StateStore/Record Contexts
- PR-8730 - KAFKA-10048: Possible data gap for a consumer after a failover when u…
- PR-9306 - KAFKA-10477: Enabling the same behavior of NULL JsonNodeType to MISSI…
- PR-9281 - KAFKA-10478: Allow duplicated ports in advertised.listeners
- PR-9354 - KAFKA-10134 Follow-up: Set the re-join flag in heartbeat failure
- PR-9356 - KAFKA-10556: NPE if sasl.mechanism is unrecognized
- PR-9309 - KAFKA-10503: MockProducer doesn’t throw ClassCastException when no
- PR-9102 - KAFKA-10326 Both serializer and deserializer should be able to see th…
- PR-9333 - KAFKA-9274: Revert deprecation of retries for producer and admin clients
- PR-9064 - KAFKA-10205: Documentation and handling of non deterministic Topologies
- PR-9186 - KAFKA-10277: Allow null keys with non-null mappedKey in KStreamKGlobalTable join
- PR-9121 - KAFKA-10351: add tests for IOExceptions for GlobalStateManagerImpl/OffsetCheckpoint
- PR-9255 - KAFKA-6585: Consolidate duplicated logic on reset tools
- PR-9266 - KAFKA-10469: Resolve logger levels hierarchically
- PR-9284 - KAFKA-10479 Throw exception if users try to update configs of existen…
- PR-9332 - KAFKA-10511; Ensure monotonic start epoch/offset updates in MockLog
- PR-8973 - KAFKA-10218: Stop reading config topic in every subsequent tick if catchup fails once
- PR-8181 - KAFKA-9584 Headers ConcurrentModificationException
- PR-9317 - KAFKA-10509: Added throttle connection accept rate metric (KIP-612)
- PR-9304 - KAFKA-10502:TimestampRouter may occur threadlocal leak
- PR-9156 - KAFKA-10077: Filter downstream of state-store results in spurious tombstones
- PR-9337 - KAFKA-10519; Add missing unit test for VotedState
- PR-9334 - KAFKA-10516; Disable automatic retry of THROTTLING_QUOTA_EXCEEDED errors in the kafka-topics command (KIP-599)
- PR-9100 - Add AlterISR RPC and use it for ISR modifications
- PR-8295 - KAFKA-9627: Replace ListOffset request/response with automated protocol
- PR-9083 - KAFKA-9450: Follow-up; Forbid process after closed
- PR-9312 - KAFKA-10505: Fix parsing of generation log string.
- PR-9323 - KAFKA-10514: Fix unit test for state directory cleanup
- PR-9130 - KAFKA-10492; Core Kafka Raft Implementation (KIP-595)
- PR-8892 - KAFKA-10068: verify assignment performance with large cluster
- PR-9251 - KAFKA-10459: Document IQ APIs where order does not hold between stores
- PR-9202 - KAFKA-10401: Fix the currentStateTimeStamp doesn’t get set correctly
- PR-9223 - KAFKA-10438 Lazy initialization of record header to reduce memory usa…
- PR-9290 - KAFKA-10487; Fetch response should return diverging epoch and end offset
- PR-9275 - KAFKA-10435; Fetch protocol changes for KIP-595
- PR-9286 - KAFKA-10292: Set min.insync.replicas to 1 of __consumer_offsets
- PR-9274 - KAFKA-10131: Remove use_zk_connection flag
- PR-9257 - KAFKA-10463 the necessary utilities in Dockerfile should include git
- PR-9272 - KAFKA-10458; Updating controller quota does not work since Token Bucket
- PR-9178 - KAFKA-8362: fix the old checkpoint won’t be removed after alter log dir
- PR-9239 - Adding reverse iterator usage for sliding windows processing (extending KIP-450)
- PR-9264 - KAFKA-5636: Add Sliding Windows documentation
- PR-9231 - KAFKA-10447: Migrate tools module to JUnit 5 and mockito
- PR-8834 - KAFKA-10134: Enable heartbeat during PrepareRebalance and Depend On State For Poll Timeout
- PR-8657 - KAFKA-8334 Make sure the thread which tries to complete delayed reque…
- PR-9157 - KAFKA-5636: Update for KIP-450 to handle early records
- PR-9221 - KAFKA-10436: Implement KIP-478 Topology changes
- PR-9182 - KAFKA-10403 Replace scala collection by java collection in generating…
- PR-9219 - KAFKA-10432: LeaderEpochCache is incorrectly recovered for leader epoch 0
- PR-9032 - KAFKA-10259: KIP-554 Broker-side SCRAM Config API
- PR-9215 - KAFKA-10133: MM2 readme update on config
- PR-9122 - KAFKA-10314: KafkaStorageException on reassignment when offline log d…
- PR-9191 - KAFKA-10355: Throw error when source topic was deleted
- PR-9232 - KAFKA-9924: Add remaining property-based RocksDB metrics as described in KIP-607
- PR-9138 - KAFKA-9929: Support backward iterator on WindowStore
- PR-9023 - KAFKA-10272: Add IBM i support to “stop” scripts
- PR-8798 - KAFKA-10098: Remove unnecessary escaping in regular expression.
- PR-9240 - KAFKA-10456: Wrong description in kafka-console-producer.sh help
- PR-8955 - KAFKA-10020: Create a new version of a scala Serdes without name clash (KIP-616)
- PR-9039 - KAFKA-5636: SlidingWindows (KIP-450)
- PR-9177 - KAFKA-9924: Add RocksDB metric num-entries-active-mem-table
- PR-9194 - KAFKA-10384: Separate converters from generated messages
- PR-9094 - KAFKA-10054: KIP-613, add TRACE-level e2e latency metrics
- PR-9112 - KAFKA-10312 Fix error code returned by getPartitionMetadata
- PR-9131 - KAFKA-10367: Allow running the Streams demo app with a config file
- PR-9136 - KAFKA-10211: Add DirectoryConfigProvider
- PR-9137 - KAFKA-9929: Support reverse iterator on KeyValueStore
- PR-9146 - KAFKA-10316 Updated Kafka Streams upgrade-guide.html
- PR-9028 - KAFKA-10035: Safer conversion of consumer timeout parameters
- PR-7222 - KAFKA-8806 Reduce calls to validateOffsetsIfNeeded
- PR-9197 - Revert KAFKA-9309: Add the ability to translate Message to JSON
- PR-9148 - KAFKA-10379: Implement the KIP-478 StreamBuilder#addGlobalStore()
- PR-9189 - KAFKA-10407: Have KafkaLog4jAppender support linger.ms and batch.size
- PR-9144 - KAFKA-9705: (part-1) add redirection fields in the request header
- PR-9174 - KAFKA-10395: relax output topic check in TTD to work with dynamic routing
- PR-9172 - KAFKA-10387: Fix inclusion of transformation configs when topic creation is enabled in Connect
- PR-8768 - KAFKA-10023: Enforce broker-wide and per-listener connection creation…
- PR-9183 - KAFKA-10404; Use higher poll timeout to avoid rebalance in testCoordinatorFailover
- PR-9184 - KAFKA-8033; Wait for NoOffsetForPartitionException in testFetchInvalidOffset
- PR-9181 - KAFKA-9516; Increase timeout in testNonBlockingProducer to make it more reliable
- PR-9108 - KAFKA-9273: Extract testShouldAutoShutdownOnIncompleteMetadata from S…
- PR-9098 - KAFKA-9924: Prepare RocksDB and metrics for RocksDB properties recording
- PR-9163 - KAFKA-10386; Fix flexible version support for records type
- PR-9170 - KAFKA-10391: Overwrite checkpoint in task corruption to remove corrupted partitions
- PR-8549 - KAFKA-9911: Add new PRODUCER_FENCED error code
- PR-9166 - KAFKA-10388; Fix struct conversion logic for tagged structures
- PR-8964 - KAFKA-9450: Decouple flushing state from commiting
- PR-9004 - KAFKA-10261: Introduce the KIP-478 apis with adapters
- PR-9140 - KAFKA-10371; Partition reassignments can result in crashed ReplicaFetcherThreads
- PR-9060 - KAFKA-9274: Remove retries from InternalTopicManager
- PR-9120 - KAFKA-10316: Consider renaming getter method for Interactive Queries
- PR-9114 - KAFKA-10162; Use Token Bucket algorithm for controller mutation quota (KIP-599, Part III)
- PR-9047 - KAFKA-9274: Remove retries for global task
- PR-9116 - KAFKA-10341: Add 2.6.0 to system tests and streams upgrade tests
- PR-9092 - KAFKA-10163; Define controller_mutation_rate as a Double instead of a Long
- PR-9008 - KAFKA-9629 Use generated protocol for Fetch API
- PR-9097 - KAFKA-10319: Skip unknown offsets when computing sum of changelog offsets
- PR-9007 - KAFKA-10120: Deprecate DescribeLogDirsResult.all() and .values()
- PR-9095 - KAFKA-10321: fix infinite blocking for global stream thread startup
- PR-9081 - KAFKA-10309: KafkaProducer’s sendOffsetsToTransaction should not block infinitively
- PR-9012 - KAFKA-10270: A broker to controller channel manager
- PR-9089 - KAFKA-10224: Update jersey license from CDDL to EPLv2
- PR-9066 - KAFKA-10287: Skip unknown offsets when computing sum of changelog offsets
- PR-9086 - FIX: Remove staticmethod tag to be able to use logger of instance
- PR-9075 - KAFKA-10306: GlobalThread should fail on InvalidOffsetException
- PR-9034 - KAFKA-10246 : AbstractProcessorContext topic() throws NPE
- PR-9071 - KAFKA-10305: Print usage when parsing fails for ConsumerPerformance
- PR-9061 - MINOR: removed incorrect deprecation annotations
- PR-9065 - KAFKA-10301: Do not clear Partition#remoteReplicasMap during partition assignment updates
- PR-9051 - KAFKA-10268: dynamic config like “–delete-config log.retention.ms” doesn’t work
- PR-8200 - KAFKA-5876: IQ should throw different exceptions for different errors(part 1)
- PR-9026 - KAFKA-10274; Consistent timeouts in transactions_test
- PR-8968 - KAFKA-10164; Throttle Create Topic, Create Partition and Delete Topic Operations (KIP-599, Part II, Admin Changes)
- PR-8933 - KAFKA-10163; Throttle Create Topic, Create Partition and Delete Topic Operations (KIP-599, Part I, Broker Changes)
- PR-8864 - KAFKA-9274: Mark retries config as deprecated and add new task.timeout.ms config
- PR-9044 - KAFKA-10279; Allow dynamic update of certificates with additional SubjectAltNames
- PR-8935 - KAFKA-10189: reset event queue time histogram when queue is empty
- PR-9043 - KAFKA-10295: Wait for connector recovery in test_bounce
- PR-9040 - KAFKA-10286: Connect system tests should wait for workers to join group
- PR-8979 - KAFKA-10223; Use NOT_LEADER_OR_FOLLOWER instead of non-retriable REPLICA_NOT_AVAILABLE for consumers
- PR-8948 - KAFKA-10174: Prefer –bootstrap-server for configs command in ducker tests
- PR-8239 - KAFKA-9666: Don’t increase transactional epoch when trying to fence if the log append fails
- PR-9021 - KAFKA-10257 system test kafkatest.tests.core.security_rolling_upgrade…
- PR-8928 - KAFKA-10192: Wait for REST API to become available before testing blocked connectors in integration tests
- PR-9003 - KAFKA-10240: Stop throwing WakeupExceptions during sink task shutdown
- PR-8672 - KAFKA-10002; Improve performances of StopReplicaRequest with large number of partitions to be deleted
- PR-9013 - KAFKA-10044 Deprecate ConsumerConfig#addDeserializerToConfig and Prod…
- PR-9009 - KAFKA-6453: Document how timestamps are computed for aggregations and joins
- PR-8994 - KAFKA-10247: Correctly reset state when task is corrupted
- PR-9010 - KAFKA-10262: Ensure that creating task directory is thread safe
- PR-9005 - KAFKA-10263: Do not assign standby for revoking stateless tasks
- PR-8996 - KAFKA-10249: don’t try to read un-checkpointed offsets of in-memory stores
- PR-8995 - Restore stream-table duality description
- PR-8902 - KAFKA-10179: Pass correct changelog topic to state serdes
- PR-8974 - KAFKA-10225 Increase default zk session timeout for system tests
- PR-8934 - KAFKA-10134: Use long poll if we do not have fetchable partitions
- PR-8990 - KAFKA-10243; ConcurrentModificationException while processing connection setup timeouts
- PR-8989 - KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups
- PR-8579 - KAFKA-9930: Prevent ReplicaFetcherThread from throwing UnknownTopicOrPartitionException upon topic creation and deletion.
- PR-8962 - KAFKA-10166: checkpoint recycled standbys and ignore empty rocksdb base directory
- PR-8983 - KAFKA-8398: Prevent NPE in forceUnmap
- PR-8479 - KAFKA-9769: Finish operations for leaderEpoch-updated partitions up to point ZK Exception
- PR-8944 - KAFKA-10209: Fix connect_rest_test.py after the introduction of new connector configs
- PR-8604 - KIP-597: MirrorMaker2 internal topics Formatters
- PR-8938 - KAFKA-10173: Use SmokeTest for upgrade system tests
- PR-8712 - KAFKA-10006: Don’t create internal topics when LeaderNotAvailableException
- PR-7496 - KAFKA-9018: Throw clearer exceptions on serialisation errors
- PR-8858 - KAFKA-10153: Error Reporting in Connect Documentation
- PR-8929 - KAFKA-4996: Fix findbugs multithreaded correctness warnings for streams
- PR-8927 - KAFKA-10200: Fix testability of PAPI with windowed stores
- PR-8683 - KAFKA-9893: Configurable TCP connection timeout and improve the initial metadata fetch
- PR-8925 - KAFKA-9974: Integration test shouldApplyUpdatesToStandbyStore; Make produce-sync flush
- PR-8917 - KAFKA-10180: Fix security_config caching.
- PR-8905 - KAFKA-10173: Fix suppress changelog binary schema compatibility
- PR-8926 - KAFKA-10166: always write checkpoint before closing an (initialized) task
- PR-7577 - KAFKA-9076: support consumer offset sync across clusters in MM 2.0
- PR-8924 - KAFKA-10198: guard against recycling dirty state
- PR-8900 - KAFKA-10169: swallow non-fatal KafkaException and don’t abort transaction during clean close
- PR-8887 - KAFKA-10135: Extract Task#executeAndMaybeSwallow to be a general utility function into TaskManager…
- PR-8845 - KAFKA-10126:Add a warning message for ConsumerPerformance
- PR-8174 - KAFKA-9439: add KafkaProducer API unit tests
- PR-8865 - KAFKA-10168: fix StreamsConfig parameter name variable
- PR-8896 - KAFKA-10185: Restoration info logging
- PR-8890 - KAFKA-9891: add integration tests for EOS and StandbyTask
- PR-8898 - KAFKA-10138: Prefer –bootstrap-server for reassign_partitions command in ducktape tests
- PR-8850 - KAFKA-10141: Add more detail to log segment delete messages
- PR-8822 - KAFKA-10113; Specify fetch offsets correctly in LogTruncationException
- PR-8876 - KAFKA-10167: use the admin client to read end-offset
- PR-8841 - KAFKA-10123 Fix incorrect value for AWAIT_RESET#hasPosition
- PR-8853 - KAFKA-10147 MockAdminClient#describeConfigs(Collection<ConfigResource…
- PR-8882 - KAFKA-10165: Remove Percentiles from e2e metrics
- PR-8856 - KAFKA-10150: task state transitions/management and committing cleanup
- PR-8880 - KAFKA-10169: Error message when transit to Aborting / AbortableError / FatalError
- PR-8872 - Fix log message for transition from standby to active
- PR-8873 - Avoid WARN log message when re-init from checkpoint skipped
- PR-8849 - KAFKA-10144: clean up corrupted standby tasks before attempting a commit
- PR-8857 - KAFKA-10157: Fix broken tests due to InterruptedException from FinalizedFeatureChangeListener
- PR-8312 - KAFKA-9432 automated protocol for DescribeConfigs
- PR-8764 - KAFKA-10049: Fixed FKJ bug where wrapped serdes are set incorrectly when using default StreamsConfig serdes
- PR-8680 - KAFKA-10027: Implement read path for feature versioning system (KIP-584)
- PR-8787 - KAFKA-10085: correctly compute lag for optimized source changelogs
- PR-8818 - KAFKA-10086: Integration test for ensuring warmups are effective
- PR-8069 - KAFKA-9374: Make connector interactions asynchronous
- PR-8829 - KAFKA-10115: Incorporate errors.tolerance with the Errant Record Reporter
- PR-8828 - KAFKA-9216: Enforce that Connect’s internal topics use compact cleanup policy
- PR-8455 - KAFKA-9845: Warn users about using config providers with plugin.path property
- PR-8502 - KAFKA-9066: Retain metrics for failed tasks
- PR-8847 - KAFKA-7833: Add missing test
- PR-8453 - KAFKA-9841: Revoke duplicate connectors and tasks when zombie workers return with an outdated assignment
- PR-8663 - KAFKA-9985: Sink connector may exhaust broker when writing in DLQ
- PR-8833 - KAFKA-9441: remove prepareClose() to simplify task management
- PR-8825 - KAFKA-7833: Add StateStore name conflict check in InternalTopologyBuilder
- PR-8630 - KAFKA-9969: Exclude ConnectorClientConfigRequest from class loading isolation
- PR-8685 - KAFKA-10014 Always try to close all channels in Selector#close
- PR-8775 - KAFKA-10079: improve thread-level stickiness
- PR-7384 - KAFKA-8938: Improve allocations during Struct validation in ConnectSchema
- PR-8827 - KAFKA-9849: Fix issue with worker.unsync.backoff.ms creating zombie workers when incremental cooperative rebalancing is used
- PR-8664 - KAFKA-9716: Clarify meaning of compression rate metrics
- PR-8805 - KAFKA-9848: Avoid triggering scheduled rebalance delay when task assignment fails but Connect workers remain in the group
- PR-8783 - KAFKA-10063 UnsupportedOperation when querying cleaner metrics after …
- PR-8803 - KAFKA-10102: update ProcessorTopology instead of rebuilding it
- PR-8676 - KAFKA-10005: Decouple RestoreListener from RestoreCallback
- PR-8807 - KAFKA-10106: log time taken to handle LeaderAndIsr request
- PR-8684 - KAFKA-10012 Reducing memory overhead associated with strings in Metri…
- PR-8270 - KAFKA-9216: Enforce internal config topic settings for Connect workers during startup
- PR-8820 - KAFKA-10097: Internalize checkpoint data
- PR-8804 - KAFKA-9851: Revoking Connect tasks due to connectivity issues should also clear the running assignment
- PR-8486 - KAFKA-9840: Skip End Offset validation when the leader epoch is not reliable
- PR-8797 - Fixing KAFKA-10094
- PR-8135 - KAFKA-9570: Define SSL configs in all worker config classes, not just distributed
- PR-8814 - KAFKA-10111: Make SinkTaskContext.errantRecordReporter() a default method
- PR-8811 - KAFKA-10110: Corrected potential NPE when null label value added to KafkaMetricsContext
- PR-8776 - KAFKA-9441: Improve Kafka Streams task management
- PR-8724 - KAFKA-10040; Make computing the PreferredReplicaImbalanceCount metric more efficient
- PR-8759 - KAFKA-10066: TestOutputTopic should pass record headers into deserializers
- PR-8755 - KAFKA-10069: Correctly remove user-defined “predicate” and “negate” configs from transformation properties
- PR-8311 - KAFKA-9434: automated protocol for alterReplicaLogDirs
- PR-8644 - KAFKA-9313: Set use_all_dns_ips as the new default for client.dns.lookup (KIP-602)
- PR-8784 - KAFKA-9788: Use distinct names for transaction and group load time se…
- PR-8793 - KAFKA-9514; The protocol generator generated useless condition when a field is made nullable and flexible version is used
- PR-8792 - KAFKA-10089 The stale ssl engine factory is not closed after reconfigure
- PR-8782 - KAFKA-10080; Fix race condition on txn completion which can cause duplicate appends
- PR-8785 - KAFKA-10084: Fix EosTestDriver end offset
- PR-8786 - KAFKA-10083: fix failed testReassignmentWithRandomSubscriptionsAndChanges tests
- PR-8737 - KAFKA-9945: TopicCommand should support –if-exists and –if-not-exists when –bootstrap-server is used
- PR-8695 - KAFKA-9320: Enable TLSv1.3 by default (KIP-573)
- PR-8777 - KAFKA-10082: Fix the failed testMultiConsumerStickyAssignment
- PR-8779 - [MINOR] Fixing spotbug fail - removing unused variable.
- PR-8774 - KAFKA-10081: Remove an unused local variable to pass spotbugsMain check
- PR-8706 - KAFKA-10030 allow fetching a key from a single partition
- PR-8760 - Kafka-10064 Add documentation for KIP-571
- PR-8668 - KAFKA-9987: optimize sticky assignment algorithm for same-subscription case
- PR-7956 - KAFKA-9392; Clarify deleteAcls javadoc and add test for create/delete timing
- PR-8769 - KAFKA-10074: Improve performance of matchingAcls
- PR-8767 - KAFKA-10060 GroupMetadataManager should not log if there are no offse…
- PR-8761 - Remove redundant code of KafkaProducer
- PR-8723 - KIP-569-KAFKA-9494: DescribeConfigsResponse - include additional metadata information
- PR-8739 - KAFKA-10056; Ensure consumer metadata contains new topics on subscription change
- PR-8705 - KAFKA-10029; Don’t update completedReceives when channels are closed to avoid ConcurrentModificationException
- PR-8238 - KAFKA-9130: KIP-518 Allow listing consumer groups per state
- PR-8248 - KAFKA-9501: convert between active and standby without closing stores
ksqlDB¶
- PR-6800 - fix: LDAP Authentication
- PR-6732 - fix: (minor) don’t use deprecated jersey calls
- PR-6702 - fix: use Java’s Base64 instead of jersey’s
- PR-6668 - fix: fix error categorization on NPE from streams (#6655)
- PR-6608 - fix: Check for index before removing value in undo of COLLECT_LIST (#…
- PR-6558 - fix: add back configs for setting TLS protocols and cipher suites
- PR-6551 - fix: avoid RUN SCRIPT to override CLI session variables/properties
- PR-6550 - fix: support joins on key formats with different default serde features
- PR-6541 - feat: CASE expressions can now handle 12+ conditions in docker + cloud cherry picking
- PR-6537 - fix: ‘-e/-f’ CLI parameters are not handling session variables correctly
- PR-6482 - fix: Internal Server Error for /healthcheck endpoint in RBAC-enabled
- PR-6467 - fix: check for nested UnspportedVersionException during auth op check
- PR-6400 - feat: Add support for ALTER STREAM|TABLE
- PR-6437 - fix: Properly clean up state when executing a command fails
- PR-6451 - fix: fixes performance regression in pull queries introduced by sessionConfig
- PR-6448 - feat: support non-primitive types as keys
- PR-6354 - feat: add aggregated rocksdb metrics
- PR-6446 - fix: support unwrapped struct value inference
- PR-6410 - fix: #6319 default port for CLI: set default port if not mentionned
- PR-6411 - feat: enable support for JSON key format
- PR-6393 - Join format mismatch
- PR-6353 - feat: Add an endpoint for returning the query limit configuration
- PR-6405 - feat: avoid spurious tombstones in table output
- PR-6397 - fix: recovery hangs when using TERMINATE ALL
- PR-6351 - refactor: split repartition node classes
- PR-6366 - Topic info cache
- PR-6370 - refactor: change QTT to store serialized data in mock Kafka
- PR-6365 - Query schemas
- PR-6346 - feat: add commandRunnerCheck to healthcheck detail
- PR-6348 - fix: backup files are re-created on every restart
- PR-6349 - feat: NONE format for key-less streams
- PR-6267 - feat: new CLI parameter to execute a command and quit (without CLI interaction)
- PR-6344 - feat: support for DELIMITED key format
- PR-6292 - Json keys
- PR-6149 - feat: Support Comparisons on complex types
- PR-6148 - feat: Add metrics for pull query request/response size in bytes
- PR-6295 - fix: JSON format to set correct scale of decimals
- PR-6278 - fix: don’t use queryId of last terminate command after restore
- PR-6253 - feat: add KSQL processing log message on uncaught streams exceptions
- PR-6240 - feat: surface error to user when command topic deleted while server running
- PR-6166 - bug: Improved error message for reserved keywords
- PR-6238 - fix: delimited format should write decimals in a format it can read
- PR-6164 - feat: CommandRunner enters degraded state when corruption detected in metastore
- PR-6194 - feat: add KEY_FORMAT and FORMAT WITH clause properties
- PR-6251 - revert: “fix: improve join predictability by increasing max task idle …
- PR-6236 - fix: scale of ROUND() return value
- PR-6245 - fix: format cast arguments with passed context
- PR-6239 - fix: improve error handling of invalid Avro identifier
- PR-6201 - feat: substantially improve avro deserialization performance
- PR-6188 - fix: register correct unwrapped schema
- PR-6173 - feat: Support IF NOT EXISTS on CREATE TYPE
- PR-6182 - fix: fail on non-string MAP keys
- PR-6036 - feat: Support for IF EXISTS on CREATE CONNECTOR
- PR-6160 - fix: delete zombie consumer groups 🧟
- PR-6172 - fix: missing topic classifier now uses MissingSourceTopicException
- PR-6163 - fix: allow expressions in flat map
- PR-6109 - feat: clarify key or value in (de)serialization processing log messages
- PR-6118 - feat: Adds support for 0x, X’…’, x’…’ type hex strings in udf:encode
- PR-6135 - fix: wrong value schema being registered for unwrapped primitives
- PR-6096 - feat: support PARTITION BY NULL for creating keyless stream
- PR-6039 - feat: add Ksql warning to KsqlResource response when CommandRunner degraded
- PR-6066 - fix: Use a SandboxedPersistentQueryMetadata to not interact with KafkStreams
- PR-6084 - feat: add serialization exceptions to processing logger
- PR-6085 - fix: properly handle restore on CREATE OR REPLACE
- PR-6067 - feat: Support IF EXISTS keyword on DROP CONNECTOR
- PR-6040 - feat: add a serverStatus to ServerInfo and display the status in the CLI
- PR-5983 - fix: Uses pull query metrics for all paths, not just /query
- PR-6068 - fix: dont cleanup external resources on CREATE OR REPLACE
- PR-6054 - Single value wrapping
- PR-5998 - feat: Support subscript and nested functions in grouping queries
- PR-6032 - feat: CommandRunner enters degraded states when it processes command with higher version than it supports
- PR-6061 - feat: hard delete schemas for push queries
- PR-6031 - feat: DistributingExecutor fails DDL statement if CommandRunner DEGRADED
- PR-6012 - feat: move command topic deserialization to CommandRunner and introduce DEGRADED CommandRunnerStatus
- PR-6048 - fix: remove unnecessary parser token (MINOR)
- PR-6005 - fix: allow implicit cast of numbers literals to decimals on insert/select
- PR-6033 - fix: protobuf format does not support unwrapping
- PR-5944 - feat(client): support describe source in Java client
- PR-6018 - fix: set restarted query as healthy during a time threshold
- PR-5984 - refactor: Call the correct pull/push query validator
- PR-5982 - feat: support UDAFs with and without init Args with same param type
- PR-5979 - fix: replace kafka-streams deprecated code
- PR-5960 - fix: NPE when udf metrics enabled
- PR-5969 - feat: add config for suppress buffer
- PR-5962 - feat: Support [IF EXISTS] on DROP TYPE command
- PR-5476 - feat: Add consumer offsets to DESCRIBE EXTENDED
- PR-5877 - feat: New ksql.properties.overrides.denylist to deny clients configs overrides
- PR-5884 - feat: add suppress functionality
- PR-5807 - feat: add service to restart failed persistent queries
- PR-5594 - fix: improve join predictability by increasing max task idle
- PR-5883 - fix: close query on invalid use of HTTP/2 with /query endpoint
- PR-5874 - fix: change default grace period to zero
- PR-5879 - fix: create the metastore backups directory if it does not exist
- PR-5849 - feat: Enable datagen to set the message timestamp
- PR-5779 - feat: add suppress node
- PR-5843 - feat: new array_remove udf
- PR-5831 - feat: Replay command topic to local file to backup KSQL Metastore
- PR-5835 - fix: configure topic retention based on retention clause for windowed tables
- PR-5823 - fix: always use the changelog subject in table state stores
- PR-5816 - feat(client): return query ID when persistent query started via Java client
- PR-5814 - feat: expose query ID in CommandStatusEntity (MINOR)
- PR-5775 - feat(client): support DDL/DML statements in Java client
- PR-5813 - feat: organize UDFs by category
- PR-5818 - fix: Allow backported ALPN to be used and warn if no OpenSSL
- PR-5769 - fix: ensure null values cast to varchar/string remain null
- PR-5743 - fix: windowed tables now have cleanup policy compact+delete
- PR-5781 - fix: circumvent KAFKA-10179 by forcing changelog topics for tables
- PR-5744 - fix: Make sure UDTF describe shows actual function description
- PR-5739 - feat: add syntax for suppression
- PR-5766 - feat: support CREATE OR REPLACE w/ config guard but w/o restrictions
- PR-5742 - fix: Reuse KsqlClient instance for inter node requests
- PR-5763 - fix: ksqlDB should not truncate decimals
- PR-5750 - fix: show overridden props in CLI
- PR-5679 - refactor: Convert remaining Kudfs
- PR-5730 - fix: make sure old query stream doesn’t block on close
- PR-5641 - feat(client): support streaming inserts in Java client
- PR-5682 - feat(client): support list queries in Java client
- PR-5680 - feat: support WINDOWEND in WHERE of pull queries
- PR-5671 - feat(client): support admin operations in Java client
- PR-5672 - fix: simplify pull query error message
- PR-5644 - fix: support GROUP BY with no source columns used
- PR-5645 - fix: ensure only deserializable cmds are written to command topic
- PR-5482 - feat: Adds SSL mutual auth support to intra-cluster requests
- PR-5474 - feat: implements ARRAY_JOIN as requested in (#5028)
- PR-5563 - feat: new split_to_map udf
- PR-5559 - feat: add CHR UDF
- PR-5597 - fix: fail AVRO/Protobuf/JSON Schema statements if SR is missing
- PR-5573 - Stream with no key should not fail if key has data
REST Proxy¶
- PR-773 - KREST-287: Syncronize resource pools/managers lazy instantiation.
- PR-756 - KREST-26: Add a blocklist config to allow disabling individual endpoints.
- PR-761 - Add toBuilder method to Cluster
- PR-755 - Update JUnit dependency to JUnit 5.
- PR-216 - MMA-7520: add CsrfTokenProtectionFilter
- PR-209 - (backport) SEC-592: Customize Jetty request queue and threadpool size (#172)
- PR-215 - ff-3928: bump version of Guava in parent of schema-registry.
- PR-210 - ST-4542: Upgrade jetty to address CVE-2020-27216
- PR-205 - CIAM-261: Add Jetty ThreadPool Metrics
- PR-194 - Fix build failure
Schema Registry¶
- PR-1747 - DG-1189 Add latest.compatibility.strict flag
- PR-1746 - DG-931: throw ConfigException in Avro/Json/Protobuf Converter
- PR-1724 - DG-1124 NPE in ProtobufMessageFormatter
- PR-1722 - DG-1122 Treat reordering Protobuf msgs as incompatible
- PR-1711 - add testUsingSubjectNameStrategy()
- PR-1709 - QEC-2994: fix bug in getting subject name for value
- PR-1700 - DG-293: Improve exception messages in JsonSchemaConverter and ProtobufConverter
- PR-1699 - DG-1040 Register ids for soft deletes
- PR-1688 - DG-1007 Preserve refs when converting schema to envelope
- PR-1687 - DG-924 Disable default validation by default
- PR-1686 - MMA-8608 Handle duplicate configs after stripping prefix
- PR-1684 - MMA-8608: parse configs before calling restService.configure()
- PR-1678 - DG-990 Allow SpecificAvroSerde to work with array/map
- PR-1676 - DG-972 Fix name collisions with field and message
- PR-1675 - DG-962 Handle const schemas in JSON Schema converter
- PR-1668 - Removed property ${confluent.maven.repo}
- PR-1658 - DG-951 Handle duplicate message names during deserialization
- PR-1654 - Adding an NPE check for permanent delete
- PR-1648 - DG-935 Treat removing Protobuf message types as incompatible
- PR-1642 - Avro Console producer should support subject name strategy option
- PR-1647 - DG-933 Resend old value when handling invalid key-value
- PR-1638 - DG-924 Enable Avro default validation for new schemas
- PR-1634 - DG-921 Fix Avro specific reader for unions
- PR-1625 - DG-410 Fix doc/defaults in Avro converter
- PR-1614 - Add failing compatibility reason in rest response
- PR-1621 - DG-848 Enable mode mutability by default, fixes #1610
- PR-1591 - Add offset checkpoints for persistent stores
- PR-1571 - Add setTenant
- PR-1559 - Add API to get subjects by predicate
- PR-1544 - DG-616 Permanent Delete support in SR client
- PR-1533 - DG-534 Enhance compat endpoint to validate schema
- PR-1532 - DG-540 Add batch get schema API
- PR-1527 - NetworkClient fix for upstream Kafka changes
- PR-1519 - #1515 - Cache Avro DatumReader and DatumWriter instances to increase performance
- PR-1498 - update README to include protobuf, json schema
- PR-1491 - Revert “Add node information to SchemaRegistryProtocol (#1485)”
Commercial Features¶
Confluent Server¶
- KSTORAGE-1348 Flush epoch cache before reinitialization in Log.renameDir
- SEC-1805: Reduce failed MDS request send log threshold from ERROR to Debug
- Fix test failure in test_kibosh_service (CPKAFKA-6180)
- SEC-1802 Check for topic creation
- SEC-1802 Don’t block TopicManager creation on describeTopics
- SEC-1770 NonBlockingKafkaExporter should notice that topics exist
- KSTORAGE-1178: Moved missing tier state file recovery code under assignTopicId (#2702)
- KSQL-5843: prevent log spam from MDS RestClient on client close
- Add back specific log message for unclean recovery test (#2938)
- SEC-1789 Bump Bouncy Castle dep to resolve CVE-2020-28052
- AUDIT-249– Added mime type “application/protobuf” supporting to BinaryDeserializer
- CPKAFKA-6413: test_offset_migration_single_group_offset_not_exists_on_dest
- CPKAFKA-6375: Decode python3 bytes object in tier fence restore test
- CPKAFKA-6374: Handle None in tier roundtrip system test
- CPKAFKA-6370: Use python3 range (versus xrange) to have BrokerStorageFailureTest work with python3
- CPKAFKA-6371: Use python3 sys.maxsize in tiered storage system tests
- KC-1275: Update KafkaReader to handle entries from multiple partitions
- CPKAFKA-6373: Fix deprecated has_key python2 method
- CPKAFKA-6376: Use range over xrange to fix TierMetadataCorruptionTest to work with python3
- KSTORAGE-1290: Bump google storage client dependency to 1.113.4
- CPKAFKA-6372: Fix bad assertion in cluster link test
- CPKAFKA-3011: fix timing issue with onConfigChange and ConfluentLeaderAndIsrRequest
- Backport of CPKAFKA-6268
- CNK-1431: Add confluent.ansible.managed property
- KC-1228 cherry pick for 6.1.x
- CPKAFKA-6018: Fix cluster link test timing issue with metrics and replica status
- KSTORAGE-1260: tiered segments do not correctly block retention for local segments (#2831)
- CNK-1325: Add a metric for initiated executions.
- CNK-819: Don’t self-heal if a rebalance doesn’t improve detection goals.
- Port fix for KSTORAGE-1230 to 6.1.x
- CPKAFKA-6126: Update commons codec (#2785)
- CNK-1210: Introduce kafka-add-brokers CLI
- CPKAFKA-6233: Update lastUpdateTimeMs before updating the current broker addition state
- CNK-1396: Register BrokerAddCount metric in CDBE
- CNK-617: Temporary pull out CloudNetworkFault + deps
- CNK-1349: Switch broker removal state from PLAN_EXECUTION_INITIATED to PLAN_COMPUTATION_INITIATED on failover
- KC-1226: Enable confluent.cluster.link.io.max.bytes.per.second to be configured at cluster-level
- KC-1225: Fix NPE on broker restart with paused links, fix blocked state
- CNK-1228: Update SBK add_broker system tests to utilize kafka-add-brokers for checking operation completion
- KC-1027 backport into 6.1.x
- Backport Update system test to use –comparator.json (#2743)
- CNK-1332: Handle TopicExistsException when creating SBK topics
- KC-1203: Stop mirroring if destination partition is under-min-isr, metrics changes
- KC-1195: Ensure connections are closed if principal cannot be built
- backport METRICS-2459: update default include list for hosted monitoring
- backport METRICS-2396: Add extra checks to ensure that we do not leak EventExporter objects
- backport METRICS-2110: document Telemetry Reporter packaging (#2710)
- backport METRICS-1805: remove duplicate resource type label (#2744, #2755)
- Add unit tests to disable tier storage at broker level
- KSTORAGE-1180 Update tier enable flag at FileTierPartitionState unconditionally
- KC-1183: Fix exception in writer metric for non-MDS brokers
- KC-1025 automatic observer promotion/demotion in MRC (#2649) cherry-pick
- CNK-1304: Update remove broker validation to use strict collections
- CNK-1331: Fix typo in SBK Persistence Store initialization
- CNK-1209: Describe AddBroker functionality and integration tests
- KC-1132: Add cluster link request metrics for source cluster
- CPKAFKA-6030: Fix broker removal system test
- CNK-1290: Update the sub task status field in BrokerAdditionDescription and BrokerRemovalDescription and CNK-1288 - add description() method to DescribeBrokerRemovalsResult for feature parity
- CNK-1233: Adds support for logical cluster type link-healthcheck.
- CNK-1208: Return a high level balancer operation status in describe broker removals
- KSTORAGE-1075: Add debug logging when PendingFetch completes
- [METRICS-2356] patch version for thread leak fix
- Increase test tolerance in non deterministic test
- CC-12141: Confluent’s WorkerConfigDecorator in Connect should handle connector validate responses with extra properties
- KSTORAGE-1004: Restore tool should be able to work standalone without comparator tool
- KC-1149: Ensure stopped mirrors are not treated as read-only partitions
- CNK-1012: Add broker state machine
- CNKAF-1236: Add startTimeMs and lastUpdateTimeMs to the describe broker removal RPC
- KC-1148: Fixing metric conflict issues where link can’t be recreated due to metric conflict issues
- Add config option to override schema validator interceptor class
- SEC-1477 Add trace logging on LDAP search success
- CCS -> CE sync upstream 28 sept 2020
- CONFLUENT: Add config for the tests that rely on MockAuditLogProvider.
- KSTORAGE-1176: chmod -R ag+w in ccloud startup script increases startup time
- CPKAFKA-4479: Make operationally important Kafka configs dynamic (fetch.max.bytes)
- KC-907: Update to cc-base 7.4.0 and remove prometheus and async profiler downloads
- CNK-1126: Request quota auto-tuning sorts by an unstable value
- CNK-374: Return task history for only configured retention period
- ce-metrics: add space to reconcileConfigs() log message
- KC-375: Include cluster-link-id in requests from destination brokers to source
- METRICS-2358 : only initialize event logger when enabled, properly close it, and add better support for reconfiguration
- CNKAF-1134: Close AdminClient, Producers and Consumers immediately
- KC-1120: Ensure that RBAC metadata write failures are reset after writer elections
- CNKAF-856: DescribeBrokerAdditions API - RPC and Admin API skeleton
- KC-938: Add config for dedicated-cluster topics for cluster-linking soak setup.
- KC-955: Adds describeMirrors API.
- CNKAF-1068: Add validation for broker removal for rf equal to alive brokers
- CNKAF-1235: Revise state machine for broker removal
- CIAM-349 CloudClusterAdmin can see Schema Registry
- KSTORAGE-1100: Offset cache can’t be disabled although we have the facility to do so
- KSTORAGE-986: Check for invalid topic partitions within the recovery tool
- CNK-1206: Update dependencies for SBK serializers
- CNK-42: Only enable Databalancer Count metric if balancer is enabled
- CNKAF-1109: Use AdminClient for Leader Elections, handle exceptions
- KSTORAGE-1160: avoid unnecessary maxCompactionDelay calculations when now < max.compaction.lag.ms
- KC-899: Monitor auto-topic creation and compaction
- Propagate CC release branch to ce-kafka-images
- CNKAF-1001: Retry plan computation in Broker Removal
- KSTORAGE-1150: Allow customers to change log.cleaner.max.compaction.lag.ms cluster default
- CIAM-371 Add Delete operation to Cluster for CloudClusterAdmin
- CNKAF-1174: improve fetch session cache performance
- KC-1017: Extends replica status to include mirror information.
- KC-44: Maintain topics per tenant in metadata cache
- KC-1021: Add RequestConvertToJson tests
- KSTORAGE-797: allow recovery path materializer to use secured listeners
- CC-11832: add connector name to metrics context when calling Kafka Cl…
- CNKAF-1062: Add validation for broker removal leading to under min isr
- CNKAF-1224: Log all uncaught SBK exception and errors
- CNKAF-920: Raise invalid broker removal exception with specific error message
- KC-1019: ScramUserCredentials matches missing in RequestConvertToJson
- [CNKAF-1195] Avoid stale cluster metadata for AnomalyDetection plans.
- CIAM-321 Add Audit Resource Type for OrgAdmin role
- Configure ListenerConnectionQuota on broker startup
- Suspending rbac system tests till we fix hang issue
- local start offset check is done outside committed offset check block.
- KSTORAGE-785: allow CCloud users to set max.compaction.lag.ms
- CNKAF-915 Improve log level in executor
- Durability Audit fixes
- KSTORAGE-1013: add broker internal log dir storage probe
- CNKAF-1075: Improve Confluent SASL secrets caching
- KC-190: add metric to count responding to disconnected clients
- cherry-pick from AK - Fix that support, resolve of “kafka.apache.org ” to 3 IP addresses (#9294)
- Reverted changes in TestUtils which added additional parameter for enabling durability audit.
- Fixed unit test in AuditManager
- KSTORAGE-1023: Durability audit follow up
- CNKAF-765; Update the TenantQuotaCallback to support Controller Mutation Quota
- CPKAFKA-6013: Skip tier state presence validation for deleted partitions
- KSTORAGE-972: Modify TopicIdPartition.toString() implementation to use ‘:’ and ‘-‘ as delimiters
- KSTORAGE-885: add lagging and realtime consumer to retention ms/bytes system test
- KC-883: Update soak-cluster configuration for source and destination.
- Migrated AuditJob away from KafkaScheduler
- KSTORAGE-973: Check for invalid topic partitions within the fencing tool
- KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline
- KSTORAGE-725: Exclude segments where the high offset is lower than the lowerBoundOffset for tiering.
- KSTORAGE-1074: retry should log at INFO level
- KSTORAGE-1082: tier replication causes incorrect HWM after restore (#2507)
- KSTORAGE-1075: Add more debug info for consumption tracking
- feat: Add load metrics for logical cluster metadata
- KSTORAGE-1086: Better validations to check if tiering is disabled
- CNKAF-1138: Emit JSONs with auto-generated schema
- CNKAF-546: Reinitialize expired partition detail sensors
- KSTORAGE-909: Follow-up PR to durability audit PR
- KSTORAGE-1054: add another Azure auth method via config file
- KC–603: Fix NPE in KafkaBasedLog during broker shutdown if license topic creation fails
- KC-902: Fix cluster link authorization, add end-to-end authorization tests
- KSTORAGE-970: Fix help screen in TierTopic tooling
- KSTORAGE-1084: use file name convention for tier_unclean_leader_election_test.py
- CPKAFKA-5114: Add group size metrics
- Sync upstream august 18 2020
- KSTORAGE-1082: tier replication causes incorrect HWM after restore
- CNKAF-1169: Correctly handle single racks and no racks when creating topics in the MT interceptor
- CPKAFKA-5114: Add group size metrics
- Update mk-include
- KSTORAGE-887: Integrate TierFetcherBenchmark into compatibility checker
- KSTORAGE-887: Make TierFetcherBenchmark configurable for Pure Storage testing
- CNKAF-1087: Fix rack aware topic placement for cloud use case
- KC-854: Allow socket buffer size configurable for high latency cluster links on Cloud
- KC-801: Reassignment succeeds with empty constraints
- [CNKAF-1144] Confluent.operator.managed property (#2484)
- [Backport]METRICS-2089 Adding the correct NOTICE / COPYRIGHT for the confluent-telemetry package
- KSTORAGE-1042: Harden compatibility framework #1
- METRICS-2089 Adding the correct NOTICE / COPYRIGHT for the confluent-telemetry package
- KSTORAGE-1067: Dump fetch context when tiered storage runs into an unexpected error
- [cnkaf-1144] Confluent.operator.managed property
- Durability Audit - Audit Manager, Materialization and Reporting and Event generation
- CNKAF-902: Add metrics to track SSL handshake failures and avg/max time
- switch to using DefaultAzureCredentials for Azure authentication
- CNKAF-108: Improve log message for dynamic certificate update
- KC-166: Add metric for controller loading time
- KSTORAGE-974: Print tiertopic partition and timestamp for fencing tool
- KSTORAGE-859: MergedLog.createUploadableSegment deleted segment check
- CNKAF-758:Add cpu load metric
- KSTORAGE-1040: Better checks to prevent tiering from being disabled dynamically
- METRICS-2159: use a loading cache to avoid expensive pattern matching calls
- METRICS-2160: optimize local exporter regex
- METRICS-2159: use a loading cache to avoid expensive pattern matching calls
- KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale.
- avoid calls to gettimeofday on every metric key in telemetry
- KC-102 Add cc-zookeeper docker build
- CPKAFKA-5882. Add copyDependantLibs dependency to ce-sbk and ce-broker-plugins module
- [CNKAF-1112 WORKAROUND] Do not allow enabling of strict quotas.
- feat: Cache auth failures
- SEC-1016: Changes to NonBlockingKafkaExporter
- CONFLUENT: Upgrade netty-tcnative to 2.0.34.Final
- CONFLUENT: Change visibility of ConfluentProvider.createRbacAuthorizer() to public
- KC-845: Ensure LogConfig includes broker defaults to pass schema config validation
- CPKAFKA-5873 Don’t load Netty classes when using default SSL engine
- CONFLUENT: Remove shading from ce-broker-plugins and ce-sbk
- [KSTORAGE-829] Storage Fault Injection: Integrate Kibosh with Ducktape
- CPKAFKA-5851: Align joda-time version with common
- KSTORAGE-1000: TierDeletedPartitionsCoordinator Handling of Emigration During Read
- CNKAF-863: Only showed broker removal status for given broker.
- KC-613: Use kafka-rest-lib instead of kafka-rest on classpath.
- KC-613: Use kafka-rest-lib instead of kafka-rest on classpath.
- Update compatibility checker README.md
- SEC-1373 EmbeddedKafkaCluster sets Audit Log replication factor to 1
- CPKAFKA-5849: Don’t log password configs in request log filter
- CONFLUENT: Remove unused jacksonDatabind version definition
- CPKAFKA-5795: Change guava dependency to 28.1-jre
- CPKAFKA-5796: Remove ZK dependency from ce-licensing
- CPKAFKA-5835: Fix regression in cluster link compatibility test
- Add secrets to default RBAC policies
- KC-810: Add cluster link destination tenant metrics suitable for billing
- METRICS-2108 Telemetry-client dependency version update from 1.174.0 to 1.192.0 to update default base URL for TelemetryHttpClient
- KC-770: Support backpressure for cluster link when disk usage is high
- METRICS-2108 Telemetry-client dependency version update from 1.174.0 to 1.192.0 to update default base URL for TelemetryHttpClient
- KSTORAGE-996 Fix logic to check bytes fetched from local log
- [CNKAF-1089][RM-32] Remove SBK dependencies on grpc-stub. (#2408)
- KSTORAGE-991 Implementing Additional Config Validations
- KSTORAGE-334, KSTORAGE-893, KSTORAGE-894: DeleteRecords, retention time and bytes system tests
- backport METRICS-2098 exclude zstd/lz4 from telemetry-clients to avoid conflicting versions
- KSTORAGE-994 Improved Configuration Documentation
- [CNKAF-1089][RM-32] Remove SBK dependencies on grpc-stub.
- Redo SEC-866
- CNKAF-655: SBK Soak cluster workload
- METRICS-2090: Remove duplicate labels on metrics
- METRICS-2097: Don’t capture ‘total_size` and ‘tier_size` metrics
- METRICS-2098 exclude zstd / lz4 from telemetry-clients to avoid conflicting versions
- KC-637 cherry pick to 6.0.x
- KC-637 Offset migration system tests.
- KSTORAGE-957: Only set SSEAwsKeyManagementParams if a customer encryption key is supplied and sseAlgorithm is aws:kms. (#2353)
- KSTORAGE-886 [MINOR] Move scripts from bin to compatibility checker directory
- SEC-866-6.0.x again
- KSTORAGE-886: Implementing basic tiering load test utils
- CONFLUENT: Remove the soak clients status reporter cronjob
- KSTORAGE-957: Only set SSEAwsKeyManagementParams if a customer encryption key is supplied and sseAlgorithm is aws:kms.
- KSTORAGE-506: Prevent accidental change to critical configurations for tier topic
- CIAM-227: Add requestor principal to AuthWriter interface methods
- METRICS-2031 Prune the hardcoded Kafka broker default include list to avoid collecting high cardinality metrics.
- METRICS-2048 Sanitize ksql service ids from their metric name to make them more practical to use. Also ignore legacy metrics.
- [CNKAF-1038] SBK Rebalance Cluster Test: add rolling bounces (#2326)
- METRICS-2031 Prune the hardcoded Kafka broker default include list to avoid collecting high cardinality metrics.
- CNKAF-1058; Ensure that Control Request Batching does not send LeaderAndIsrRequest with yet unknown broker(s)
- CNKAF-896: Avoid expensive calls to MetadataCache#getNonExistingTopics
- KSTORAGE-947: batch tier segment deletion delays to improve deletion SLA (#2309)
- METRICS-2048 Sanitize ksql service ids from their metric name to make them more practical to use. Also ignore legacy metrics.
- KC-238: add max rebalance time gauge
- CNKAF-1028:Passed SNI host name to OAuthBearerValidatorCallbackHandler
- CNKAF-1039: Update error for insufficient metrics to compute plan
- backport METRICS-2010: create a new ‘releaseTarGz’ task for ce-metrics (#2282)
- METRICS-2010: create a new ‘releaseTarGz’ task for ce-metrics
- CPKAFKA-2864: Update base docker image for new tooling
- CONFLUENT: Allow both AK and Confluent class for system test process
- CPKAFKA-5762: Fix topic deletion in system tests
- CNKAF-1039: Update error for insufficient metrics to compute plan
- KC-671: SBK system test with CL
- METRICS-1988: add confluent-telemetry directory to kafka classpath
- CONFLUENT: Bump cc-trogdor chart docker image to cc-trogdor:v0.207.0-5.3.0-ce-SNAPSHOT
- KAFKA-10371; Partition reassignments can result in crashed ReplicaFetcherThreads
- [Cherry 2367] [SEC-1345] Read HTTP Reason phrase when connection closed
- CC-4539: Use plugin.path mechanism to load rest servlet initializers in Connect
- Revert “SEC-866 - Migrate audit log “other” category into “management” category. (#2330)”
- [SEC-1345] Read HTTP Reason phrase when connection closed
- backport METRICS-2037 remove non-relocated dependencies from confluent-metrics fat jar (#2332)
- [CLIENTS-1633] Schedule MDS url refresher asynchronously
- Increase root FS to 20GB
- METRICS-1763, METRICS-1767 Configure telemetry metrics whitelist for Connect and MQTT Proxy Proactive Support (#2241)
- SEC-866 - Migrate audit log “other” category into “management” category.
- METRICS-1763,METRICS-1767 Configure telemetry metrics include list for Connect and MQTT Proxy Proactive Support
- SEC-866 - Migrate audit log “other” category into “management” category.
- KC-391: Source topic deletion system test for CL
- METRICS-2037 remove non-relocated dependencies from confluent-metrics fat jar
- KSTORAGE-947: batch tier segment deletion delays to improve deletion SLA
- METRICS-2056 Update api.secret datatype from String to password in event’s HttpExporterConfig.
- KC-727: Prevent source topic config mirroring from violating alter config policies
- [CLIENTS-1633] Authorizer RestClient should fail updating active node…
- KAFKA-10282; Remove Log metrics immediately when deleting log (#2342)
- [CNKAF-1038] SBK Rebalance Cluster Test: add rolling bounces
- CNKAF-1036: Fix issue where dynamic throttle applies for the current rebalance only - cherry pick to 6.0
- Rebalance Cluster System Test
- KAFKA-10282: Remove Log metrics immediately when deleting log
- CNKAF-1008: Add Broker/Remove Broker proposal logging
- CNKAF-1036: Fix issue where dynamic throttle applies for the current rebalance only
- KC-633: Auth failures system test for Cluster Linking
- Add CloudNetworkFault to Trogdor
- [CHERRY-2308] Move kafka-http-server project into clients project
- METRICS-2007 Updating Telemetry Reporter to exclude exporter configs or disabled exporters with no type set.
- Move kafka-http-server project into clients project
- KC-638: System tests with tier storage enabled on destination
- KC-754: Cluster link compatibility system tests
- KC-631 Good path cluster linking test using SASL_SSL
- backport METRICS-1992: Fix schema registry cluster id resource label (#2285)
- SBK: System test for cluster rebalancing
- METRICS-1760 Configuring additional Kafka Broker include list metrics for Proactive Support.
- KC-749: initial single-topic setup for cluster-linking soak clusters
- KC-755: Fix NoSuchElementException warnings for consumer groups with no new offsets
- CONFLUENT: Fix broken cluster link system tests in 6.0.x
- CNKAF-481: Oauth validation of SNI HostName
- METRICS-1950 Adding integration test to verify proxy configurations are propagated correctly.
- CNKAF-481: Passed down SNI hostname and enable parsing in transport layer.
- CNKAF-1008: Log full proposal dump for add/remove
- METRICS-1992: Follow up comment based on review
- CNKAF-1029: Ensure Executor state is reset after unexpected exceptions
- KC-735: Improve cluster link addition.
- KSTORAGE 806: Tiering Functionality Tests
- KC-726: Cluster link config support for CCloud.
- CNKAF-1035: Roll cluster while a new broker is getting added
- clean up auto aborting input stream
- KSTORAGE-888: Add tier fetches metrics for tiered storage
- METRICS-1998: add rbac metrics to default metrics list (#2269)
- METRICS-1992: Fix schema registry cluster id resource label
- backport METRICS-1993: add connect.cluster.id label (#2264)
- Updated default topic replicas factor to 3 for audit log.
- KC-741: Don’t include passwords in cluster link request logs
- KC-736: Allow deletion of cluster links if mirrored topics are marked for deletion
- KSTORAGE-896: fix tier deletion manager system test (#2256)
- CNKAF-822: Cherry pick add broker system test to 6.0.x
- KC-725: Backport of KC-725 to 6.0.x
- METRICS-1993: add connect.cluster.id to telemetry resource labels
- CNKAF-822: Adds an SBK AddBroker system test
- METRICS-1998: add rbac metrics to default metrics list
- KC-725: Fixed issue where metrics were not cleaned up when cluster links were deleted
- KSTORAGE-823: Log.maybeIncrementLogStartOffset should not flush leader epoch cache
- backport METRICS-1986 Fix to give higher precedence to “confluent.telemetry.metrics.collector.whitelist” over default provider whitelist. (#2266)
- backport METRICS-1985 Add missing metrics to the _local exporter for SBK feature to work as expected.
- CONFLUENT: Make RbacClusters.java to work with Writer/AuthWriter interfaces
- METRICS-1985 Add missing metrics to the _local exporter for SBK feature to work as expected.
- KC-728: Return INVALID_CONFIG error for invalid link configs, log unexpected errors
- METRICS-1986 Fix to give higher precedence to “confluent.telemetry.metrics.collector.whitelist” over default provider whitelist.
- KSTORAGE-920: TierTopicConsumer primary consumer long poll throttles catchup (#2253)
- KSTORAGE-864 Fetch requests to return NotLeaderForPartition during ULE recovery
- CNKAF-819: Test SBK broker removal with roll
- METRICS-1762 Configure telemetry metrics whitelist for KSQL Proactive Support
- KSTORAGE-896: fix tier deletion manager system test
- KSTORAGE-920: TierTopicConsumer primary consumer long poll throttles catchup
- CIAM-12: update mk-include
- METRICS-1762 Configure telemetry metrics whitelist for KSQL Proactive Support
- Backporting fix for METRICS-1764 and METRICS-1766
- update kafkatest template for balancer replication factor (for < 3-node clusters)
- CNKAF-1015: Fix issue in SBK startup by moving CruiseControl instantiation after the startup components
- KSTORAGE-876: Adding Tiered Storage TotalSizeAggregate Metric
- KC-638: Cluster linking system test with tiered storage enabled in source cluster
- KSTORAGE-890: Initial commit with the AuditJob infrastructure
- CONFLUENT: Fix reassign partitions system tests
- METRICS-1764 Configure telemetry metrics whitelist for Schema Registry Proactive Support
- METRICS-1766 Configure telemetry metrics whitelist for Rest proxy Proactive Support
- CNKAF-825: Introduce a system test for replica placement constraints
- CNKAF-1014: Rename usages of SBK in logs to DataBalancer
- CNKAF-613: Introduce a broker removal system test
- METRICS-1894:Telemetry Reporter is missing labels configured in prope…
- METRICS-1817: add include regex default value to Provider interface
- METRICS-1938: upgrade jose4j version (#2219)–backport
- METRICS-1929: use advertised listeners/host-port when deriving local producer config
- METRICS-1938: upgrade jose4j version
- [CNKAF-963] Switch AddBroker plan computation to wait METADATA_TTL.
- CNKAF-954: Removes GoalOptimizer proposal caching
- KSTORAGE-892 Re add couple of lines of ULE code that got deleted duri…
- Backporting config/events Telemetry Reporter changes along with all shading issues.
- Added one more toJson method in protoserde for webhook alert
- (backport) MMA-8432: fix wrong package name
- MMA-8432: fix wrong package name
- KC-428: Always log authorize results to file based on log4j settings
- KAFKA-10279; Allow dynamic update of certificates with additional SubjectAltNames
- CNKAF-1004: Don’t describe removed broker configs
- METRICS-1894:Telemetry Reporter is missing labels configured in properties file
- KC-714: SslTransportLayer write should invoke SSLEngine.wrap once with all the buffers
- CNKAF-1004: Don’t describe removed broker configs
- CNKAF-762: Abstract CruiseControl startup and config translation logic into own package
- METRICS-1895: Update telemetry client version
- KC-705: Retry metadata topic consumer.position() until authorizer startup times out
- METRICS-1918: bump gson from 2.8.5 -> 2.8.6
- METRICS-1895: upgrade telemetry client version
- (backport) prevent ‘io.confluent.controlcenter’ from being renamed by shadow plugin
- KSTORAGE-526 Recover from corrupt or missing TierState file during bootup
- KSTORAGE-822: Expose Tiered Storage configs for GA in 6.0
- KSTORAGE-822: Expose Tiered Storage configs for GA in 6.0
- (Backport) include UnderMinIsrPartitionCount in the default broker metrics list
- Add UnderMinIsrPartitionCount to default broker metrics list
- CNKAF-481: Parse SNI in sslTransportLayer
- CPKAFKA-2852: CCloud metadata listener thread should not die on unexpected exception
- METRICS-1875 fix shading issues with cloudevent proto generated classes
- SEC-1276 Allow empty resourceNames in CRNs
- MINOR: Improve AuthenticationException logging
- SEC-384: Use the correct log level for debug output
- KC-634: Cluster link system tests with broker failures
- CIAM-110 Associate RBAC Roles with organizations
- METRICS-1875: Shade generated ce-events protobuf
- prevent ‘io.confluent.controlcenter’ from being renamed by shadow plugin
- CNKAF-985: Bump leader distribution goal priority
- KSTORAGE-500 Remove TierTopicAppender.becomeArchiver from the interface and inline the implementation
- KC-614: Check that topic name is not empty before tenant prefixing
- CNKAF-964: Shade opencensus protobufs into SBK jar (#2155)
- CONFLUENT: Update log statements in alterBrokerConfigs/alterTopicConfigs methods
- SBK: cherry pick javadoc updates for removeBroker
- METRICS-1878: Shade telemetry client classes
- Add KafkaRest Metrics Provider
- KSTORAGE-845 Check if tier topic exists already before calling AdminZkClient#createTopic
- CNKAF-952: Update javadoc for removeBroker ConfluentAdmin API with InvalidBrokerRemovalException
- CNKAF-851: log goals skipped during detection in one line
- Add KafkaRest Metrics Provider
- CNKAF-964: Shade opencensus protobufs into SBK jar
- KSTORAGE-806 Modifying Correctness Framework to accept .properties
- KC-630: Cluster link system test support with good path mirror test
- CONFLUENT: Prevent log start offset overrun during retention
- METRICS-1862 Remove config events proto defition from ce-kafka and use the one checked in into Schroedinger instead.
- KSTORAGE-861: fatal NPE in TierTopicConsumer thread caused by initialization order (#2133)
- Back port [METRICS-1815] Connect client metrics missing some resource level tag…
- KSTORAGE-806 Patch to add Confluent License Information
- KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups (#8989)
- CPKAFKA-5456: Shade telemetry-api classes
- KSTORAGE-806: Implementing a CLI for compatibility correctness tests & building object store correctness tests
- CNKAF-936: Remove dot-handling logic in metrics processor (#2121)
- KSTORAGE-861: fatal NPE in TierTopicConsumer thread caused by initialization order
- METRICS-1833 Special handling of KSQL metrics in telemetry reporter that are getting mangled.
- [METRICS-1815] Connect client metrics missing some resource level tags
- METRICS-1833 Special handling of KSQL metrics in telemetry reporter that are getting mangled.
- CNKAF-936: Remove dot-handling logic in metrics processor
- Azure block blob implementation for tiered storage
- Deprecate KafkaHttpServlet in favour of rest-utils Application. (#2135)
- CLIENTS-1599: Deprecate KafkaHttpServlet in favour of rest-utils Application.
- KC-532: Support observers in replica assignment tool
- CONFLUENT: Remove unused val and stale documentation in ReplicaManager
- [METRICS-1844, METRICS-1839, METRICS-1756,METRICS-1853] - Reconcile configs in Telemetry Reporter before creating a ConfluentTelemetryConfig object. Use password object for api.secret in ConfluentTelemetryConfig.java.
- [CNKAF-885, CNKAF-946, CNKAF-947] Set SBK Default Throttle
- [METRICS-1844, METRICS-1839, METRICS-1756, METRICS-1853] - Reconcile configs in Telemetry Reporter before creating a ConfluentTelemetryConfig object. Use password object for api.secret in ConfluentTelemetryConfig.java.
- fix: Make EmbeddedKafka threadsafe
- METRICS-1635: Generate and send config events
- METRICS-1801: align telemetry domain metric names
- METRICS-1835 Shade confluent-serializers dependencies in confluent-metrics (#2111)
- [CNKAF-885, CNKAF-946, CNKAF-947] Set SBK Default Throttle, handle absent brokers when setting throttle.
- METRICS-1835 Shade ce-serializer dependencies in ce-metrics
- CNKAF-481: Verify Cluster Id in FileBasedPlainSaslAuthenticator
- KSTORAGE-708: Durability Topic Partition management
- METRICS-1733 replace ‘whitelist’ with ‘include’ for telemetry and metrics reporter (#2096)
- METRICS-1810 use SBK topic replicas config if present (#2109)
- METRICS-1733 replace ‘whitelist’ with ‘include’ for telemetry and metrics reporter
- CNKAF-481: Allow embedding of logical cluster name in advertised listener config
- METRICS-1810 use SBK topic replicas config if present
- fix: fixes exception in server shutdown due to metrics registration
- METRICS-1809: add telemetry settings to default server.properties file
- KC-681: Make num.partitions a dynamic config
- CNKAF-849: Sanitize log messages when checking for startup conditions
- METRICS-1819: support using port 0 for telemetry local exporter
- [CNKAF-927] Fix typo in server.properties. (#2090)
- [CNKAF-924, CNKAF-926] Fix Databalancer Topic RF config, remove Teleme…
- [CNKAF-927] Fix typo in server.properties.
- CONFLUENT: Added check for CL metrics
- CNKAF-918: Relax GoalUtils.eligibleBrokers criteria to allow leadership movements off observers.
- CONFLUENT: Added check for CL metrics for 6.0.x
- CIAM-109 Add option to fetch role bindings for multiple scopes
- [CNKAF924, CNKAF-926] Fix Databalancer Topic RF config, remove Telemetry RF dependency
- 6.0.x
- CPKAFKA-4955: Fix failing testTopicDeletion
- CIAM-109 Add ALL DescribeAccess and AlterAccess for resource specific grants
- cherry-pick PR 2047, 2095 from 6.0.x
- KC-655 Offset migration configs with racially neutral terms.
- Removed two testing files accidentally added.
- SEC-1130-Inject ConfluentAdmin from ce-kafka
- backport METRICS-1820: use password object for telemetry api secret
- CNKAF-893: Update SBK to work with secure Zookeeper
- CIAM-96 Added cloud-cluster scope and describe access for higher levels.
- [METRICS-1820] use password object for telemetry api secret
- SEC-1034: use confluent-log4j instead of log4j
- METRICS-1816, METRICS-1705 local telemetry exporter metrics whitelist configuration for SBK (#2031)
- CONFLUENT: deleting extra topic config change in cluster linking test
- KC-505: Integration tests for CL metrics
- CNKAF-909: Add direct dependency on opencensus for SBK (#2057)
- KAFKA-10149: Allow auto preferred leader election when partitions are…
- CNKAF-918: Relax GoalUtils.eligibleBrokers criteria to allow leadership movements off observers.
- CNKAF-909: Add direct dependency on opencensus for SBK
- Add RackAwareGoal to goal list
- Add RackAwareGoal to goal list
- CNKAF-884: Remove unnecessary metrics (#2059)
- Migrate MDS to run on KafkaHttpServer implementation
- KAFKA-10149: Allow auto preferred leader election when partitions are…
- Fix ce-http-server import control violation for SslContextFactory
- KC-505: integration tests for CL metrics for 6.0.x
- [METRICS-1816] [METRICS-1705] Fixing some metric names in default whitelist and configured local telemetry exporter metrics whitelist configuration for SBK
- [CNKAF-867] Handle empty proposals for addBroker
- METRICS-1717 use shared domain for jvm level metrics (#1949)
- Migrate MDS to KafkaHttpServlet implementation
- CNKAF-854: Disabling SBK should persist the broker removal cancellation status
- KC-138: Implements cluster link pausing.
- [CNKAF-873] Don’t require network capacity configs. (#1996)
- CNKAF-884: Remove unnecessary metrics
- SEC-1183 - Treat empty authority in CRN pattern as a wildcard
- CNKAF-882: Add proposal logging for add/remove
- CNKAF-837: Add broker should be able to add removed brokers.
- METRICS-1717 use shared domain for system level metrics
- KC-393: Integration tests for mirroring compacted topics
- Add inter broker listener endpoint information to HTTP server binder.
- Add inter broker listener endpoint information to HTTP server binder.
- CNKAF-773: Do not throw an exception when the broker is not present in the cluster model
- CNKAF-847: Complete removal callback when there are no proposals to execute
- KSTORAGE-802: Allow specifying AWS credentials through configuration file
- [METRICS-1552][METRICS-1719] remove telemetry broker assumptions and use metrics context for broker check
- [CNKAF-848] Set preferred user SBK config in server.properties. (#1999)
- [CNKAF-873] Don’t require network capacity configs.
- [CNKAF-719] Add option to disable dynamic throttle setting when stati…
- KSTORAGE-819: Force roll segments only for partitions eligible for tiering
- KSTORAGE-799: reduce Log.lock contention in ArchiveTask segment selection logic
- KSTORAGE-774: Fix active segment roll logic when roll.min.bytes setting is enabled
- KSTORAGE-766: Ensure delete segment throttler does not block active segment rolls
- KSTORAGE-769: Fix to remove the costly toSet conversions in disk throttler
- KSTORAGE-758: Include ERROR status in TierPartitionStatus.isOpen()
- KSTORAGE-595 NPE in TierDeletedPartitionsCoordinator after tier topic is reassigned / deleted from broker
- [CNKAF-848] Set preferred user SBK config in server.properties.
- KSTORAGE-754: KafkaMetrics Whitelist Creation and new KafkaMetricsWrapper util (#1906)
- KSTORAGE-754: KafkaMetrics Whitelist Creation and new KafkaMetricsWrapper util
- Revert “KC-138: Implements cluster link pausing. (#1994)”
- KC-138: Implements cluster link pausing.
- KC-611: Fix mirror topic create processing incorrect config.
- KSTORAGE-547: state recovery injection tooling
- [METRICS-1235] implement exporter level whitelist
- CONFLUENT: Fix Confluent provider start future completion, add unit tests
- [CNKAF-881] SBK: Don’t treat all new brokers as empty. (#2005)
- SBK/CruiseControl: low utilization fix
- CNAKF-833, CNKAF-842: Fix missing request rate metric and enable tele…
- KC-521: Add per listener authentication audit log enable config
- CNAKF-833, CNKAF-842: Fix missing request rate metric and enable telemetry reporter sampler by default
- KSTORAGE-547: state recovery injection tooling
- CIAM-94 Decouple scope path keys and Role scope type
- RackAwareGoal should allow for uneven replica distribution based on t…
- [CNKAF-881] SBK: Don’t treat all new brokers as empty.
- [CNKAF-719] Add option to disable dynamic throttle setting when stati…
- KC-365: Add Jmh benchmarks for Authentication Audit Events
- CNKAF-788: Introduce proper API exceptions in the broker removal API
- KSTORAGE-706: Durability events and database.
- METRICS-1708: add telemetry version header for forward-compatibility (#1883)
- Allowing user to disable tiered storage safely - Phase 1
- CONFLUENT: Use undeprecated ciphers for TLSv1.3 cipher change test
- Port from CC: Fix low utilization threshold computation (#92)
- CNKAF-791: Change data format of failedBrokers in ZK
- CNKAF-807: Remove zkclient from gradle dependencies.
- CNKAF-575: Remove IOtec:ZkClient dependency.
- [CNKAF-776] Switch SBK Maven publish group to “io.confluent”
- [CNKAF-776] Make SBK publish to “io.confluent”
- RackAwareGoal should allow for uneven replica distribution based on t…
- [METRICS-1609] dynamically add telemetry reporter at runtime
- CONFLUENT: Move clusterId set on AuditLogProvider Instance to AuditLogProviderFactory
- KC-389: Retry segment deletion in cluster link test to fix timing issue
- KC-389: Cluster link integration tests with tier storage enabled
- CPKAFKA-5239 - Fix AddPartitionsToTxn audit event category
- CPKAFKA-5207: Complete authorizer start future only when sufficient nodes are ready
- CNKAF-843: Fix remove broker validation check
- Persist the broker removal state and handle failover
- CNKAF-773: Do not throw exception when there are no replicas to move on broker removal
- CNKAF-428: Integration test for replica placement self-healing
- Add retries to telemetry sampler integration test (#1922)
- [CNKAF-803] Add a metric to track the number of pending add-broker requests
- KC-72: Add topic listing for cluster links.
- KC-519: Support Authentication events in audit log routing
- CPKAFKA-5207: Fail fast during broker start up if multiple brokers use same MDS URL
- KC-536 bug fix to make offsets/acls configs dynamic (#1893)
- [CNKAF-803] Add a metric to track the number of pending add-broker requests.
- CNKAF-428: Update ReplicaPlacementGoal to handle replication factor change
- KC-545: Allow configuring ssl cipher suites at the cluster level
- fix telemetry reporter reconfiguration
- KSTORAGE-623: added disk throttling metrics for producer and follower throttle rates
- KC-536 bug fix to make offsets/acls configs dynamic
- [KSTORAGE-610] Add metric for tier materialization lag
- update ksql and streams provider
- CNKAF-649: Handle broker restart during broker removal
- KC-390: Cluster link authorization tests
- KC-366: Add authentication audit logging support to Selector and authentication implementations
- KC-594: use BufferSupplier GroupMetadataManager.scheduleLoadGroupAndOffsets
- METRICS-1708: add telemetry version header for forward-compatibility
- backport [METRICS-1564] telemetry - derive interbroker client configs (#1816)
- ksql and streams provider use configs for building labels
- (Backport) make commit-id optional, update whitelist to include C3 metrics
- make commit-id optional, update whitelist to include C3 metrics
- SEC-1083 - AuditAdmin role’s scopeType = Cluster
- SEC-1083 - AuditAdmin role’s scopeType = Cluster
- KC-544: Alter configs from external listener should only update said listener configs
- CNKAF-428: Update ReplicaPlacementGoal to handle replication factor change
- KC-392: Integration tests for cluster link quotas on source and destination
- KC-582: Add multi-tenant support for alterMirrors.
- KSTORAGE-705-Define per broker durability metrics and its management
- CNKAF-795: Ensure SBK clients have bootstrap.servers configured
- [METRICS-1564] telemetry - derive interbroker client configs
- CIAM-71 Handle RBAC access policies at different scopeLevels
- fix checkstyle error
- [KAFKA-8528] Expose Trogdor-specific JMX metrics for Tasks and Agents
- Backport PR 1897 to 6.0.x
- update ksql and streams provider
- SEC-1083 - Add AuditAdmin role
- CNKAF-794: Properly wrap broker removal exceptions in listener and improve remove broker test
- Add schemas topic label to SchemaRegistryProvider
Security¶
- Fix auth for SchemasResource
- Fix SR security test classpath
- Implement asychronous schema fetching in LruSchemaRegistryClient
- SEC-1478 Update from JUnit4 to JUnit5
- Implements FF-873 in a generic way by supporting wildcards *
- Check authentication.roles in KSQL bearer token requests
- Include Kafka dependencies to KSQL packaging
- DG-540 Add batch get schema API
- fix: Make rest-utils compile scope as ksqlDB no longer pulls this in
MQTT Proxy¶
- Fix telemetry reporter test on 6.0.x
Metadata Service¶
- SEC-1565: fix unit test based on SEC-1565 on 5.4.x
- SEC-1565: Incorrect implementation of equals in ManagedRoleBindings
- SEC-1566: Switch to https for packages.confluent.io
- SEC-1507 cluster information migration script should validate properties file
- [MMA-9298]: bump up the version for nice name feature
- [SEC-1506]: cluster-information-migration-script - allow user to specify curl options
- [SEC-1514]: cluster-information-migration-script - suppress progress but show errors
- [SEC-1499]: Add –help option to cluster registry import/export script
- CIAM-399: decouple rbac-db from rbac-api-server
- SEC-1494: Update to junit5
- CIAM-338 - EnvAdmin can see more that it should in cli
- [SEC-1471]: Backport Friendly cluster name from master to 6.0.x
- CIAM-371 Add Delete operation to Cluster for CloudClusterAdmin
- CIAM-338 - Test MyRoleBindings endpoints
- CIAM-325 Add ON_ERROR_STOP inside rbac_schema_postgres.sql
- CIAM-318 Define cc-rbac Postgres users in SQL
- SEC-1452: refactor out cluster name validation to its own validator
- CIAM-121 Environment variable is named DBMIGRATE_SHARE_DIR
- SEC-1419: fix script to exit with 0 after successful execution
- CIAM-121 Support DB Migrations
- CIAM-313 give USAGE privilege to RBAC DB users
- Add implementation for newly added “describe and alter SCRAM user” API
- SEC-1402 Cluster Registry Unicode Cluster Names
- CIAM-257: Add db healthcheck skip option for cpd
- CIAM-269 Terraform to create schema and DB roles
- CIAM-295 - Verify MDS behavior with CCloud setup and ANON super.user
- Restore Cluster Registry import script in master
- CIAM-241 - Port v2 tests to demonstrate calls to DB
- MMA-7359 - Space in Principal Breaks Control Center Authorization
- CIAM-248 - Parity Suite Infrastructure in MDS
- CIAM-269 Terraform for CC RBAC RDS
- CIAM-261: Add metrics for all rest end points
- CIAM-262 Use connection pool for RBAC DB
- CIAM-222: Config support for db
- CIAM-134 Add test to demonstrate /lookup/rolebindings/principal
- CIAM-245: Add DbAuthCache/DbAuthWriter metrics
- SEC-1382: validate groups in “super.users”
- CIAM-236 Add test for removal of env describe when cluster admin removed
- CIAM-221 Implement list rolebindings queries
- CIAM-102: Add caching to DBAuthCache findRule method
- SEC-1372 - MDS : Address V1 Visibility Testing Gap found by C3
- CIAM-227: (follow-up) Add DbAuthStoreConfig and minor cleanups to DbAuthWriter/RbacOrmService
- CIAM-237 rename resource_id to id
- CIAM-227: Implement resource RoleBinding methods and add validations to DBAuthWriter
- CIAM-223: Support add, delete, lookup rolebindings from DB
- CIAM-82 Cloud RBAC Schema
- CIAM-232 Add endpoint to delete all role bindings for a principal
- CIAM-82 Data generator
- CIAM-119 - Rbac DB Stubs
- CIAM-119 - DB Auth Maven Module and Stubs
- CIAM-162 Provide /lookup/rolebindings endpoint
- CIAM-86: Add perf tests for RBAC control plane authorizations
- CIAM-110 Fix yaml indentation
- CIAM-140 Add tests for /principals endpoints
- CIAM-110 Associate RBAC Roles with Namespaces
- CIAM-59: Readme update
- CIAM-59: Update mds-in-a-box to support v2alpha1 rbac
- SEC-1089 Fix v2 yaml MdsScope
- CIAM-145 Update sdk yaml to use lookup role endpoint, make bindWithRe…
- Fix aquirePort typo in test helper function
- CIAM-138 Test /roles endpoint
- SEC-1226 - Updated audit log api code after ce-audit added a new category event authentication
- CIAM-132 Don’t allow empty principal name
- SEC-1229 - MINOR: Disable getUpdatedAuditRouteConfigTest and dependents because …
- SEC-1205 Gracefully handle roles request for nonexistent role
- CIAM-59: Update MdsWritesFilter pathMatcher for v2alpha1
- CIAM-109 fixed checkstyle
- CIAM-109 add hierarchy traversal for managed resource endpoint
- Have ce-kafka inject the (internal) ConfluentAdmin directly, instead …
- Cherry pick 297: Move MDS to KafkaHttpServlet
- CIAM-96 fix unit tests based on changes in cloud cluster admin role
- CIAM-126 Make /roles/{roleName} and /roleNames v2 endpoints available to mds sdk
- CIAM-125 Update Roles object
- SEC-1127 - Fixed a defect that AuditLogConfigSpec didn’t get updated
- CIAM-59: Disable token endpoints if no user store configured
- CIAM-100 Allow scopes with SR cluster but no Kafka cluster
- CIAM-27 Lookup scope includes contained scope
- CIAM-94 Upstream interface changes
- CIAM-59: Config to skip userstore setup/basicauth fallback
- CIAM-79 Update MDS for changed interfaces in CIAM-71
- removed required kafka-cluster in Scope
Replicator¶
- REPL-1310 fix for monitor thread failures in start stop start cycle
- REP-1300: Null out licenseManager on connector stop to avoid hang on restart
- REPL-1303: Setting deadline correctly when trying to resume partitions
- FF-3928:Exclude test dependency javassisst from curator-test to be assembled
- FF-3928: Bump the version of Guava
- REPL-1263: Supporting source url configs for active active SR
- REPL-1204: Enable retries on cluster id fetch for Replicator task start
- REP-1193: Removed error logging for failed executable connector submission
- REP-1056: Handle empty assignments
- REP-1046: Improve logging on conversion error
- REP-917: Executable should not log an errors when attempting to start connectors on followers
- REP-762: Add a READ ACLs check in verifier tool for destination topics
- REP-763: Improve WARN log to explain if the error is on source or dest location