Confluent Platform Component Changelogs¶
This topic provides changelogs for the individual Confluent Platform components.
Version 6.0.13¶
Released March 2023
Community Features¶
Common¶
Kafka¶
REST Proxy¶
Schema Registry¶
- PR-2575 - DGS-6331 Handle javaType for oneOfs during JSON deserialization
- PR-2568 - DGS-6306 Allow multiple oneofs in Protobuf converter
- PR-2540 - DGS-6075: Add explicit definition of snakeyaml.
- PR-2519 - DGS-6014 cherry pick Maven plugin enhancement to 5.5.x
- PR-2514 - DGS-5950: Allow optional map key in Avro
Commercial Features¶
Confluent Server¶
- KMETA-478 Enable zk audit logs by default
- KC-2328: Update yaml.v2 to 2.2.4
- KC-2333: Update io.netty libraries to 4.1.86.Final
- CPKAFKA-3810: Report observers in ReplicaStatus.
- CPKAFKA-3852: Add/remove listener metrics during dynamic listener update
- KAFKA-9038: [WIP] Allow creating partitions for topics partitions not in reassignment
- CONFLUENT: Use single audit log provider for MDS and its hosting broker
- MINOR: Optimize MetricsUtils.fullMetricName by precompiling regexes
- KAFKA-9039: Optimize ReplicaFetcher fetch path (#7443)
Security¶
- SEC-5350: Update netty-codec-http to 4.1.86.Final
Secret Registry¶
- Remove unused variables
- Update RestService.java
Control Center¶
- KC-2261 timeout websocket connection via jwt token
- Update CODEOWNERS to match latest definition from master
- Refactoring- made code more functional and improved readability.
- Updated NotFoundException message in scenario where topicName provided returns 0 no consumergroupdata.
- Updated code based on PR feedback.
- Updated consumerGroupId API to also support topicName queryParam.
- Improving test cases with scenarios of single consumergroup covering multiple topics. Updated consumerGroupId to not support topicName queryParam, it is not required.
- Adding unit test for CachedConsumerOffsetsResource
- Adding support for topicName in query param to return only consumer groups for that topic
- Reading password using getPassword instead of getString, this fixes the casting exception
- Fixing failing test case, due to merge issue
- Renaming Health_check to health check
- Adding config based functionality to force http1 for armeria health checks
- MINOR: Update repo to use mvn-wrapper to speed up builds
MQTT Proxy¶
- Remove the pin of netty version to use the (current) version defined in common
Metadata Service¶
- CIAM-2615 - Bump TestNG for 7.5 - which is the last version to support jdk8
- Increase buffer size for Proxy Servlet
Version 6.0.11¶
Released December 2022
Community Features¶
Common¶
Kafka¶
REST Proxy¶
Schema Registry¶
- PR-2440 - Update the Snakeyaml dependency
- 8e5ce168 - Unpin Protobuf version.
- PR-2414 - DGS-5253 Handle cycles when resolving references
- PR-2405 - DGS-4754 Handle empty record default at field level
- PR-2399 - DGS-5084 Ignore compat check in IMPORT mode
- PR-2395 - DGS-4971 Handle map types with enhanced.protobuf.schema.support
Commercial Features¶
Confluent Server¶
- CONFLUENT: Skip publishing for projects with no scala suffix when the scala version is not the default
- DP-9030 - Switch to codeartifact repo
- KENGINE-283 Topic ID Mismatch Issue - Exception added to disallow Alt
- Increase vagrant aws timeout for system test
- BouncyCastle Issue fix for connect service [6.0.x]
- KSECURITY-792: Upgrade from Scala 2.13.8 to 2.13.10
- KSECURITY-788: Upgrade jackson-databind version to 2.13.4.2
- KSECURITY-832: Upgrade bc fips drivers
- CIAM-2424 - Name RestClient threads for stackdump identification (#7887)
- KSECURITY-794: Upgrade to protobuf version 3.19.6
- CONFLUENT: Remove “principal.builder.class” config from Broker Client configs
- Cherry-pick KAFKA-12153
- Add stripped down transaction tool which allows aborting hanging transactions
- CIAM-2304: Add SecurityMetadata:Describe to SecurityAdmin
- RCCA-8564: log a warning if LDAP login fails becuase of network issues
Security¶
- Exclude bcpkix-jdk15on as compile dep and add as test dependency
- Pin bc-nonfips dep as a test dependency
- Pin bouncycastle fips dependency in confluent-security-plugins
- Removed bcfips from common
- excluded bcfips jars
- AUTHN-1087: Upgrade vertx to 3.9.14
- AUTHN-1325: Update bc-fips version to 1.0.2.3
Secret Registry¶
- Exclude Non-FIPS dep and use bc-fips dependency
Control Center¶
- MMA-9023: memory bounding for streams applications [5.4.x, master]
- Revert “MMA-12206 upgrade io.netty:netty-codec-http”
MQTT Proxy¶
- Fixing the tests and config validator, to incorporate new changes in validation of MqttDecoder brought in by netty upgrade
- Upgrade netty
Metadata Service¶
- Implement writeTransactionMarkers in DelegateConfluentAdmin
- CIAM-2304: Add unit tests for SecurityAdmin
Replicator¶
- REPL-2055-: Log WakeupException at debug level
- RCCA-7678: Reverse proxy header check added
Version 6.0.10¶
Released August 2022
Community Features¶
Common¶
Kafka¶
Commercial Features¶
Confluent Server¶
- [METRICS-3888][METRICS-4448] update to telemetry-client without shaded Netty for 6.0.x
- KAFKA-14107: Upgrade Jetty version.
- [METRICS-4448][METRICS-3338] upgrade telemetry to pick up netty not shaded change for 6.0.x
- Fix exception handling in RBAC writer coordinator
- Migrating log4j12 to reload4j, slf4j-log4j12 to slf4j-reload4j.jar (6.0.x)
- Avoid sending partial updates during LDAP group manager startup
- Upgrading google-cloud-storage to v2.6.1 (6.0.x) #6992
- Upgrading netty version to 4.1.79.Final (backport 6.0.x)
- Upgrading aws-java-sdk-s3 to v1.12.268 (6.0.x)
- Update bouncy castle to 1.70 (back port 6.0.x)
Security¶
- add auth token provider to ksql security extensions
- revert pom
- AUTHN-1085: Fix NPE in debug log
- APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Secret Registry¶
- Migrate from log4j to reload4j for connect-plugin
- CC-17641,CC-17644 Bump jetty version
- APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Control Center¶
- MMA-12201 remove slf4j from transitive deps
- MMA-12206 upgrade io.netty:netty-codec-http
- MMA-12201 Exclude slf4j in transitive dependency
- MMA-12201: Use reload4j version properties from common
- MMA-8781: Respect SR Config and handle nulls in CachingSchemaRegistryClientSupplier
- MMA-12201 Replace log4j with reload-4j
- MMA-12127: Change type from String to Password for CONTROL_CENTER_EMAIL_PASSWORD
MQTT Proxy¶
- CC-17620: remove dependency on slf4j-log4j12 and confluent-log4j
Metadata Service¶
- CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS
Replicator¶
- Migrate confluent-log4j to reload4j.
Version 6.0.9¶
Released July 2022
Community Features¶
Schema Registry¶
- 31183d32 - Fix merge issue
- ce22bb34 - Fix merge issue
- a930a7b6 - Fix merge issue
- 6c78adf4 - Fix merge issue
- d46bebb5 - Fix merge issue
- PR-2328 - DGS-4162 Handle refs at different levels in Protobuf converter
- PR-2327 - DGS-1648 Allow Protobuf msg fullname to be passed for console producer
- PR-2318 - DGS-3974 Fix StackOverflow with recursive JSON schema in converter
Version 6.0.8¶
Released June 2022
Commercial Features¶
Confluent Server¶
- build: kafka system tests should support larger EBS volumes for newer instances (#6371) (#6406)
Metadata Service¶
- RCCA-6743: Test that we can allow rest client to disable hostname verification
- This release version includes a fix to prevent stack trace exposure from getting surfaced in HTTP responses for malformed requests.
Version 6.0.7¶
Released June 2022
Community Features¶
Common¶
Kafka¶
- PR-703 - KSECURITY-220: Update jackson packages to 2.13.2
- PR-712 - Fix ducker-ak
- PR-11946 - KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (#11946)
- PR-11872 - KAFKA-12879: Remove extra sleep (#11872)
- PR-11871 - KAFKA-12879: Addendum to reduce flakiness of tests (#11871)
- PR-11797 - KAFKA-12879: Revert changes from KAFKA-12339 and instead add retry capability to KafkaBasedLog (#11797)
- PR-10311 - KAFKA-12462: proceed with task revocation in case of thread in PENDING_SHUTDOWN (#10311)
REST Proxy¶
Schema Registry¶
- PR-2290 - DGS-3674 Upgrade to Protobuf 3.19.4
- PR-2285 - Merge Conflict Resolution (from 5.4.x to 5.5.x)
- PR-2283 - DGS-3659 Handle null default in JSON Schema converter
- a5c4a3b7 - Override dependency of kotlin-scripting-compiler-embeddable to Kotlin version
- bd9cb11e - bumping kotlin components version to 1.6 for kafka-connect-protobuf-converter
- 96471721 - Upgrade kotlin to 1.6.0
- PR-2259 - Fix upgrade to Jackson 2.13
- PR-2257 - DGS-3548 Optimize cache for JSON Schema converter
- PR-2253 - DGS-3535: Remove enforcement of building with JDK8
- PR-2241 - DGS-3484 Allow NaN when parsing JSON
- PR-2227 - DGS-3417 Fix default value conversions in JsonSchemaConverter
- 7b222add - Updating snakeyaml
- PR-2215 - DGS-3250 Return 422 instead of 500 if schema lookup is invalid
- PR-2214 - DGS-3248 Return 422 instead of 500 if subject is READONLY during DELETE
- PR-2201 - DGS-3066 Perf optimizations for AvroSchema
- PR-2200 - DGS-3133: Compare canonical string instead of jsonNode when comparing JsonSchema
Commercial Features¶
Confluent Server¶
- Upgrade Gson
- use correct cp urls in dockerfile - 6.0.x
- KSTORAGE-47: Metrics - measure broker consumer lag in time
- KSECURITY-221: Update jackson packages to 2.13.2 version
- cherrypick two ducker changes from ccs 2.6 to ce 6.0.x
- CPKAFKA-3509, CPKAFKA-3510: API on AuthCache to get known scopes and bindings
- KSECURITY-198: Update protobuf-java packages to 3.19.4
- CPKAFKA-5991: Fix Cluster Link System Test Failure
Control Center¶
- Bump maven-shade-plugin to 3.3.0
- Fix build break
- LOG_DIR to match rest of confluent components
Metadata Service¶
- CIAM-2019 - Fix 6.x MDS builds from Jackson update
- MINOR SEC-3087 SEC-3028 Avoid NPE in integration test
Version 6.0.6¶
Released March 2022
Community Features¶
Common¶
Kafka¶
REST Proxy¶
- PR-963 - KREST-4113 Don’t delete all ACLs if an empty DELETE request is received.
- PR-969 - KREST-3836 update maven to pull from https
- 5514b21c - MMA-11468: Add option to enable nosniff protection
- PR-305 - Remove Redundant Netty Definition
- PR-302 - APIF-2027: Upgrade jetty-server to 9.4.44.v20210927.
- PR-301 - APIF-2005: Change the Maven repository protocol: http → https
Schema Registry¶
- PR-2187 - DGS-3066 Handle unions in AvroSchema.metaEquals
- PR-2186 - DGS-3061 Only cache top-level schemas in AvroData
- PR-2175 - DGS-2990: Backport PR 2174 to 5.2.x
- PR-2174 - DGS-2990 Removing message value from exception to avoid logging custo
- PR-2164 - DGS-2937 Use default scale of 0 in Avro converter
- PR-2151 - DGS-2875: Return correct default value for connect union
- PR-2148 - DGS-2830 Check for addition/removal of required Protobuf fields
- PR-2140 - DGS-2814 Ensure id generation is thread-safe in converters
- PR-2126 - DGS-2719: add explicit dependency for kotlin-scripting-jvm
- PR-2125 - DGS-2727 Handle multiple refs to same Protobuf enum
- PR-2122 - DGS-2707 Ensure Protobuf reserved range uses exclusive end
- PR-2108 - DGS-2646 Support name method on ProtobufSchema with only enums
- PR-2107 - DGS-2636 Collect properties for allOf in JSON Schema converter
- PR-2100 - Backport PR2039 to 5.5.x branch
- PR-2103 - DGS-2626 Handle combined schema with title and/or object
- PR-2089 - DGS-2541 Fallback to simple name comp in Protobuf compat
Commercial Features¶
Confluent Server¶
- Cherry-pick “Disabled bubbling errors from JsonSerializer and Deserializers for connect that print the message in stack trace (#5558)” onto 6.0.x
- [CLI-1430] Add GCM functionality to DecryptionEngine (#5563)
- Upgrade TCNative
- APPSEC-860: Upgrade Telemetry to adopt new Netty version.
- Bump Protobuf Dependency Version
- CPKAFKA-3442: Disable collection of all per-partition throughput metrics
- KC-1933; Upgrade confluent-log4j to 1.2.17-cp2.2 (6.0.x)
Security¶
- Added fix for empty bulkAuthorize requests
- DG-1249, DG-1257, DG-1353, DG-1423, DG-1462, DG-1491, DGS-2950: Fix testGetLastOffsetWithExistingMessages
- APPSEC-827:Upgrade bc-fips
Secret Registry¶
Control Center¶
- MMA-11603: Update protobuf version
- MMA-11121: Fix compatibility issues for partition size metric
- MMA-11478: enable cors check on websocket
- MMA-11451: fix @JsonProperty usage to yield expected output
- (MINOR): update armeria version and exclude netty artifacts
- (MINOR): fix merge
- MMA-11468: Add option to enable nosniff protection
- MMA-10986: fix health check resource for connect
- MMA-10571: add SR basic auth to armeria healthcheck
- (MINOR): fix armeria test
- MMA-9126: Fix broken RocksDB stats endpoint
- MMA-9292: unit test for ClusterDataExporter extraction of security protocol
- MMA-9292: control-center-export script reads security.protocol from prop file
- MMA-10114: Change default values for command topic configuration
- MMA-9479: Remove JDK dependency from control center stop script
- Added support for bigint format for numbers coming from ksql JSON
MQTT Proxy¶
- [CC-16389] Changing the maven URL in pom.xml from http to https
- Revert “[CC-16389] Changing the maven URL in pom.xml from http to https”
- [CC-16389] Changing the maven URL in pom.xml from http to https
Metadata Service¶
- MDS-33 - Add AES GCM enable variable
- SEC-238 MDS Test : Silent Proxy of Writes
- SEC-2805 - Manually tested fix for MDS forwarding logic to handle unicode and request with query params
Version 6.0.5¶
Released December 2021
Community Features¶
Kafka¶
REST Proxy¶
Schema Registry¶
- PR-2086 - Update netty to 4.1.68.Final
- PR-2079 - DGS-2507 Use fullName if enhanced.protobuf.schema.support=true
- PR-2075 - DGS-2491 Ensure ProtobufSchema.name returns qualified name
- PR-2070 - DGS-2401 Add converter flag to allow old doc/default behavior
- PR-2036 - Fix default value for array of records
- PR-2068 - Demostrate wrong cache hit for union type reader schema
- PR-2061 - DGS-2417 Change moving Protobuf messages as compatible
- PR-2058 - DGS-2393 Fix special chars in options for oneof/enum
- PR-2052 - DGS-2315: Only tombstone previous schemas during hard deletion
- PR-2043 - DGS-2259 Omit defaults for fields with type Message
- PR-2026 - DGS-2078 Fix optional Avro schema generation
- PR-2024 - DGS-2076 Fix NPE when cross referencing schema w/o name
- PR-2023 - DGS-2073 Fix Protobuf converter for multiple map refs
Commercial Features¶
Confluent Server¶
- CONFLUENT: Upgrade netty to 4.1.68.Final and disable io.netty.handler.ssl.openssl.useTasks
- KAFKALESS-521: Don’t join empty throttled replicas with new throttled replicas
- SEC-2731 - Improve Ldap handling for stale connections (#4497)
- CC-13747 CIAM-1263 FF-4419: Limit max retries in MDS URL Refresh loop
- CIAM-1253 Fix Logging of various timeouts in ce-rest-auth RestClient
- SEC-1537 RestClient Retry Logic Fixed (#4395)
- KC-1878 Debug Logging for LDAP Context creation
Security¶
- Upgrade Netty
- Upgrade Vertx
Control Center¶
Make non-idempotent request http connection short-lived. Allow user to configure setMaxRequestsQueuedPerDestination and setIdleTimeout.
Metadata Service¶
- Override commons-codec to 1.13
Replicator¶
- Upgrade jakarta.el:jakarta.el-api
- REPL-1843: Track uncommited records given to Connect framework.
Confluent Platform Docker Images¶
- Confluent Platform Docker Images will no longer ship with Yum/Apt configuration that allowed users to update running Confluent Platform containers to the next Confluent Platform release. Confluent recommends upgrading Confluent Platform service components running in containers by replacing the running container with the next Confluent Platform released Docker image and using Docker volumes to persist data between upgrades.
- Confluent Platform Docker images formerly shipped with world writable data and configuration directories (
chmod a+w). Those permissions are now removed, and adjustments made to make those directories writable from the image’sappuserUSER. - Confluent Platform Docker images are now built with
yum --nodocs ...to prevent installing Red Hat Documentation. This slims down the images slightly. Confluent documentation from Confluent packages are still installed.
Version 6.0.4¶
Released September 2021
Community Features¶
Common¶
Kafka¶
- c18bfc49 - KAFKA-13209: Upgrade jetty-server
- PR-11098 - KAFKA-13099; Transactional expiration should account for max batch size (#11098)
- PR-10919 - KAFKA-12985: Upgrade jetty to 9.4.42 (#10919)
- PR-9541 - KAFKA-10675: Add schema name to ConnectSchema.validateValue() error message (#9541)
- PR-10922 - KAFKA-12790: Remove SslTransportLayerTest.testUnsupportedTlsVersion (#10922)
- PR-10834 - KAFKA-12904: Corrected the timeout for config validation REST API resource (#10834)
- PR-10014 - KAFKA-12252 and KAFKA-12262: Fix session key rotation when leadership changes (#10014)
REST Proxy¶
Schema Registry¶
- PR-2016 - DGS-2026 Allow generating JSON schemas with additionalProperties=true
- PR-2014 - DGS-2020 Allow optional to be overridden in SchemaWrapper
- PR-2012 - DGS-2018 Ensure names in Avro union are scrubbed
- PR-1995 - Corrected Typo in ProtobufConverter and JsonSchemaConverter
- PR-1990 - DGS-1879 changed timeout parameter
- PR-1989 - Upgrade commons-compress to 1.21 (5.4.x and later)
- PR-1986 - DGS-1915: Upgrade commons-compress version
- PR-1985 - Add confluent-log4j as dependency
- PR-1977 - DGS-1909 Fix AVRO_FIELD_DEFAULT_FLAG_PROP in AvroData
- PR-1972 - DGS-1902 Use fully qualified names for Protobuf Map types
- PR-1969 - SEC-2492 exclude guava version with CVE
- PR-1968 - DGS-1880 Allow additionalProperties to be set in converter
- PR-1967 - Logging schema size when register
- PR-1965 - Remove use of WrappedParsedSchema in mock client
- PR-1956 - DGS-1853 Fix handling of slashes with refs in maven plugin
- PR-1954 - DGS-1850 Prevent duplicate imports in Protobuf converter
- PR-1947 - DGS-1829 Handle duplicate refs during Protobuf Schema compat check
- PR-1943 - DGS-1826 Fix NPE when struct default has null
- PR-1945 - DGS-1825 Fix DataException when scrubbing invalid names (Avro)
- PR-1942 - DGS-1825 Fix DataException when scrubbing invalid names (Protobuf)
- PR-1939 - DGS-1820 Pass ObjectMapper to JSON Schema generator
- PR-1937 - DGS-1820 JSON Schema validation should take a JsonNode
- PR-1931 - DGS-1794 Don’t return read-only byte buffers from Protobuf converter
- PR-1928 - DGS-1758: Allow AvroData take JSON null as default
- PR-1923 - DGS-1737: MockSchemaRegistryClient doesn’t generate new schema id with protobuf
- PR-1920 - DGS-404: Improving error messages - adding 40408 and 40409 for subject level compatibility and mode error
- PR-1907 - DGS-1658: Preserve field names when rendering Protobuf objects as JSON
Commercial Features¶
Confluent Server¶
- Change Guava version from andriod to jre.
- KC-1800: Upgrade Guava to 30.0-android in 6.0.x
- KAFKA-13209: Upgrade jetty-server
- DGS-1903: (backport to 6.0.x) Add config option for missing schemas TTL (#4118)
- KC-1809 Upgrade transitive hibernate-validator dependency
- KC-1810: Upgrade commons.io in ce-kafka
- KAFKALESS-523: Topic placement constraints not used in CreatePartitions
- KC-1784: Use URI instead of URL for RBAC writer comparison to avoid hostname resolution issues
- KC-1317: Fix incrementalAlterConfig for list property with default null
- KSTORAGE-1626: Fix logging during topic creation
- Fix logging during exception in DynamicBrokerConfig
- KC-1625: Fix logging in DynamicBrokerConfig
- KAFKA-10570; Rename JMXReporter configs for KIP-629
- KC-1752 Remove public facing String redact API which is not currently used
- KC-1752 Fix potential alter/update config related logging
Security¶
- APPSEC-287: Override commons-codec
- fix formatting issues with newer checkstyle versions
- fix: Checkstyle issues in AuthorizationFilter.java
- SEC-1034: log4j migration to confluent repackaged version
- DGS-1903: Add missing schema cache to LruSchemaRegistryClient
- SEC-2533 upgrade commons-io to 2.11.0
- SEC-2493 Remove bouncy-castle explicit dependency
Secret Registry¶
- CC-15204: Upgrade bcprov-jdk15on to 1.68
Control Center¶
- MMA-7623: Hide stack trace in HTML output for failed responses
- MMA-10895: Add support for prefixed ‘auth.bearer.roles.claim’ property
- Upgrade Apache io-common to v2.7
- DGS-1654: Upgrade Protobuf to v3.17.0
MQTT Proxy¶
- CCMSG-1251: log4j migration to confluent repackaged version
Metadata Service¶
- SEC-2440 - MDS - Fix MdsScope Validation and messages and add tests
- SEC-1034: log4j migration to confluent repackaged version
- Fixed issue where users with ResourceOwner role, could not see rolebindings of other users on resources they own
Replicator¶
- No Changes
Version 6.0.3¶
Released June 2021
Community Features¶
Common¶
- PR-339 - update spotbugs and jacoco for improved jdk support
- PR-337 - DGS-1551: upgrade jackson-bom to 2.9.10.20210106
- PR-334 - CC-14690 : Define com.fasterxml.jackson.dataformat:jackson-dataformat-cbor.
- PR-332 - Update dependency-check to 6.1.6
- PR-326 - KREST-1066 Upgrade Guava to the latest version to patch CVE-2020-8908.
- PR-344 - APPSEC-110: Exclude Netty in zookeeper 3.4.14
- PR-342 - Fix jacksondatabind and jackson-cbor version
- PR-343 - Change zookeeper version to 3.4.14
- PR-340 - Fix zookeeper, jacksondatabind and jackson-cbor version
Kafka¶
- 7e2fb89f - KAFKA-10570; Rename JMXReporter configs for KIP-629
- 746954dd - KC-1752 Remove public facing String redact API which is not currently used
- PR-574 - KC-1746: Upgrade jetty, jersey, commons-codec
- 25f4b233 - KAFKA-12820: Upgrade maven-artifact dependency to resolve CVE-2021-26291
- PR-564 - APPSEC-60: Upgrade org.apache.maven:apache-maven
- PR-10611 - KAFKA-12730; Avoid duplicate logout if Kerberos login fails (#10611)
- PR-9605 - KAFKA-10727; Handle Kerberos error during re-login as transient failure in clients (#9605)
- PR-10536 - KAFKA-12667: Fix incorrect error log on StateDirectory close (#10536)
- PR-10526 - KAFKA-12655 Update Jetty: 9.4.38.v20210224 → 9.4.39.v20210325 (#10526)
- PR-10474 - KAFKA-12602: Fix LICENSE file (#10474)
- PR-540 - SEC-1883: Upgrade Netty version to 4.1.62.Final
- PR-10396 - KAFKA-12474: Handle failure to write new session keys gracefully (#10396)
- PR-10452 - KAFKA-12593: Fix Apache License headers (#10452)
- PR-10397 - KAFKA-12508: Disable KIP-557 (#10397)
- PR-10360 - KAFKA-12508: Emit records with same value and same timestamp (#10360)
- PR-10118 - KAFKA-10192: Increase max time to wait for worker to start in some integration tests (#10118)
- PR-10260 - Remove ReassigningPartitions metric when shutting down ReplicaManager (#10260)
- 8d258649 - KAFKA-12400: Upgrade jetty to fix CVE-2020-27223
- 04d3bd9a - KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
- PR-10016 - KAFKA-10340: Proactively close producer when cancelling source tasks (#10016)
- 79447588 - KAFKA-12211: don’t change perm for base/state dir when no persistent store
ksqlDB¶
REST Proxy¶
- PR-857 - KREST-1263 Exclude a vulnerable jackson-mapper-asl dependency from avro 1.8.2 in 5.3.x.
- d7a631a4 - KREST-1029 Remove a new occurrence of SLF4J versioning in kafka-rest.
- 40e94513 - KREST-1029 Remove SLF4J versioning from kafka-rest.
- PR-831 - KREST-865 Remove usage of the deprecated SimpleAclAuthorizer.
- PR-234 - Update Jetty: 9.4.36.v20210114 → 9.4.38.v20210224
- 66aa0c15 - KREST-1030 Upgrade hibernate-validator to fix CVE-2020-10693.
- 6b8327ac - KREST-1030 Upgrade Jersey to fix a number of CVEs.
- 2e5ff421 - KREST-1030 Upgrade Jetty to fix a number of CVEs.
- PR-251 - bump guava version to 30.1.1-jre
- PR-249 - DGS-1621 Upgrade Jetty to 9.4.40
- PR-246 - fix: CVE-2021-28168
- PR-244 - Upgrade Jetty to 9.4.39.v20210325
- PR-243 - DGS-1546 Map JsonMappingException to 400 response
- PR-242 - Upgrade Jetty to 9.4.39.v20210325
- PR-239 - KREST-860 Fix deprecated usages of org.apache.kafka.common.Metric.value()
- PR-237 - Define Netty version to be shared to downstreams.
- PR-229 - KC-1456: Update Jetty to 9.4.36.v20210114
- PR-231 - UIF-298: Upgrade Jetty to 9.4.35.v2020112
Schema Registry¶
- PR-1902 - DGS-1727 Don’t throw exception when comparing array item
- 290762a9 - Fix Jetty method
- PR-1897 - DGS-1716 Remove logging of configs
- PR-1896 - DGS-1704 override commons-compress
- PR-1895 - DGS-1704 Upgrade jackson-mapper-asl
- PR-1890 - DGS-1698 Handle recursive JSON schemas in Converter
- PR-1889 - DGS-49 Fix StackOverFlowException with cyclic JSON Schema
- PR-1885 - DGS-1677 Normalize Protobuf schemas during validation
- PR-1879 - DGS-1551 Upgrade hibernate-validator
- PR-1881 - DGS-469 Reuse MetricsContext object in KafkaGroupLeaderElector
- PR-1877 - DGS-1664 Allow URL encoded subject names in maven plugin
- PR-1873 - DGS-1657 Add converter flag to enable scrubbing invalid names
- PR-1870 - DGS-1659 Be more lenient with non-string enums
- PR-1863 - DGS-1631 Ensure enums are emitted after message types
- PR-1852 - DGS-1595 Ignore invalid Avro defaults in Avro converter
- PR-1847 - DGS-1570 Handle optional integer/number with format in JSON Schema
- PR-1846 - DGS-1569 Ensure recursive types are not built prematurely
- PR-1844 - DGS-1562 Don’t require refs be declared with root schema
- PR-1839 - DGS-1550 Add cache for transformed schemas
- PR-1833 - DGS-1534 Use raw Avro schema to check doc equality
- PR-1821 - DGS-1469 Add server-side schema cache
- PR-1816 - DGS-1391 fix Avro Convertor for GenericData of Enum Union with names
- PR-1815 - DGS-1451 Add Jackson annotations for object evolution
- PR-1808 - DGS-1403 Allow register different schema types if compatibility level
- PR-1807 - UIF-302 Add Guava declaration in dependency.
- PR-1806 - DGS-1402 Flag to set non-required fields to optional in JSON Schema
- PR-1805 - DGS-1400 Escape Special Character in Protobuf Option
- PR-1804 - DGS-1392 Allow removing false from closed content model in JSON Schema
- PR-1802 - DGS-1398 Allow adding empty schema to open content model for JSON Schema
- PR-1801 - DGS-1396 Fix optionality of ref schemas in JSON Schema converter
- PR-1799 - DGS-1391 fix Avro Convertor for GenericData of union with null, string, enum
- PR-1795 - DGS-1221 Ensure AvroSchema.hashCode is consistent with equals
- PR-1791 - DGS-1364 Fix compatibility for JSON empty schema
- PR-1784 - DGS-1221 Fix AvroSchema.equals
- PR-1323 - API update for Jetty 9.4.26.v20200117
Commercial Features¶
Confluent Server¶
- Correct syntax for password based schema registry config
- KC-1625: DynamicBrokerConfig must not log sensitive config keys
- METRICS-3248: Upgrade telemtry-client version
- METRICS-3264 Make TELEMETRY_PROXY_PASSWORD from STRING type to PASSWO
- CPKAFKA-3263: set number of tier topic partitions to 50
- KC-1741: Sync Jetty, Jersey, Netty, etc versions from 6.2.x
- KAFKA-12820: Upgrade maven-artifact dependency to resolve CVE-2021-26291
- SEC-298 - Log errors when MDS can not start
- KC-1610: Upgrade dataformat-cbor version
- CONFLUENT: Fix inconsistent range exception when determining unique log segments
- METRICS-2934: Do not log configuration values on failed Provider validation
- METRICS-2907: Make HttpExporterConfig proxy passwords Password objects
- MMA-9397: Filter out confluent.ssl configs when describing topic configs
- KAFKA-12400: Upgrade jetty to fix CVE-2020-27223
- KAFKA-12389: Upgrade of netty-codec due to CVE-2021-21290
- KC-1522: Do not log parsed configs containing passwords in CruiseControlStartable
- Limit awscli to version 1.18.* to be compatible with Python 3.4
- KAFKA-12211: don’t change perm for base/state dir when no persistent store
- KAFKA-12324: Upgrade jetty to fix CVE-2020-27218
Security¶
- DGS-1691: Perform license checks in background thread
- KQE-229: Update vertx-core to 3.9.7
- Pins Ksql to a jetty-jaas 9.4.28.v20200408 (#365) (#366)
- DGS-1463: Modify config types for sensitive SR validator plugin configs
Secret Registry¶
- SEC-2105: Update guava to version 30.1.1-jre
MQTT Proxy¶
- CCMSG-1101: Upgrade Netty to 4.1.63.Final
- CC-14445: Remove compile dependency on Kafka broker
- SEC-2022: Upgrade Netty to 4.1.62.Final.
Metadata Service¶
- SEC-2191: Properly handle unicode characters when rewriting urls
Replicator¶
- REPL-1410: Changed logging for WakeupException to DEBUG
- REPL-1386: Adding clear logging statements for src and dest operations
Version 6.0.2¶
Released February 2021
Community Features¶
Common¶
Kafka¶
- PR-10152 - KAFKA-12339: Add retry to admin client’s listOffsets
- PR-10158 - KAFKA-12343: Handle exceptions better in TopicAdmin, including UnsupportedVersionException
- PR-10153 - KAFKA-12340: Fix potential resource leak in Kafka*BackingStore
- PR-10122 - KAFKA-12326: Corrected regresion in MirrorMaker 2 executable introduced with KAFKA-10021
- PR-10098 - KAFKA-12321 the comparison function for uuid type should be ‘equals’ …
- PR-10103 - KAFKA-12272: Fix commit-interval metrics
- PR-9780 - KAFKA-10021: Changed Kafka backing stores to use shared admin client to get end offsets and create topics
- PR-9977 - KAFKA-12310: Update zookeeper to 3.5.9
- PR-10060 - KAFKA-10716: persist UUID in state directory for stable processId across restarts - 2.7
- PR-10061 - KAFKA-12193: Re-resolve IPs after a client disconnects
- PR-10032 - KAFKA-12270: Handle race condition when Connect tasks attempt to create topics
- PR-10040 - KAFKA-12259: Use raw config to infer connector type when returning connect status response
- PR-9987 - KAFKA-10895: Gracefully handle invalid JAAS configs (follow up fix)
- PR-9319 - KAFKA-10413: Allow for even distribution of lost/new tasks when multiple Connect workers join at the same time
- PR-9247 - KAFKA-10362: When resuming Streams active task with EOS, the checkpoint file is deleted
- PR-10020 - KAFKA-10689: fix windowed FKJ topology and put checks in assignor to avoid infinite loops
- PR-9671 - KAFKA-10793: move handling of FindCoordinatorFuture to fix race condition
- PR-9525 - KAFKA-10658 ErrantRecordReporter.report always return completed futur…
- PR-9983 - KAFKA-8930: MirrorMaker v2 documentation (#324)
- PR-9765 - KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations
- PR-9968 - KAFKA-12190: Fix setting of file permissions on non-POSIX filesystems
- PR-9669 - KAFKA-10792: Prevent source task shutdown from blocking herder thread
- PR-9806 - KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect
- PR-9783 - KAFKA-10815 EosTestDriver#verifyAllTransactionFinished should break l…
- PR-9678 - KAFKA-10798; Ensure response is delayed for failed SASL authentication with connection close delay
- PR-9702 - CVE-2020-25649: bumping jackson to patched version 2.10.5.1
- PR-9698 - KAFKA-10811: Correct the MirrorConnectorsIntegrationTest to correctly mask the exit procedures
- PR-8965 - KAFKA-8147: Add changelog topic configuration to KTable suppress
- PR-9648 - KAFKA-10758: ProcessorTopology should only consider its own nodes when updating regex source topics
- PR-9633 - KAFKA-10706; Ensure leader epoch cache is cleaned after truncation to end offset
- PR-9634 - KAFKA-10755: Should consider commit latency when computing next commit timestamp
- PR-9583 - [KAFKA-10705]: Make state stores not readable by others
ksqlDB¶
- PR-7052 - fix: incompatible schema causing KsqlEngineTest.shouldNotFailIfAvroSchemaEvolvable failure
- PR-7032 - fix: format cast arguments with passed context (6.0.x)
- PR-6976 - fix: Changes a query validation tests after upgrade to antlr
- PR-6935 - fix: upgrade httpclient version
- PR-6934 - fix: upgrade antlr to 4.7.1
- PR-6800 - fix: fixes LDAP login authentication method
- PR-6732 - fix: (minor) don’t use deprecated jersey calls
- PR-6702 - fix: use Java’s Base64 instead of jersey’s
REST Proxy¶
- PR-228 - MMA-9930: Use compatible version of Guava
- PR-222 - UIF-300: Upgrade org.apache.httpcomponents.httpclient
- PR-799 - KREST-558 Backport the fix for KREST-524 (fixing a concurrent test failure) to 5.5.x and 6.0.x.
- PR-223 - UIF-302: Upgrade guava for 5.4.x and older.
- PR-224 - upgrade hibernate-validator to 6.1.7.Final
- PR-216 - MMA-7520: add CsrfTokenProtectionFilter
- PR-209 - (backport) SEC-592: Customize Jetty request queue and threadpool size (#172)
- PR-215 - ff-3928: bump version of Guava in parent of schema-registry.
- PR-210 - ST-4542: Upgrade jetty to address CVE-2020-27216
Schema Registry¶
- PR-1779 - DGS-772 Add API annotations for /mode
- PR-1777 - Upgrade to everit 1.12.2
- PR-1639 - Upgrade to wire 3.6.0
- PR-1776 - MINOR: remove duplicate guava entries from pom.xml
- PR-1775 - Fix Avro compat check
- PR-1774 - Add config to remove Java props from Avro schemas
- PR-1770 - DGS-1315: upgrade HibernateValidator
- PR-1769 - DGS-1314: update JetBeanKotlin to 1.4.21
- PR-1760 - DGS-1244 Resolve dependencies before caching
- PR-1757 - DG-1239 Fix Avro Converter for union with null, string, enum
- PR-1739 - DG-980 Support ISO 8601 for JSON date/time
- PR-1754 - DG-1237 Handle missing fields in JsonSchemaConverter
- PR-1753 - DG-1236 Improve oneOf handling in JsonSchemaConverter
- PR-1748 - DG-424: prevent import different schema with same id
- PR-1747 - DG-1189 Add latest.compatibility.strict flag
- PR-1746 - DG-931: throw ConfigException in Avro/Json/Protobuf Converter
- PR-1724 - DG-1124 NPE in ProtobufMessageFormatter
- PR-1722 - DG-1122 Treat reordering Protobuf msgs as incompatible
- PR-1700 - DG-293: Improve exception messages in JsonSchemaConverter and ProtobufConverter
- PR-1699 - DG-1040 Register ids for soft deletes
Commercial Features¶
Confluent Server¶
- SEC-1805: Reduce failed MDS request send log threshold from ERROR to Debug
- CONFLUENT: Update Zookeeper node start timeout in system tests
- KAFKA-10727; Handle Kerberos error during re-login as transient failure
- CPKAFKA-3077: Handle exceptions when uploading a segment that may have been deleted
- KC-1459: Reducing logging level for license expiry from INFO to DEBUG
- KC-924: Add event processing metrics for ZK ACLs and MDS records
- KC-922, KC-1238: Handle writer stop exceptions to ensure new writer is re-elected
- CONFLUENT: Make RbacClusters.java to work with Writer/AuthWriter interfaces
- KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations
- KSTORAGE-1348 Flush epoch cache before reinitialization in Log.rename
- KSQL-5843: prevent log spam from MDS RestClient on client close
- CNK-1206: Update dependencies for SBK serializers (#2634)
- SEC-1789 Bump Bouncy Castle dep to resolve CVE-2020-28052
- CPKAFKA-3004: System tests for client compatibility and replication with tiering enabled
- KC-1275: Update KafkaReader to handle entries from multiple partitions
- KSTORAGE-1290: Bump google storage client dependency to 1.113.4
- CPKAFKA-3016: Use regular offsets out of range handling for tiered storage
- KSTORAGE-1000: TierDeletedPartitionsCoordinator Handling of Emigration During Read
- KSTORAGE-1255: Reduce default interval to detect deleted partitions
- KC-428: Always log authorize results to file based on log4j settings
- CPKAFKA-6126: Update commons codec
Secret Registry¶
- CC-9785: Fix build by unifying Bouncy Castle dependencies (#36)
MQTT Proxy¶
- Fix telemetry reporter test on 6.0.x
Replicator¶
- REPL-1409: Added WARN logging if destination SR deemed not ready
- fix: upgrade airlist:airline to upgrade guava
- MINOR: fixed curator cve
- REPL-1396: Fix CVE for HttpClient
- SEC:1820: Upgrade codec version to fix CVE
- REPL-1396: Fix CVE for aws-java-sdk-ecr
- REPL-1310: Fix for monitor thread failures in start stop start cycle
- REP-1300: Fix issue where a stopped connector would hang on restart
- REPL-1303: Setting deadline correctly when trying to resume partitions
- FF-3928: Bump the version of Guava to fix CVE
Version 6.0.1¶
Released December 2020
Community Features¶
Kafka¶
- PR-9468 - KAFKA-10454 / (2.6) Update copartitionSourceGroups when optimization algorithm is triggered
- PR-9515 - KAFKA-10651: read offsets directly from checkpoint for uninitialized tasks
- PR-9534 - KAFKA-10664: Delete existing checkpoint when writing empty offsets
- PR-9467 - KAFKA-10515: Properly initialize nullable Serdes with default values
- PR-9270 - KAFKA-10284: Group membership update due to static member rejoin should be persisted
- PR-9431 - KAFKA-10426: Deadlock on session key update.
- PR-9383 - KAFKA-10455: Ensure that probing rebalances always occur
- PR-9343 - KAFKA-10332: Update MM2 refreshTopicPartitions() logic
- PR-9425 - KAFKA-10600: Connect should not add error to connector validation values for properties not in connector’s ConfigDef
- PR-9422 - KAFKA-10602: Make RetryWithToleranceOperator thread safe
- PR-9149 - KAFKA-10340: Improve the trace logging under connector based topic creation
- PR-9375 - KAFKA-10574: Fix infinite loop in Values::parseString
- PR-9408 - KAFKA-10598: Improve IQ name and type checks
- PR-9020 - KAFKA-10271 Performance regression while fetching a key from a single partition
- PR-9373 - KAFKA-10564: only process non-empty task directories when internally cleaning obsolete state stores
- PR-8910 - KAFKA-10188: Prevent SinkTask::preCommit from being called after SinkTask::stop
- PR-9320 - KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale.
- PR-9323 - KAFKA-10514: Fix unit test for state directory cleanup
- PR-9347 - KAFKA-10531: Check for negative values to Thread.sleep call
- PR-9306 - KAFKA-10477: Enabling the same behavior of NULL JsonNodeType to MISSI
- PR-9354 - KAFKA-10134 Follow-up: Set the re-join flag in heartbeat failure
- PR-8973 - KAFKA-10218: Stop reading config topic in every subsequent tick if catchup fails once
- PR-8181 - KAFKA-9584 Headers ConcurrentModificationException
- PR-9312 - KAFKA-10505: Fix parsing of generation log string.
- PR-9202 - KAFKA-10401: Fix the currentStateTimeStamp doesn’t get set correctly
- PR-9286 - KAFKA-10292: Set min.insync.replicas to 1 of __consumer_offsets
- PR-8834 - KAFKA-10134: Enable heartbeat during PrepareRebalance and Depend On State For Poll Timeout
- PR-9219 - KAFKA-10432: LeaderEpochCache is incorrectly recovered for leader epoch 0
- PR-9174 - KAFKA-10395: relax output topic check in TTD to work with dynamic routing
- PR-9197 - Revert KAFKA-9309: Add the ability to translate Message to JSON
- PR-9172 - KAFKA-10387: Fix inclusion of transformation configs when topic creation is enabled in Connect
- PR-9183 - KAFKA-10404; Use higher poll timeout to avoid rebalance in testCoordinatorFailover
- PR-9184 - KAFKA-8033; Wait for NoOffsetForPartitionException in testFetchInvalidOffset
- PR-9181 - KAFKA-9516; Increase timeout in testNonBlockingProducer to make it more reliable
- PR-8938 - KAFKA-10173: Use SmokeTest for upgrade system tests
ksqlDB¶
- PR-6559 - fix: add back configs for setting TLS protocols and cipher suites (6.0.x)
- PR-6482 - fix: Internal Server Error for /healthcheck endpoint in RBAC-enabled
- PR-6472 - fix: check for nested UnspportedVersionException during auth op check
- PR-6608 - fix: Check for index before removing value in undo of COLLECT_LIST (#…
- PR-6607 - chore: remove CLI version checker agent
Schema Registry¶
- PR-1676 - DG-972 Fix name collisions with field and message
- PR-1675 - DG-962 Handle const schemas in JSON Schema converter
- PR-1668 - Removed property ${confluent.maven.repo}
- PR-1658 - DG-951 Handle duplicate message names during deserialization
- PR-1648 - DG-935 Treat removing Protobuf message types as incompatible
- PR-1647 - DG-933 Resend old value when handling invalid key-value
- PR-1634 - DG-921 Fix Avro specific reader for unions
- PR-1625 - DG-410 Fix doc/defaults in Avro converter
- PR-1684 - MMA-8608: parse configs before calling restService.configure()
- PR-1679 - MINOR: log CachedSchemaRegistryClient configs
- PR-1678 - DG-990 Allow SpecificAvroSerde to work with array/map
- PR-1688 - DG-1007 Preserve refs when converting schema to envelope
- PR-1686 - MMA-8608 Handle duplicate configs after stripping prefix
Commercial Features¶
Confluent Server¶
- CNK-1332: Handle TopicExistsException when creating SBK topics
- KC-1195: Ensure connections are closed if principal cannot be built
- SEC-1474–Upgraded bcpkix-jdk15on to 1.66
- Backport METRICS-1805: remove duplicate resource type label (#2744, #2755)
- Add unit tests to disable tier storage at broker level
- KSTORAGE-1180 Update tier enable flag at FileTierPartitionState unconditionally
- KSTORAGE-506: Prevent accidental change to critical configurations for tier topic (#1414)
- CNK-1304: Update remove broker validation to use strict collections
- CNKAF-1068: Add validation for broker removal for rf equal to alive brokers
- CNKAF-1062: Add validation for broker removal leading to under min isr
- CNKAF-920: Raise invalid broker removal exception with specific error message
- CNK-1331: Fix typo in SBK Persistence Store initialization
- backport METRICS-2396: Add extra checks to ensure that we do not leak EventExporter objects
- backport METRICS-2110: document Telemetry Reporter packaging
- KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline
- KSTORAGE-725: Exclude segments where the high offset is lower than the lowerBoundOffset for tiering.
- KSTORAGE-1004: Restore tool should be able to work standalone without comparator tool
- KSTORAGE-986: Check for invalid topic partitions within the recovery tool
- KSTORAGE-972: Modify TopicIdPartition.toString() implementation to use ‘:’ and ‘-‘ as delimiters
- KSTORAGE-973: Check for invalid topic partitions within the fencing tool
- KSTORAGE-974: Print tiertopic partition and timestamp for fencing tool
- KSTORAGE-1100: Offset cache can’t be disabled although we have the facility to do so
- KSTORAGE-1074: retry should log at INFO level
- KSTORAGE-970: Fix help screen in TierTopic tooling
- KSTORAGE-1067: Dump fetch context when tiered storage runs into an unexpected error
- CNKAF-915 Improve log level in executor
- METRICS-1819: support using port 0 for telemetry local exporter
- CNK-374: Return task history for only configured retention period
- CNKAF-1134: Close AdminClient, Producers and Consumers immediately
- CNK-42: Only enable Databalancer Count metric if balancer is enabled
- CNKAF-1109: Use AdminClient for Leader Elections, handle exceptions
- CNKAF-1114: Shutdown CruiseControl at startup if exception is thrown
- CNKAF-1001: Retry plan computation in Broker Removal
- CNKAF-1224: Log all uncaught SBK exception and errors
- [CNKAF-1195] Avoid stale cluster metadata for AnomalyDetection plans.
- CNKAF-1015: Fix issue in SBK startup by moving CruiseControl instantiation after the startup components
- [CNKAF-963] Switch AddBroker plan computation to wait METADATA_TTL.
- CNKAF-954: Removes GoalOptimizer proposal caching
- CNKAF-762: Abstract CruiseControl startup and config translation logic into own package
- CNKAF-985: Bump leader distribution goal priority
- CNKAF-849: Sanitize log messages when checking for startup conditions
- [CNKAF-867] Handle empty proposals for addBroker
- Backports [METRICS-2356] patch version for thread leak fix (#2681)
- CC-12141: Confluent’s WorkerConfigDecorator in Connect should handle connector validate responses with extra properties
- Backport CC-11832: add connector name to metrics context when calling Kafka Client
- backport ce-metrics: add space to reconcileConfigs() log message
- backport METRICS-2358: only initialize event logger when enabled, properly close it, and add better support for reconfiguration (#2644)
- KC-1120: Ensure that RBAC metadata write failures are reset after writer elections
- KSTORAGE-797: allow recovery path materializer to use secured listeners (#2572)
- KSTORAGE-1260: tiered segments do not correctly block retention for local segments (#2831)
- KSTORAGE-835: producer state is loaded twice in MergedLog/Log init (#2104)
Security¶
- Implement asychronous schema fetching in LruSchemaRegistryClient
Metadata Service¶
- SEC-1507 cluster information migration script should validate properties file
- [MMA-9298]: bump up the version for nice name feature
- [SEC-1506]: cluster-information-migration-script - allow user to specify curl options
- [SEC-1514]: cluster-information-migration-script - suppress progress but show errors
- [SEC-1499]: Add –help option to cluster registry import/export script
- [SEC-1471]: Backport Friendly cluster name from master to 6.0.x
- CIAM-338 - Test MyRoleBindings endpoints
- SEC-1565: Incorrect implementation of equals in ManagedRoleBindings
Replicator¶
- REPL-1204 enable retries on cluster id fetch for Replicator task start
- REPL-1263: Supporting source url configs for active active SR
Version 6.0.0¶
Released September 2020
Community Features¶
Common¶
- PR-294 - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- PR-278 - SEC-1334: update confluent-log4j version
- PR-277 - KC-613: Update slf4j version to match AK’s.
- PR-274 - ST-3988: Update jackson version to 2.10.5
- PR-275 - SEC-1305: update jackson-bom version
- PR-272 - METRICS-1918: bump gson from 2.8.5 -> 2.8.6
- PR-270 - SEC-1034: log4j migration to confluent repackaged version
- PR-269 - Upgrade Jackson to 2.10.2 and Avro to 1.9.2
- PR-265 - downgrade maven-deploy-plugin to 2.8.2
- PR-258 - update various maven plugins
- PR-257 - ST-1323: Enable source jars if Apache or Confluent license files are found
- PR-262 - fix: Pin the Scala version
- PR-261 - Update ZooKeeper to 3.5.8
- PR-260 - Update Scala version to 2.13
- PR-259 - add property for jackson-jsonschema version
- PR-248 - Upgrade Avro to 1.9.2
Kafka¶
- PR-9197 - Revert KAFKA-9309: Add the ability to translate Message to JSON
- PR-9172 - KAFKA-10387: Fix inclusion of transformation configs when topic creation is enabled in Connect
- PR-392 - KAFKA-2715: Removed previous system_test folder
- PR-384 - KAFKA-2714: Added integration tests for exceptional cases in fetching
- PR-9089 - KAFKA-10224: Update jersey license from CDDL to EPLv2
- PR-9066 - KAFKA-10287: Skip unknown offsets when computing sum of changelog offsets
- PR-9086 - FIX: Remove staticmethod tag to be able to use logger of instance
- PR-9075 - KAFKA-10306: GlobalThread should fail on InvalidOffsetException
- PR-9065 - KAFKA-10301: Do not clear Partition#remoteReplicasMap during partition assignment updates
- PR-9051 - KAFKA-10268: dynamic config like “–delete-config log.retention.ms” doesn’t work
- PR-9026 - KAFKA-10274; Consistent timeouts in transactions_test
- PR-8979 - KAFKA-10223; Use NOT_LEADER_OR_FOLLOWER instead of non-retriable REPLICA_NOT_AVAILABLE for consumers
- PR-8239 - KAFKA-9666: Don’t increase transactional epoch when trying to fence if the log append fails
- PR-9009 - KAFKA-6453: Document how timestamps are computed for aggregations and joins
- PR-8994 - KAFKA-10247: Correctly reset state when task is corrupted
- PR-9010 - KAFKA-10262: Ensure that creating task directory is thread safe
- PR-9005 - KAFKA-10263: Do not assign standby for revoking stateless tasks
- PR-8996 - KAFKA-10249: don’t try to read un-checkpointed offsets of in-memory stores
- PR-8934 - KAFKA-10134: Use long poll if we do not have fetchable partitions
- PR-8989 - KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups
- PR-8962 - KAFKA-10166: checkpoint recycled standbys and ignore empty rocksdb base directory
- PR-8983 - KAFKA-8398: Prevent NPE in forceUnmap
- PR-8858 - KAFKA-10153: Error Reporting in Connect Documentation
- PR-8917 - KAFKA-10180: Fix security_config caching.
- PR-8905 - KAFKA-10173: Fix suppress changelog binary schema compatibility
- PR-8926 - KAFKA-10166: always write checkpoint before closing an (initialized) task
- PR-8924 - KAFKA-10198: guard against recycling dirty state
- PR-8900 - KAFKA-10169: swallow non-fatal KafkaException and don’t abort transaction during clean close
- PR-8887 - KAFKA-10135: Extract Task#executeAndMaybeSwallow to be a general utility function into TaskManager…
- PR-8896 - KAFKA-10185: Restoration info logging
- PR-8890 - KAFKA-9891: add integration tests for EOS and StandbyTask
- PR-8822 - KAFKA-10113; Specify fetch offsets correctly in LogTruncationException
- PR-8876 - KAFKA-10167: use the admin client to read end-offset
- PR-8841 - KAFKA-10123 Fix incorrect value for AWAIT_RESET#hasPosition
- PR-8853 - KAFKA-10147 MockAdminClient#describeConfigs(Collection<ConfigResource…
- PR-8882 - KAFKA-10165: Remove Percentiles from e2e metrics
- PR-8856 - KAFKA-10150: task state transitions/management and committing cleanup
- PR-8880 - KAFKA-10169: Error message when transit to Aborting / AbortableError / FatalError
- PR-8872 - Fix log message for transition from standby to active
- PR-8873 - Avoid WARN log message when re-init from checkpoint skipped
- PR-8849 - KAFKA-10144: clean up corrupted standby tasks before attempting a commit
- PR-8764 - KAFKA-10049: Fixed FKJ bug where wrapped serdes are set incorrectly when using default StreamsConfig serdes
- PR-8787 - KAFKA-10085: correctly compute lag for optimized source changelogs
- PR-8818 - KAFKA-10086: Integration test for ensuring warmups are effective
- PR-8069 - KAFKA-9374: Make connector interactions asynchronous
- PR-8828 - KAFKA-9216: Enforce that Connect’s internal topics use compact cleanup policy
- PR-8829 - KAFKA-10115: Incorporate errors.tolerance with the Errant Record Reporter
- PR-8455 - KAFKA-9845: Warn users about using config providers with plugin.path property
- PR-8502 - KAFKA-9066: Retain metrics for failed tasks
- PR-8847 - KAFKA-7833: Add missing test
- PR-8453 - KAFKA-9841: Revoke duplicate connectors and tasks when zombie workers return with an outdated assignment
- PR-8663 - KAFKA-9985: Sink connector may exhaust broker when writing in DLQ
- PR-8833 - KAFKA-9441: remove prepareClose() to simplify task management
- PR-8825 - KAFKA-7833: Add StateStore name conflict check in InternalTopologyBuilder
- PR-8630 - KAFKA-9969: Exclude ConnectorClientConfigRequest from class loading isolation
- PR-8775 - KAFKA-10079: improve thread-level stickiness
- PR-7384 - KAFKA-8938: Improve allocations during Struct validation in ConnectSchema
- PR-8827 - KAFKA-9849: Fix issue with worker.unsync.backoff.ms creating zombie workers when incremental cooperative rebalancing is used
- PR-8805 - KAFKA-9848: Avoid triggering scheduled rebalance delay when task assignment fails but Connect workers remain in the group
- PR-8783 - KAFKA-10063 UnsupportedOperation when querying cleaner metrics after shutdown
- PR-8803 - KAFKA-10102: update ProcessorTopology instead of rebuilding it
- PR-8676 - KAFKA-10005: Decouple RestoreListener from RestoreCallback
- PR-8684 - KAFKA-10012 Reducing memory overhead associated with strings in SelectorMetrics
- PR-8270 - KAFKA-9216: Enforce internal config topic settings for Connect workers during startup
- PR-8820 - KAFKA-10097: Internalize checkpoint data
- PR-8804 - KAFKA-9851: Revoking Connect tasks due to connectivity issues should also clear the running assignment
- PR-8486 - KAFKA-9840: Skip End Offset validation when the leader epoch is not reliable
- PR-8135 - KAFKA-9570: Define SSL configs in all worker config classes, not just distributed
- PR-8814 - KAFKA-10111: Make SinkTaskContext.errantRecordReporter() a default method
- PR-8811 - KAFKA-10110: Corrected potential NPE when null label value added to KafkaMetricsContext
- PR-8776 - KAFKA-9441: Improve Kafka Streams task management
- PR-8759 - KAFKA-10066: TestOutputTopic should pass record headers into deserializers
- PR-8755 - KAFKA-10069: Correctly remove user-defined “predicate” and “negate” configs from transformation properties
- PR-8644 - KAFKA-9313: Set use_all_dns_ips as the new default for client.dns.lookup (KIP-602)
- PR-8784 - KAFKA-9788: Use distinct names for transaction and group load time sensors
- PR-8792 - KAFKA-10089 The stale ssl engine factory is not closed after reconfigure
- PR-8782 - KAFKA-10080; Fix race condition on txn completion which can cause duplicate appends
- PR-8785 - KAFKA-10084: Fix EosTestDriver end offset
- PR-8786 - KAFKA-10083: fix failed testReassignmentWithRandomSubscriptionsAndChanges tests
- PR-8737 - KAFKA-9945: TopicCommand should support –if-exists and –if-not-exists when –bootstrap-server is used
- PR-8695 - KAFKA-9320: Enable TLSv1.3 by default (KIP-573)
- PR-8777 - KAFKA-10082: Fix the failed testMultiConsumerStickyAssignment
- PR-8706 - KAFKA-10030 allow fetching a key from a single partition
- PR-8760 - Kafka-10064 Add documentation for KIP-571
- PR-8668 - KAFKA-9987: optimize sticky assignment algorithm for same-subscription case
- PR-7956 - KAFKA-9392; Clarify deleteAcls javadoc and add test for create/delete timing
- PR-8769 - KAFKA-10074: Improve performance of matchingAcls
- PR-8723 - KIP-569-KAFKA-9494: DescribeConfigsResponse - include additional metadata information
- PR-8739 - KAFKA-10056; Ensure consumer metadata contains new topics on subscription change
- PR-8705 - KAFKA-10029; Don’t update completedReceives when channels are closed to avoid ConcurrentModificationException
- PR-8238 - KAFKA-9130: KIP-518 Allow listing consumer groups per state
- PR-8248 - KAFKA-9501: convert between active and standby without closing stores
- PR-8699 - KAFKA-9673: Filter and Conditional SMTs
- PR-8720 - KAFKA-9971: Error Reporting in Sink Connectors (KIP-610)
- PR-8736 - KAFKA-9802; Increase transaction timeout in system tests to reduce flakiness
- PR-8731 - KAFKA-10050: kafka_log4j_appender.py fixed for JDK11
- PR-8589 - KAFKA-9146: KIP-571 Add option to force delete active members in StreamsResetter
- PR-8691 - KAFKA-9960: implement KIP-606 to add metadata context to MetricsReporter
- PR-8709 - KAFKA-9952; Remove immediate fetch completion logic on high watermark updates
- PR-8692 - KAFKA-10018: Change command line tools from /bin/sh to /bin/bash
- PR-8697 - KAFKA-9983: KIP-613, add INFO level e2e latency metrics
- PR-8376 - KAFKA-9724 Newer clients not always sending fetch request to older brokers
- PR-8504 - KAFKA-9298: reuse mapped stream error in joins
- PR-8716 - KAFKA-6145: KIP-441: Fix assignor config passthough
- PR-6824 - KAFKA-7523: Add ConnectedStoreProvider to Processor API
- PR-8221 - KAFKA-9561: update task input partitions after rebalance
- PR-7886 - KAFKA-9353: Add groupInstanceId to DescribeGroup for better visibility
- PR-8722 - KAFKA-5295: Allow source connectors to specify topic-specific settings for new topics (KIP-158)
- PR-8395 - Added doc for KIP-535 and updated it for KIP-562
- PR-8118 - KAFKA-9472: Remove deleted Connect tasks from status store
- PR-2604 - KAFKA-4794: Add access to OffsetStorageReader from SourceConnector
- PR-8698 - KAFKA-10022:console-producer supports the setting of client.id
- PR-6284 - KAFKA-6755: Allow literal value for MaskField SMT
- PR-8357 - KAFKA-9767: Add logging to basic auth rest extension
- PR-8620 - KAFKA-9944: Added supporting customized HTTP response headers for Kafka Connect.
- PR-8511 - KAFKA-9888: Copy connector configs before passing to REST extensions
- PR-8654 - KAFKA-9931: Implement KIP-605 to expand support for Connect worker internal topic configurations
- PR-8713 - KAFKA-6145: Add unit tests for assignments of only stateless tasks
- PR-8696 - KAFKA-6145: KIP-441: Enforce Standby Task Stickiness
- PR-8679 - KAFKA-10003: Mark KStream.through() as deprecated
- PR-8658 - KAFKA-9980: Fix bug where alterClientQuotas could not set default client quotas
- PR-8379 - KAFKA-9780: Deprecate commit records without record metadata
- PR-8651 - kafkatest: Deploy VerifiableClient in constructor to avoid test timeouts
- PR-8472 - KAFKA-9855 - return cached Structs for Schemas with no fields
- PR-8608 - KAFKA-9950: Construct new ConfigDef for MirrorTaskConfig before defining new properties
- PR-8444 - KAFKA-8869: Remove task configs for deleted connectors from config snapshot
- PR-7942 - KAFKA-9409: Supplement immutability of ClusterConfigState class in Connect
- PR-8661 - KAFKA-9603: Do not turn on bulk loading for segmented stores on stand-by tasks
- PR-8671 - KAFKA-9859 / Add topics generated by KTable FK join to internal topic matching logic
- PR-8681 - KAFKA-10010: Should make state store registration idempotent
- PR-8682 - KAFKA-10011: Remove task id from lockedTaskDirectories during handleLostAll
- PR-8673 - KAFKA-9992: EmbeddedKafkaCluster.deleteTopicAndWait not working with kafka_2.13
- PR-8689 - KAFKA-6145: Add unit tests to verify fix of bug KAFKA-9173
- PR-8675 - KAFKA-10004: ConfigCommand fails to find default broker configs without ZK
- PR-8569 - KIP-551: Expose disk read and write metrics
- PR-8678 - Update Gradle to 6.4.1
- PR-8659 - KAFKA-9617 Replica Fetcher can mark partition as failed when max.message.bytes is changed
- PR-8618 - KAFKA-9955: Prevent SinkTask::close from shadowing other exceptions
- PR-8558 - KAFKA-8611 / KIP-221 documentation
- PR-8670 - KAFKA-10001: Should trigger store specific callback if it is also a listener
- PR-8090 - KAFKA-9537 - Cleanup error messages for abstract transformations
- PR-8674 - KAFKA-9996: Upgrade zookeeper to 3.5.8
- PR-8667 - KAFKA-9994: Handle task migrated inside corruption path
- PR-8665 - KAFKA-9984 Should fail the subscription when pattern is empty
- PR-8222 - KAFKA-9650: include human readable units in ms and bytes configs
- PR-8588 - KAFKA-6145: KIP-441: Improve assignment balance
- PR-313 - KAFKA-2641: Upgrade path for ZK authentication
- PR-8550 - KAFKA-9850 Move KStream#repartition operator validation during Topology build process
- PR-8648 - KAFKA-9966: Add internal assignment listener to stabilize EOS version 2 upgrade test
- PR-8596 - KAFKA-9821: persist followup rebalance in assignment and consolidate rebalance triggering mechanisms
- PR-8647 - KAFKA-9669; Loosen validation of inner offsets for older message formats
- PR-8254 - KIP-557: Add Emit On Change Support
- PR-8626 - KAFKA-9921: explicit handling of null values with retainDuplicates
- PR-8643 - KAFKA-9956: Authorizer APIs may be invoked more than once for a given request
- PR-8632 - KAFKA-9972: Only commit tasks with valid states
- PR-8628 - KAFKA-9942: Fixes ConfigCommand client quotas w/ default users.
- PR-8613 - KAFKA-6145: Set HighAvailabilityTaskAssignor as default in streams_upgrade_test.py
- PR-8230 - KAFKA-9667: Connect JSON serde strip trailing zeros
- PR-8483 - KAFKA-9865: Expose output topic names from TopologyTestDriver
- PR-8360 - KAFKA-9768: Fix handling of rest.advertised.listener config
- PR-7950 - KAFKA-9419: Integer Overflow Possible with CircularIterator
- PR-8602 - KAFKA-9947; Ensure proper shutdown of components in TransactionsBounceTest
- PR-8591 - KAFKA-6342: Move workaround for JSON parsing of non-escaped strings
- PR-8609 - KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
- PR-8496 - KAFKA-9748: Add Streams EOS version 2 integration test
- PR-8607 - KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
- PR-8527 - Remove dependencies on deprecated –zookeeper command flags in junit tests
- PR-8551 - KAFKA-9918 SslEngineFactory is NOT closed when channel is closing
- PR-8565 - KAFKA-9798: Send one round synchronously before starting the async producer
- PR-8533 - KAFKA-9589: Fixed bug in V2 log validator tests
- PR-8577 - use appropriate fn for readability. (maybe)
- PR-8586 - KAFKA-9939; Fix overcounting delayed fetches in request rate metrics
- PR-8595 - KAFKA-8410: Revert Part 1: processor context bounds (#8414)
- PR-8597 - KAFKA-6145: KIP 441 remove balance factor
- PR-8590 - KAFKA-6145: Remove check to reuse previous assignment
- PR-8204 - KAFKA-9633: Ensure ConfigProviders are closed
- PR-8554 - KAFKA-9919: Add logging to KafkaBasedLog::readToLogEnd
- PR-8567 - KAFKA-9652: Fix throttle metric in RequestChannel and request log due to KIP-219
- PR-8559 - KAFKA-9922:Update examples README
- PR-8442 - KAFKA-9830: Implement AutoCloseable in ErrorReporter and subclasses
- PR-8578 - KAFKA-9875: Make integration tests more resilient
- PR-8582 - KAFKA-9932: Don’t load configs from ZK when the log has already been loaded
- PR-8574 - KAFKA-9925: decorate pseudo-topics with app id
- PR-8580 - KAFKA-9832: fix attempt to commit non-running tasks
- PR-8540 - KAFKA-9127: don’t create StreamThreads for global-only topology
- PR-8568 - KAFKA-9176: Retry on getting local stores from KafkaStreams
- PR-8548 - KAFKA-9823: Follow-up, check state for handling commit error response
- PR-8541 - KAFKA-6145: KIP-441: Add TaskAssignor class config
- PR-8564 - KAFKA-9921: disable caching on stores configured to retain duplicates
- PR-8566 - Fix minor code issue
- PR-8525 - KAFKA-9885; Evict last members of a group when the maximum allowed is reached
- PR-8524 - KAFKA-9866: Avoid election for topics where preferred leader is not in ISR
- PR-8509 - KAFKA-9839: Broker should accept control requests with newer broker epoch
- PR-8224 - KAFKA-9704: Fix the issue z/OS won’t let us resize file when mmap.
- PR-8326 - KAFKA-8639: Replace AddPartitionsToTxn with Automated Protocol
- PR-8445 - KAFKA-9823: Remember the sent generation for the coordinator request
- PR-8536 - KAFKA-9883: Add better error message when REST API forwards a request and leader is not known
- PR-8537 - KAFKA-9907: Switch default build to Scala 2.13
- PR-8464 - KAFKA-9852: Change the max duration that calls to the buffer pool can block from 2000ms to 10ms
- PR-8417 - KAFKA-8955: Add an AbstractResponse#errorCounts(Stream) and tidy
- PR-8530 - KAFKA-9388: Refactor integration tests to always use different application ids
- PR-8482 - KAFKA-9863: update the deprecated –zookeeper option in the documentation into –bootstrap-server
- PR-8497 - KAFKA-6145: KIP-441 Build state constrained assignment from balanced one
- PR-8520 - Add explicit grace period to tumbling window example
- PR-8522 - KAFKA-9868: Reduce transaction log partitions for embed broker
- PR-8523 - Adding github whitelist
- PR-8475 - KAFKA-6145: KIP-441: Add test scenarios to ensure rebalance convergence
- PR-8501 - KAFKA-9881: Convert integration test to verify measurements from RocksDB to unit test
- PR-8448 - KAFKA-9796; Broker shutdown could be stuck forever under certain conditions
- PR-8471 - KAFKA-9854 Re-authenticating causes mismatched parse of response
- PR-8476 - KAFKA-9838; Add log concurrency test and fix minor race condition
- PR-8378 - KAFKA-9779: Add Stream System Test 2.5
- PR-6210 - KAFKA-7885: TopologyDescription violates equals-hashCode contract.
- PR-8092 - KAFKA-9540: Move “Could not find the standby task while closing it” log to debug level
- PR-8256 - KAFKA-9675: Fix bug that prevents RocksDB metrics to be updated
- PR-8440 - KAFKA-9832: extend Kafka Streams EOS system test
- PR-8492 - KAFKA-9739: Fix for 2.5 branch
- PR-8403 - KAFKA-9797; Fix TestSecurityRollingUpgrade.test_enable_separate_interbroker_listener
- PR-8477 - KAFKA-9864: avoid expensive QuotaViolationException usage
- PR-8469 - [KAFKA-9826] Handle an unaligned first dirty offset during log cleaning.
- PR-8470 - KAFKA-8611 / Refactor KStreamRepartitionIntegrationTest
- PR-8257 - KAFKA-9539; Add leader epoch in StopReplicaRequest (KIP-570)
- PR-8353 - KAFKA-9764: Remove stream simple benchmark suite
- PR-8457 - KAFKA-9842; Add test case for OffsetsForLeaderEpoch grouping in Fetcher
- PR-8077 - KAFKA-9583: use topic-partitions grouped by node to send OffsetsForLeaderEpoch requests
- PR-8474 - KAFKA-9853: Improve performance of Log.fetchOffsetByTimestamp
- PR-7015 - KAFKA-8436: use automated protocol for AddOffsetsToTxn
- PR-8399 - KAFKA-3720: Change TimeoutException to BufferExhaustedException when no memory can be allocated for a record within max.block.ms
- PR-8443 - KAFKA-9832: Extend Streams system tests for EOS version 2
- PR-8436 - KAFKA-6145: KIP-441 avoid unnecessary movement of standbys
- PR-7170 - KIP-221 / Add KStream#repartition operation
- PR-7844 - KAFKA-9309: Add the ability to translate Message classes to and from JSON
- PR-8441 - KAFKA-9748: Extend Streams integration tests for EOS version 2
- PR-8212 - KAFKA-9642: Change “BigDecimal(double)” constructor to “BigDecimal.valueOf(double)”
- PR-8317 - KAFKA-9691: Fix NPE by waiting for reassignment request
- PR-8315 - KAFKA-9433: Use automated protocol for AlterConfigs request and response
- PR-8425 - KAFKA-6145: KIP-441 Move tasks with caught-up destination clients right away
- PR-8451 - KAFKA-9835; Protect FileRecords.slice from concurrent write
- PR-8409 - KAFKA-6145: KIP-441 Pt. 6 Trigger probing rebalances until group is stable
- PR-8338 - KAFKA-8890: KIP-519- Make SSL context/engine configuration extensible
- PR-8405 - KAFKA-9801: Still trigger rebalance when static member joins in CompletingRebalance phase
- PR-8414 - KAFKA-8410: Part 1: processor context bounds
- PR-8423 - KAFKA-9818: improve error message to debug test
- PR-8420 - KAFKA-9815; Ensure consumer always re-joins if JoinGroup fails
- PR-8371 - KAFKA-9753: A few more metrics to add
- PR-8402 - KAFKA-9793: Expand the try-catch for task commit in HandleAssignment
- PR-8383 - KAFKA-9784: Add OffsetFetch to group concurrency test
- PR-8422 - KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates unused iterator
- PR-8418 - KAFKA-9807; Protect LSO reads from concurrent high-watermark updates
- PR-8408 - KAFKA-9810: Document Connect Root REST API on /
- PR-8412 - KAFKA-9750; Fix race condition with log dir reassign completion
- PR-8411 - KAFKA-9812: fix infinite loop in test code
- PR-8400 - KAFKA-9739: Fixes null key changing child node
- PR-8375 - KAFKA-9776: Downgrade TxnCommit API v3 when broker doesn’t support
- PR-8407 - KAFKA-9809: Shrink transaction timeout for streams
- PR-8359 - KAFKA-9778: Add validateConnector functionality to the EmbeddedConnectCluster
- PR-8337 - KAFKA-6145: Pt. 5 Implement high availability assignment
- PR-8331 - KAFKA-9748: extend EosIntegrationTest for EOS version 2
- PR-7755 - KAFKA-9233: Fix IllegalStateException in Fetcher retrieval of beginni…
- PR-8269 - KAFKA-9659: Add more log4j when updating static member mappings
- PR-8356 - KAFKA-9441: Cleanup Streams metrics for removed task commit latency metrics
- PR-8370 - KAFKA-9753: Add active tasks process ratio
- PR-8389 - KAFKA-9777; Remove txn purgatory to fix race condition on txn completion
- PR-8279 - KAFKA-9706: Handle null keys/values in Flatten transformation
- PR-8367 - KAFKA-9719: Streams with EOS version 2 should fail fast for older brokers
- PR-8388 - KAFKA-9759: Add doc change for KIP-562
- PR-8350 - KAFKA-9760: Add KIP-447 protocol change to upgrade notes
- PR-8382 - KAFKA-9685: PT2, avoid unnecessary set creation in ACL matching
- PR-8164 - KAFKA-9600: EndTxn should enforce strict epoch checking if from client
- PR-8368 - KAFKA-9770: Close underlying state store also when flush throws
- PR-8369 - KAFKA-9771: Port patch for inter-worker Connect SSL from Jetty 9.4.25
- PR-8358 - KAFKA-9756: Process more than one record of one task at a time
- PR-8320 - KAFKA-8470: State change logs should not be in TRACE level
- PR-8280 - KAFKA-9707: Fix InsertField.Key should apply to keys of tombstone records
- PR-8332 - KAFKA-9688: kafka-topic.sh should show KIP-455 adding and removing replicas
- PR-8346 - KAFKA-9373: Reduce shutdown time by avoiding unnecessary loading of indexes
- PR-6741 - Improve performance of checkpointHighWatermarks, patch 1/2
- PR-8138 - KAFKA-9573: Fix JVM options to run early versions of Kafka on the latest JVMs
- PR-8343 - KAFKA-9758: Doc changes for KIP-523 and KIP-527
- PR-8290 - KAFKA-9677: Fix consumer fetch with small consume bandwidth quotas
- PR-8339 - KAFKA-9752; New member timeout can leave group rebalance stuck
- PR-8285 - KAFKA-9700:Fix negative estimatedCompressionRatio issue
- PR-8287 - KAFKA-9711 The authentication failure caused by SSLEngine#beginHandsh…
- PR-8334 - KAFKA-6145: Add balanced assignment algorithm
- PR-8336 - KAFKA-9749: TransactionMarkerRequestCompletionHandler should treat KAFKA_STORAGE
- PR-8327 - KAFKA-9743: Catch commit offset exception to eventually close dirty tasks
- PR-7939 - KAFKA-9407: Return Immutable List from SchemaSourceTask
- PR-8330 - KAFKA-9742: Fix broken StandbyTaskEOSIntegrationTest
- PR-8226 - KAFKA-9651: Fix ArithmeticException (÷ by 0) in Partitioner impls
- PR-8318 - KAFKA-9451: Enable producer per thread for Streams EOS
- PR-8205 - KAFKA-9634: Add note about thread safety of ConfigProvider
- PR-8252 - KAFKA-6145: Pt 2.5 Compute overall task lag per client
- PR-8325 - KAFKA-9741: Update ConsumerGroupMetadata before calling onPartitionsRevoked()
- PR-8324 - KAFKA-9701 (fix): Only check protocol name when generation is valid
- PR-8307 - KAFKA-9727: cleanup the state store for standby task dirty close and check null for changelogs
- PR-8262 - KAFKA-6145: Add constrained balanced assignment algorithm
- PR-8319 - KAFKA-9734: Fix IllegalState in Streams transit to standby
- PR-8155 - KAFKA-9553: Improve measurement for loading groups and transactions
- PR-8244 - KAFKA-8820: kafka-reassign-partitions.sh should support the KIP-455 API
- PR-8223 - KAFKA-9654 ReplicaAlterLogDirsThread can’t be created again if the pr…
- PR-8284 - KAFKA-9225: Bump rocksdb 5.18.3 to 5.18.4
- PR-8218 - KAFKA-9441: Unify committing within TaskManager
- PR-7972 - KAFKA-9435: DescribeLogDirs automated protocol
- PR-7039 - KAFKA-8618: Replace Txn marker with automated protocol
- PR-8299 - KAFKA-9568: enforce rebalance if client endpoint has changed
- PR-8313 - KAFKA-5604: Remove the redundant TODO marker on the Streams side
- PR-8253 - KAFKA-9656: Return COORDINATOR_NOT_AVAILABLE for older producer clients
- PR-8161 - KAFKA-9047: Making AdminClient group operations respect retries and backoff
- PR-7936 - KAFKA-9404: Use ArrayList instead of LinkedList in Sensor Class
- PR-8260 - KAFKA-9625: Fixing IncrementalAlterConfigs with respect to Broker Configs
- PR-8278 - KAFKA-8803: Remove timestamp check in completeTransitionTo
- PR-8289 - KAFKA-9712: Catch and handle exception thrown by reflections scanner
- PR-8236 - KAFKA-9670: Reduce allocations in Metadata Response preparation
- PR-8083 - KIP-546 (1/2): Implements describeClientQuotas() and alterClientQuotas().
- PR-8267 - KAFKA-6647: Do note delete the lock file while holding the lock
- PR-8246 - KAFKA-6145: Pt 2. Include offset sums in subscription
- PR-8293 - KAFKA-9715: Eliminate reference to interBrokerProtocolVersion in TransactionStateManager
- PR-8294 - KAFKA-9718; Don’t log passwords for AlterConfigs in request logs
- PR-7957 - KAFKA-8768: DeleteRecords request/response automated protocol
- PR-8265 - KAFKA-9657: Throw upon offset fetch unsupported stable flag protocol
- PR-8266 - KAFKA-9695; Handle null config values for createTopics, alterConfigs
- PR-8272 - KAFKA-9701: Add more debug log on client to reproduce the issue
- PR-8177 - KAFKA-9605: Do not attempt to abort batches when txn manager is in fatal error
- PR-7876 - KAFKA-9344:Logged consumer config does not always match actual config…
- PR-8232 - KAFKA-9658: Fix removing user quotas
- PR-8255 - KAFKA-9686 MockConsumer#endOffsets should be idempotent
- PR-8235 - KAFKA-9176: Do not update limit offset if we are in RESTORE_ACTIVE mode
- PR-283 - KAFKA-2391: add timeout to KafkaConsumer blocking calls
- PR-282 - KAFKA-2428
- PR-8220 - KAFKA-9645: Fallback to unsubscribe during Task Migrated
- PR-8228 - KAFKA-9663: Doc some null returns in KafkaStreams
- PR-284 - KAFKA-2620: Introduce Scalariform
- PR-8233 - KAFKA-9668: Iterating over KafkaStreams.getAllMetadata() results in ConcurrentModificationException
- PR-279 - KAFKA-2479: Transient exception
- PR-8227 - KAFKA-9662: Wait for consumer offset reset in throttle test to avoid losing early messages
- PR-8121 - KAFKA-6145: Pt 1. Bump protocol version and encode task lag map
- PR-8213 - KAFKA-9615: Clean up task/producer create and close
- PR-8229 - KAFKA-9661: Propagate includeSynonyms option to AdminClient in ConfigCommand
- PR-254 - KAFKA-2588 ReplicaManager partitionCount metric should actually be re…
- PR-277 - Kafka-2587: Only notification handler will update the cache and all verifications will use waitUntilTrue.
- PR-8209 - KAFKA-9632; Fix MockScheduler synchronization for safe use in Log/Partition tests
- PR-8186 - KAFKA-9618: Directory deletion failure leading to error task RocksDB open
- PR-275 - KAFKA-2527; System Test for Quotas in Ducktape
- PR-8208 - KAFKA-8895: delete all topics before recreating
- PR-8197 - KAFKA-9573: Fix VerifiableProducer and VerifiableConsumer to work with older Kafka versions
- PR-1286 - KAFKA-3618: Handle ApiVersionsRequest before SASL authentication
- PR-8087 - KAFKA-9525: add enforceRebalance method to Consumer API
- PR-8190 - KAFKA-9623: Keep polling until the task manager is no longer rebalancing in progress
- PR-8187 - KAFKA-9620: Do not throw in the middle of consumer user callbacks
- PR-273 - KAFKA-2517; Performance Regression post SSL implementation (zero copy)
- PR-8168 - KAFKA-9607: Do not clear partition queues during close
- PR-8169 - KAFKA-9610: Do not throw illegal state when remaining partitions are not empty
- PR-8173 - KAFKA-9614: Not initialize topology twice in StreamTask
- PR-8165 - KAFKA-9601: Stop logging raw connector config values
- PR-8166 - KAFKA-9602: Close the stream internal producer only in EOS
- PR-8062 - KAFKA-9498; Topic validation during the topic creation triggers unnecessary TopicChange events
- PR-8029 - KAFKA-8147: Add changelog topic configuration to KTable suppress
- PR-8167 - KAFKA-9533: Revert ValueTransform forwards null
- PR-8009 - KAFKA-9308: Reworded the ssl part of the security documentation
- PR-8159 - KAFKA-9599 create unique sensor to record group rebalance
- PR-8145 - KAFKA-9581: Remove rebalance exception withholding
- PR-8142 - KAFKA-9577: SaslClientAuthenticator incorrectly negotiates SASL_HANDSHAKE version
- PR-8105 - KAFKA-9441: Add internal StreamsProducer
- PR-263 - KAFKA 2578 Client Metadata internal state should be synchronized
- PR-8143 - KAFKA-9582: Do not abort transaction in unclean close
- PR-8111 - KAFKA-9206: throw a KafkaException when encountering CORRUPT_MESSAGE
- PR-8058 - KAFKA-9481: Graceful handling TaskMigrated and TaskCorrupted
- PR-8116 - KAFKA-9562: part 1: ignore exceptions while flushing stores in close(dirty)
- PR-7337 - KAFKA-8910: Incorrect javadoc at KafkaProducer.InterceptorCallback#onCompletion
- PR-8108 - KAFKA-9533: ValueTransform forwards null values
- PR-7913 - KAFKA-9343: Add ps command for Kafka and zookeeper process on z/OS.
- PR-8119 - KAFKA-9558: Fixing retry logic for getListOffsetsCalls
- PR-8089 - KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset
- PR-8125 - KAFKA-9515: Upgrade ZooKeeper to 3.5.7
- PR-8106 - KAFKA-9319: Fix generation of CA certificate for system tests.
- PR-8109 - KAFKA-9545: Fix subscription bugs from Stream refactoring
- PR-8088 - KAFKA-9535: Update metadata upon retrying partitions for ListOffset
- PR-7389 - KAFKA-8805: Bump producer epoch on recoverable errors
- PR-8060 - KAFKA-9274: Gracefully handle timeout exception
- PR-7781 - KAFKA-8904: Improve producer’s topic metadata fetching.
- PR-8085 - KAFKA-9556: Fix two issues with KIP-558 and expand testing coverage
- PR-8112 - KAFKA-9557: correct thread process-rate sensor to measure throughput
- PR-8023 - KAFKA-8507 kip 499 Unify connection name flag for command line tool
- PR-7674 - KAFKA-9106 make metrics exposed via jmx configurable
- PR-7731 - KAFKA-9204: allow ReplaceField SMT to handle tombstone records
- PR-7733 - KAFKA-9192: fix NPE when for converting optional json schema in structs
- PR-8000 - KAFKA-9417: New Integration Test for KIP-447
- PR-8031 - KAFKA-9447: Add new customized EOS model example
- PR-8091 - KAFKA-6607: Commit correct offsets for transactional input data
- PR-8053 - KAFKA-9499; Improve deletion process by batching more aggressively
- PR-7955 - KAFKA-9423: Refine layout of configuration options on website and make individual settings directly linkable
- PR-8015 - KAFKA-9500: Fix topology bug in foreign key joins
- PR-8065 - KAKFA-9503: Fix TopologyTestDriver output order
- PR-7996 - KAFKA-9355: Fix bug that removed RocksDB metrics after failure in EOS
- PR-8054 - KAFKA-9390: Make serde pseudo-topics unique
- PR-8024 - KAFKA-9483; Add Scala KStream#toTable to the Streams DSL
- PR-8040 - KAFKA-6607: Commit correct offsets for transactional input data
- PR-8059 - KAFKA-7052 Avoiding NPE in ExtractField SMT in case of non-existent fields
- PR-8061 - KAFKA-9517: Fix default serdes with FK join
- PR-8033 - KAFKA-9487: Follow-up PR of Kafka-9445
- PR-8081 - KAFKA-9523: Migrate BranchedMultiLevelRepartitionConnectedTopologyTest into a unit test
- PR-8018 - KAFKA-9480: Fix bug that prevented to measure task-level process-rate
- PR-8039 - KAFKA-9505: Only loop over topics-to-validate in retries
- PR-8027 - KAFKA-7658: Follow up to original PR
- PR-8057 - KAFKA-9507 AdminClient should check for missing committed offsets
- PR-8056 - KAFKA-9519: Deprecating ZK for ConfigCommand
- PR-8048 - KAFKA-9509: Fixing flakiness of MirrorConnectorsIntegrationTest.testReplication
- PR-252 - KAFKA-2586; Enable SSL for inter-broker communication when SSL is enabled in tests
- PR-255 - KAFKA-2593 Key value stores can use specified serializers and deserializers
- PR-7770 - KAFKA-9261; Client should handle unavailable leader metadata
- PR-7997 - KAFKA-9113: Clean up task management and state management
- PR-8037 - KAFKA-9491; Increment high watermark after full log truncation
- PR-8007 - KAFKA-9477 Document RoundRobinAssignor as an option for partition.assignment.strategy
- PR-7568 - KAFKA-9074: Correct Connect’s Values.parseString to properly parse a time and timestamp literal
- PR-8030 - KAFKA-9492; Ignore record errors in ProduceResponse for older versions
- PR-8028 - KAFKA-9490: Fix generics for Grouped
- PR-8026 - Bump trunk to 2.6.0-SNAPSHOT
- PR-7725 - KAFKA-9027, KAFKA-9028: Convert create/delete acls requests/response to use generated protocol
ksqlDB¶
- PR-6126 - fix: create /var/lib/kafka-streams directory on RPM installation
- PR-6165 - fix: allow expressions in flat map (6.0.x)
- PR-5952 - METRICS-1988: add confluent-telemetry directory to ksql-run-class
- PR-5937 - fix: avoid losing cause of processing errors
- PR-5895 - fix: adds a handler to gracefully shutdown
- PR-5885 - fix: close query on invalid use of HTTP/2 with /query endpoint (6.0.x)
- PR-5871 - fix: remove schema compat check if schema exists
- PR-5844 - fix: Reuse KsqlClient instance for inter node requests (#5742)
- PR-5837 - fix: always use the changelog subject in table state stores
- PR-5786 - fix: better error on DECIMAL type with PROTOBUF format
- PR-5818 - fix: Allow backported ALPN to be used and warn if no OpenSSL
- PR-5783 - fix: SEC-1034: log4j migration to confluent repackaged version
- PR-5788 - fix: circumvent KAFKA-10179 by forcing changelog topics for tables
- PR-5763 - fix: ksqlDB should not truncate decimals
- PR-5761 - support UDTFs on streams without keys
- PR-5756 - fix: do not overwrite schemas from a CREATE STREAM/TABLE
- PR-5730 - fix: make sure old query stream doesn’t block on close
- PR-5684 - fix: join queries with SELECT * and WHERE fail
- PR-5703 - fix: support CASE statements returning NULL
- PR-5656 - fix: allow empty structs in schema inference
- PR-5645 - fix: ensure only deserializable cmds are written to command topic
- PR-5644 - fix: support GROUP BY with no source columns used
- PR-5638 - feat: implements ARRAY_JOIN as requested in (#5028) (#5474)
- PR-5617 - fix: don’t fail if broker does not support AuthorizedOperations
- PR-5621 - fix: /inserts-stream endpoint now supports nested types
- PR-5563 - feat: new split_to_map udf
- PR-5559 - feat: add CHR UDF
- PR-5597 - fix: fail AVRO/Protobuf/JSON Schema statements if SR is missing
- PR-5591 - fix: make stream and column names case-insensitive in /inserts-stream
- PR-5592 - fix: disallow requests to /inserts-stream if insert values disabled
- PR-5576 - fix: Stream with no key should not fail if key has data
- PR-5568 - fail aggregations that don’t select any value columns
- PR-5516 - feat: reload TLS certificate without restarting server
- PR-5560 - fix: set retention.ms to -1 instead of Long.MAX_VALUE
- PR-5548 - feat: new UDFs for set-like operations on Arrays
- PR-5552 - fix: improve print topic format detection
- PR-5528 - feat: create MetricsContext for ksql metrics reporters
- PR-5534 - feat: new CONCAT_WS UDF
- PR-5531 - fix: inserts now support zero-valued decimals
- PR-5536 - feat: new UDFs for working with Maps
- PR-5546 - feat: new string UDFs LPad, RPad
- PR-5523 - feat: Add encode udf
- PR-5542 - feat: support TIMESTAMP being a key column
- PR-5535 - feat: new UUID UDF
- PR-5532 - fix: Prevent memory leaks caused by pull query logging
- PR-5533 - feat: keys can now be named using the KEY syntax
- PR-5529 - fix: deals with issue #5521 by adding more descriptive error message
- PR-5513 - fix: update Concat UDF to new framework and make variadic
- PR-5504 - feat: Adds UDF regexp_replace
- PR-5501 - feat: Adds udf regexp_split_to_array
- PR-5495 - feat: turn on snappy compression for produced data
- PR-5515 - fix: Retry on connection closed
- PR-5505 - feat: new UDFs for array max/min/sort
- PR-5507 - feat: Adds UDF regexp_extract_all
- PR-4909 - feat: add extra log messages for pull queries
- PR-5506 - fix: allow dynamic construction of an ARRAY of STRUCTS with duplicate values #5436
- PR-5461 - fix: KSQL does not accept queries when running QueryLimit - 1 queries
- PR-5497 - build: reference plugin URL as property (MINOR)
- PR-5478 - fix: Remove stacktrace from error message
- PR-5435 - fix(client): fail queryID future if executeQuery() request fails (MINOR)
- PR-5469 - fix: /inserts-stream endpoint now accepts complex types
- PR-5477 - fix: Improve pull query error logging
- PR-5470 - ST-1323: Add license file so sources get attached
- PR-5459 - fix: avoid unnecessary warnings from PRINT
- PR-5448 - feat(client): support (non-streaming) insert into in Java client
- PR-5455 - fix: allow setting auto.offset.reset=latest on /query-stream endpoint
- PR-5425 - fix: exclude window bounds from persistent query value & schema match
- PR-5431 - fix: fail on WINDOW clause without matching GROUP BY
- PR-5340 - feat: Expose Vert.x metrics
- PR-5444 - Fix markdown for function references
- PR-5385 - feat: INSTR function #881
- PR-5412 - feat: introduce RegexClassifier for classifying errors via cfg
- PR-5392 - feat: Pull Queries: QPS check utilizes internal API flag to determine if forwarded
- PR-5355 - How-to guides: first set
- PR-5391 - fix(client): expose query ID from executeQuery() return type
- PR-5386 - fix: Block writer thread if response output buffer is full
- PR-5374 - feat: expose JMX metric that classifies an error state
- PR-5363 - feat: drop WITH(KEY) syntax
- PR-5371 - feat(client): support push query termination in Java client
- PR-5382 - MINOR: Use kafka.scala.version variable in pom file
- PR-5380 - MINOR: Update mbknor-jackson-jsonschema_2.12 artifact to mbknor-jackson-jsonschema_2.13
- PR-5381 - fix: Override Scala version in our build
- PR-5360 - fix(client): Java client error handling
- PR-5270 - fix: remove leading zeros when casting decimal to string
- PR-5273 - feat: implement earliest_by_offset() UDAF
- PR-4975 - fix: fix NPE in latest_by_offset if first element being processed has null…
- PR-4782 - feat: Implement latest_by_offset() UDAF
- PR-5212 - feat: Adds the ability have internal endpoints listen on ksql.internal.listener
- PR-5332 - fix: Allow value delimiter to be specified for datagen
- PR-5093 - Any key name
- PR-5200 - feat(client): Java client with push + pull query support
- PR-5287 - fix: allow structs in schema provider return types
- PR-5205 - fix: /query-stream endpoint should serialize Struct (MINOR)
- PR-5239 - fix: Move Cors handler in front of /chc handlers
- PR-5124 - Backport array len
- PR-5213 - fix: use sr’s jackson-jsonschema version
- PR-5158 - feat: Remove dependency to ksqldb-common
- PR-5198 - fix: use schema in annotation as schema provider if present
- PR-5193 - feat: Do not allow access to new streaming endpoints using HTTP1.x
- PR-5189 - build: pin streams to 5.5.0 instead of 5.5.0-SNAPSHOT
- PR-5162 - bug: stop multi-way joins failing on ambiguous column errors
- PR-5150 - feat: klip-14 - rowtime as pseduocolumn
- PR-5145 - fix: do not allow implicit casting in UDAF function lookup
- PR-5122 - Event-driven microservice tutorial
- PR-5104 - feat: fail startup if command contains incompatible version
- PR-5105 - feat: transient queries added to show queries output
- PR-5137 - BREAKING CHANGE: minor: allow more precise configuration of rocks bounds
- PR-5142 - fix: support quotes in explain statements (MINOR)
- PR-5126 - feat: Implement cluster health check endpoints
- PR-5123 - refactor: Migrate old websocket query endpoint
- PR-5139 - fix: include lower-case identifiers among those that need quotes (#3723)
- PR-5081 - feat: add multi-join expression support
- PR-5075 - fix: replace ‘null’ in explain plan with correct op type (MINOR)
- PR-5023 - fix: Improve error message for where/having type errors
- PR-5064 - feat: support multi-way joins
- PR-5054 - fix: add content type header to ported endpoints response (MINOR)
- PR-5069 - fix: make endpoints available while waiting for precondition
- PR-5048 - fix: reject requests to new API server if server is not ready
- PR-5047 - fix: make endpoints available while waiting for precondition
- PR-5059 - fix: Make sure internal client is configured for TLS
- PR-4985 - fix: Don’t set chunked transfer encoding for HTTP2 proxy
- PR-4974 - fix: Catch server startup exceptions
- PR-4945 - feat: Support wildcards for unauthed paths
- PR-4944 - feat: CORS support for Vert.x based ksqlDB
- PR-4851 - feat: Migrate /ksql and /ksql/terminate endpoints to Vert.x
- PR-5019 - fix: improve handling of NULLs
- PR-5052 - fix: json creator for QueryStatusCount should match getter (MINOR)
- PR-5029 - feat: add KsqlQueryStatus to decouple from KafkaStreams.State
- PR-4921 - Two new tutorials: materialized views + streaming ETL pipelines
- PR-5013 - feat: support more advanced suite of LIKE expressions
- PR-4952 - Support aliases in PARTITION BY and GROUP BY
- PR-5002 - feat: speed up restarts by not building topologies for terminated queries
- PR-4986 - Distinguish keys for tables by naming them PRIMARY KEY
- PR-4875 - feat: scatter gather query status from all servers in cluster for ‘SHOW QUERIES [EXTENDED]’ statement
- PR-4993 - fix: Allows unclosed quote characters to exist in comments
- PR-5012 - fix: allow trailing slash in listeners config (MINOR)
- PR-5005 - fix: query id for TERMINATE should be case insensitive
- PR-4977 - fix: generated aliases for struct field access no longer require quoting
- PR-5006 - Remove unused / duplicate methods (MINOR)
- PR-4951 - feat: Adds rate limiting to pull queries
- PR-4972 - fix: When pull query metrics enabled, throws NPE
- PR-4956 - fix: output valid multiline queries when running SHOW QUERIES
- PR-4966 - fix: Pass correct request configs when making pull query request
- PR-4942 - fix: do not allow grouping sets
- PR-4926 - fix: throw error when column does not exist during INSERT INTO
- PR-4940 - fix: do not allow GROUP BY and PARTITION BY on boolean expressions
- PR-4908 - fix: Don’t wait for streams thread to be in running state
- PR-4878 - fix: handle leap days correctly during timestamp extraction
- PR-4893 - fix: Infer TLS based on scheme of server string
- PR-4889 - fix: If startup hangs (esp on preconditions), shutdown server correctly
- PR-4872 - fix: ‘drop (stream|table) if exists’ fails if source does not exist
- PR-4827 - fix: avoid duplicate column name errors from auto-generated aliases
- PR-4822 - fix: Stop PARTITION BY and UDTFs that fail from terminating the query
- PR-4850 - feat: expand IFNULL to support all types
- PR-4859 - fix: Filter out hosts with no lag info by default
- PR-4868 - test: add test coverage to ensure that historical commands from previous versions always work in the current version
- PR-4837 - fix: Returns empty lag info for a dead host rather than last received lags
- PR-4867 - fix: avoid long possible format lists on PRINT TOPIC for nulls
- PR-4829 - feat: add COALESCE function
- PR-4653 - fix: CommandRunner metric has correct metric displayed when thread dies
- PR-4823 - fix: do not filter out rows where PARTITION BY resolves to null
- PR-4814 - fix: use describeTopics() for Kafka healtcheck probe
- PR-4771 - feat: add internal request to KsqlRequestConfig and SessionProperties
- PR-4816 - fix: do not lose old key values on repartition
- PR-4830 - fix: add deserializer for SqlType
- PR-4809 - fix: Removes unnecessary error logging for heartbeat since this is ex…
- PR-4754 - fix: better error code on shutdown
- PR-4805 - fix: only create processing log stream if it doesn’t exist
- PR-4743 - fix: make CLI unit-tests platform independent
- PR-4785 - fix: docker image directories in light of renames (MINOR)
- PR-4608 - feat: Add metrics for pull queries endpoint
- PR-4778 - feat: Security plumbing
- PR-4777 - fix: add transfer-encoding:chunked header if Jetty omits this
- PR-4733 - feat: support for tunable retention, grace period for windowed tables
- PR-4728 - feat: add REGEXP_EXTRACT UDF
- PR-4717 - fix: create schemas at topic creation
- PR-4709 - fix: stop logging about command topic creation on startup if exists (MINOR)
- PR-4725 - feat: add ARRAY_LENGTH UDF
- PR-4713 - feat: Integrate new API with existing security framework
- PR-4714 - test: avoid NPE if user puts input topic name in output
- PR-4685 - feat: basic auth for new API server
- PR-4681 - fix: fix the error message for failing to wait for CommandRunner to catch up in DistributingExecutor
- PR-4632 - fix: change default exception handling for timestamp extractors to log-and-continue
- PR-4675 - fix: change configOverrides back to streamsProperties
- PR-4597 - fix: added special handling for forwarded pull query request
- PR-4650 - fix: Return immutable list of streams metadata
- PR-4643 - fix: idempotent terminate that can handle hung streams
- PR-4489 - fix: csas/ctas with timestamp column is used for output rowtime
- PR-4625 - fix: support partial schemas
- PR-4509 - feat: display errors in CLI in red text
- PR-4621 - fix: patch KafkaStreamsInternalTopicsAccessor as KS internals changed
- PR-4606 - fix: StoreQueryParameters name change
- PR-4573 - feat: add the topic name to deserialization errors
- PR-4575 - feat: log groupby errors to processing logger
- PR-4590 - feat: Simple perf tool for new API
- PR-4581 - feat: Provide upper limit on number of push queries
- PR-4511 - [SECURITY] Use HTTPS to resolve dependencies in Maven Build
- PR-4512 - feat: Plug insert streams into backend
- PR-4551 - feat: enhance PRINT TOPIC’s format detection
REST Proxy¶
- PR-738 - KC-613: Remove share/java/kafka-rest dependencySet.
- PR-737 - KC-613: Split share/java/kafka-rest in kafka-rest-bin and kafka-rest-lib.
- PR-734 - CLIENTS-1659: Remove V1 API Kafka core dependency and associated Zookeeper Client configs
- PR-731 - CLIENTS-1657: Explicitly depend on Guava.
- PR-729 - METRICS-1988: add confluent-telemetry directory to kafka-rest-run-class
- PR-723 - Refactor requests to fix go sdk generation bug and add parameterized servers to openapi-spec
- PR-726 - Add [Consumer|Producer]Performance tools back
- PR-722 - Add reassignment API to openapi spec.
- PR-721 - Add ACL APIs to openapi spec.
- PR-719 - Add consumers (admin) APIs to openapi spec.
- PR-718 - CNKAF-1017: Update v3 error response format
- PR-715 - Add config endpoints to openapi spec.
- PR-714 - OpenAPI Spec for a small subset of the V3 APIs.
- PR-712 - Move MetricsReporter Config extraction to rest-utils
- PR-711 - CLIENTS-1613: Add configuration to enable/disable V2 and V3 APIs.
- PR-710 - Remove cluster-wide topic configs.
- PR-709 - KC-613: Use slf4j version from common pom.xml.
- PR-707 - Change resource label resource.kafka.cluster.id to resource.cluster.id
- PR-703 - Enable setting cluster_id on KafkaRestMetricsContext
- PR-706 - CNKAF-951: Add catchingCompose utility to handle exceptions.
- PR-704 - CLIENTS-1599: Add constructor accepting path to KafkaRestApplication.
- PR-697 - CLIENTS-1597: Add relationship reassignment and refactor reassignment resource
- PR-701 - Rename alter config UPDATE to SET.
- PR-700 - Add Batch Alter (UPDATE/DELETE) Broker/Cluster/Topic configs.
- PR-696 - Add LIST/GET Consumer Group/Consumer/Consumer Assignment.
- PR-693 - Add search/create/delete ACLs APIs.
- PR-687 - CNKAF-739: Reassignment endpoint for a given topic name
- PR-685 - CNKAF-739 - Reassignments endpoint for given topic name and partition id
- PR-692 - Remove AdminClientWrapper.
- PR-691 - Migrate off JSON:API cleanup.
- PR-682 - CNKAF-739: Add Rest endpoint for list all partition reassignments
- PR-690 - Switch from application/vnd.api+json to application/json.
- PR-689 - KIP-606 MetricsContext
- PR-686 - Migrate off JSON:API.
- PR-683 - KIP-226: Add cluster-wide dynamic default configs.
- PR-678 - ST-1323: Add license file so sources get attached
- PR-679 - Migrate internal entities to Google’s Auto-Value.
- PR-680 - Add static factory to Cluster.
- PR-677 - KIP-226: Add support for configs source and synonyms.
- PR-675 - Give more unique names to the testing containers.
- PR-674 - Open visibility of configurePre/PostResourceHandling.
- PR-672 - Fix REST Proxy injection-based authentication.
- PR-671 - Make ParititonData public
- PR-670 - Add JsonCreator to v3 entities
- PR-667 - Copy admin client config from REST Proxy. Need for authentication tests.
- PR-658 - Remove all methods from AdminClientWrapper.
- PR-665 - CPKAFKA-4979: Remove non-determinism from searchReplicasByBroker.
- PR-662 - Add find partition replicas by broker id endpoint.
- PR-660 - CLIENTS-1505: Add broker config delete endpoint
- PR-659 - CLIENTS-1504: Add update endpoint for broker config
- PR-654 - Remove AdminClientWrapper.getBrokerIds() and add JSON:API errors.
- PR-656 - Fix default Content-Type, when none or any (/) is provided.
- PR-655 - Remove spurious topic/partitions existence test when producing.
- PR-647 - Remove V1 APIs and all their dependencies.
- PR-652 - CLIENTS-1501: Add broker configs endpoints
- PR-617 - Fix compile error due to KIP-515 (ZK TLS)
- PR-191 - SEC-1034: log4j migration to confluent repackaged version
- PR-199 - Rest CORS option skip will now default to true
- PR-195 - Add a config flag to skip OPTION when CORS is enabled
- PR-193 - Removed OPTION omission for CORS
- PR-192 - All original configs must be passed to MetricReporters
- PR-190 - Use prefix to propagate metrics reporter configs
- PR-186 - Extensible Application Metrics instance configuration
- PR-180 - KIP-606 MetricsContext
- PR-183 - ST-1323: Add license file so sources get attached
- PR-182 - Add getter for Application’s Metrics object
- PR-178 - Migrate rest-utils to use AK library instead of common-metrics
- PR-177 - SEC-929-Customized HTTP response headers.
- PR-172 - SEC-592: Customize Jetty request queue and threadpool size
- PR-174 - Catch invalid format exception: #501
Schema Registry¶
- PR-1624 - DG-857 Fix Avro converter for top-level bytes object
- PR-1616 - DG-845 Have typeName override Object.class type; fixes #1615
- PR-1611 - Correct License in Client library
- PR-1598 - CP-644: Include CP jars required for Security and monitoring
- PR-1601 - DG-820: Fix telemetry reporter handling of leadership changes
- PR-1590 - DG-810: Explicitly add kafkastore.topic config to MetricsReporter instances
- PR-1594 - DG-813 If use.latest.version is true, use on Protobuf dependencies
- PR-1588 - DG-811 Allow JSON schema spec version to be specified
- PR-1589 - DG-812 Derive file extension during download
- PR-1587 - Upgrade to Kotlin 1.4.0
- PR-1586 - DG-800 Cache ProtobufSchema using name in Converter
- PR-1528 - SEC-1034: log4j migration to confluent repackaged version
- PR-1580 - DG-789 Fix for CVE-2020-15824
- PR-1579 - DG-788 Allow anyOf to be backward compat with oneOf/allOf
- PR-1578 - DG-767 Calculate proper defaults for client ssl configs
- PR-1576 - DG-767 Use passed in defaults for client ssl configs
- PR-1577 - DG-774 restore ssl client configs
- PR-1575 - DG-780 Add support for number with format for JSON Schema
- PR-1569 - METRICS-1988: add confluent-telemetry directory to schema-registry-run-class
- PR-1574 - DG-774 Avoid regenerating default names for Protobuf data
- PR-1567 - DG-740 Convert connect defaults for JSON Schema
- PR-1563 - DG-713: Disable automatic setup of telemetry reporting
- PR-1561 - DG-723 Enable defaults for JSON Schema
- PR-1556 - DG-715: Change telemetry cluster label names
- PR-1550 - DG-695 Normalize JSON ref schemas before comparing
- PR-1548 - ST-3988: update swagger-core to 1.6.2
- PR-1547 - ST-3988: update swagger-core to 1.6.2
- PR-1538 - DG-670 Treat anyOf same as oneOf in JSON Schema converter
- PR-1537 - DG-652 Fix missing negation in conditional
- PR-1535 - DG-652 Handle imports during protobuf compat checks
- PR-1531 - DG-628 Allow changes to id during json schema compat checks
- PR-1530 - DG-607 Add lenient processing for numbers in oneof
- PR-1523 - DG-607 Ensure fields for union are optional
- PR-1520 - Handle null error stream in RestService
- PR-1522 - DG-606 Add option to not generate oneof for nullable fields
- PR-1517 - Upgrade maven-plugin dependencies to 3.6 in 5.4.x
- PR-1510 - Add metric name test
- PR-1509 - align naming conventions
- PR-1508 - Add telemetry reporter SR config option
- PR-1507 - Use config originals for retrieving TR enablement
- PR-1493 - Use MetricsContext to set JMX prefix
- PR-1502 - Split telemetry cluster label
- PR-1500 - Only add TelemetryReporter if explicitly enabled in config
- PR-1483 - Telemetry reporter integration
- PR-1497 - DG-554 Pass refs when forwarding register requests
- PR-1496 - DG-535 Handle recursive Protobuf schemas
- PR-1495 - DG-402 Missed some places where dotted names can occur
- PR-1492 - Fix NPE during tombstone metric update
- PR-1485 - Add node information to SchemaRegistryProtocol
- PR-1487 - DG-527 Omit defaults for proto 3 schemas (fixed in wire)
- PR-1482 - DG-499 Fix JSON Schema compat checks for partial models
- PR-1484 - DG-524 Catch NPE during Avro compat check
- PR-1480 - DG-27 Use leader/follower terminology
- PR-1477 - Allow custom update handlers; fixes #382
- PR-1476 - Add unit test for mvn validate action
- PR-1468 - Implement new Schema Registry Metrics
- PR-1474 - Add validate action for SR maven plugin
- PR-1473 - Add ability to use latest version during serialization
- PR-1471 - DG-436 Pass existing Avro schema instead of regenerating
- PR-1470 - Allow unknown properties while deserializing SchemaRegistryProtocol.Assignment
- PR-1469 - Fix message name on schema when version is included; fixes #1467
- PR-1466 - DG-450 Perform proper matching of subschemas
- PR-1461 - DG-268 Deprecate ZK-based coordination
- PR-1455 - Fix Scala 2.13 deprecations
- PR-1450 - DG-400 Support enums using allOf in JSONSchema converter
- PR-1451 - Upgrade swagger deps
- PR-1449 - DG-402 Add support for packages in ProtobufConverter
- PR-1440 - Fix null schema type
- PR-1433 - use property for jackson-jsonschema
- PR-1436 - feat: adds subjectName in the exception for incompatible schema
- PR-1437 - Fix cyclic schema references
- PR-1431 - Cherry-pick for permanent delete
- PR-1426 - DG-377 Allow generated classes to use refs during serialization
- PR-1427 - DG-399 Use envelope in JsonSchemaUtils.toObject
- PR-1417 - Allow id to be passed to console prod/cons; fixes #160
- PR-1422 - Fix cycles in Avro schemaEquals
- PR-1420 - SEC-929–Added one testing for verifying a feature added to rest-utils repo.
- PR-1416 - DG-399 Use envelope in JsonSchemaUtils.toObject
- PR-1397 - DG-91 Schema permanent delete.
- PR-1389 - DG-91 Ability to permanently delete schemas
- PR-1369 - DG-267 - Fix Protobuf converter enum handling
- PR-1368 - Performance: Use singleton ObjectMapper
- PR-1344 - Enhanced serialization exception
- PR-1334 - DG-237: Migrate SchemaRegistry CP KafkaMetrics to AK KafkaMetrics
- PR-1326 - Add API to retrieve supported schema types
Commercial Features¶
Control Center¶
- Introduced self-balancing cluster options on the UI, enabling users to easily balance workloads and dynamically expand and shrink clusters.
- Introduced Tiered Storage options on the UI, giving users the option to add and scale storage.
- Enabled auto-updates to the user interface by default.
Confluent Server¶
- CPKAFKA-6013: Skip tier state presence validation for deleted partitions
- KSTORAGE-1082: tier replication causes incorrect HWM after restore (#2507)
- KSTORAGE-1086: Better validations to check if tiering is disabled
- [CNKAF-1144] Confluent.operator.managed property (#2484)
- [Backport]METRICS-2089 Adding the correct NOTICE / COPYRIGHT for the confluent-telemetry package
- KSTORAGE-1040: Better checks to prevent tiering from being disabled dynamically
- METRICS-2159: use a loading cache to avoid expensive pattern matching calls
- CPKAFKA-5882. Add copyDependantLibs dependency to ce-sbk and ce-broker-plugins module
- KC-845: Ensure LogConfig includes broker defaults to pass schema config validation
- CPKAFKA-5873 Don’t load Netty classes when using default SSL engine
- CONFLUENT: Remove shading from ce-broker-plugins and ce-sbk
- CPKAFKA-5851: Align joda-time version with common
- CNKAF-863: Only showed broker removal status for given broker.
- KC-613: Use kafka-rest-lib instead of kafka-rest on classpath.
- CPKAFKA-5849: Don’t log password configs in request log filter
- CONFLUENT: Remove unused jacksonDatabind version definition
- CPKAFKA-5795: Change guava dependency to 28.1-jre
- CPKAFKA-5796: Remove ZK dependency from ce-licensing
- CPKAFKA-5835: Fix regression in cluster link compatibility test
- METRICS-2108 Telemetry-client dependency version update from 1.174.0 to 1.192.0 to update default base URL for TelemetryHttpClient
- [CNKAF-1089][RM-32] Remove SBK dependencies on grpc-stub. (#2408)
- backport METRICS-2098 exclude zstd/lz4 from telemetry-clients to avoid conflicting versions
- METRICS-2090: Remove duplicate labels on metrics
- METRICS-2097: Don’t capture ‘total_size` and ‘tier_size` metrics
- KC-637 cherry pick to 6.0.x
- KSTORAGE-957: Only set SSEAwsKeyManagementParams if a customer encryption key is supplied and sseAlgorithm is aws:kms. (#2353)
- SEC-866-6.0.x again
- CONFLUENT: Remove the soak clients status reporter cronjob
- METRICS-2031 Prune the hardcoded Kafka broker default include list to avoid collecting high cardinality metrics.
- METRICS-2048 Sanitize ksql service ids from their metric name to make them more practical to use. Also ignore legacy metrics.
- [CNKAF-1038] SBK Rebalance Cluster Test: add rolling bounces (#2326)
- CNKAF-1058; Ensure that Control Request Batching does not send LeaderAndIsrRequest with yet unknown broker(s)
- KSTORAGE-947: batch tier segment deletion delays to improve deletion SLA (#2309)
- backport METRICS-2010: create a new ‘releaseTarGz’ task for ce-metrics (#2282)
- CPKAFKA-5762: Fix topic deletion in system tests
- CNKAF-1039: Update error for insufficient metrics to compute plan
- KC-671: SBK system test with CL
- METRICS-1988: add confluent-telemetry directory to kafka classpath
- CONFLUENT: Bump cc-trogdor chart docker image to cc-trogdor:v0.207.0-5.3.0-ce-SNAPSHOT
- [Cherry 2367] [SEC-1345] Read HTTP Reason phrase when connection closed
- Revert “SEC-866 - Migrate audit log “other” category into “management” category. (#2330)”
- backport METRICS-2037 remove non-relocated dependencies from confluent-metrics fat jar (#2332)
- [CLIENTS-1633] Schedule MDS url refresher asynchronously
- Increase root FS to 20GB
- METRICS-1763, METRICS-1767 Configure telemetry metrics whitelist for Connect and MQTT Proxy Proactive Support (#2241)
- SEC-866 - Migrate audit log “other” category into “management” category.
- KC-391: Source topic deletion system test for CL
- KAFKA-10282; Remove Log metrics immediately when deleting log (#2342)
- CNKAF-1036: Fix issue where dynamic throttle applies for the current rebalance only - cherry pick to 6.0
- Rebalance Cluster System Test
- CNKAF-1008: Add Broker/Remove Broker proposal logging
- KC-633: Auth failures system test for Cluster Linking
- [CHERRY-2308] Move kafka-http-server project into clients project
- KC-638: System tests with tier storage enabled on destination
- KC-754: Cluster link compatibility system tests
- KC-631 Good path cluster linking test using SASL_SSL
- backport METRICS-1992: Fix schema registry cluster id resource label (#2285)
- CONFLUENT: Fix broken cluster link system tests in 6.0.x
- CNKAF-1029: Ensure Executor state is reset after unexpected exceptions
- KC-735: Improve cluster link addition.
- CNKAF-1035: Roll cluster while a new broker is getting added
- METRICS-1998: add rbac metrics to default metrics list (#2269)
- backport METRICS-1993: add connect.cluster.id label (#2264)
- Updated default topic replicas factor to 3 for audit log.
- KC-741: Don’t include passwords in cluster link request logs
- KC-736: Allow deletion of cluster links if mirrored topics are marked for deletion
- KSTORAGE-896: fix tier deletion manager system test (#2256)
- CNKAF-822: Cherry pick add broker system test to 6.0.x
- KC-725: Backport of KC-725 to 6.0.x
- backport METRICS-1986 Fix to give higher precedence to “confluent.telemetry.metrics.collector.whitelist” over default provider whitelist. (#2266)
- backport METRICS-1985 Add missing metrics to the _local exporter for SBK feature to work as expected.
- KSTORAGE-920: TierTopicConsumer primary consumer long poll throttles catchup (#2253)
- KSTORAGE-864 Fetch requests to return NotLeaderForPartition during ULE recovery
- CNKAF-819: Test SBK broker removal with roll
- METRICS-1762 Configure telemetry metrics whitelist for KSQL Proactive Support
- Backporting fix for METRICS-1764 and METRICS-1766
- update kafkatest template for balancer replication factor (for < 3-node clusters)
- KC-638: Cluster linking system test with tiered storage enabled in source cluster
- CNKAF-825: Introduce a system test for replica placement constraints
- CNKAF-1014: Rename usages of SBK in logs to DataBalancer
- CNKAF-613: Introduce a broker removal system test
- METRICS-1894:Telemetry Reporter is missing labels configured in prope…
- METRICS-1817: add include regex default value to Provider interface
- METRICS-1938: upgrade jose4j version (#2219)–backport
- METRICS-1929: use advertised listeners/host-port when deriving local producer config
- KSTORAGE-892 Re add couple of lines of ULE code that got deleted duri…
- Backporting config/events Telemetry Reporter changes along with all shading issues.
- Added one more toJson method in protoserde for webhook alert
- (backport) MMA-8432: fix wrong package name
- CNKAF-1004: Don’t describe removed broker configs
- METRICS-1895: Update telemetry client version
- KC-705: Retry metadata topic consumer.position() until authorizer startup times out
- METRICS-1918: bump gson from 2.8.5 -> 2.8.6
- (backport) prevent ‘io.confluent.controlcenter’ from being renamed by shadow plugin
- KSTORAGE-822: Expose Tiered Storage configs for GA in 6.0
- (Backport) include UnderMinIsrPartitionCount in the default broker metrics list
- CPKAFKA-2852: CCloud metadata listener thread should not die on unexpected exception
- SEC-1276 Allow empty resourceNames in CRNs
- SEC-384: Use the correct log level for debug output
- KC-634: Cluster link system tests with broker failures
- CNKAF-964: Shade opencensus protobufs into SBK jar (#2155)
- Add KafkaRest Metrics Provider
- KSTORAGE-845 Check if tier topic exists already before calling AdminZkClient#createTopic
- CNKAF-952: Update javadoc for removeBroker ConfluentAdmin API with InvalidBrokerRemovalException
- CNKAF-851: log goals skipped during detection in one line
- KC-630: Cluster link system test support with good path mirror test
- CONFLUENT: Prevent log start offset overrun during retention
- KSTORAGE-861: fatal NPE in TierTopicConsumer thread caused by initialization order (#2133)
- Back port [METRICS-1815] Connect client metrics missing some resource level tag…
- CNKAF-936: Remove dot-handling logic in metrics processor (#2121)
- METRICS-1833 Special handling of KSQL metrics in telemetry reporter that are getting mangled.
- Deprecate KafkaHttpServlet in favour of rest-utils Application. (#2135)
- [METRICS-1844, METRICS-1839, METRICS-1756,METRICS-1853] - Reconcile configs in Telemetry Reporter before creating a ConfluentTelemetryConfig object. Use password object for api.secret in ConfluentTelemetryConfig.java.
- [CNKAF-885, CNKAF-946, CNKAF-947] Set SBK Default Throttle, handle ab…
- METRICS-1801: align telemetry domain metric names
- METRICS-1835 Shade confluent-serializers dependencies in confluent-metrics (#2111)
- METRICS-1733 replace ‘whitelist’ with ‘include’ for telemetry and metrics reporter (#2096)
- METRICS-1810 use SBK topic replicas config if present (#2109)
- fix: fixes exception in server shutdown due to metrics registration
- METRICS-1809: add telemetry settings to default server.properties file
- [CNKAF-927] Fix typo in server.properties. (#2090)
- [CNKAF-924, CNKAF-926] Fix Databalancer Topic RF config, remove Teleme…
- CONFLUENT: Added check for CL metrics for 6.0.x
- CPKAFKA-4955: Fix failing testTopicDeletion
- Removed two testing files accidentally added.
- SEC-1130-Inject ConfluentAdmin from ce-kafka
- backport METRICS-1820: use password object for telemetry api secret
- CNKAF-893: Update SBK to work with secure Zookeeper
- METRICS-1816, METRICS-1705 local telemetry exporter metrics whitelist configuration for SBK (#2031)
- CONFLUENT: deleting extra topic config change in cluster linking test
- CNKAF-909: Add direct dependency on opencensus for SBK (#2057)
- KAFKA-10149: Allow auto preferred leader election when partitions are…
- CNKAF-918: Relax GoalUtils.eligibleBrokers criteria to allow leadership movements off observers.
- SEC-1034: use confluent-log4j instead of log4j
- Add RackAwareGoal to goal list
- CNKAF-884: Remove unnecessary metrics (#2059)
- Migrate MDS to run on KafkaHttpServer implementation
- KC-505: integration tests for CL metrics for 6.0.x
- METRICS-1717 use shared domain for jvm level metrics (#1949)
- CNKAF-854: Disabling SBK should persist the broker removal cancellation status
- [CNKAF-873] Don’t require network capacity configs. (#1996)
- SEC-1183 - Treat empty authority in CRN pattern as a wildcard
- CNKAF-882: Add proposal logging for add/remove
- CNKAF-837: Add broker should be able to add removed brokers.
- KC-393: Integration tests for mirroring compacted topics
- Add inter broker listener endpoint information to HTTP server binder.
- CNKAF-773: Do not throw an exception when the broker is not present in the cluster model
- CNKAF-847: Complete removal callback when there are no proposals to execute
- KSTORAGE-802: Allow specifying AWS credentials through configuration file
- [CNKAF-848] Set preferred user SBK config in server.properties. (#1999)
- [CNKAF-719] Add option to disable dynamic throttle setting when stati…
- KSTORAGE-819: Force roll segments only for partitions eligible for tiering
- KSTORAGE-799: reduce Log.lock contention in ArchiveTask segment selection logic
- KSTORAGE-774: Fix active segment roll logic when roll.min.bytes setting is enabled
- KSTORAGE-766: Ensure delete segment throttler does not block active segment rolls
- KSTORAGE-769: Fix to remove the costly toSet conversions in disk throttler
- KSTORAGE-758: Include ERROR status in TierPartitionStatus.isOpen()
- KSTORAGE-595 NPE in TierDeletedPartitionsCoordinator after tier topic is reassigned / deleted from broker
- KSTORAGE-754: KafkaMetrics Whitelist Creation and new KafkaMetricsWrapper util (#1906)
- KC-611: Fix mirror topic create processing incorrect config.
- KSTORAGE-547: state recovery injection tooling
- CONFLUENT: Fix Confluent provider start future completion, add unit tests
- [CNKAF-881] SBK: Don’t treat all new brokers as empty. (#2005)
- SBK/CruiseControl: low utilization fix
- CNAKF-833, CNKAF-842: Fix missing request rate metric and enable tele…
- RackAwareGoal should allow for uneven replica distribution based on t…
- CNKAF-788: Introduce proper API exceptions in the broker removal API
- METRICS-1708: add telemetry version header for forward-compatibility (#1883)
- CNKAF-791: Change data format of failedBrokers in ZK
- CNKAF-807: Remove zkclient from gradle dependencies.
- CNKAF-575: Remove IOtec:ZkClient dependency.
- [CNKAF-776] Switch SBK Maven publish group to “io.confluent”
- [METRICS-1609] dynamically add telemetry reporter at runtime
- KC-389: Retry segment deletion in cluster link test to fix timing issue
- KC-389: Cluster link integration tests with tier storage enabled
- CPKAFKA-5207: Complete authorizer start future only when sufficient nodes are ready
- CNKAF-843: Fix remove broker validation check
- Persist the broker removal state and handle failover
- CNKAF-773: Do not throw exception when there are no replicas to move on broker removal
- CNKAF-428: Integration test for replica placement self-healing
- Add retries to telemetry sampler integration test (#1922)
- [CNKAF-803] Add a metric to track the number of pending add-broker requests
- KC-72: Add topic listing for cluster links.
- CPKAFKA-5207: Fail fast during broker start up if multiple brokers use same MDS URL
- KC-536 bug fix to make offsets/acls configs dynamic (#1893)
- CNKAF-428: Update ReplicaPlacementGoal to handle replication factor change
- fix telemetry reporter reconfiguration
- update ksql and streams provider
- CNKAF-649: Handle broker restart during broker removal
- KC-390: Cluster link authorization tests
- backport [METRICS-1564] telemetry - derive interbroker client configs (#1816)
- (Backport) make commit-id optional, update whitelist to include C3 metrics
- SEC-1083 - AuditAdmin role’s scopeType = Cluster
- KC-392: Integration tests for cluster link quotas on source and destination
- KC-582: Add multi-tenant support for alterMirrors.
- CNKAF-795: Ensure SBK clients have bootstrap.servers configured
- Backport PR 1897 to 6.0.x
- SEC-1083 - Add AuditAdmin role (#1861)
- SEC-1039 - TopicSpec builder sets partitions and RF.
- CNKAF-794: Properly wrap broker removal exceptions in listener and improve remove broker test
- Removed disk throttling dependency on produce backpressure config
- Jvm Collector for Telemetry Reporter
- METRICS-1698 Setting default telemetry.enabled and exporter.enabled values to false.
- KC-224: make leader epoch cache flush asynchronous when stamping new leader epoch
- KC-537: apply topic-create-policy when auto-creating topics while fetching metadata
- CNKAF-782: MetricsReporterSamplers are configured in an odd manner.
- CONFLUENT: Fix cipher name for ChaCha20
- CNKAF-428: Replica placement
- METRICS-1152 document how to enable the telemetry-reporter in CPD
- KC-534: Adds forceful cluster link deletion.
- KCORE-54: Controller should accumulate metadata into fewer requests to mitigate handling delays
- CNAKF-645: Add metrics sampler to consume from telemetry reporter
- Generic telemetry reporter
- KSTORAGE-649: Remove duplicate library dependencies accidentally included in merge.
- [cnkaf-730] part 2: Completion callbacks for add and remove brokers
- [CNKAF-730] Minimal addBrokers support for SBK
- KC-207: Destination CL Metrics
- METRICS-1362 Updating telemetry dependency version from 1.87 to 1.98 to set new content-type
- KSTORAGE-649 and KSTORAGE-660: Support sts::AssumeRole and SSE-KMS
- CNKAF-697 - High bandwidth throttles during cluster roll
- Make ClusterLinkIntegrationTest run as integration test
- CNKAF-760: Integrate api state persistence with remove broker
- Upgrade cloudevents from v0.3 to v1.0
- CC-8138: Add Connect feature to inject Confluent license properties into connector properties before startup
- KC-571: Move tenant prefix to cluster link metadata instead of configs
- CIAM-69 Use authorizer setting to configure CC RBAC roles
- [CNKAF-759] Deal with non-Anomaly executions.
- SEC-994: Update Cluster Registry doc
- KC-387: Multi-tenant integration tests for security updates, offset migration
- KSTORAGE-752: removed disk throttler dependency on logDir existence at server startup
- KC-518: Use cluster link ID as primary key for cluster link metadata.
- Refactor Event Logger to make it generic
- KSTORAGE-699 Rationalize ArchiveTask/DeletionTask Exceptions
- CNKAF-649: Introduce integration test for SBK’s AddBroker, RemoveBroker and DescribeBroker APIs
- Add confluent metrics labels for connect metrics
- CNKAF-649 - Compose the broker removal operation into multiple interruptible tasks
- KSTORAGE-740: Add log closing time metrics
- SEC-951: Upgraded bc-fips to 1.0.2
- Add Confluent metrics tag Kafka server and client
- KSTORAGE-730: Allow max.partition.fetch.bytes to be configurable for tier fetches
- CNKAF-752: Store and fetch remove broker api state
- KSTORAGE-746; Make timing in TierTopicListenersTest deterministic
- CPKAFKA-4744; Fix log cleaner test failure caused by concurrent cleaning
- KC-407: Micro-benchmark for ClusterLinkFetcherThread based on ReplicaFetcherThreadBenchmark
- Introducing disk-usage based throttling for producers and replication followers
- cherry-pick upstream KAFKA-9960: implement KIP-606 to add metadata context to MetricsReporter (#8691)
- KCORE-142 new tests and bug fix for offset command config
- [CNKAF-741] Handle duplicate metrics in the DataBalancerMetricsRegistry
- Add domain to Context class
- CNKAF-702: At SBK startup check for api state persistence topic
- KC-382: Allow alter cluster configs for subset of configs in CCloud
- KC-533: Allow headers to be injected into RBAC records
- CIAM-64 Allow running other Providers with the MultiTenantAuthorizer
- KSTORAGE-741: Deserialize errorOffsetAndEpoch from Header during scan and init
- CNKAF-649 - Broker Removal operation with in-memory state
- CNKAF-617: Complete the DescribeBrokerRemovals API handler
- KSTORAGE-743: prevent tier.feature use with IBP < 2.4
- METRICS-1555 Exposing proxy configs as top level telemetry configs and making them dynamically configurable
- KCORE-142 - Offset Migration for Cluster Linking
- [CNKAF-637] Config variable for auto-healing
- KSTORAGE-425 Add tiered storage metrics to MetricsReporter
- CONFLUENT: Bump major CCloud version #major
- KC-370: Support Authentication events in ConfluentAuditLogProvider
- CPKAFKA-3743; Consumer group tool does not work on internal endpoint
- KSTORAGE-720: Always return tiered storage related configurations in DescribeConfigResponse
- [CNKAF-742] Detect new, empty brokers in controller; call into SBK
- KC-120: Integration Tests for ZK-Based ACLs for Periodic ACL Sync
- CONFLUENT: Add PowerMockIgnore for Scram API in RBAC test to fix classloader issues in subsequent tests
- SEC-959 Cluster Registry - Full Cluster Info CRUD Gatekeeping
- KC-517: Update cluster link integration test to use admin client.
- SEC-1015: Added kafka Metrics support to Confluent Authorizer, RBAC and AuditLog.
- CNKAF-649: Revise SBK Executor’s locking strategy
- KCORE-154: add tiered storage and CP to downgrade system test
- KC-128: Validate source topic read access on mirror topic creation.
- KCORE-329 (2/2): Adds ability to alter a topic’s mirror in TopicCommand.
- Created Broker removal CLI
- KSTORAGE-726: handle offset gaps in tier fetches
- CPKAFKA-2789; Add support-metrics-fullcollector to ce-kafka
- KC-506: Use cluster link tags for cluster link fetcher metrics
- KSTORAGE-627: TierMetadataComparator tool for tier-topic recovery pipeline
- KCORE 506: migrate secrets to Vault
- Spelling fix in LdapGroupManager
- KCORE-95: Encrypt sensitive cluster link configs stored in ZooKeeper
- KCORE-437: Refactor ClusterLink managers to cleanly disable/enable
- CONFLUENT: Improve performance of matchingAcls and AclSeqs.find
- KCORE-49: Disables mirror topic partition and restricted config updates.
- KCORE-485: Process mirror state during replication only on active destination leader
- KCORE-487: Add AuthenticationEvent and update Audit log event schema
- CONFLUENT: Introduce separate methods in KafkaApis for consumer and follower fetch handling
- Update Netty to 4.1.50 and clean up logging of NettySslEngineFactory
- CIAM-45 Allow RoleBindings at path scopes
- CNKAF-636: SBK broker failure detection
- KSTORAGE-627: TierPartitionForceRestore, TierPartitionState hotswap and tier task recovery
- CNKAF-701: Refactor SampleStore topic existence check
- Added tests for KafkaCruiseControl.
- CNKAF-649: Add BrokerRemoval state machine
- KSTORAGE-573: epoch state cache is flushed multiple times in recovery
- [METRICS-1234] telemetry - add support for multiple exporters of the same type
- KCORE-389: Add initial version of Audit API and update existing code
- CNKAF-677: Remove ZK fallback for SBK and move methods from ExecutorUtils to Executor
- CNKAF-684: Honor low utilization threshold settings for self-healing …
- CIAM-28 Add ConfluentCloudCrnAuthority with special treatment of path
- KSTORAGE-717: Filter all Confluent-specific topic configs from DescribeConfigsResponse
- KSTORAGE-652: FileTierPartitionState base path is not updated correctly
- CNKAF-699: Skip num cores or disk capacity update if node missing
- KCORE-340: Propagate new partitions in source cluster to controller
- KSTORAGE-653: re-add check_cluster_state check to tier upgrade test
- KCORE-141: Tenant prefixed cluster links and topic mirroring for Cloud
- KCORE-331: _confluent-license topic exponential backoff
- fix some downstream validation issue
- Add shutdown hook to KafkaHttpServlet interface
- SEC-1008 Do not close null AdminClient
- CONFLUENT: Remove ConfluentLeaderAndIsr and TierListOffset requests/responses
- CNKAF-704: Remove tenant prefix from CreatePartition responses
- CNKAF-314: Add DescribeBrokerRemovals{Request,Response} RPC and Admin Client API
- CONFLUENT: Set Scala 2.13 as the default Scala version
- KCORE-436: Client interceptor for destination tenant prefixing in cluster link clients
- CIAM-25 Allow CRNs with Paths but no Clusters
- emit deltas as cumulative metrics
- CNKAF-649: (part 1/2) Introduce ShutdownManager for SBK broker removal
- CNKAF-697 (part1): Increase min broker quota constants
- CONFLUENT: Improve error message when cluster linking is disabled
- KSTORAGE-412 Server side handling for the unclean leader flag from LeaderAndIsr re…
- METRICS-1514 collect one-minute rates from yammer meters
- Upgrade telemetry api to latest version
- CONFLUENT: Enable fatal warnings with Scala 2.13 and fix warnings
- CONFLUENT: Make createSslEngine(String, String) public again
- KCORE-329: Alter mirrors API.
- KCORE-426: Server should not resolve DNS on establishment of SASL connection
- CPKAFKA-4923: Disable audit logging in RbacTest::test_ldap_to_rbac_authorizer_upgrade
- CNKAF-632: RemoveBrokers api plumbing
- switch downstream validation
- [MINOR] Remove scoverage plugin from SBK build.
- Bump Chart versions
- Merge changes from Apache Kafka
- CONFLUENT: Always close consumer in ConsumeBenchWorker
- KCORE-438 (1/2): Remove topic read test on cluster link creation.
- CNKAF-638: Introduce SBK internal RPC client
- KCORE-41: Disable local produce requests for mirror topics.
- CONFLUENT: Fix build warnings in new ACL sync code
- CLIENTS-1476: Create Kafka HTTP Server interfaces and loading mechanism.
- KCORE-99: Periodic ACL Migration
- CNKAF-622: Properly handle dynamic config changes.
- KCORE-435: Propagate cluster link create/delete using ZK change notification
- KCORE-427: Add broker config to enable cluster linking feature
- CONFLUENT: Don’t reach into java.lang.reflect.Field internals
- KSTORAGE-683: Alter fencing logic for recovery work
- CPKAFKA-4982 -Change Trogdor’s Helm chart to deploy in parallel.
- KAFKA-9731: Disable immediate fetch response for hw propagation if replica selector is not defined
- KCORE-167: Process replication failures with retry timeout to move partition to failed state
- CONFLUENT: Fix locking during metadata restart for link reconfiguration
- KSTORAGE-677: FIX: Reset archiver max lag whenever there is no lag
- remove unnecessary labels from kafka server resource
- CNKAF-629: Add InitiateShutdown Request
- KCORE-55: Adds mirrorTopic and clusterLink to topic creation API.
- KCORE-53: Dynamic security config update for inter-cluster connections
- CNKAF-483: Added static and dynamic configs for request backpressure tuning
- KSTORAGE-26: Fix non-deterministic TierEpochStateReplicationTest
- CNKAF-593: Consolidate all thread creation in KafkaCruiseControlThreadFactory
- KCORE-104: Fix truncation during unclean leader election for linked partition
- CONFLUENT: Simplify Gauge definitions in Partition
- CNKAF-610: Rename StartRebalance RPC and DrainBroker Admin API to RemoveBrokers
- [CNKAF-606] Stop proposal execution on Cruise Control shutdown/SBK deactivation
- [CNKAF-558] Handle SSL interbroker listeners for SBK clients.
- CONFLUENT: Ensure license validator is closed in test to avoid impacting subsequent test
- KSTORAGE-648: remove S3TierObjectStore multi-part log segment upload support
- CNKAF-596: Add a test to verify SBK API access with RBAC
- KSTORAGE-628: Implement TierPartitionFence injector tool for recovery
- KSTORAGE-63: pt2, add TierTopicListeners, add getAndRemoveAll and tests
- KSTORAGE-63: pt1, add TierPartitionForceRestore metadata
- SEC-912 - Allow MDS to make changes to dynamic configs
- KCORE-341: Fix permission type in authorizer debug log entry
- SOAK: Revise soak workloads
- [SEC-912] Allow MDS to make changes to dynamic configs
- KSTORAGE-631: ducker up should not require GCS credentials to be activated in Dockerfile
- CNKAF-577: Handle null config values when describing static throttle configs
- CONFLUENT: Add support for Scala 2.13
- WFLOW-51: Implement a beginShutdown hook which allows for triggering ControlledShutdown
- KCORE-42 (4/4): Adds mirror SyncTopicsConfigs task.
- KCORE-48: Broker implementation for cluster linking APIs.
- KCORE-48 (partial): Adds ClusterLinkCommand.
- KCORE-330: Allow brokers to complete start up with configured license
- [CNKAF-594] Support broker shutdown for SBK
- [CNKAF-492] Set some more config parameters for SBK
- Update cloud Dockerfile to use JDK 14.0.1
- CNKAF-313: Migrate SBK to use DataBalancerMetricsRegistry
- CNKAF-549: Add metric for self healing enabled
- Instructions for exporting metrics to Sandbox env
- KSTORAGE-644: Separate fencing error metrics for primary & catchup consumers
- CONFLUENT: Update JDK to 11.0.7 by bumping cc-base to 5.1.0
- KSTORAGE-634 Added stronger bounds to ensure tiered log deletion
- KCORE-137 (partial): Expands per-topic cluster link state.
- [SEC-895] ConfluentAuditLogProvider should handle reconfiguration wit…
- CONFLUENT: Reduce allocations in requests via buffer caching
- KAFKA-9864: Avoid expensive QuotaViolationException usage (#8477)
- CNKAF-50: Add metrics for replication throttling
- KCORE-51: Adds cluster link metadata object.
- CNKAF-585; Multi tenant interceptor rejects create topic requests with NO_REPLICATION_FACTOR or NO_NUM_PARTITIONS when quotas are enabled
- KCORE-316: Don’t load configs from ZK when the log has already been loaded
- Fix MultiTenantTest
- CPKAFKA-1836: Simplify forwarding scripts by just adding .sh suffix
- CONFLUENT: add utility method to write public keys to pem files (#19)
- CAAS-1960: Record authenticated tenant connection/disconnection metrics
- KAFKA-7304: Limit connection accept rate to prioritize other channels (#29)
- CONFLUENT: Add Aegis Proxy
- CONFLUENT: Update cc-base to v2.3.0
- Merge changes from apache kafka
- CPKAFKA-2164: Ensure that we specify a charset when writing out and reading string bytes
- CONFLUENT: Fix unused compiler warnings
- [SEC-901] Speed up SameClusterTest
- KCORE-42 (partial): Adds functions for merging source + mirror topics…
- SEC-728 - MDS Cluster Registry - Enable Dynamic Configuration
- KCORE-48 (partial): Adds ClusterLinkClientManager for managing admin …
- Towards KCORE-165 (partial): Adds ClusterLinkScheduler for handling cluster l…
- Towards KCORE-136: Cluster linking APIs client+RPC wiring.
- [CNKAF-582] Handle topic creation during checkStartupComponents
- CNKAF-372: Disk capacity sizing
- CONFLUENT: Remove unnecessary allocations in hot matchingAclExists
- METRICS-1415 fix collection of integer gauges
- CNKAF-313: Adds a MetricsRegistry wrapper class for SBK
- [CNKAF-370] Support dynamic enable/disable of SBK
- [CNKAF-492] Adjust the default parameters of the KafkaCruiseControlConfig.
- KSTORAGE-634 Added stronger bounds to ensure tiered log deletion
- KSTORAGE-628: Introduce RecoveryUtils class with recovery tooling related functions
- CPKAFKA-4647: Disable audit logging in RbacTest::test_broker_failure test
- KSTORAGE-628: Introduce TierPartitionFence event type + fencing logic
- add AppInfo version+commitId to metrics messages
- KSTORAGE-577: System test for fencing logic, along with CLI tool to trigger fencing
- KSTORAGE-632: Better logging for replica fetcher state transitions for tiered storage
- KSTORAGE-381 Tooling around TierMetadataValidator to use tier-validator as a cc-service
- CNKAF-536: Reduce number of warnings when SBK creating clients
- [METRICS-1361] telemetry-api -> opencensus-proto
- CNKAF-537: Add drainbrokers adminClientAPI
- CNKAF-562: Change log level for self-healing being triggered from WARN to INFO.
- KSTORAGE-613: fix log start offset initialization on full truncation
- KSTORAGE-626: Remove metrics reporter configuration when configuring internal tier topic clients
- [CNKAF-528] Use the inter-broker listener for SBK clients.
- Prevent warning message on every client metadata update
- KSTORAGE-580: Return a generic error message when compaction is enabled on a tiered topic
- KCORE-40: Inter-cluster fetcher for cluster linking
- Update role, Owner tags for cost reporting
- KCORE-297: Fix wildcard principals for centralized ACLs and no.acl.found for ConfluentServerAuthorizer
- CNKAF-541: Reduce the number of tests run in RandomClusterTest
- METRICS-1232 - Whitelist check for individual and derived metrics
- [METRICS-1233] make telemetry metrics whitelist a dynamic config
- CNKAF-548: Skip checking quotas when recording replication traffic
- Set TopicExclusion string in SBK-generated CruiseControl config.
- KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates unused iterator (#8422) (#1477)
- KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates unused iterator (#8422)
- KSTORAGE-331, KSTORAGE-392: add and improve tier component listener metrics and heartbeats
- CPKAFKA-4753: Add metrics tracking total txn time, txn timeouts, and illegal state errors
- CPKAFKA-4649: Ensure LDAP group manager is recreated on writer failover
- SBK MINOR: Drain permits in startUp
- CNKAF-319: Add Start rebalance protocol
- modify kafka_path to only set project for confluentplatform
- CNKAF-520, CNKAF-525: Wait for KafkaSampleStore check to complete at startup
- KSTORAGE-411: Pass a hint down from controller to the leader in case of an unclean leader election
- KSTORAGE-453: Persist leader epoch in tier checkpoint file
- re-schedule pr scans to weekly
- [CNKAF-492] SBK: CruiseControl Property defaults
- Publish preview artifacts with preview release settings
- CNKAF-50: Add static broker-level replication throttles
- SBK: Implement KafkaDataBalancer startup/shutdown methods
- CPKAFKA-4772: Fix PhysicalClusterMetadataTest’s tests
- METRICS-1292 Dynamic Update of API key and secret in Kafka.
- [CNKAF-216] Derive SBK capacity values from config parameters directly.
- CONFLUENT: Make Rest Client Http Authenticator configurable
- CNKAF 362: Topic exclusion logic for SBK in KafkaDataBalancer
- CNKAF-305: Dynamic configs for sbk - enable/disable, throttle
- CONFLUENT: Update cc-trogdor-service and cc-soak-clients to 0.803.0-6.0.0-ce-SNAPSHOT
- KSTORAGE-545: Handle failed deletions from GcsTierObjectStore
- CONFLUENT :HttpBearerCredentialProvider must implement configurable
- ST-3292: migrate to semaphore2
- CNKAF-19: ADB: Soak test self-healing
- KSTORAGE-579: FIX fencing advancement issues
- KSTORAGE-574: producer and epoch state restore is not resilient to failures
- KSTORAGE-399 : Crash broker if FileTierPartitionState encounters corrupted flushed state file during initialization.
- SEC-162: RestClient refactor regression
- [CNKAF-479] Support executing ce-sbk objects with the rest of Kafka
- [CNKAF-491] Remove clusterConfigs functionality from the TopicConfigProvider.
- Revert “KCORE-114: Allow placement-constraints to be configured for internal topics (offsets + transactions) (#1319) (#1401)”
- KSTORAGE-482 Enhancing TierMetadataValidator tool to verify objects in S3 TierObjectStore
- KSTORAGE-559: add maximum eligible segments deleted on every cleanup run
- CNKAF-486: Add filter to exclude internal topics
- Put an initial implementation of the DataBalancer interface in ce-sbk.
- KCORE-114: Allow placement-constraints to be configured for internal topics (offsets + transactions) (#1319)
- KSTORAGE-568: bound hotset retention by LSO
- KCORE-114: Allow placement-constraints to be configured for internal topics (offsets + transactions)
- CPKAFKA-4670 Fix for system tests running 0.8.2.2 version producers
- CNKAF-324: Allow configuring a delay before broker registration with ZK.
- CNKAF 447: Update SBK mode config to a flag
- [CNKAF-477] Restore ConfluentMetricsReporterSampler
- KSTORAGE-572: FIX: Access fenced partitions metric under lock in TierTopicConsumer
- CPKAFKA-4676: TierRoundtripTest.test_tier_roundtrip timeout increase
- KSTORAGE-548: Check for missing init leader messages
- KCORE-234: Upgrade netty to 4.1.48.Final and netty-tcnative to 2.0.29.Final
- MCM-965: Reduced physical cluster metadata reload period to 2 mins
- CC-7979: Add new topic tracking operations of KIP-558 to default RBAC roles
- KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer (cherry-pick)
- Bump up rollingupgrade deb package from 0.5.0 to 0.9.0
- [CPKAFKA-4658] Restore removed SchemaRegistryUrl Config.
- KCORE-225: Schedule periodic license validation task at fixed rate
- DG-274, DG-133: Add configs for Basic Auth and SSL for embedded Schema Registry client (#1374)
- DG-274, DG-133: Add configs for Basic Auth and SSL for embedded Schema Registry client
- KSTORAGE-465: GCS tier deletion manager system test support
- v5.5.x KSTORAGE-524: Fix potential NPE in DelayedFetch
- KCORE-199: Set default values for schema configs on topics to avoid NPE
- CNKAF 289: Increase partition count in soak cluster
- [CNKAF-283] Put heavily trimmed CruiseControl into ce-kafka/ce-sbk
- KSTORAGE-550 Narrow scope of KafkaException to avoid handling of any unwanted bug.
- CONFLUENT MINOR: Revert #799, remove batch verifier from eos topic consumers
- KCORE-111: Default replica placement constraints for external topics
- CONFLUENT: fix exhaustive match warnings in TierArchiver
- KSTORAGE-354: avoid roundtrip and connection in GCS fetch path
- KCORE-175: Extend TopicPlacement to include assignment validation logic
- SEC-159: RestRequest pluggable HTTP auth
- KSTORAGE-543: Fix AutoAbortingS3InputStream: account for # read bytes being -1
- KSTORAGE-214, KSTORAGE-407: Bump FileTierPartitionState version
- Tier Storage system test for Consumer.offsetForTimes API
- Refactoring FIPS implementation
- KSTORAGE-389: Stronger detection of duplicate message processing for tiered storage
- CPKAFKA-2336: Log layer metrics for tiered storage
- CNKAF-305: Remove topic config from DynamicBalancer config
- METRICS-1138 Removing shading of transitive dependencies due to telemetry-client jar
- KCORE-190: Add max LSO lag metric
- KSTORAGE-468 : Metadata states validation insync with log start offset.
- CNKAF-304: Default value for rack aware flag
- KCORE-104: Re-instate PartitionLockTest.testAppendReplicaFetchWithUpdateIsr
- KSTORAGE-524: Fix potential NPE in DelayedFetch
- SEC-804: Wait for config update before verification in audit config change test
- CPKAFKA-1953: resume materialization from TierTopicManager commit positions
- Log TierPartitionStatus to differentiate online vs catchup consumer
- KSTORAGE-450: default confluent.tier.fetcher.num.threads = 4
- KSTORAGE-457 ListOffsetRequest v0 does not account for tiered section…
- KSTORAGE-330, KSTORAGE-328, KSTORAGE-325 TierStorageMaterialization recovery and validation tool.
- KSTORAGE-452: Improve README for system tests w/ Tiered Storage
- KSTORAGE-400: Handle duplicate PartitionDeleteInitiate messages
- KSTORAGE-420: missing toString on MergedLog
- KSTORAGE-460: Disable preferred tier reads for replica fetch
- CPKAFKA-4369. Cherry pick of #1228 into 5.5.x
- KSTORAGE-214, KSTORAGE-407: Fence user partition failures when materializing Tier Topic events
- KSTORAGE-510: Ensure that PendingFetch::finish() stops the PendingFetch
- Reduced test strength to ignore consumer data coverage caused by catc…
- CPKAFKA-4126: Add prefix to system test config to match MDS changes
- SEC-147: Refactor RestClient
- KSTORAGE-519: Fix: Ensure unknown exceptions from DeletionTask are isolated to user partition
- KSTORAGE-511: Add size asserts for TierPartitionStateEntry
- KCORE-183: Avoid new allocations in MultiTenantRequestContext.filteredMetadataResponse call
- KSTORAGE-344: TierMaterializationValidation as service for System Tests
- CONFLUENT: Fix bug in result handling from readFromLocalLog
- CNKAF-323: Add metric with per-tenant partition count
- KCORE-151: Allow some reassignment of offline brokers
- KSTORAGE-508: System test s3 prefix random generation incorrectly app…
- KSTORAGE-505: Tier deletion manager system test failing in vagrant runs
- KSTORAGE-424: Allow use of SSL for internal tier topic clients
- KSTORAGE-384 Adding system test for tiered state file corruption
- Merge branch ‘ccs-master’
- KSTORAGE-502: Improve documentation for tiered storage configurations
- CNKAF-305: Dynamic configs for SBK - kafka-side changes
- KCORE-93: Removing ZK utils from ce-kafka
- KCORE-110: Introducing new flag to display partitions violating topic replica constraints
- CNKAF-304: Update network config name
- CNKAF-304: Add static configs for self-balancing kafka
- KCORE-104: Re-introduce PartitionLockTest
- CONFLUENT: Update protobuf to 3.11.4 (#1237)
- Update protobuf to 3.11.4
- CPKAFKA-2700; Disable license validator in MultiTenantAuthorizer
- Define a DataBalancer object in the Kafka Controller.
- SEC-744 Make Audit Log Configs visible to AdminClient
- METRICS-582: Adding tracking for kafka.broker.rack in telemetry
- SEC-752 AuditLog FetchFollower
- KSTORAGE-163: Initial implementation of TierFetcher memory bounding.
- CPKAFKA-2705: Added retention to soak topics
- KSTORAGE-479: Revert “KSTORAGE-464: Fix: ensure a queued TierTopic ev…
- Add manual testing sections to ce-metrics README
- SEC-746 Ignore unknown JSON properties for forwards compatibility
- KSTORAGE-464: Fix: ensure a queued TierTopic event future throws when replaced
- KSTORAGE-337/KSTORAGE-348: tier deletion manager system test
- CONFLUENT: Add check in consume test util to verify records were consumed
- METRICS-541 increase metric reporter default publishing time from 15 se…
- CPKAFKA-3878: Re-enable scala inlining
- CPKAFKA-4369. Copy Kafka configs when creating LogConfig- Part 1 of 2
- CONFLUENT: Log better message on soak clients task creation 409 error response
- TierArchiver: sleep outside of synchronized block in main loop
Security¶
- Validate KSQL/RBAC cannot initialize on non-SASL protocols
- Check authentication.roles in KSQL bearer token requests
- CLIENTS-1658: Remove kafka core dependency from kafka-rest-security-plugin
- SEC-1034: log4j migration to confluent repackaged version
- remove test scope from connect-runtime dependency
- DG-706: fix bug in schema registry client tests, as well as some refactor
- DG-708: Skip Schema Validation query range check for first lookup
- Include Kafka dependencies to KSQL packaging
- Bump Vert.x version in confluent-ksql-security-plugin to 3.9.1
- fix: Make rest-utils compile scope as ksqlDB no longer pulls this in
- SEC-1091 - CR C3 - Implement MDS 1.0 Features Endpoint
- Remove AdminClientWrapper.
- Switch from application/vnd.api+json to application/json.
- Migrate off JSON:API.
- SEC-951: Upgraded bc-fips jars to 1.0.2
- Add authorizations for mode endpoints
- Add test that reproduces rest-proxy authorization bug.
- Fix authentication for non-producer POST requests in kafka-rest.
- Fix issue SSL class could not be found
- Remove obsolete kafka-rest dependencies.
- Fix bugs in Vertx auth plugin loginservice initialisation and unimplemented response method
- Fix Content-Type for produce request.
- Fix test due to ksql refactoring.
- Remove kafka-rest V1 APIs and all their dependencies.
- DG-403 Add missing method to retrieve schema refs
- Vertx auth adaptors
- SEC-850 - CSP : Make code path from BEARER Auth to BEARER Login explicit
- Add confluent-security-plugins-common module back
- Remove support-metrics-common dependency
- KCORE-82: Removing ZK licensing code and introducing LicenseManager
- Use kafka.metrics.KafkaYammerMetrics instead of com.yammer.metrics.Metrics
- Fixing compiling error due to KIP-515
MQTT Proxy¶
- CC-11294: Bumped up Netty version to be consistent with Kafka and rest of CP.
- METRICS-1989: confluent-kafka-mqtt depends on confluent-telemetry
- CC-10900: Add confluent-telemetry to production jars.
- CC-9739: Pass producer-striped configs and bootstrap.servers to AdminClient when getting Kafka cluster id.
- CC-9786: Depend on confluent-metrics in tests only.
- fix for METRICS-1733 change telemetry config constant names
- CC-8363: Configure telemetry reporter.
Metadata Service¶
- SEC-1421 - Set default replicator factor when creating audit log topic
- SEC-1369 - Update MDS to non-broken version of Maven Assembly plugin
- SEC-1034: log4j migration to confluent repackaged version
- SEC-866 - move audit log “other” to “management”
- [SEC-1345] Read HTTP Reason phrase when connection closed
- Move kafka-http-server project into clients project
- SEC-1323 - Make MDS Api doc not have everything be Bold
- Fixed code which hard code security protocl to SASL_PLAINTEXT, should…
- SEC-1268 - Backout confluent.security.event.logger.destination.log.* properties
- SEC-1259 - Clarify audit log config API spec
- SEC-777 - Cluster Registry & MDS should be lenient with it’s JSON Parsing
- SEC-777 Cluster Registry & MDS should be lenient with it’s JSON Parsing
- SEC-1222 Role Resources For Principal API throw a null pointer exception if ClusterName is passed in ResourceRequest
- SEC-1089 CR C3 - Return Nice Names for all other Private UI endpoints
- Fixed defect that RbacApiAppConfig missed all audit config with prefix confluent.security.
- CLIENTS-1599: Convert KafkaHttpServlet to rest-utils Application.
- SEC-1091 - CR C3 - Implement MDS 1.0 Features Endpoint
- SEC-1212 - Do not create a new admin client if admin config not changed
- SEC-1097: Disallow reassignment of name <=> scope.
- SEC-1088 CR C3 - Return Nice Names for C3 visibility endpoint
- MMA-7805: Fix for correctly packaging cluster-information-migration-script
- MMA-7805 Script to import and export cluster information data to and from Registry
- SEC-1191 - Assign default topics to produce, consume and other category
- SEC-1021: updated validation + make error messages useful
- SEC-1130 - Inject ConfluentAdmin from ce-kafka
- Migrate MDS to KafkaHttpServlet
- Added more testing cases for verifying audit log backend API.
- SEC-1127 - Fixed a defect that AuditLogConfigSpec didn’t get updated in DynamicC…
- SEC-988 Cluster Registry Fix to make CLI backwards compatible
- SEC-1122 - Fix 6.0.x build break
- SEC-986: Lockdown cluster name
- SEC-940 - Make clusters optional in MdsScope in swagger spec
- SEC-1037: Return sorted ‘full’ clusters followed by sorted ‘redacted’ clusters for cluster registry
- SEC-1003 Cluster Registry : Migrate from “name” to “clusterName”
- SEC-1020 - Cluster Info Protocol Validation
- SEC-982: Nice name gatekeeping
- CIAM-12 - Try loading token.key.path as a public key for cloud
- SEC-731: Single shot overwrite of multiple clusters
- SEC-959: Full Cluster Info CRUD Gatekeeping for Cluster Registry
- SEC-1019 Cluster Registry : Introduce security protocol field to ClusterInfo
- SEC-985: MDS CR NiceNames : Update basic RBAC endpoints
- SEC-1015: Fixed compile error due to interface changed by SEC-1015 in ce-kafka.
- For backward compatibility we name the json scope object as ‘scope’
- Add support for Cluster Scope object with clusters or paths to MdsSco…
- CIAM-49 Spec file for MDS API v2alpha1 - fix whitespace
- CIAM-49 Spec file for MDS API v2alpha1
- SEC-1001: Require at least one host in a Cluster
- SEC-926: Using dynamic config for cluster registry backend
- SEC-941 Add Http api that returns an array of Mds Scope object.
- SEC-942: MDS CR NiceNames : Introduce MdsScope Object
- SEC-965: Sorted-ness of ClusterRegistry
- SEC-943 - MDS CR : Investigate creating ResourceType for CR CRUD
- Fix OpenAPI specification format
- SEC-730: Cluster Registry - Cluster Level CRUD “API stub”
- SEC-730: Cluster Registry - Cluster Level CRUD “API stub”
- SEC-949 - MDS CR : Update CR given Audit logs fix
- SEC-728 - Cluster Registry - Make the backend be a broker Dynamic Config
- SEC-891 - MDS : Short term fix flakey Token tests by isolating them
- [SEC-844] restore /security/1.0 prefix
- SEC-871 - MDS Test : Test / document OOTB Ldap and RBAC case sensitivity
- SEC-850 - CSP : Make code path from BEARER Auth to BEARER Login explicit
- SEC-800 - MDS : Remove MdsLoginService.roles
- SEC-751 MDS - Clean up the openapi yaml spec for CLI
- SEC-751 - MDS - Clean up the openapi yaml spec for the CLI
- SEC-751 - MDS - Clean up the openapi yaml spec for the CLI
- SEC-727 - Cluster Registry - Lock down the basic list of clusters
- SEC-726 - Service Reg - Basic list of clusters
Replicator¶
- REP-1176 Log error when connector submission fails for Replicator executable
- REP-1131: Add support for Replicator with regex router
- REP-1055 Enable converter configs for Replicator connector
- REP-787: Synchronize fetching end offsets from source cluster to calculate lag
- REP-824: Changed JMX initialization to use method from KIP-606
- REP-721: Delete old license validation code from codebase
- REP-795 moved translated records log to
DEBUGand only print if records were translated - REP-761: ensure
offset.topic.commitis false iftopic.preserve.partitionsis false - REP-755: retrieve port from –connect-url if specified
- REP-743: Fix issue where we don’t properly log the topic that got an
UnknownTopicPartitionException - REP-754 fix for offset commit failure resulting in loss of offsets
- REP-689: Reduce log level of offset commit exception from
ERRORtoWARN - REP-665: Deleting all ZK dependencies from code base
- REP-673: Port ZK dependencies for replicator-verifier into the verifier module
- REP-664: Remove ZK dependencies from
ReplicatorSourceConnectorandReplicatorSourceTask