Important
You are viewing documentation for an older version of Confluent Platform. For the latest, click here.
Confluent Platform Component Changelogs¶
This topic provides changelogs for the individual Confluent Platform components.
Contents
Confluent Platform Changelog¶
Version 5.4.11¶
Commercial Features¶
Confluent Server¶
- correct bad cherry-pick
- DP-9030 - Switch to codeartifact repo
- KENGINE-283 Topic ID Mismatch Issue - Exception added to disallow Alt
- Increase vagrant aws timeout for system test
- Bouncycastle issue fix for connect service [5.4.x]
- APIF-3112: Upgrade io.prometheus.jmx:jmx_prometheus_javaagent to 0.17.2
- KSECURITY-788: Upgrade jackson-databind version to 2.13.4.2
- KSECURITY-832: Upgrade bc fips drivers
- CIAM-2424 - Name RestClient threads for stackdump identification (#7887)
- KSECURITY-794: Upgrade to protobuf version 3.19.6
- CIAM-2304: Add SecurityMetadata:Describe to SecurityAdmin
- RCCA-8564: log a warning if LDAP login fails becuase of network issues
Security¶
- Pin bc-nonfips dep as a test dependency
- Pin bouncycastle fips dependency in confluent-security-plugins
- Removed bcfips from common
- excluded bcfips jars
- AUTHN-1325: Update bc-fips version to 1.0.2.3
Control Center¶
- MMA-9023: memory bounding for streams applications [5.4.x, master]
- Revert “MMA-12206 upgrade io.netty:netty-codec-http”
MQTT Proxy¶
- Fixing the tests and config validator, to incorporate new changes in validation of MqttDecoder brought in by netty upgrade
- Upgrading netty
Metadata Service¶
- CIAM-2304: Add unit tests for SecurityAdmin
Connectors¶
Elasticsearch Connector¶
HDFS Connector¶
JDBC Connector¶
- PR-1272 - Revert “Allow close() on CachedConnectionProvider while attempting to connect…”
- PR-1206 - Allow close() on CachedConnectionProvider while attempting to connect
- 09922baf - Fixing https://confluentinc.atlassian.net/browse/CC-17951
JMS Connector¶
Replicator Connector¶
- REPL-2055-: Log WakeupException at debug level
- RCCA-7678: Reverse proxy header check added
S3 Connector¶
- 34e4a9e4 - restoring 5.4.x
- f5c2066f - [maven-release-plugin] prepare for next development iteration
- 26054f26 - [maven-release-plugin] prepare release kafka-connect-storage-cloud-5.4.11
- 66b1148e - Upgrading the common to 5.5.12
- 6131dc42 - Correcting ‘kafka.connect.storage.common.version’
- ae2bc6ba - Explicitly add woodstox with 5.4.0 as hadoop-common extracts 5.3.0 version
- 754539b4 - Explicitly add woodstox with 5.4.0 as hadoop-common extracts 5.3.0 version
- 43cec916 - Fixing release issues for 5.4.x
- 189cc353 - Updated tag
- c52d8be8 - fixing release issue
- 22079e1f - fixing release issue
- 9242e8c1 - fixing release issue
- 9c03516e - fixing build issue
- 63e79c19 - correcting common version
- 1db7f7e8 - CCMSG-2293 : bumping hadoop version
- d9488782 - bumping hadoop and jettison version
- ca35a6c5 - bumping hadoop and jettison version
Version 5.4.10¶
Community Features¶
Common¶
Kafka¶
Commercial Features¶
Confluent Server¶
- [METRICS-3888][METRICS-4448] update to telemetry-client without shaded Netty for 5.4.x
- KAFKA-14107: Upgrade Jetty version.
- [METRICS-4448] upgrade telemetry to pick up netty upgrade change for 5.4.x
- Migrating log4j12 to reload4j, slf4j-log4j12 to slf4j-reload4j.jar (5.4.x)
- Upgrading google-cloud-storage to v2.6.1 (5.4.x) #6994
- Upgrading aws-java-sdk-s3 to v1.12.268 (5.4.x)
- Upgrading netty version to 4.1.79.Final (backport 5.4.x)
- Update bouncy castle to 1.70 (back port 5.4.x)
Security¶
- add auth token provider to ksql security extensions
- AUTHN-1085: Fix NPE in debug log
- APIF-2714: Migrate from confluent-log4j to slf4j-reload4j.
Control Center¶
- MMA-12206 upgrade io.netty:netty-codec-http
- MMA-12201 Exclude slf4j in transitive dependency
- MMA-12201: Use reload4j version properties from common
- MMA-8781: Respect SR Config and handle nulls in CachingSchemaRegistryClientSupplier
- MMA-12201 Replace log4j with reload-4j
MQTT Proxy¶
- CC-17620: remove dependency on slf4j-log4j12 and confluent-log4j
Metadata Service¶
- CIAM-2164 - Migrate from confluent-log4j to reload4j for MDS
Connectors¶
HDFS Connector¶
JMS Connector¶
- CC-17744:Switch from confluent-log4j to reload4j
Replicator Connector¶
- Migrate confluent-log4j to reload4j.
S3 Connector¶
Version 5.4.9¶
Community Features¶
Version 5.4.8¶
Commercial Features¶
Confluent Server¶
- APPSEC-1184: Enforce Gson version in all projects.
- Upgrade Gson
- KSECURITY-244 RCCA-6743: Set hostname verifier to accept everything when identifica
- KSECURITY-221: Update jackson packages to 2.13.2 version
- cherrypick two ducker changes from ccs 2.6 to ce 6.0.x
- MINOR. Skip –bootstrap-server config when testing with server not th
- CPKAFKA-3509, CPKAFKA-3510: API on AuthCache to get known scopes and bindings
- KSECURITY-198: Update protobuf-java packages to 3.19.4
Control Center¶
- Bump maven-shade-plugin to 3.3.0
- Fix build break
- MMA-11901: suppress error message altogether
- LOG_DIR to match rest of confluent components
- APIF-2311: Remove graphql module
Metadata Service¶
- This release version includes a fix to prevent stack trace exposure from getting surfaced in HTTP responses for malformed requests.
JDBC Connector¶
JMS Connector¶
- CCMSG-1656 excluded protobuf
- CCMSG-1814: Excluding protobuf from confluent-licensing
Replicator Connector¶
- REPL-1970: Remove source record logging
Version 5.4.7¶
Commercial Features¶
Confluent Server¶
- Cherry-pick “Disabled bubbling errors from JsonSerializer and Deserializers for connect that print the message in stack trace (#5558)” onto 5.4.x
- Upgrade TCNative
- Bump Protobuf Dependency Version
- CPKAFKA-3454: Add ACL storage support to KafkaAuthStore
- KC-1933; Upgrade confluent-log4j to 1.2.17-cp2.2 (5.4.x)
Security¶
- DG-1249, DG-1257, DG-1353, DG-1423, DG-1462, DG-1491, DGS-2950: Fix testGetLastOffsetWithExistingMessages
- APPSEC-827:Upgrade bc-fips
Control Center¶
- MMA-11603: Update protobuf version
- MMA-11478: enable cors check on websocket
- (MINOR): update armeria version and exclude netty artifacts
- MMA-11468: Add option to enable nosniff protection
- MMA-10986: fix health check resource for connect
- MMA-10571: add SR basic auth to armeria healthcheck
- MMA-9126: Fix broken RocksDB stats endpoint
- MMA-10114: Change default values for command topic configuration
- MMA-9479: Remove JDK dependency from control center stop script
MQTT Proxy¶
- [CC-16389] Changing the maven URL in pom.xml from http to https
- Revert “[CC-16389] Changing the maven URL in pom.xml from http to https”
- [CC-16389] Changing the maven URL in pom.xml from http to https
Metadata Service¶
Connectors¶
Elasticsearch Connector¶
JDBC Connector¶
- PR-1166 - Upgrade org.postgresql:postgresql
- PR-1153 - CCDB-4512: Fix handling of PostgreSQL NUMERIC/DECIMAL types with unspecified scale
- PR-1148 - Revert “MINOR: call setCharacterStream API with length (#1130)”
- PR-1145 - CCDB-4476: Prevent blocking on database connection in JdbcSourceConnector::taskConfigs
- PR-1143 - CCDB-4454: Automatically truncate long table names to respect PostgreSQL length limits
- PR-1144 - CCDB-4470: Handle case mismatch between user-specified and table-provided timestamp column names
JMS Connector¶
- Update CODEOWNERS to connect-team1
S3 Connector¶
- PR-473 - CCMSG-1582 Fix bug that generates a small files when scheduled rotation has expired
Storage Common¶
Version 5.4.6¶
Commercial Features¶
Confluent Server¶
- CONFLUENT: Upgrade netty to 4.1.68.Final and disable io.netty.handler.ssl.openssl.useTasks (#4606)
- SEC-2731 - Improve Ldap handling for stale connections (#4497)
- CC-13747 CIAM-1263 FF-4419: Limit max retries in MDS URL Refresh loop
- CIAM-1253 Fix Logging of various timeouts in ce-rest-auth RestClient
- RestClient test should not use Jupiter in backport
- SEC-1537 RestClient Retry Logic Fixed (#4395)
- KC-1878 Debug Logging for LDAP Context creation
Metadata Service¶
- Override commons-codec to 1.13
Connectors¶
JMS Connector¶
- Start fresh batching everytime waitForAllPendingToCommit turns true (independent of commitRecords())
- CCLOG-1054: pinned netty-codec version
- Fix display names of configs JMS Source connector
Replicator Connector¶
- Upgrade jakarta.el:jakarta.el-api
- REPL-1843: Track uncommited records given to Connect framework.
Storage Common¶
- PR-216 - Upgrade netty version
Confluent Platform Docker Images¶
- Confluent Platform Docker Images will no longer ship with Yum/Apt configuration that allowed users to update running Confluent Platform containers to the next Confluent Platform release. Confluent recommends upgrading Confluent Platform service components running in containers by replacing the running container with the next Confluent Platform released Docker image and using Docker volumes to persist data between upgrades.
- Confluent Platform Docker images formerly shipped with world writable data and configuration directories (
chmod a+w). Those permissions are now removed, and adjustments made to make those directories writable from the image’sappuserUSER. - Confluent Platform Docker images are now built with
yum --nodocs ...to prevent installing Red Hat Documentation. This slims down the images slightly. Confluent documentation from Confluent packages are still installed.
Version 5.4.5¶
Community Features¶
Common¶
- PR-372 - Backport changes in https://github.com/confluentinc/common/pull/370
- PR-357 - log4j migration to confluent repackaged version
- PR-361 - Upgrade bcprov-jdk15on to 1.68
- PR-365 - Upgrade log4j2 to 2.13.2
- PR-309 - SEC-1789: Upgrade bouncycastle to 1.68
- PR-344 - APPSEC-110: Exclude Netty in zookeeper 3.4.14
- PR-342 - Fix jacksondatabind and jackson-cbor version
- PR-343 - Change zookeeper version to 3.4.14
- PR-340 - Fix zookeeper, jacksondatabind and jackson-cbor version
- PR-339 - update spotbugs and jacoco for improved jdk support
Kafka¶
- 9e50cc96 - KAFKA-13209: Upgrade jetty-server
- PR-609 - [CVE-2021-35516] Upgrade org.apache.commons:commons-compress
- PR-10919 - KAFKA-12985: Upgrade jetty to 9.4.42 (#10919)
- PR-10922 - KAFKA-12790: Remove SslTransportLayerTest.testUnsupportedTlsVersion (#10922)
- PR-3849 - KSTORAGE-1626: Fix logging during topic creation (#3849)
- d3d4f087 - Fix logging during exception in DynamicBrokerConfig
- d50fde2f - KC-1625: Fix logging in DynamicBrokerConfig
- 1844221e - KAFKA-10570; Rename JMXReporter configs for KIP-629
- 4f3fb7ae - KC-1752 Remove public facing String redact API which is not currently used
- 4c75baf7 - KC-1752 Fix potential alter/update config related logging
- PR-573 - KC-1744: Upgrade Jetty and Commons-codec
- PR-564 - APPSEC-60: Upgrade org.apache.maven:apache-maven
KSQL¶
REST Proxy¶
- PR-885 - KREST-1577 Replace a manually set Avro version with the Avro version variable.
- PR-870 - Use a more lenient test for testBackoffMsControlsPollCalls.
- 0c83d02d - SEC-1034: log4j migration to confluent repackaged version
- d0b719bf - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- 19154f1d - SEC-1034: log4j migration to confluent repackaged version
- 8a9c055f - build: backport formatting checkstyle issues from #873
- PR-857 - KREST-1263 Exclude a vulnerable jackson-mapper-asl dependency from avro 1.8.2 in 5.3.x.
- PR-854 - security patch release support
- d7a631a4 - KREST-1029 Remove a new occurrence of SLF4J versioning in kafka-rest.
- 40e94513 - KREST-1029 Remove SLF4J versioning from kafka-rest.
- PR-280 - KREST-578: Add configurable DosFilter support. (#238)
- PR-275 - Fix rest utils checkstyle and log4j updates
- PR-270 - Upgrade Jetty to 9.4.43.v20210629
- 66aa0c15 - KREST-1030 Upgrade hibernate-validator.
- 6b8327ac - KREST-1030 Upgrade Jersey.
- 2e5ff421 - KREST-1030 Upgrade Jetty.
- bfb547cc - KREST-1030 Upgrade hibernate-validator.
- 0ea562be - KREST-1030 Upgrade Jersey.
- 8ab657af - KREST-1030 Upgrade Jetty.
- PR-251 - bump guava version to 30.1.1-jre
Schema Registry¶
- PR-1990 - DGS-1879 changed timeout parameter
- PR-1989 - Upgrade commons-compress to 1.21 (5.4.x and later)
- PR-1992 - Fixing checkstyle complaints 5.4.x
- PR-1986 - DGS-1915: Upgrade commons-compress version
- PR-1984 - Add confluent-log4j as dependency
- PR-1977 - DGS-1909 Fix AVRO_FIELD_DEFAULT_FLAG_PROP in AvroData
- PR-1969 - SEC-2492 exclude guava version with CVE
- PR-1943 - DGS-1826 Fix NPE when struct default has null
- PR-1896 - DGS-1704: override commons-compress
- PR-1895 - DGS-1704: Upgrade jackson-mapper-asl
- PR-1887 - security patch release support
- PR-1884 - Upgrade maven-plugin dependencies to 3.6 in 5.4.x (#1517)
- PR-1879 - DGS-1551 Upgrade hibernate-validator
Commercial Features¶
Confluent Server¶
- KAFKA-13209: Upgrade jetty-server
- AUDIT-143: Fix kafka exporter race condition which resulted in granted audit events appearing in the rejected topic
- CPKAFKA-3372: Fix namespace syntax error in cc-soak-clients template
- DGS-1903: (backport to 5.4.x) Add config option for missing schemas TTL (#4118)
- KC-1809 Upgrade transitive hibernate-validator dependency
- KC-1800: Upgrade guava to 30.0 (#4094)
- KC-1784: Use URI instead of URL for RBAC writer comparison to avoid hostname resolution issues
- KSTORAGE-1626: Do not log sensitive configs during topic creation (#3849 )
- KSTORAGE-1626: Do not log sensitive configs during topic creation
- Hide secrets when logging during exception in DynamicBrokerConfig
- KC-1625: DynamicBrokerConfig must not log sensitive config values
- KAFKA-10570; Rename JMXReporter configs for KIP-629
- KC-1752 Remove public facing String redact API which is not currently used
- KC-1752 Fix potential alter/update config related secret logging leakage
- Correct syntax for password based schema registry config
- Fix secret type for Schema registry client
- KC-1625: DynamicBrokerConfig must not log sensitive config keys
- KC-1752 Fix potential alter/update config related secret logging leakage
- DGS-1701 Fix potential secret logging leakage in embedded Schema Regi
- DGS-1701 Fix potential secret logging leakage in embedded Schema Regi
- CONFLUENT: Sync from confluentinc/kafka 2.4
- KC-1744: Upgrade Netty
- CPKAFKA-3265: Fix the cloud CreateTopicPolicy
- SEC-298 - Log errors when MDS can not start
Security¶
- APPSEC-287: Override commons-codec
- fix formatting issues with newer checkstyle versions
- fix: Checkstyle issues in AuthorizationFilter.java
- SEC-1034: log4j migration to confluent repackaged version
- DGS-1903: Add missing schema cache to LruSchemaRegistryClient
- SEC-2533 upgrade commons-io to 2.11.0
- SEC-2493 Remove bouncy-castle explicit dependency
- Fix BasicAuthorizerIntegrationTest
MQTT Proxy¶
- CCMSG-1251: log4j migration to confluent repackaged version
- CCMSG-1101: Upgrade Netty to 4.1.63.Final
Metadata Service¶
- SEC-1034: log4j migration to confluent repackaged version
- Fixed issue where users with ResourceOwner role, could not see rolebindings of other users on resources they own
Connectors¶
Elasticsearch Connector¶
HDFS Connector¶
JDBC Connector¶
JMS Connector¶
- SEC-1034: log4j migration to confluent repackaged version
- CC-15203: Pin bcprov-jdk15 verison to resolve CVE
- ST-5057: security patch release support
- CC-14525, CC-14537: Move connect-runtime to test scope to address CVEs
Replicator Connector¶
- REPL-1667: Fixing CVEs for 5.3.x and beyond
S3 Connector¶
Storage Common¶
- PR-212 - Upgrade Jetty server dependency to 9.4.43.v20210629
- PR-210 - CC-15305: Upgrade transitive dependency commons-compress to 1.21
- PR-201 - Upgrade commons-io to 2.7
- PR-200 - Upgrade libthrift to 0.13.0
- PR-186 - Upgrade jackson-mapper-asl to 1.9.14.jdk17-redhat-00001
- PR-185 - CC-14522: resolve hadoop, jackson, thrift, log4j, commons-collection, bean-utils, netty, zookeeper, commons-codec, and httpclient CVEs
- d0c4526d - Set version to 5.3.5-SNAPSHOT
Version 5.4.4¶
Community Features¶
Common¶
- PR-326 - KREST-1066 Upgrade Guava to the latest version to patch CVE-2020-8908.
- PR-314 - MMA-9930: Sync Google Guava version.
- PR-309 - SEC-1789: Upgrade bouncycastle to 1.68
- PR-305 - SEC-1474: Upgraded bcpkix-jdk15on to 1.66
- PR-337 - DGS-1551: upgrade jackson-bom to 2.9.10.20210106
- PR-334 - CC-14690: Define com.fasterxml.jackson.dataformat:jackson-dataformat-cbor.
- PR-332 - Update dependency-check to 6.1.6
Kafka¶
- PR-561 - backport KAFKA-9777 to 2.4
- PR-8289 - KAFKA-9712: Catch and handle exception thrown by reflections scanner (#8289)
- PR-552 - fix: CVE-2021-28168
- PR-551 - Upgrade org.reflections:reflections to 0.9.12
- PR-549 - Upgrade Jetty to 9.4.39.v20210325
- PR-541 - SEC-2022: Upgrade netty to 4.1.62.Final
- PR-524 - KC-1458: Update jacksonDatabind to 2.10.5.1
- eafebafa - KAFKA-12324: Upgrade jetty to fix CVE-2020-27218
- PR-518 - KC-1457: Update Httpclient to 4.5.13
- PR-10065 - KAFKA-12193: Re-resolve IPs after a client disconnects (#9902) (#10065)
- PR-9977 - KAFKA-12310: Update zookeeper to 3.5.9 (#9977)
- PR-9987 - KAFKA-10895: Gracefully handle invalid JAAS configs (follow up fix) (#9987)
- PR-9319 - KAFKA-10413: Allow for even distribution of lost/new tasks when multiple Connect workers join at the same time (#9319)
- PR-9765 - KAFKA-10763: Fix incomplete cooperative rebalances preventing connector/task revocations (#9765)
- PR-9806 - KAFKA-10895: Attempt to prevent JAAS config from being overwritten for basic auth filter in Connect (#9806)
- PR-8357 - KAFKA-9767: Add logging to basic auth rest extension (#8357)
- PR-9633 - KAFKA-10706: Ensure leader epoch cache is cleaned after truncation to end offset (#9633)
- PR-9481 - KAFKA-10284: Disable static membership test in 2.4 (#9481)
- PR-9431 - KAFKA-10426: Deadlock in DistributedHerder during session key update. (#9431)
- PR-9375 - KAFKA-10574: Fix infinite loop in Values::parseString (#9375)
- PR-9307 - Backported changes to fix tests that resolve to multiple addresses. (#9307)
- PR-9320 - KAFKA-10439: Connect’s Values to parse BigInteger as Decimal with zero scale. (#9320)
- PR-9306 - KAFKA-10477: Fix JsonConverter regression to treat MISSING nodes as NULL nodes (#9306)
KSQL¶
REST Proxy¶
- PR-831 - KREST-865 Remove usage of the deprecated SimpleAclAuthorizer.
- 8bafcb7d - RM-39: Backport #766 to 5.4.x.
- PR-249 - DGS-1621 Upgrade Jetty to 9.4.40
- PR-246 - fix: CVE-2021-28168
- PR-244 - Upgrade Jetty to 9.4.39.v20210325
- PR-237 - Define Netty version to be shared to downstreams.
- PR-229 - KC-1456: Update Jetty to 9.4.36.v20210114
- PR-231 - UIF-298: Upgrade Jetty to 9.4.35.v20201120
- PR-228 - MMA-9930: Use compatible version of Guava
- PR-222 - UIF-300: Upgrade org.apache.httpcomponents.httpclient
- PR-223 - UIF-302: Upgrade guava for 5.4.x and older.
- PR-224 - upgrade hibernate-validator to 6.1.7.Final
- PR-216 - MMA-7520: add CsrfTokenProtectionFilter
- PR-209 - (backport) SEC-592: Customize Jetty request queue and threadpool size (#172)
- PR-210 - ST-4542: Upgrade jetty to address CVE-2020-27216
Schema Registry¶
- PR-1807 - UIF-302: Add Guava declaration in dependency.
- PR-1737 - Backport #1733 to 5.4.x
- PR-1779 - DGS-772 Add API annotations for /mode
- PR-1770 - DGS-1315: upgrade HibernateValidator
- PR-1748 - DG-424: prevent import different schema with same id
- PR-1699 - DG-1040 Register ids for soft deletes
- PR-1686 - MMA-8608 Handle duplicate configs after stripping prefix
- PR-1684 - MMA-8608: parse configs before calling restService.configure()
- PR-1668 - Removed property ${confluent.maven.repo}
- PR-1647 - DG-933 Resend old value when handling invalid key-value
Commercial Features¶
Confluent Server¶
- KC-1630: Upgrade telemetry version
- Upstream sync from confluentinc/kafka July 3 2019
- backport KAFKA-9777 to 5.4.x
- CVE: Force jacksonCbor version
- KC-1610: Upgrade dataformat-cbor version
- CPKAFKA-2773 : Dynamic ssl certs update
- Dynamic ssl certificate updates
- Add IncrementalAlterConfigs to MultitenantRequestContext
- SEC-2022: Upgrade netty to 4.1.62 Final.
- Correct Downstream validation parameters
- CONFLUENT: Prevent multiple invocations for computing tier lag
- MMA-9397: Filter out confluent.ssl configs when describing topic configs
- MINOR: Allow packaging to skip including cloud-only makefile parts
- KAFKA-12324: Upgrade jetty to fix CVE-2020-27218
- KC-1457 : Update Httpclient to 4.5.13 and add explicit HttpClient dependency
- CONFLUENT: Change the archiver ForkJoin pool to a fixed thread pool.
- CONFLUENT: Update Zookeeper node start timeout in system tests
- KSQL-5843: prevent log spam from MDS RestClient on client close
- SEC-1789 Bump Bouncy Castle dep to resolve CVE-2020-28052
- KSTORAGE-1290: Bump google storage client dependency to 1.113.4
- RM-42: KAFKA-10727; Handle Kerberos error during re-login as transient failure in clients
- CPKAFKA-6126: Update commons codec (#2785)
- Upgraded bcpkix-jdk15on to 1.66 (#2769)
Security¶
- DGS-1463: Modify config types for sensitive SR validator plugin configs
MQTT Proxy¶
- SEC-2022: Upgrade Netty to 4.1.62.Final.
Connectors¶
Elasticsearch Connector¶
- PR-506 - CCMSG-835, CCMSG-834: Update outdated commons-codec to fix CVE
- PR-495 - CC-13829: Upgrade httpclient to 4.5.13
- PR-466 - adding a null check when validating list of urls
- PR-463 - CC-10923: add reasonable validators for number configs (#456)
- PR-459 - CC-12621: Add config combo validator for ssl configs
- PR-454 - CC-12447: add comprehensive ITs to ES
- PR-455 - CC-12530: do not throw exception on flush after task close
- PR-449 - CC-12129: catch resource_already_exists_exception when creating indices
HDFS Connector¶
JDBC Connector¶
- PR-1059 - CC-14541 : Exclude zk and netty from packaging
- PR-1042 - Upgrade org.postgresql:postgresql to version 42.2.19
- PR-1004 - CCDB-3681: pick binding method wisely according to column def
- PR-959 - CC-12626: validate topic.prefix to disallow spaces in between
- PR-947 - CC-12468: Periodically return control flow to WorkerSourceTask
- PR-942 - Maven: Removed build property ${confluent.maven.repo}.
- PR-938 - Remove Connect-utils and Licensing-Extensions Deps
JMS Connector¶
- CC-14525, CC-14537: Move connect-runtime to test scope to address CVEs
- Upgrade Active MQ client to 5.15.14 and fix CVE
- CC-12824: Handle WebLogic Server clients, which include on messages a destination that is both a Queue and Topic
- CC-11937: Correct the maximum retry interval in the JMS source task
Replicator Connector¶
- REPL-1410: Changed logging for WakeupException to DEBUG
- REPL-1386: Adding clear logging statements for src and dest operations
- REPL-1409: Added WARN logging if destination SR deemed not ready
- SEC:1820: Upgrade codec version to remediate CVEs
- FF-3928: Bump the version of Guava to remediate CVEs
- REPL-1204 enable retries on cluster id fetch for Replicator task start
S3 Connector¶
- PR-414 - CCMSG-1021: Upgrade AWS SDK to 1.11.1015 and exclude jackson-dataformat-cbor
- PR-399 - SEC-1996: Replace commons-httpclient with apache httpclient
- PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common
- PR-378 - CC-12695: stop throwing retriable exceptions on non-retriable S3 errors
Storage Common¶
- PR-176 - CC-13830: Upgrade org.apache.httpcomponents:httpclient version to 4.5.13 to address CVE
- PR-165 - Fix time-zone aware calculation for scheduled rotation between daylight savings and standard time
- PR-151 - Correct SR schemas cache size property name for AvroDataConfig
- PR-156 - Update commons codec to 1.15 to resolve CVE and exclude zookeeper test jar
Version 5.4.3¶
Commercial Features¶
Security¶
- Validate KSQL/RBAC cannot initialize on non-SASL protocols
- SEC-1034: log4j migration to confluent repackaged version
- DG-706: fix bug in schema registry client tests, as well as some refactor
- DG-708: Skip Schema Validation query range check for first lookup
Confluent Server¶
- CONFLUENT: Allow super.users access to authorize and update resource roles
- KC-801: Reassignment succeeds with empty constraints
- CPKAFKA-5849: Don’t log password configs in request log filter
- METRICS-2005: Upgrade telemetry dependencies to fix Netty CVEs
- formalize C3 resource names (C3 -> ControlCenter)
- CPKAFKA-2852: CCloud metadata listener thread should not die on unexpected exception
- SEC-384: Use the correct log level for debug output
- Merge fix kafka.log.LogManagerTest compilation issue
- KSTORAGE-559: add maximum eligible segments deleted on every cleanup
- [CPKAFKA-4964] Handle nonActiveLogSegments when minCompactionLag > 0
Metadata Service¶
- SEC-1034: log4j migration to confluent repackaged version
Community Features¶
Common¶
Schema Registry¶
- PR-1625 - DG-410 Fix doc/defaults in Avro converter
- PR-1624 - DG-857 Fix Avro converter for top-level bytes object
- PR-1611 - Correct License in Client library
- PR-1598 - CP-644: Include CP jars required for Security and monitoring
- PR-1528 - SEC-1034: log4j migration to confluent repackaged version
- PR-1578 - DG-767 Calculate proper defaults for client ssl configs
- PR-1547 - ST-3988: update swagger-core to 1.6.2
- PR-1520 - Handle null error stream in RestService
- PR-1517 - Upgrade maven-plugin dependencies to 3.6 in 5.4.x
- PR-1470 - Allow unknown properties while deserializing SchemaRegistryProtocol.Assignment
Kafka¶
- PR-414 - KAFKA-2697: client-side support for leave group
- PR-8973 - KAFKA-10218: Stop reading config topic in every subsequent tick if catchup fails once
- PR-8181 - KAFKA-9584 Headers ConcurrentModificationException
- PR-9202 - KAFKA-10401: Fix the currentStateTimeStamp doesn’t get set correctly
- PR-400 - KAFKA-2719: Use wildcard classpath for dependant-libs
- PR-9106 - KAFKA-10146, KAFKA-9066: Retain metrics for failed tasks (backport to 2.4)
- PR-9089 - KAFKA-10224: Update jersey license from CDDL to EPLv2
- PR-9086 - FIX: Remove staticmethod tag to be able to use logger of instance
- PR-9043 - KAFKA-10295: Wait for connector recovery in test_bounce
- PR-9040 - KAFKA-10286: Connect system tests should wait for workers to join group
- PR-8921 - KAFKA-10160: Kafka MM2 consumer configuration
- PR-8989 - KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups
- PR-8905 - KAFKA-10173: Fix suppress changelog binary schema compatibility
- PR-8455 - KAFKA-9845: Warn users about using config providers with plugin.path property
- PR-8663 - KAFKA-9985: Sink connector may exhaust broker when writing in DLQ
- PR-8453 - KAFKA-9841: Revoke duplicate connectors and tasks when zombie workers return with an outdated assignment
- PR-8630 - KAFKA-9969: Exclude ConnectorClientConfigRequest from class loading isolation
- PR-8827 - KAFKA-9849: Fix issue with worker.unsync.backoff.ms creating zombie workers when incremental cooperative rebalancing is used
- PR-8805 - KAFKA-9848: Avoid triggering scheduled rebalance delay when task assignment fails but Connect workers remain in the group
- PR-8270 - KAFKA-9216: Enforce internal config topic settings for Connect workers during startup
- PR-8804 - KAFKA-9851: Revoking Connect tasks due to connectivity issues should also clear the running assignment
- PR-8135 - KAFKA-9570: Define SSL configs in all worker config classes, not just distributed
- PR-8786 - KAFKA-10083: fix failed testReassignmentWithRandomSubscriptionsAndChanges tests
- PR-8777 - KAFKA-10082: Fix the failed testMultiConsumerStickyAssignment
- PR-8779 - [MINOR] Fixing spotbug fail - removing unused variable.
- PR-8668 - KAFKA-9987: optimize sticky assignment algorithm for same-subscription case
- PR-8739 - KAFKA-10056; Ensure consumer metadata contains new topics on subscription change
- PR-8118 - KAFKA-9472: Remove deleted Connect tasks from status store
- PR-8511 - KAFKA-9888: Copy connector configs before passing to REST extensions
- PR-8608 - KAFKA-9950: Construct new ConfigDef for MirrorTaskConfig before defining new properties
- PR-8444 - KAFKA-8869: Remove task configs for deleted connectors from config snapshot
- PR-8673 - KAFKA-9992: EmbeddedKafkaCluster.deleteTopicAndWait not working with kafka_2.13
- PR-8618 - KAFKA-9955: Prevent SinkTask::close from shadowing other exceptions
- PR-8674 - KAFKA-9996: Upgrade zookeeper to 3.5.8
- PR-8090 - KAFKA-9537 - Cleanup error messages for abstract transformations
- PR-8647 - KAFKA-9669; Loosen validation of inner offsets for older message formats
- PR-8616 - KAFKA-9127: don’t create StreamThreads for global-only topology (2.4)
- PR-8230 - KAFKA-9667: Connect JSON serde strip trailing zeros
- PR-8360 - KAFKA-9768: Fix handling of rest.advertised.listener config
- PR-7950 - KAFKA-9419: Integer Overflow Possible with CircularIterator
- PR-8204 - KAFKA-9633: Ensure ConfigProviders are closed
- PR-8554 - KAFKA-9919: Add logging to KafkaBasedLog::readToLogEnd
- PR-8442 - KAFKA-9830: Implement AutoCloseable in ErrorReporter and subclasses
Connectors¶
Elasticsearch Connector¶
- PR-435 - CC-9455: address SslFactor backwards/forwards compatibility
- PR-433 - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- PR-428 - CC-4774: Check for unexpected datatypes in response from server
- PR-413 - CC-9373: Skip only exceptions during conversion when drop.invalid.message is true
- PR-412 - CC-9347: Inspect interrupted state before next attempt
- PR-411 - CC-9261: call get() on futures so that we properly handle exceptions …
- PR-410 - CC-9314: Allow disabling hostname verification
HDFS Connector¶
- PR-510 - Resolving netty CVEs
- PR-507 - [ci-skip] bump hadoop version to address beanutils CVE
- PR-501 - CC-7360: Recover gracefully if closing a temporary file fails
- PR-506 - CC-9755: Validate timezone configuration with rotate.schedule.interva…
- PR-503 - CC-9710: Retry effectively on exceptions from time-based commits
- PR-31 - Fix security bugs
JDBC Connector¶
- PR-918 - CC-8794: MS SQL Server datetime give repeated messages due to rounding error
- PR-925 - CC-11940: fix a bug that happens when merging long text to oracle table
- PR-910 - CC-11507: Always parse Date in UTC
- PR-906 - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- PR-692 - Enable CURRENT_TIMESTAMP for SAP HANA JDBC
- PR-866 - CC-8750: Changed the dialects to sanitize all URL properties that case-insensitively contain “password”
- PR-844 - CC-8898: Add better error messages when consumed records don’t match connector config
JMS Connector¶
- Address CVEs by upgrading activemq client to 1.15.13
- SEC-1034: log4j migration to confluent repackaged version
- SEC-1307: Backport “log4j replacement with confluent repackaged version”
- CC-3439: Update MQ Source Connector to better handle errors and auto reconnect when MQ manager is restarted
Replicator Connector¶
- REP-1176 Log error when connector submission fails for Replicator executable
- REP-787: Synchronize
endOffsetsConsumerinFetchEndOffsetsService - REP-743: Log the topic that got an
UnknownTopicPartitionException
S3 Connector¶
- PR-357 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3 (back port to 5.0.x)
- PR-352 - SEC-1307: Backport “log4j replacement with confluent repackaged version”
- PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms
- PR-328 - CC-9198 Move null value handling earlier in SinkTask put
Version 5.4.2¶
Commercial Features¶
Security¶
- DG-274: Basic auth support for schema validation
- DG-133: Add SSL configs to schema validation’s SR client
- SEC-251: Don’t treat Basic authentication as JWT Bearer Token
Confluent Server¶
- CONFLUENT: Ensure license validator is closed in test to avoid impacting subsequent test
- KSTORAGE-502: Improve documentation for tiered storage configurations for 5.4.x
- KCORE-341: Fix permission type in authorizer debug log entry
- KCORE-316: Don’t load configs from ZK when the log has already been loaded
- KCORE-330: Allow brokers to complete start up with configured license
- add AppInfo version+commitId to metrics messages
- KCORE-297: Fix wildcard principals for centralized ACLs and no.acl.found for ConfluentServerAuthorizer
- CPKAFKA-4649: Ensure LDAP group manager is recreated on writer failover
- KCORE-234: Upgrade netty to 4.1.48.Final and netty-tcnative to 2.0.29.Final
- KCORE-225: Schedule periodic license validation task at fixed rate
- DG-274, DG-133: Add configs for Basic Auth and SSL for embedded Schema Registry client (#1374)
- SEC-159: RestRequest pluggable HTTP auth
- CPKAFKA-1955; Wait for SCRAM credentials to be propagated in tests
MQTT Proxy¶
- Upgrade netty version to 4.1.48
- CC-8256: Don’t pull in incorrect netty version
Metadata Service¶
- SEC-251 - Don’t treat Basic authentication as JWT Bearer Token
Community Features¶
Common¶
- PR-254 - ST-3339: Make dependency-check execution a JDK8 activated profile
- PR-252 - ST-3255: Parameterize the option in docker plugin to always pull latest base image
- PR-251 - ST-3227: Pull down the latest base image version when building docker images with the maven plugin
- PR-250 - Upgrade to junit 4.13
- PR-249 - Update ZooKeeper to 3.5.7
KSQL¶
- PR-5139 - fix: include lower-case identifiers among those that need quotes (#3723)
- PR-5076 - fix: support trailing slashes in listener URLs
- PR-4978 - test: fix issue with unresolvable host test
- PR-4936 - fix: remove immutable properties from headless mode
- PR-4935 - fix: handle leap days correctly during timestamp extraction (#4878)
- PR-4917 - fix: logic when closing ksqlEngine fixed
- PR-4857 - fix: don’t cleanup topics on engine close (#4658)
- PR-4871 - Junit upgrade 5.1.x
- PR-4819 - fix: backport fixes from query close
- PR-4769 - fix: fix test compile goal
- PR-4767 - fix: build
- PR-4764 - Fix broken 5.3.x
- PR-4759 - build: disable -Werror compiler argument in tests
- PR-4716 - fix: decimals in structs should display as numeric (#4165)
- PR-4723 - fix: don’t display decimals in scientific notation in CLI
- PR-4742 - test: include rows in row count mismatch error message in RQTT
- PR-4704 - fix: backport fixes from query close
Schema Registry¶
- PR-1437 - Fix cyclic schema references
- PR-1422 - Fix cycles in Avro schemaEquals
- PR-1403 - CC-8525: add equals implementation for CyclicSchemaWrapper
- PR-1398 - DG-349 Ensure schema key version matches schema value
- PR-1387 - DG-305 Fix ClassCastException for proxy.port
- PR-1381 - CC-7027: Revert to legacy serialization of Decimal logical type on incompatible scale/precision
- PR-1378 - fix: connect.default on long avro field
Kafka¶
- PR-8574 - KAFKA-9925: decorate pseudo-topics with app id
- PR-8543 - KAFKA-9826: Handle an unaligned first dirty offset
- PR-8562 - Test compilation fixes for Scala 2.11
- PR-8536 - KAFKA-9883: Add better error message when REST API forwards a request and leader is not known
- PR-8476 - KAFKA-9838: Add log concurrency test and fix minor race condition
- PR-8471 - KAFKA-9854: Re-authenticating causes mismatched parse of response
- PR-8451 - KAFKA-9835: Protect FileRecords.slice from concurrent write
- PR-8420 - KAFKA-9815: Ensure consumer always re-joins if JoinGroup fails
- PR-8418 - KAFKA-9807: Protect LSO reads from concurrent high-watermark updates
- PR-8408 - KAFKA-9810: Document Connect Root REST API on /
- PR-8412 - KAFKA-9750: Fix race condition with log dir reassign completion
- PR-8416 - KAFKA-9739: Fixes null key changing child node
- PR-8279 - KAFKA-9706: Handle null keys/values in Flatten transformation
- PR-8280 - KAFKA-9707: Fix InsertField.Key should apply to keys of tombstone records
- PR-8290 - KAFKA-9677: Fix consumer fetch with small consume bandwidth quotas
- PR-8339 - KAFKA-9752: New member timeout can leave group rebalance stuck
- PR-8336 - KAFKA-9749: TransactionMarkerRequestCompletionHandler should treat KAFKA_STORAGE_ERROR as retriable
- PR-8205 - KAFKA-9634: Add note about thread safety of ConfigProvider
- PR-8223 - KAFKA-9654: ReplicaAlterLogDirsThread can’t be created again
- PR-8256 - KAFKA-9675: Fix bug that prevents RocksDB metrics to be updated
- PR-8232 - KAFKA-9658: Fix removing user quotas
- PR-8208 - KAFKA-8895: Delete all topics before recreating in Stream integration tests
Connectors¶
HDFS Connector¶
JDBC Connector¶
- PR-839 - CC-8823: Add support for the JDBC sink to write to views in addition to tables
JMS Connector¶
- CC-8250: Revert JMS Retry Functionality
Replicator Connector¶
- REP-754 Fix for offset commit failure resulting in loss of offsets
- REP-700 Fix for bad handling of UnknownTopicOrPartitionException
- REP-685: Fixing NullPointerException for null timestamp
- REP-677: Adding default connect.protocol=eager for Replicator Executable
S3 Connector¶
Version 5.4.1¶
Commercial Features¶
Confluent Server¶
- KAFKA-9601: Stop logging raw connector config values (#8165)
- CONFLUENT: Fix exception on trying to retrieve tierable segments
- KAFKA-9599 create unique sensor to record group rebalance (#8159)
- CPKAFKA-2705: Added retention to soak topics
- KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset … (#8136)
- KAFKA-9533: ValueTransform forwards null values (#8108)
- KAFKA-8025: Fix flaky RocksDB test (#8126)
- CONFLUENT: Add KeyPair PEM reader/writer to PemUtils
- KAFKA-8245: Fix Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups (#8032)
- KAFKA-9515: Upgrade ZooKeeper to 3.5.7 (#8125)
- KAFKA-9535; Update metadata before retrying partitions when fetching offsets (#8088)
- CONFLUENT: Migrate from Nexus to Artifactory
- KAFKA-9355: Fix bug that removed RocksDB metrics after failure in EOS (#7996)
- ST-3015: Switching from nexus to artifactory repo
- allow ReplaceField SMT to handle tombstone records (#7731)
- KAFKA-9192: fix NPE when for converting optional json schema in structs (#7733)
- KAFKA-9181; Maintain clean separation between local and group subscriptions in consumer’s SubscriptionState (#7941)
- KAFKA-9500: Fix FK Join Topology (#8015)
- Kafka-9503: Fix TopologyTestDriver output order (#8065)
- KAFKA-9390: Make serde pseudo-topics unique (#8054)
- KAFKA-9540: Move “Could not find the standby task while closing it” log to debug level (#8092)
- KAFKA-7052 Avoiding NPE in ExtractField SMT in case of non-existent fields (#8059)
- KAFKA-9517: Fix default serdes with FK join (#8061)
- KAFKA-9507; AdminClient should check for missing committed offsets (#8057)
- Initial commit to migrate Docker build from cc-kafka
- SEC-713 Refresh Audit Log topic metadata
- KAFKA-9491; Increment high watermark after full log truncation (#8037)
- KAFKA-9074: Correct Connect’s Values.parseString to properly parse a time and timestamp literal (#7568)
- KAFKA-9492; Ignore record errors in ProduceResponse for older versions (#8030)
- KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning (#7932)
- KSTORAGE-409: lazily initialize consumer lag histogram to address all…
- CONFLUENT: Bump up version to 5.4.1
- CPKAFKA-2388; Add RBAC system test to verify role-based authorization
- KAFKA-9254; Overridden topic configs are reset after dynamic default change (#7870)
- KAFKA-9143: Log task reconfiguration error only when it happened (#7648)
- KAFKA-9024: Better error message when field specified does not exist (#7819)
- KAFKA-9083: Various fixes/improvements for Connect’s Values class (#7593)
- KAFKA-9449; Adds support for closing the producer’s BufferPool. (#7967)
- KAFKA-9235; Ensure transaction coordinator is stopped after replica deletion (#7963)
- KAFKA-9144; Track timestamp from txn markers to prevent early producer expiration (#7687)
- KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore (#7908)
- KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
- KAFKA-9065; Fix endless loop when loading group/transaction metadata (#7554)
- KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
- HOTFIX: fix system test race condition (#7836)
- KAFKA-9293; Fix NPE in DumpLogSegments offsets parser and display tombstone keys (#7820)
- KAFKA-8933; Fix NPE in DefaultMetadataUpdater after authentication failure (#7682)
- KAFKA-9232; Coordinator new member timeout does not work for JoinGroup v3 and below (#7753)
- KAFKA-9307; Make transaction metadata loading resilient to previous errors (#7840)
- fix rows (#7837)
- KAFKA-9011: Removed multiple calls to supplier.get() in order to avoid multiple transformer instances being created. (#7685)
- KAFKA-9011: Scala bindings for flatTransform and flatTransformValues in KStream (#7520)
- KAFKA-9310: Handle UnknownProducerId from RecordCollector.send (#7845)
- KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
- KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
- KAFKA-8705: Remove parent node after leaving loop to prevent NPE (#7117)
- port paragrpah from CP docs (#7808)
- fix typo in processor-api developer guide docs (#7689)
Control Center¶
- [MMA-6375, MMA-6372] update license viewing permissions for OTA
- MMA-6007: Fix for invalid follower and observer count values
- [MMA-5879] smart proxy via service healthchecks
- Sort SR urls for multi-cluster SR hash
- support for multiple SR clients
- refactor kafka-rest proxy to handle async registration
- actually use caching SR client supplier
- chore: update @confluent/control-center to 3.91.43-cp-5.4.x.0
- chore: update @confluent/control-center to 3.91.41-cp-5.4.x.0
- chore: update @confluent/control-center to 3.91.40-cp-5.4.x.0
- chore: update @confluent/control-center to 3.91.39-cp-5.4.x.0
- chore: update @confluent/control-center to 3.35.31-cp-5.3.x.0
- chore: update @confluent/control-center to 3.91.38-cp-5.4.x.0
- update @confluent/control-center to 3.91.37-cp-5.4.x.0
- chore: upgrade @confluent/control-center to v3.91.34-cp-5.4.x.0
- chore: upgrade @confluent/control-center to v3.91.33-cp-5.4.x.0
- chore: update @confluent/control-center to v3.91.32-cp-5.4.x.0
Confluent Hub Client¶
- Update upstream to ce-kafka
Kafka Connect Elasticsearch¶
- CC-7147: set the max connection per route config for client
Kafka Connect JDBC Connector¶
- CC-7637: change from DEBUG level to TRACE level
- CC-8090: Validate connection.attempts property in config class
- CC-8087: Remove plaintext logging of task configs
- [jdbc-source-connector] Fixed issue in CachedConnectionProvider, to enable retries when a connection is failed
- Make metadata allFields ordered consistently with the valueSchema.
- CC-7628 Set conservative timeout for JDBC to stop trying to reach drive
- CC-5047: connect jdbc url field should have example text
- Move setFetchSize to GenericDatabaseDialect
- CC-7637: added logging of query to be executed at DEBUG level
- Handling deletes with schema !null
- Fix Postgres OOM by limiting fetch size
Kafka Connect JMS Source¶
- MINOR: Fix Kafka Scala version in all modules
- CC-3439: Update MQ Source Connector to better handle errors and autoreconnect when MQ manager is restarted
- CC-6503: Capping batch size at 10000 for IBM MQ connector
- MINOR: maven compiler plugin no longer requires Java 8
- fix dependency to not use hardcoded scala
- CC-7957: corrected pom version and confluent-licensing-new version
- CC-8052: corrected pom version and confluent-licensing-new version
- CC-7984: corrected pom version
- Revert “Revert “CC-3937: repackaging base classes””
- Revert “CC-3937: repackaging base classes”
- CC-3937: repackaging base classes
- CC-7926: Corrected logic to prevent infinite loops during poll()
- CC-7866: Added more and improved existing logging in the base JMS source task and connector
- CC-7866: Added more and improved existing logging in the base JMS source task and connector
- Update upstream to ce-kafka
Confluent Replicator¶
- REP-638: Ensuring translatorMonitor is not null before closing it out
- REP-637: Ensuring timestampsCommitter is not null before closing it out
Community Features¶
Kafka¶
- PR-8188 - throttle consumer timeout increase (#8188)
- PR-8165 - KAFKA-9601: Stop logging raw connector config values (#8165)
- PR-270 - Update to jackson databind 2.9.10.3
- PR-8159 - KAFKA-9599 create unique sensor to record group rebalance (#8159)
- PR-263 - Changes to migrate to Artifactory
- PR-8136 - KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset … (#8136)
- PR-8108 - KAFKA-9533: ValueTransform forwards null values (#8108)
- PR-8126 - KAFKA-8025: Fix flaky RocksDB test (#8126)
- PR-259 - Update to jackson databind 2.9.10.2
- PR-8032 - KAFKA-8245: Fix Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups (#8032)
- PR-8125 - KAFKA-9515: Upgrade ZooKeeper to 3.5.7 (#8125)
- PR-8088 - KAFKA-9535; Update metadata before retrying partitions when fetching offsets (#8088)
- PR-7996 - KAFKA-9355: Fix bug that removed RocksDB metrics after failure in EOS (#7996)
- PR-7731 - allow ReplaceField SMT to handle tombstone records (#7731)
- PR-7733 - KAFKA-9192: fix NPE when for converting optional json schema in structs (#7733)
- PR-7941 - KAFKA-9181; Maintain clean separation between local and group subscriptions in consumer’s SubscriptionState (#7941)
- PR-8015 - KAFKA-9500: Fix FK Join Topology (#8015)
- PR-8065 - Kafka-9503: Fix TopologyTestDriver output order (#8065)
- PR-8054 - KAFKA-9390: Make serde pseudo-topics unique (#8054)
- PR-8092 - KAFKA-9540: Move “Could not find the standby task while closing it” log to debug level (#8092)
- PR-8059 - KAFKA-7052 Avoiding NPE in ExtractField SMT in case of non-existent fields (#8059)
- PR-8061 - KAFKA-9517: Fix default serdes with FK join (#8061)
- PR-8057 - KAFKA-9507; AdminClient should check for missing committed offsets (#8057)
- PR-1 - Kafka 2257
- PR-8037 - KAFKA-9491; Increment high watermark after full log truncation (#8037)
- PR-7568 - KAFKA-9074: Correct Connect’s Values.parseString to properly parse a time and timestamp literal (#7568)
- PR-8030 - KAFKA-9492; Ignore record errors in ProduceResponse for older versions (#8030)
- PR-7932 - KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning (#7932)
- PR-248 - CONFLUENT: Bump up version to 5.4.1
- PR-7870 - KAFKA-9254; Overridden topic configs are reset after dynamic default change (#7870)
- PR-7648 - KAFKA-9143: Log task reconfiguration error only when it happened (#7648)
- PR-7819 - KAFKA-9024: Better error message when field specified does not exist (#7819)
- PR-7593 - KAFKA-9083: Various fixes/improvements for Connect’s Values class (#7593)
- PR-7967 - KAFKA-9449; Adds support for closing the producer’s BufferPool. (#7967)
- PR-7963 - KAFKA-9235; Ensure transaction coordinator is stopped after replica deletion (#7963)
- PR-7687 - KAFKA-9144; Track timestamp from txn markers to prevent early producer expiration (#7687)
- PR-7908 - KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore (#7908)
- PR-7554 - KAFKA-9065; Fix endless loop when loading group/transaction metadata (#7554)
- PR-7904 - KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
- PR-7836 - HOTFIX: fix system test race condition (#7836)
- PR-7820 - KAFKA-9293; Fix NPE in DumpLogSegments offsets parser and display tombstone keys (#7820)
- PR-7682 - KAFKA-8933; Fix NPE in DefaultMetadataUpdater after authentication failure (#7682)
- PR-7753 - KAFKA-9232; Coordinator new member timeout does not work for JoinGroup v3 and below (#7753)
- PR-7840 - KAFKA-9307; Make transaction metadata loading resilient to previous errors (#7840)
- PR-7837 - fix rows (#7837)
- PR-7685 - KAFKA-9011: Removed multiple calls to supplier.get() in order to avoid multiple transformer instances being created. (#7685)
- PR-7520 - KAFKA-9011: Scala bindings for flatTransform and flatTransformValues in KStream (#7520)
- PR-7845 - KAFKA-9310: Handle UnknownProducerId from RecordCollector.send (#7845)
- PR-7839 - KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
- PR-7117 - KAFKA-8705: Remove parent node after leaving loop to prevent NPE (#7117)
- PR-7808 - port paragrpah from CP docs (#7808)
- PR-7689 - fix typo in processor-api developer guide docs (#7689)
KSQL¶
- PR-4662 - fix: backport fixes from query close
- PR-4626 - docs: document healthcheck for ksqlDB headless containers (DOCS-3066)
- PR-4630 - docs: add VALUE_DELIMITER property to serialization topic (DOCS-3320)
- PR-4613 - docs: fix broken link in ksqlDB architecture topic (DOCS-2864)
- PR-4527 - test: data providers should be ordered (MINOR)
- PR-4544 - docs: pin Pygments version to 2.4.2 (DOCS-3389)
- PR-4520 - docs: add new config settings (DOCS-3380)
- PR-4519 - docs: link to MD blog post on ksqlDB tutorials page (DOCS-3379)
- PR-4470 - docs: clarify that DROP STREAM/TABLE deletes source topic (DOCS-2386)
- PR-4446 - docs: fix typo on ksqlDB installation index page (DOCS-3332)
- PR-4438 - docs: update tutorial stack file for v0.6.0/5.4.0 (DOCS-3330)
- PR-4438 - docs: update tutorial stack file for v0.6.0/5.4.0 (DOCS-3330)
- PR-4431 - docs: add note about KSQL headless mode and Avro schemas (DOCS-1487)
- PR-4361 - docs: add note about ksqlDB headless mode and Avro schemas (DOCS-1487)
- PR-4272 - fix: add ksql-test-runner deps to ksql package lib
- PR-4314 - 5.4 pending shutdown fix
- PR-4271 - fix: pull queries should work across nodes (#4169)
- PR-4377 - docs: add missing KSQL/CCloud Docker parameter (DOCS-2550)
- PR-4350 - docs: add workaround to docs for ksql-test-runner jars (DOCS-3266)
- PR-4343 - docs: add CP release version token to markdown docs (DOCS-3259)
- PR-4342 - docs: update release branch token in ksqlDB markdown docs (DOCS-3258)
- PR-4341 - docs: remove inline struct content from 5.4.x branch (DOCS-3256)
- PR-4340 - docs: remove inline struct content from 5.4.0-post branch (DOCS-3256)
- PR-4338 - docs: add readthedocs.yml to 5.4.0-post branch (DOCS-3255)
- PR-4337 - docs: add docs-md directory to the new 5.4.0-post branch (DOCS-3254)
- PR-4297 - fix: deadlock when closing transient push query
- PR-4299 - build: Do not run twist lock scan and other docker image operations since we no longer build images in this repo
MQTT Proxy¶
- Update upstream to ce-kafka
Schema Registry¶
- PR-1357 - DG-263: Fix Avro defaults of type record
- PR-1268 - Add null checks
- PR-1324 - Update Metadata::bootstrap signature
- PR-1300 - CC-7692 / ISSUE-1197: Fix configuration of basic auth credentials provider
- PR-1296 - Adds always-true hostname verifier if ssl.endpoint.identification.algorithm is set to null or empty string
Version 5.4.0¶
Commercial Features¶
Confluent Server¶
- KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
- CPKAFKA-4272: revert KafkaServer shutdown order to match AK
- CPKAFKA-4178: replication system test should wait for hotset retention on local segments
- CPKAFKA-4247: resign as controller on KafkaController.shutdown
- KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
- SEC-701: Change confluent.http.server.listeners default host to 0.0.0.0
- CPKAFKA-4236: Check for empty observer list in addition to None (#1082)
- SEC-699: Audit Log timestamps always include milliseconds
- KSTORAGE-339: Tiered storage system tests flaky due to tiering_completed check (#1050)
- MINOR: Hardcoded fix for us-east-1 buckets returning value of “US” (#…
- SEC-698 Change audit log topic prefix to not be internal
- CPKAFKA-4189: Add trogdor test classes for Acl performance tests
- KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795)
- KSTORAGE-349: Synchronize log deletion and tier partition state flush
- MINOR: Bump system test version from 2.2.1 to 2.2.2 (#7765)
- MINOR: Increase the timeout in one of Connect’s distributed system tests (#7789)
- CPKAFKA-4202: Use older AdminClient API in LicenseStore for connector compatibility
- MINOR: Convert last streams join test to TTD (#7777)
- KAFKA-9184: Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771)
- KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
- MINOR: Convert Stream-StreamJoin Integration Test to TTD (#7752)
- KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
- KAFKA-9190: Close connections with expired authentication sessions (#7723)
- KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
- KAFKA-9258: Check Connect Metrics non-null in task stop (#7768)
- KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
- KAFKA-9231: Streams Threads may die from recoverable errors with EOS enabled (#7748)
- CPKAFKA-4114: Improve ReplicaStatusCommand output.
- KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
- CPKAFKA-4167: Pass custom configs to the ChannelBuilder implementations
- KSTORAGE-319 Expose tiered storage configs in docs
- KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
- KAFKA-9244: Update FK reference should unsubscribe old FK (#7758)
- CPKAFKA-4165: Propagate schema registry URL to metadata server
- SEC-595: Revert the changes related to adding new prefix for RestClientConfig
- KAFKA-9123: Test a large number of replicas (#7621)
- KAFKA-8981: Add rate limiting to NetworkDegradeSpec (#7446)
- CPKAFKA-4107: Make client configs for license and RBAC consistent
- CPKAFKA-4127: Make MDS host check in brokers reliable
- SEC-595: Pass RestClientConfig to the token callback handler
- CPKAFKA-4024: Make SslEngineBuilder usable without netty
- KSTORAGE-329: Remove log metrics on deletion
- KAFKA-8981: Add rate limiting to NetworkDegradeSpec (#7446)
- MINOR: Remove unused import in ReplicaStatus
- CPKAFKA-3945: Remove deprecated ReplicaStatus API
- MINOR: Updated StreamTableJoinIntegrationTest to use TTD (#7722)
- CPKAFKA-4119: Ensure state is properly reset on writer rebalance
- MINOR: Fix –enable-autocommit flag in verifiable consumer (#7743)
- CPKAFKA-4111: Detaches the MetadataServer life-cycle from RBAC provider
- KAFKA-8509: Add downgrade system test (#7724)
- KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
- CPKAFKA-3865: Clean up AdminClient changes in CE Kafka
- KSTORAGE-311: Ensure offset cache is maintained accurately when batches are skipped
- KSTORAGE-323: invalid handling of SegmentDeleteInitiate for fenced segments
- MINOR: Fix Streams EOS system tests by adding clean-up of state dir (#7693)
- KSTORAGE-323: metadata read should only return valid segments (#1008)
- KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
- CPKAFKA-4080: Don’t flag ReplicaStatus as an internal API
- CPKAFKA-4090: License manager should never configure metrics reporter
- HOTFIX: safely clear all active state in onPartitionsLost (#7691)
- MINOR: Ignore avro-generated support-metrics-fullcollector files
- CPKAFKA-4072: Make sure audit log exporter has metadata (simpler version)
- HOTFIX: safely clear all active state in onPartitionsLost (#7691)
- CPKAFKA-4086: Fix configuration in reassign_partitions_test.py
- DG-120: Move the step of retrieving SR URL to copyKafkaConfigToLog to address 5.4 blocker (#967)
- CONFLUENT: Assert bucket access via more constrained S3 policy action
- HOTFIX: Fix unit tests that failed when executed from IDE (#7707)
- KAFKA-9198: Complete purgatory operations on receiving StopReplica (#7701)
- MINOR: Fetch only from leader should be respected in purgatory (#7650)
- KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
- KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
- KAFKA-9196: Update high watermark metadata after segment roll (#7695)
- CPKAFKA-3866: Simplify tests to remove order-based flakiness
- KSTORAGE-309: invalid handling of full fetch sessions for tiered fetches (#941)
- KAFKA-9072: Add Topology naming to the dev guide (#7629)
- CONFLUENT: CC-6522: Use non-null default Kafka cluster ID if herder cannot deduce one
- KAFKA-9175: Update MirrorMaker 2 topic/partition metrics (#7688)
- CPKAFKA-4039: Include authorizationInfo subtypes in Audit Log Entry
- CPKAFKA-3996: Fix LogDirFailureTest to work when license topic is present
- MINOR: refactor replica last sent HW updates due to performance regression (#7671)
- KSTORAGE-252: change to info log level (from debug) when endoffset materialized and…
- KSTORAGE-252: update partition state endOffset on deleteInitiate (#820)
- KAFKA-9102: Increase default zk session timeout and replica max lag [KIP-537] (#7596)
- CPKAFKA-4034: Retry on UnsupportedVersionException for license topic create
- KAFKA-9171: Handle ReplicaNotAvailableException during DelayedFetch (#7678)
- CPKAFKA-3999: Disable hostname validation for license client in error system test
- CPKAFKA-3894: Fix flaky test KafkaAuthWriterTest.testWriterElection
- CONFLUENT: Check if license topic exists before attempting to create
- MINOR: Change topic-exists log for CreateTopics from INFO to DEBUG (#7666)
- KSTORAGE-232: use ListOffsetRequest rather than TierListOffsetRequest when tiering enabled (#934)
- CONFLUENT: Remove usage of ThrowingRunnable
- MINOR: Update Epoch field descrition in FetchRequest api doc (#7673)
- CPKAFKA-3848: Add reassign validation for observer and topic placement
- KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
- SEC-658 Disable metrics reporter for audit log producer
- DG-65: Integrate KIP 467 changes in AK
- CONFLUENT: Add static method to NewPartitionReassignment for Observers
- KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
- CPKAFKA-3949: Reset metadata writer executor after writer rebalance
- CPKAFKA-3985: Rename RBAC and ACL providers to CONFLUENT and ZK_ACL
- KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
- CPKAFKA-3955: Validate placement constraint when altering configs
- MINOR: Return null in key mapping of committed (#7659)
- CNKAF-103: KIP-496 does not work in multi-tenant clusters
- KSTORAGE-286: Better handling of Archiver Exceptions from race conditi…
- CONFLUENT: Add unit tests to check observers tagged field
- CPKAFKA-3932: Use IBP to gate observer usage and remove feature flag
- CPKAFKA-3951: Change audit log config name to “enabled” for consistency
- KSTORAGE-307: memory leak in Tiered Storage CancellationContext(s)
- CPKAFKA-3942: Support multi-login-module static JAAS for license clients in system tests
- CPKAFKA-3859: Improve ReplicaStatus information
- DG-66: Throw an error when broker-side schema validation is configured improperly
- KAFKA-9140: Also reset join future when generation was reset in order to re-join (#7647)
- CONFLUENT: Add broker ACLs for license topic in system tests
- MINOR: Rework NewPartitionReassignment public API (#7638)
- HOTFIX: remove reference to unused Assignment error code (#7645)
- MINOR: Rework NewPartitionReassignment public API (#7638)
- CPKAFKA-3853: Use Kafka protocol for centralized ACL update from brokers
- HOTFIX: fix bug in VP test where it greps for the wrong log message (#7642)
- CONFLUENT: Quote sasl.jaas.config option names used for license client
- CPKAFKA-3904 Route Audit Log messages without building CloudEvent
- CPKAFKA-3901 Audit Log Router cache should store negative result
- HOTFIX: Remove from restoringByPartition once restored (#7631)
- HOTFIX: Remove from restoringByPartition once restored (#7631)
- CPKAFKA-3783: Set observers in CreatePartitons call (#879)
- KAFKA-8972 (2.4 blocker): TaskManager state should always be updated after rebalance (#7620)
- KSTORAGE-288: fix dynamic updating of proprietary log configs (#885)
- KSTORAGE-290: retry archiving for general S3 AmazonClientException (#862)
- CPKAFKA-3877: Add backoff and timeout for license retry, set replication factor for tests
- KAFKA-8972 (2.4 blocker): bug fix for restoring task (#7617)
- CPKAFKA-3831: Fix ReplicaStatus IsInSync reporting incorrect value
- CPKAFKA-3855: Don’t allow produce logging to the audit log topic
- KSTORAGE-230: TierFetcher offset/batch size cache (#719)
- KAFKA-8972 (2.4 blocker): clear all state for zombie task on TaskMigratedException (#7608)
- MMA-5596]: Add ‘ObserverReplicasCount’ to monitored metrics list
- CPKAFKA-3718: Add ObserverReplicasCount metric
- CPKAFKA-3849: Update leader for static observers
- KSTORAGE-82: Tiered storage topic deletion support
- KAFKA-8972 (2.4 blocker): correctly release lost partitions during consumer.unsubscribe() (#7441)
- MINOR: improve logging of tasks on shutdown (#7597)
- CPKAFKA-3718: Add ObserverReplicasCount metric
- CPKAFKA-3843: Observers should be passed through in ListPartitionReassignment
- CONFLUENT: Update observers_test for static observers
- CPKAFKA-3719: Observer support in the controller
- CPKAFKA-3721: Update topic describe to use metadata for observers
- CPKAFKA-3816: Use CaughtUpReplicasCount for consistency
- CPKAFKA-3552: Move license validator out of authorizer and make broker wide
- CPKAFKA-3852: Add/remove listener metrics during dynamic listener update
- KAFKA-9105: Add back truncateHead method to ProducerStateManager (#7599)
- MINOR: disable use of ConfluentLeaderAndIsrRequest in 5.4.x
- KAFKA-8992: Redefine RemoveMembersFromGroup interface on AdminClient (#7478)
- KAFKA-9038: Allow creating partitions while a reassignment is in progress (#7582)
- KAFKA-8972: Need to flush state even on unclean close (#7589)
- KAFKA-9089: Reassignment should be resilient to unexpected errors (#7562)
- KAFKA-8729: Add upgrade docs for KIP-467 on augmented produce response (#7522)
- MINOR: Re-implement NewPartitionReassignment#of() (#7592)
- MINOR: Add documentation for foreign-key joins (KIP-213) (#7535)
- METRICS-734: Categorize Kafka API calls for Audit Logs
- CONFLUENT: Disable audit logger for CCloud
- METRICS-736: Log Error for audit log of consumption on audit log topic
- MINOR: Use LeaderAndIsr instead of ConfluentLeaderAndIsr when tiered is enabled
- CPKAFKA-3830: Include observer information in MetadataResponse
- CPKAFKA-3730: Fix createTopicPolicy incompatibility in ce-kafka
- CPKAFKA-3723: Message changes to add observer tagged field
Confluent Cloud Plugins¶
- CC-6911: Fix healthcheck integration tests by replacing deprecated configs
- MINOR: fix shaded use of guava
- DG-78: Clear quota manager during IMPORT
- MINOR: Add maxId to TenantIncrementalIdGenerator
- CC-4841: Use CE Kafka across the repo
- CC-4345: Implement multitenant store
Confluent Rebalancer¶
- CPKAFKA-3550: Add topic placement constraint support (#21)
- MINOR: Force broker config updates when using rebalancer’s KafkaAdmin…
- MINOR: Use latest NewPartitionReassignment API
- MINOR: Update usage of NewPartitionReassignment class
- CPKAFKA-3814 (3/3): Implements rebalancer’s KafkaAdminClient
- CPKAFKA-3814 (1/3): Adds KafkaAdmin interface to abstract admin imple…
- CPKAFKA-3292: Remove usages of ZkUtils and org.I0Itec.ZkClient in favor of KafkaZkClient, AdminZkClient
Confluent Security Plugins¶
- MMA-6119: Fixed error that C3 cannot view topic messages through UX
- SEC-672: Removed authentication for endpoint /v1/metadata/id
- SEC-693: Added ServerMetadataResource to initializeSchemaRegistryResourceActio…
- MINOR: Modify schema validation integration tests to use correct exception
- CC-7272: Corrected ConnectSecurityExtensionConfig to also support old prefixes
- MINOR: Pass custom metadata configs to underlyring Rest clients
- SEC-595: Pass all configs to underlying RestClient
- CPKAFKA-4100: Update the cached KafkaRestContext.getConfig() with latest token details
- CC-6522: Use Kafka cluster of ‘UNKNOWN’ if one cannot be found by the herder
- DG-118: Catch exception when constructing error message
- SEC-649: Added BC non-FIPS into package of package-schema-registry-security-pl…
- MINOR: Add connect-runtime as test dependency required by LicenseManager
- MINOR: Change Kafka test dependencies from CCS to CE
- MINOR: Fix SslPrincipalAliasTest by setting replication factor to 1 for the license topic
- MINOR: fix shaded use of guava
- CC-6671: Serve JSON-parseable error messages instead of raw text
- CC-6154: Add RBAC credentials injection to Connect security extension
- SEC-614: add debug logging to jwtloginservice
- DG-72: Stop DDoS from a batch of bad records to schema validation plugin
- DG-46: Add /v1/metadata/id endpoint to display SR cluster information
- Sec-237: Remove www-authenticate from MDS authenticate responses
- SEC-366: Excluded Bouncy Castle FIPS jars in every module under repo confluent-security-plugins
- MINOR: Override getAllSubjectsById in AbstractSchemaRegistryClient
- MINOR: Test to check if describeConfigs doesn’t include broker-related configs for a topic
- MINOR: distinguish between REST client and server errors
- SEC-323: Instantiate JwtLoginService with preloaded PublicKey
- CPKAFKA-3170, pt. 2: Add integration tests for schema validation on broker side
- CPKAFKA-3166: Check the configs for validate key and / or value
- MINOR: use configs from the broker
- CC-5404: Change 401 responses to 403 in Schema Registry security plugin
- CPKAFKA-3169: Add a broker interceptor for schema validation
- CPKAFKA-3239: Set token credentials in MdsBearerAuthProvider Initialization
- MINOR: Remove kafka-security-plugins since this is now included in confluent-server
- CC-5038: Add additional args to account for upstream changes
Control Center¶
- MMA-6158: use jetty HttpClient instead of jersey Client
- MMA-6196: Call AdminClient.close in a separate thread
- MMA-6165: Remove multi-cluster SR associated default for 5.4.x release
- MMA-5947: support for SR url kafka rest api
- CPKAFKA-4127: Changes to match broker config update for metadata apps
- MMA-5925: only add schema registry config sets with valid url field
- MMA-5938: Revert “change the min.isr from 2 to 1 for internal topics (#1948)”
- MMA-4739: add RBAC props to each properties file
- MMA-5602: alerting endpoints for topic/group access lists
- CPKAFKA-3580: Add get replica status REST endpoint
- METRICS-315: change min.isr from 2 to 1 for internal topics
- MMA-5424: MDS visibility endpoint was moved
- MMA-5422: Add ui autoupdate default c3 config property
- MMA-5408: Add a flag for replicator monitoring feature
- MMA-5300: Make the cluster list page usable for large # of clusters
- MMA-4991: Show offsets and size per partition
- MMA-4990: Fetch under replicated metric time-series data as well
- MMA-4014:(Broker metrics): Expose new endpoint for broker partition and replicas
- MMA-4094: Topic replica and partition time series data
- MMA-4016:(feat) Broker metrics page - System
- MMA-4015:(feat) Broker metrics page - Disk usage
- MINOR: Use confluent-security-plugins as the parent pom to use CE Kafka
Confluent Platform Documentation¶
- DOCS-3265 - Fixed bad anchors in Upgrade topic
- DOCS-3257 - Fix typos on Control Center Manage Schemas for Topics
- Remove RBAAC “coming soon” from Cloud limits
- DOCS-3229 - Update docs and changelog for GCP, Azure - functions connectors
- DOCS-3252 - Operator 5.4 updates (upgrades, migrating Helm 2.0 to 3.0, migrating 5.3 to 5.4, Quick Start)
- DOCS-3211 - Remove warnings re: License tab availability for expired trial license 5.4.1 release
- DOCS-2290 - Update Confluent Control Center alerts nav
- Fix indentation for Operator
- Update RBAC quick start
- Add correct upstream Kafka version
- Updated confluent CLI requirement
- DOCS-3173 - More MRR failover and replica placement details
- DOCS-2990 - 5.4.0 release notes
- 5.4.0 changelog
- DOCS-3215 - Adding details around the 5.4 Kubernetes Operator and tiered storage support
- DOCS-3198 - Add RBAC perms for tables created with persistent query
- DOCS-1120 - Rev’d title and cleaned up some grammar, spelling error
- DOCS-1845 - Fixed typo in include file
- DOCS-3188 - Added example to config audit log to another cluster
- DOCS-3114 - Update ZooKeeper upgrade docs
- DOCS-3181 - Attribute name correction
- Update screenshot and description of metrics
- Adding changelog for Replicator for 5.4
- Fixed dates on site footer
- Centralized ACL limitations
- Fix include path for 5.4.x
- Add CLI Windows support info to docs
- Import Kafka 2.4 configs
- Update changelogs for Confluent Platform connectors
- Add 5.4 changelog for KSQL
- Updated interoperability information for 5.4
- Add info to clients doc about producer
- Update quick starts for 5.4.x
- mTLS plus RBAC, update confluent authorizer values
- Add Control Center changelog for 5.4.0
- Schema Registry Control Center screen snaps, workflow update for 5.4.x
- 5.4.x Upgrade Doc
- Control Center Connector RBAC Security tab Creds caveats
- Add 5.4.0 changelog for Schema Registry
- In https MDS config how certs are passed in CLI login
- Note not to use keytool to gen PEM files
- Add summary for Streams API changes
- REST Proxy SASL should use SASL_SSL in bootstrap.servers
- Revised ACL section in Security tutorial for centralized ACLs
- Tiered Storage Preview Docs
- Update license screenshots for version # and new OTA updates tab
- Topic Viewer Audit Log search needs to use
User:<username>instead of<username> - Remove preview from RBAC entry in license graphic
- Added MDS-related error code
- Control Center RBAC Manage Roles
- RBAC limitations
- New troubleshooting token auth section in MDS config
- Add link from Control Center configs to Replicator monitoring topic in Control Center
- Fix docs formatting Streams
- Port over of Kafka docs naming DSL topologies for Streams
- Add upgrade from docs for Kafka 2.4
- Audit logs
- Config Kafka to connect to MDS
- MDS - Adapt Swagger spec for Sphinx doc tool
- Clarify source and replicated topics, origin/destination, respectively
- Replicator fix sed commands and port numbers in tutorial
Kafka Connect¶
- REP-609: Fixing potential null pointer exception
- REP-600: Update admin client to support default number of partitions and replication factors when creating topics
- REP-24: ZK license manager change
- REP-381: Replicator should not automatically create source topics
- REP-475: do not add connect worker configurations if dest.kafka.bootstrap.servers is peresent
- REP-462: Resolve Replicator Master build failure
- REP-450: filters default configs out when applying topic configs to dest
- REP-30: Refactor the “admin-client-with-zookeeper” test to NOT have a Kafka binary dependency
- REP-379: added configuration provider that can infer dest.kafka.* config from the Connect Worker
- REP-436: refactor non timestamp specific class to a new replicator-common module
- REP-30: Better identify errors during startup()/shutdown() from KafkaServerStartable within test
- REP-322: REP-323 REST extensions for Replicator monitoring
- REP-231: Refactor ConsumerTimestampInterceptor out into a separate JAR
- REP-321: Initial commit for JMX metrics implementation
- CC-5364: Remove all package excludes except connect-api
- CC-5364: Exclude core kafka from package
- CC-4930: Adjust expected calls in case schedulePeriodicRefreshMetadata gets scheduled by NewReplicatorAdminClient
- MINOR: Upgrade bugfix versions of dependencies to avoid CVEs
- MINOR: Pin hadoop-common version to 2.7.7 and exclude htrace-core dependency
- CC-6565: Use parquet.codec and add compression options for Parquet format
- CC-6766 Handle null-valued records in all three output types
- CC-4969 MINOR: Remove Unused S3Storage object from TopicPartitionWriter
- CC-6801: Retry effectively on retriable failures
- CC-6872: Modify GCS credentials logic to follow same pattern as BigQuery
- CC-6766 Handle null-valued records in all three output types
- CC-5186: Uncaught errors breaking connector config
- CC-4686: Add credentials provider method to GCS config
- CC-6565: Add parquet.codec property to enable compression in storage sink connectors
Metadata Service¶
- SEC-701: Change confluent.http.server.listeners default host to 0.0.0.0
- SEC-696: MDS - Adapt Swagger spec for Sphinx doc tool
- CPKAFKA-4191: Make ClusterId json consistent with other components
- MINOR: Use config prefix “confluent.metadata.” in MdsKafkaTokenIntegrationTest
- CPKAFKA-4165: Add endpoint for schema registry URLs on default HTTP server
- MINOR: Remove –confluent.metadata.rest.” prefix usage from MdsKafkaTokenIntegrationTest
- SEC-655: MDS - Actually fill out / write the swagger spec
- SEC-595: Test Token callback handlers with MDS over tls
- CPKAFKA-4127: Changes to match broker config update for metadata apps
- SEC-655 - MDS: Actually fill out / write the swagger spec
- CPKAFKA-4130: Change MDS listeners default value to match ce-kafka
- SEC-678: mds-in-a-box is not supposed to be part of CP 5.4 tarball
- CPKAFKA-4111: Detaches the MetadataServer life-cycle from RBAC provider
- SEC-651: Add waitForCondition check in AclMigrationClientTest
- SEC-667: MDS - Remove unused endpoint - list groups for user
- SEC-655: MDS - Actually fill out / write the swagger spec
- MINOR: Define the JVM memory setting to be used by Maven
- CPKAFKA-3985: Rename RBAC and ACL providers to CONFLUENT and ZK_ACL
- MINOR: Put back test dependency on Audit logs
- SEC-601: MDS : Remove 5.3 preview moved endpoints
- SEC-655: MDS - Actually fill out / write the swagger spec
- SEC-652: Optimize visibility lookup endpoint implementation in metada…
- SEC-653: MDS : Suppress output of empty path elements in Scopes
- HOTFIX: Remove duplicate methods from Stub classes
- CPKAFKA-3853: Update ACL tests as per changes in CPKAFKA-3853
- SEC-635: MDS : MDS in a Box
- SEC-636: MDS - Remove Zookeeper dependency from build / artifacts
- SEC-439: MDS Swagger : Make swagger available w/out extra jars
- SEC 632: Excluded BC non-FIPS jars from packages
- SEC-623: MDS Depend on ce-audit log code and test bootstrap issue
- SEC-600: Improve error messaging for empty Scope body
- SEC-575: - Rbac Api : Rolebinding Ability per Type
- SEC-620: Added resource pattern validations
- SEC-574: Rbac List Users and Groups
- SEC-228: Add Scope ResourceType check
- MINOR: Update tests to verfify SSL configuration in RestAuthorizer/MdsAclClient
- SEC-229: Add validation for Role to ResourceTypes
- SEC-227: Add basic validators for roleName, operation and resource type
- SEC-615: Fix any timing issue involved in AclMigrationClientTest
- SEC-227: Add basic validators for roleName, operation and resource type
- CPKAFKA-3456: Add integration test for Acl Migration
- SEC-355: MDS : Cleanup auth logic
- SEC-437: Rbac Management: REST API to return all principal-managed role-bindings resources
- SEC-576: MDS Define valid resourceTypes per clusterType
- SEC-597: MDS : Move endpoints for 5.4
- SEC-438: Rbac Management: REST API to return guidelines to create role bindings
- CPKAFKA-3758: Move Acl API Request/Response classes to ce-kafka and add Integration tests
- MMA-5319: changed logout path for clearing cookies
- MINOR: Update tests to await Kafka/MDS bootup correctly
- CPKAFKA-3677: Fix the tests as per changes in CPKAFKA-3677
- MINOR: Comment out flaky test
- SEC-435: Rbac Management: REST API to return all principal-managed clusters
- MINOR: Update to new AclRule and Authorizer logic
- MMA-3278: add support for cookie authentication to mds
- SEC-436: migrate mds jenkins bot to mds-alerts
- CPKAFKA-3457: Mark optional fields as not required
- MINOR: Increase test timeout on more tests
- SEC-333: MDS : Resource Create Specific Endpoint for the UI
- SEC-367: Rbac Management : My Rolebindings Api
- MINOR: Increase test timeout
- MINOR: Fix timing flaky test by adding an Await
- MINOR: Update SecurityMetadataAuthorizerTest to work with upstream
- CPKAFKA-3457: Add APIs for Acl management
- SEC-366: Configure compile and build environment for FIPS
- SEC-356: MDS : Supress setting of the Token Public key for MDS
- SEC-296: Ignore WS context, CPKAFKA-3316 optimize configuration for token auth (default)
- SEC-233: MDS Test Infra : Find / build a better testing LDAP
- SEC-300: MetadataService pom.xml unnecessarily overriding versions from common pom
- Minor: Update Stub Server given change to token.auth.enabled
- SEC-268: Increase default token lifetime to 6 hours
- CONFLUENT: Rename SuperAdmin role to SystemAdmin
- CONFLUENT: Update SuperUser to SuperAdmin, remove references to superuser
Secrets¶
- CC-7331: Update name of secrets topic
- CC-7305: close admin client in tests to reduce flakiness
- CC-5587: Remove use of ZkUtils from secret registry
Community Features¶
Apache Kafka¶
- PR-7904 - KAFKA-9335: Fix StreamPartitionAssignor regression in repartition topics counts (#7904)
- PR-7839 - KAFKA-9306: The consumer must close KafkaConsumerMetrics (#7839)
- PR-7795 - KAFKA-9212: Ensure LeaderAndIsr state updated in controller context during reassignment (#7795)
- PR-7765 - MINOR: Bump system test version from 2.2.1 to 2.2.2 (#7765)
- PR-7789 - MINOR: Increase the timeout in one of Connect’s distributed system tests (#7789)
- PR-7777 - MINOR: Convert last streams join test to TTD (#7777)
- PR-7771 - KAFKA-9184: Redundant task creation and periodic rebalances after zombie Connect worker rejoins the group (#7771)
- PR-7763 - KAFKA-9251: Describing a non consumer group with the Admin API hangs forever (#7763)
- PR-7752 - MINOR: Convert Stream-StreamJoin Integration Test to TTD (#7752)
- PR-7723 - KAFKA-9190: Close connections with expired authentication sessions (#7723)
- PR-7773 - KAFKA-9265: Fix kafka.log.Log instance leak on log deletion (#7773)
- PR-7768 - KAFKA-9258 Check Connect Metrics non-null in task stop (#7768)
- PR-7772 - KAFKA-9261: Client should handle inconsistent leader metadata (#7772)
- PR-7748 - KAFKA-9231: Streams Threads may die from recoverable errors with EOS enabled (#7748)
- PR-7769 - KAFKA-9203: Revert “MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)” (#7769)
- PR-7760 - KAFKA-9156: Fix LazyTimeIndex & LazyOffsetIndex concurrency (#7760)
- PR-7758 - KAFKA-9244: Update FK reference should unsubscribe old FK (#7758)
- PR-7621 - KAFKA-9123 Test a large number of replicas (#7621)
- PR-7446 - KAFKA-8981 Add rate limiting to NetworkDegradeSpec (#7446)
- PR-7722 - MINOR: Updated StreamTableJoinIntegrationTest to use TTD (#7722)
- PR-7532 - KAFKA-9051: Prematurely complete source offset read requests for stopped tasks (#7532)
- PR-7729 - MINOR: Rat should ignore generated directories (#7729)
- PR-7693 - MINOR: Fix Streams EOS system tests by adding clean-up of state dir (#7693)
- PR-6031 - KAFKA-1714: Fix gradle wrapper bootstrapping (#6031)
- PR-7691 - HOTFIX: safely clear all active state in onPartitionsLost (#7691)
- PR-7707 - HOTFIX: Fix unit tests that failed when executed from IDE (#7707)
- PR-7701 - KAFKA-9198: Complete purgatory operations on receiving StopReplica (#7701)
- PR-7650 - MINOR: Fetch only from leader should be respected in purgatory (#7650)
- PR-7704 - KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)
- PR-7695 - KAFKA-9196: Update high watermark metadata after segment roll (#7695)
- PR-7629 - KAFKA-9072: Add Topology naming to the dev guide (#7629)
- PR-7688 - KAFKA-9175: Update MirrorMaker 2 topic/partition metrics (#7688)
- PR-7671 - MINOR: refactor replica last sent HW updates due to performance regression (#7671)
- PR-7680 - HOTFIX: Fix compilation error in DelayedFetchTest (#7680)
- PR-7678 - KAFKA-9171: Handle ReplicaNotAvailableException during DelayedFetch (#7678)
- PR-7666 - MINOR: Change topic-exists log for CreateTopics from INFO to DEBUG (#7666)
- PR-7673 - MINOR: Update Epoch field descrition in FetchRequest api doc (#7673)
- PR-7662 - KAFKA-9133: Cleaner should handle log start offset larger than active segment base offset (#7662)
- PR-7613 - KAFKA-8677: Simplify the best-effort network client poll to never throw exception (#7613)
- PR-7659 - MINOR: Return null in key mapping of committed (#7659)
- PR-7647 - KAFKA-9140: Also reset join future when generation was reset in order to re-join (#7647)
- PR-7645 - HOTFIX: remove reference to unused Assignment error code (#7645)
- PR-7638 - MINOR: Rework NewPartitionReassignment public API (#7638)
- PR-7642 - HOTFIX: fix bug in VP test where it greps for the wrong log message (#7642)
- PR-7631 - HOTFIX: Remove from restoringByPartition once restored (#7631)
- PR-7620 - KAFKA-8972 (2.4 blocker): TaskManager state should always be updated after rebalance (#7620)
- PR-7617 - KAFKA-8972 (2.4 blocker): bug fix for restoring task (#7617)
- PR-7608 - KAFKA-8972 (2.4 blocker): clear all state for zombie task on TaskMigratedException (#7608)
- PR-7441 - KAFKA-8972 (2.4 blocker): correctly release lost partitions during consumer.unsubscribe() (#7441)
- PR-7597 - MINOR: improve logging of tasks on shutdown (#7597)
- PR-7599 - KAFKA-9105: Add back truncateHead method to ProducerStateManager (#7599)
- PR-7478 - KAFKA-8992: Redefine RemoveMembersFromGroup interface on AdminClient (#7478)
- PR-7582 - KAFKA-9038: Allow creating partitions while a reassignment is in progress (#7582)
- PR-7589 - KAFKA-8972: Need to flush state even on unclean close (#7589)
- PR-7562 - KAFKA-9089: Reassignment should be resilient to unexpected errors (#7562)
- PR-7522 - KAFKA-8729: Add upgrade docs for KIP-467 on augmented produce response (#7522)
- PR-7592 - MINOR: Re-implement NewPartitionReassignment#of() (#7592)
- PR-7535 - MINOR: Add documentation for foreign-key joins (KIP-213) (#7535)
- PR-7579 - MINOR: Add toString to PartitionReassignment (#7579)
- PR-7570 - MINOR: Rename brokers to replicas in the reassignment API (#7570)
- PR-7574 - MINOR: Check against empty replicas in AlterPartitionReassignments (#7574)
- PR-7550 - MINOR:Upgrade guide updates for KIP-479 (#7550)
- PR-7547 - MINOR: KIP-307 upgrade guide docs (#7547)
- PR-7557 - MINOR: don’t require key serde in join materialized (#7557)
- PR-7564 - KAFKA-8943: Move SecurityProviderCreator to org.apache.kafka.common.security.auth package (#7564)
- PR-7542 - KAFKA-9041: Flaky Test LogCleanerIntegrationTest#testIsThreadFailed (#7542)
- PR-7527 - MINOR: Add metrics in operations doc for KIP-429 / KIP-467 (#7527)
- PR-7553 - MINOR: fix typo in TestInputTopic.getTimestampAndAdvance (#7553)
- PR-7421 - KAFKA-8962: Use least loaded node for AdminClient#describeTopics (#7421)
- PR-7541 - KAFKA-9058: Lift queriable and materialized restrictions on FK Join (#7541)
- PR-7500 - MINOR: Add ability to wait for all instances in an application to be RUNNING (#7500)
- PR-7451 - MINOR: Augment log4j to add generation number in performAssign (#7451)
- PR-7531 - KAFKA-9004: Prevent older clients from fetching from a follower (#7531)
- PR-7544 - MINOR: Upgrade zk to 3.5.6 (#7544)
- PR-7513 - MINOR: AbstractRequestResponse should be an interface (#7513)
- PR-7511 - Kafka-8950: Fix KafkaConsumer Fetcher breaking on concurrent disconnect (#7511)
- PR-7534 - MINOR: log reason for fatal error in locking state dir (#7534)
- PR-7309 - KAFKA-8884: class cast exception improvement (#7309)
- PR-7460 - KAFKA-8104: Consumer cannot rejoin to the group after rebalancing (#7460)
- PR-7517 - KAFKA-9000: fix flaky FK join test by using TTD (#7517)
- PR-7529 - KAFKA-8496: System test for KIP-429 upgrades and compatibility (#7529)
- PR-7537 - KAFKA-9053: AssignmentInfo#encode hardcodes the LATEST_SUPPORTED_VERSION (#7537)
- PR-7536 - MINOR: Improve FK Join docs and optimize null-fk case (#7536)
- PR-7315 - KAFKA-8340, KAFKA-8819: Use PluginClassLoader while statically initializing plugins (#7315)
- PR-6514 - KAFKA-7981: Add fetcher and log cleaner thread count metrics (#6514)
- PR-7515 - MINOR: Update Kafka Streams upgrade docs for KIP-444, KIP-470, KIP-471, KIP-474, KIP-528 (#7515)
- PR-7518 - KAFKA-9032: Bypass serdes for tombstones (#7518)
- PR-7483 - KAFKA-8897: Warn about no guaranteed backwards compatibility in RocksDBConfigSetter (#7483)
- PR-7488 - MINOR: Provide better messages when waiting for a condition in test (#7488)
- PR-7392 - KAFKA-8945/KAFKA-8947: Fix bugs in Connect REST extension API (#7392)
- PR-7491 - KAFKA-9014: Fix AssertionError when SourceTask.poll returns an empty list (#7491)
- PR-7514 - KAFKA-9033: Use consumer/producer identity in generated clientId (#7514)
- PR-7501 - KAFKA-9030: Document client-level (a.k.a. instance-level) metrics (#7501)
- PR-7462 - MINOR: code and JavaDoc cleanup (#7462)
- PR-7470 - KAFKA-8122: Fix Kafka Streams EOS integration test (#7470)
- PR-7521 - MINOR: remove unused import in QueryableStateIntegrationTest (#7521)
- PR-7519 - KAFKA-4422 / KAFKA-8700 / KAFKA-5566: Wait for state to transit to RUNNING upon start (#7519)
- PR-7305 - KAFKA-8813: Refresh log config if it’s updated before initialization (#7305)
- PR-7475 - KAFKA-8725: Improve LogCleanerManager#grabFilthiestLog error handling (#7475)
- PR-7497 - MINOR: Increase spotBugs max heap to 2 GB (#7497)
- PR-7437 - KAFKA-8671: NullPointerException occurs if topic associated with GlobalKTable changes (#7437)
- PR-7463 - MINOR: unify calls to get committed offsets and metadata (#7463)
- PR-7510 - MINOR: clarify wording around fault-tolerant state stores (#7510)
- PR-7495 - KAFKA-9020: Streams sub-topologies should be sorted by sink -> source relationship (#7495)
- PR-7503 - KAFKA-9029: Flaky Test CooperativeStickyAssignorTest.testReassignmentWithRand: bump to 4 (#7503)
- PR-7497 - MINOR: Increase spotBugs max heap to 2 GB (#7497)
- PR-7484 - MINOR: Improve assert in testCreateTopicsResponseMetadataAndConfig (#7484)
- PR-7494 - HOTFIX: fix checkstyle in Streams system test (#7494)
- PR-7494 - HOTFIX: fix checkstyle in Streams system test (#7494)
- PR-7472 - KAFKA-8743: Flaky Test Repartition{WithMerge}OptimizingIntegrationTest (#7472)
- PR-7472 - KAFKA-8743: Flaky Test Repartition{WithMerge}OptimizingIntegrationTest (#7472)
- PR-7486 - MINOR: ListPartitionReassignmentsResponse should not be entirely failed when a topic-partition does not exist (#7486)
- PR-7486 - MINOR: ListPartitionReassignmentsResponse should not be entirely failed when a topic-partition does not exist (#7486)
- PR-7167 - KAFKA-8729, pt 3: Add broker-side logic to handle the case when there are record_errors and error_message (#7167)
- PR-7167 - KAFKA-8729, pt 3: Add broker-side logic to handle the case when there are record_errors and error_message (#7167)
- PR-7479 - MINOR: fix compatibility-breaking bug in RequestHeader (#7479)
- PR-7479 - MINOR: fix compatibility-breaking bug in RequestHeader (#7479)
- PR-7458 - MINOR: Change topic zNode’s new reassigning fields names to snake case in order to be consistent with other fields (#7458)
- PR-7438 - MINOR: Fix stale comment in partition reassignment javadoc (#7438)
- PR-7458 - MINOR: Change topic zNode’s new reassigning fields names to snake case in order to be consistent with other fields (#7458)
- PR-7438 - MINOR: Fix stale comment in partition reassignment javadoc (#7438)
- PR-7451 - MINOR: Augment log4j to add generation number in performAssign (#7451)
- PR-7469 - MINOR: Just one put and flush to generation rocksDB File in RocksDBStoreTest (#7469)
- PR-7452 - KAFKA-8991: Enable scalac optimizer (#7452)
- PR-7406 - KAFKA-8954: Topic existence check is wrongly implemented in the DeleteOffset API (KIP-496) (#7406)
- PR-7406 - KAFKA-8954: Topic existence check is wrongly implemented in the DeleteOffset API (KIP-496) (#7406)
- PR-7464 - KAFKA-8932: Add tag for CreateTopicsResponse.TopicConfigErrorCode (KIP-525) (#7464)
- PR-7464 - KAFKA-8932: Add tag for CreateTopicsResponse.TopicConfigErrorCode (KIP-525) (#7464)
- PR-7393 - KAFKA-8944: Fixed KTable compiler warning. (#7393)
- PR-7429 - KAFKA-8964: Rename tag client-id for thread-level metrics and below (#7429)
- PR-7468 - MINOR: remove unused imports in Streams system tests (#7468)
- PR-7388 - KAFKA-7190: Retain producer state until transactionalIdExpiration time passes (#7388)
- PR-7388 - KAFKA-7190: Retain producer state until transactionalIdExpiration time passes (#7388)
- PR-7449 - KAFKA-8983: AdminClient deleteRecords should not fail all partitions unnecessarily (#7449)
- PR-7449 - KAFKA-8983: AdminClient deleteRecords should not fail all partitions unnecessarily (#7449)
- PR-7461 - MINOR: Modified Exception handling for KIP-470 (#7461)
- PR-7461 - MINOR: Modified Exception handling for KIP-470 (#7461)
- PR-7105 - KAFKA-7245: Deprecate WindowStore#put(key, value) (#7105)
- PR-7105 - KAFKA-7245: Deprecate WindowStore#put(key, value) (#7105)
- PR-7459 - HOTFIX: Hide built-in metrics version (#7459)
- PR-7453 - KAFKA-8985: Add flexible version support to inter-broker APIs (#7453)
- PR-7386 - KAFKA-8179: Part 7, cooperative rebalancing in Streams (#7386)
- PR-7386 - KAFKA-8179: Part 7, cooperative rebalancing in Streams (#7386)
- PR-7453 - KAFKA-8985: Add flexible version support to inter-broker APIs (#7453)
- PR-7455 - MINOR: Bump version to 2.5.0-SNAPSHOT (#7455)
- PR-7454 - MINOR: Update documentation.html to refer to 2.4 (#7454)
- PR-7378 - KAFKA-8233: TopologyTestDriver test input and output usability improvements (#7378)
- PR-7325 - KAFKA-8885: The Kafka Protocol should Support Optional Tagged Fields (#7325)
- PR-7383 - MINOR: Gracefully handle non-assigned case in fetcher metric (#7383)
- PR-7445 - MINOR: Rework error messages in ConsumerGroupCommand (#7445)
- PR-7416 - KAFKA-8934: Introduce instance-level metrics for streams applications (#7416)
- PR-7447 - MINOR: just remove leader on trunk like we did on 2.3 (#7447)
- PR-7442 - KAFKA-8974: Trim whitespaces in topic names in sink connector configs (#7442)
- PR-5527 - KAFKA-3705 Added a foreignKeyJoin implementation for KTable. (#5527)
- PR-7371 - KAFKA-6290: Support casting from logical types in cast transform (#7371)
- PR-6914 - KAFKA-8523 Enabling InsertField transform to be used with tombstone events (#6914)
- PR-7405 - MINOR: Make TopicDescription’s other constructor public (#7405)
- PR-7432 - MINOR: Start correlation id at 0 in SaslClientAuthenticator (#7432)
- PR-7285 - KAFKA-8558: Add StreamJoined config object to join (#7285)
- PR-6843 - KAFKA-8447: New Metric to Measure Number of Tasks on a Connector (#6843)
- PR-7310 - KAFKA-8804: Secure internal Connect REST endpoints (#7310)
- PR-7430 - KAFKA-5609: Connect log4j should also log to a file by default (KIP-521) (#7430)
- PR-7403 - KAFKA-7772: Dynamically Adjust Log Levels in Connect (#7403)
- PR-7436 - HOTFIX: don’t throw if upgrading from very old versions (#7436)
- PR-7423 - KAFKA-8649: Send latest commonly supported version in assignment (#7423)
- PR-7431 - MINOR: Do not falsely log that partitions are being reassigned on controller startup (#7431)
- PR-7407 - MINOR: Add provider name of SSLContext to debug log. (#7407)
- PR-7417 - MINOR: Shutdown RockDB metrics recording trigger thread (#7417)
- PR-7404 - KAFKA-8887: Use purgatory for ACL updates using async authorizers (#7404)
- PR-7354 - KAFKA-8595: Support deserialization of JSON decimals encoded in NUMERIC (#7354)
- PR-7418 - KAFKA-8807: Flaky GlobalStreamThread test (#7418)
- PR-7377 - KAFKA-8896: Check group state before completing delayed heartbeat (#7377)
- PR-7412 - MINOR: Mark RocksDBStoreTest as integration test (#7412)
- PR-7150 - KAFKA-8729 pt 2: Add error_records and error_message to PartitionResponse (#7150)
- PR-7376 - KAFKA-8927: Deprecate PartitionGrouper interface (#7376)
- PR-7401 - KAFKA-8609: Add rebalance-latency-total (#7401)
- PR-7419 - MINOR: Adjust logic of conditions to set number of partitions in step zero of assignment (#7419)
- PR-7352 - KAFKA-8911: Using proper WindowSerdes constructors in their implicit definitions (#7352)
- PR-7411 - MINOR: Update Jackson to 2.10.0 (#7411)
- PR-7408 - MINOR: Add missing + in LogSegment.toString (#7408)
- PR-7353 - KAFKA-8471: Replace control requests/responses with automated protocol (#7353)
- PR-7353 - KAFKA-8471: Replace control requests/responses with automated protocol (#7353)
- PR-7394 - MINOR: Don’t generate unnecessary strings for debug logging in FetchSessionHandler (#7394)
- PR-7402 - MINOR:fixed typo and removed outdated varilable name (#7402)
- PR-7397 - KAFKA-8934: Create version file during build for Streams (#7397)
- PR-7382 - KAFKA-8319: Make KafkaStreamsTest a non-integration test class (#7382)
- PR-7380 - KAFKA-8907: Return topic configs in CreateTopics response (KIP-525) (#7380)
- PR-7379 - MINOR: Address review comments for KIP-504 authorizer changes (#7379)
- PR-7372 - MINOR: add versioning to request and response headers (#7372)
- PR-6362 - KAFKA-7273: Extend Connect Converter to support headers (#6362)
- PR-7340 - MINOR: improve the Kafka RPC code generator (#7340)
Rest Utils¶
- PR-166 - MINOR: Log kafka exceptions for debugging
- PR-160 - Enable support for multiple servlets on a single Jetty Server (#155)
- PR-156 - Enable auto ssl keystore reload
- PR-157 - Move shutdownLatch.countDown back to Application for better readability
- PR-155 - Enable Rest Utils to provide multiple servlet contexts
- PR-154 - Add error log with thrown exception
- PR-151 - Update Jetty version to 9.4.20.v20190813 and Jersey version to 2.28
- PR-149 - Include separate metrics for each HTTP error code
- PR-147 - CC-5576: Use SslContextFactory.Server to disable server-side hostname verification
- PR-144 - Suppress the Jetty server version header
Confluent REST Proxy Changelog¶
Version 5.4.0¶
- PR-546 - CPKAFKA-2590: Bump ZK connection timeout to avoid test flakes
- PR-580 - MINOR: Include zkclient dependency to kafka-rest-scala-consumer shaded jar
- PR-587 - Add API to get beginning offset, end offset and offset for time
- PR-601 - CLIENTS-1454: Propagate Schema Registry properties to KafkaAvroDecoder
Version 5.1.0¶
- PR-511 - CPKAFKA-1876: Log cause of failure to produce messages in tests
- PR-512 - Fix race condition when expiration thread iterates over partially-initialized consumer
- PR-503 - Fix location of jars for dev testing
- PR-481 - CPKAFKA-1681: Add long polling functionality
- PR-508 - CPKAFKA-1868: Add more consumer tests
- PR-507 - CPKAFKA-1871: Fix problem where ReadTaskSchedulerThread would loop too often
- PR-504 - CPKAFKA-1865: Performance degrades drastically after running for extended periods
- PR-495 - CPKAFKA-1794: Assign leader ID -1 on offline partitions in the topic describe API
- PR-477 - Fix SimpleConsumerManager properly return requested offsets for compressed topic
- PR-474 - MINOR: Fix checkstyle path
- PR-466 - Change Consumer threading model to better handle rebalances
- PR-459 - MINOR: Add utility createConsumerProperties() test method since it was removed from AK
- PR-452 - Fix checkstyle to pass on 8.5 after upgrade from 6.19
- PR-439 - Temporarily pin Kafka and Confluent versions
- PR-438 - Don’t use deprecated RecordMetadata constructor
Version 5.0.0¶
Version 3.3.0¶
Version 3.2.2¶
- PR-303 - CLIENTS-304: Fix AvroConsumerRecord constructors and handling of topic for v1 consumer API.
Version 3.2.1¶
Version 3.2.0¶
- PR-256 - Update ClusterTestHarness to use o.a.k.common.utils.Time.
- PR-255 - clarified the key/value format limitations based on user questions
- PR-264 - Fix build to work post KIP-103
- PR-265 - Follow up for KIP-103 changes that fixes ProducerPool’s extraction of endpoints to use the security protocol in the URL instead of the listener name.
- PR-268 - modified readme to use v2. also bumped installation up for usability
- PR-271 - Updated quickstart and configuration docs for V2 and for security
- PR-258 - REST Proxy security
- PR-274 - added field ‘topic’ to tools/ConsumerPerformance.java for system test
- PR-272 - added description of JVM parameters for SASL configuration
Version 3.1.2¶
No changes
Version 3.1.1¶
No changes
Version 3.1.0¶
- PR-239 - Require bash since we use some bashisms and fix a copyright.
- PR-235 - fix typo command line options
-helpcheck - PR-222 - fixing issue #91
- PR-230 - Issue #229: Add blurb on Jetty jmx metrics
- PR-214 - Small readme.md fix
- PR-127 - Fix to ConsumerManager tests AND then handling of max bytes
- PR-203 - Fix a typo that cause non-corresponding logger name
- PR-202 - fix the implementation of topicExists
Version 3.0.0¶
- PR-164 - 2.x merge to master
- PR-166 - Bump version to 3.0.0-SNAPSHOT and Kafka dependency to 0.10.0.0-SNAPSHOT
- PR-171 - Fix build to handle rack aware changes in Kafka.
- PR-174 - Update CoreUtils usage to match kafka updates
- PR-189 - Minor fixes for compatibility with newest 0.10.0 branch.
- PR-192 - Minor fixes for compatibility with newest 0.10.0 branch.
- PR-202 - fix the implementation of topicExists
- PR-205 - Rearrange quickstart and use topics from earlier steps in requests for metadata so the example output will exactly match real output when starting from an empty cluster.
Version 2.0.1¶
Version 2.0.0¶
- PR-64 - Reduce integration test time.
- PR-66 - Add support for SimpleConsumer-like access (Issue #26)
- PR-67 - Handle conflicting IDs and separate IDs used in the REST Proxy and by Kafka’s consumer implementation.
- PR-78 - Remove kafka from list of production directories to include in CLASSPATH.
- PR-89 - JSON message support
- PR-96 - Fixed log4j and daemon flag bugs in kafka-rest-run-class based on fix from schema-registry.
- PR-99 - Require Java 7
- PR-101 - rest-utils updates
- PR-103 - Issue 94 rename main
- PR-108 - Clarify partitioning behavior for produce requests
- PR-117 - Update to Kafka 0.9.0.0-SNAPSHOT and make adjustments to work with updated ZkUtils.
- PR-122 - Use x.y.z versioning scheme (i.e. 2.0.0-SNAPSHOT)
- PR-123 - Updated args for JaasUtils.isZkSecurityEnabled()
- PR-125 - Use Kafka compiled with Scala 2.11
Control Center Changelog¶
Version 5.4.5¶
- Fixed an issue with topic pattern matching on trigger forms
- Fixed an issue with parsing large numbers in topic messages
- Added SSL SMTP support for email alerts
Version 5.4.0¶
- Added UI for managing RBAC role assignments.
- Added Replicator monitoring.
- Added Auto-update UI feature.
- Added Cluster Overview page.
- Added Broker Metrics page.
- Added ability to delete topics.
- Added ability to upload and download connector configurations.
- Added partition size to topic overview page.
- Added time series data for offline topic partitions, under replicated partitions, and out-of-sync replicas.
- Various enhancements and bug fixes to RBAC enforcement, Connect, KSQL, and License Manager.
- Bug fixes for backend HTTP client.
Version 5.3.8¶
- Added a new property that controls the segment file size for the log for command topic
- Updated the default value of the property that controls the retention period for command topic
- The X-Content-Type-Options response HTTP header is set to nosniff by default to prevent attacks based on MIME sniffing in the browser
- When confluent.controlcenter.rest.access.control.allow.origin property is enabled, CORS check is extended to WebSocket connection as well.
Version 5.3.6¶
- Fixed topic messages viewer losing precision on big numbers.
Version 5.3.0¶
- Redesigned the Control Center UI, including KSQL and Connect enhancements. New pages show cluster overview, topics pages, and consumers index.
- RBAC enforcement.
- Topic message browser enhancements, including jump to timestamp and download results in JSON format.
- Alert enhancements that added Slack and PagerDuty webhook notifications, a new Consumer lead alert, pause and resume global alerts actions, and fast navigation to Alerts features from the Alerts bell icon.
Version 5.2.0¶
- Added multiple Connect support.
- Added multiple KSQL support.
- Added broker configs editing.
- Added Schema Registry editing.
- Added license management UI.
- Improved message browsing.
- Improved KSQL UI.
- Improved scalability.
Version 5.1.0¶
- Browser support for Chrome, Firefox, and Safari. For more information, see Web Browsers.
- A new, more readable light theme.
- Many bug fixes and UX improvements.
- Added configuration confluent.controlcenter.license.manager=_confluent-controlcenter-license-manager-5-1-0. (It was confluent.controlcenter..license.manager = _confluent-controlcenter-license-manager5-1-0 in first 5.1.x release, but it is fixed now.) No need for customers to override this configuration.
Version 5.0.0¶
- Added KSQL interface to create/view streams, tables, and queries.
- Added topic inspection to read messages from a topic.
- Added consumer group message lag viewing and alerting.
- Added Schema Registry viewing.
- Added read only view of broker configurations.
- Added feature flags for variety of user accessible features.
Version 4.1.0¶
- Added ability to create and edit topics in Control Center.
- Added topic filtering and ability to hide Apache Kafka®/Confluent Platform internal topics from the UI.
- Added new alert to warn when Control Center stops receiving data from a Kafka cluster.
- Added a restricted user type that prevents editing or creating of topics, alerts, or connectors.
Version 4.0.1¶
- Made calls to Connect more resilient.
- Fixed an issue that would prevent chart highlighting on System Health charts.
- Fixed an issue that could cause Control Center to improperly warn about improper log dirs on brokers.
- Fixed an issue that could cause Control Center to improperly alert about active controller count.
Version 4.0.0¶
- Fixed an issue that could cause long request times if a cluster had brokers with noncontiguous IDs.
- Additional logging during startup.
- Fixed an issue that could cause Control Center to show blank graphs when data was present.
- Set and enforce TimestampType for internal Control Center topics at startup.
- Fixed an issue that caused us to link to an invalid consumer group on Streams Monitoring.
- Fixed an issue where Broker Monitoring could check for data last update time on wrong cluster.
- Allow overriding ConfluentMetricsReporter default producer settings.
- Allow using a file for
confluent.licenseconfig. - Docker image: allow passing license using CONTROL_CENTER_CONFLUENT_LICENSE environment variable.
- Fixed an issue that resulted in Offline and Under Replicated Partitions data to be swapped.
- Fixed an issue that could cause Control Center to shutdown if it lost connection to Kafka.
- Fixed an issue that could cause Control Center to hang if it encountered an error creating topics on startup.
Version 3.3.1 (EOSL)¶
- Allow text input in date picker.
- Fixed an issue that could cause unsubmittable forms.
- Show renamed cluster value rather than base clusterid on alerts page.
- Added alert banner on Broker Monitoring to indicate a cluster may be down.
- Scalability increase for Broker Monitoring to reduce number of messages sent to Kafka.
- Fixed an issue where 100% Network Pool Usage was displayed instead of 0%.
- Allow graceful shutdown of Control Center during startup.
- Fixed a race condition that could happen if you start Control Center and your Kafka broker at the same time.
Version 3.3.0 (EOSL)¶
- Sped up restore time.
- Reduced the size of old data over long periods of time.
- Upgraded to use new admin client (KIP-117).
- Alert UI should render better alert information.
- Triggers are createable directly though the UI on System Health.
- Massive performance improvements to Stream Monitoring UI.
- Fixed various time range display/selection bugs; persist time ranges though navigation.
- Nonconfigurable form values being sent to Connect via UI should remain and be visible prior to submit.
Version 3.2.2 (EOSL)¶
- Added shutdown script.
- Force faster compaction and cleanup on changelog topics.
- Several Streams improvements to reduce state-store restoration time after unclean shutdown.
- Reduce the volume of data to be restored to state-stores on startup.
- Fixed possible concurrency problem in setting RocksDB configuration options.
- Fixed possible deadlock on exception after streams startup.
Version 3.2.1 (EOSL)¶
- Enabled HTTP compression by default.
- Fixed possible deadlock on streams exception.
- Fixed fd leak when communicating with Kafka.
- Update defaults for monitoring producers to be more resilient.
- Prevent inadvertently creating misconfigured topics during startup during a failure.
- Check Kafka version at startup.
- Reduce minimum startup time.
- More efficient duplicate checking to reduce write amplification.
- Handle ‘hidden’ connect configs. This allows Control Center to manage Connectors that do not fully enumerate their configs.
- Ensure users only see invalid license warning if their license is invalid.
- Fixed topic display bug that would prevent topic details from showing.
- Reduced logging in js.
- Allow users to close missing data modal dialogs.
- Better rendering of System Health metrics especially when some data is missing.
- Better validation of triggers, actions, and connector configs.
- Faster processing of trigger and action creation and update.
- Corrected some misleading labels on System Health.
- Better error handling around Connect communications.
- Change default timespan in UI to 4 hours.
- Check and warn about errors on Topic Management page.
Version 3.2.0 (EOSL)¶
- Add capabilities to monitor brokers instrumented with Confluent Metrics Reporter.
- Enable alerting on broker and topic metrics.
- Add UI for viewing topic information.
- Add status UI for tracking internal progress.
- Fixed bug where we could inadvertently show license expired message.
- Fixed bugs with alert trigger creation in UI.
- Fixed issue with parsing certain error conditions from Connect.
Version 3.1.0 (EOSL)¶
- Enable gathering stream monitoring data from multiple clusters.
- Support alerting on stream monitoring conditions.
- Ability to restrict UI access via user login.
Version 3.0.1 (EOSL)¶
- Improved data fetching from the browser.
- Handle authenticated/authorized Kafka clusters.
- Bug fix for running in daemon mode.
- More efficient topic creation at startup.
- Reduced number of necessary topics/rocksdbs for aggregations.
- Reduced total amount of stored data.
- Fixed timing issues when receiving delayed data.
- Configs updates to prefix Kafka/rest configs.
- Added rate limit for logging to prevent logsize blowup.
- Allow deleting connector in UI.
Release notes for Control Center UI auto-updates¶
Release notes for each interim version will be listed here by their release version ID as they become available. You can also view the release notes for an auto-update in the Updates tab.
Version 5.4.0¶
KSQL Changelog¶
Version 5.4.0¶
Detailed Changlog¶
- PR-3980 - feat : Porting #3879, #3696 fixes for pull queries against secure clusters
- PR-3952 - fix: fix issues with multi-statement requests failing to validate
- PR-3819 - fix: /query rest endpoint should return valid JSON (BREAKING CHANGE)
- PR-3820 - fix: pull queries available on /query rest & ws endpoint
- PR-3778 - feat: add flag to disable pull queries (MINOR)
- PR-3724 - feat: Added UDFs ENTRIES and GENERATE_SERIES
- PR-3730 - fix: apply filter before flat mapping in logical planner
- PR-3716 - feat: Implement describe and list functions for UDTFs
- PR-3690 - feat: Implement schemaProvider for UDTFs
- PR-3703 - fix: Remove duplicate ksql-common dependency in ksql-streams pom
- PR-3695 - fix: quoted identifiers for source names
- PR-3687 - feat: Implement user defined table functions
- PR-3691 - fix: allow streams topic prefixed configs
- PR-3683 - feat: Implement complex expressions for table functions
- PR-3694 - fix: fix broken map coersion test case
- PR-3685 - feat: Support multiple table functions in queries
- PR-3639 - Don’t use rewrites to handle struct field access
- PR-3589 - feat: Implement EXPLODE(ARRAY) for single table function in SELECT
- PR-3588 - feat: support numeric json serde for decimals
- PR-3613 - Combine rowkey rowtime steps in KS topology
- PR-3592 - feat: Udf Invoker benchmark
- PR-3612 - feat: expression support for insert values
- PR-3609 - fix: Created new test for running topology generation manually from t…
- PR-3605 - fix: do not allow inserts into tables with null key
- PR-3599 - fix: be more lax on validating config
- PR-3601 - fix: fix parsing issue with LIST property overrides
- PR-3528 - fix: drop TopicDescriptionFactory class
- PR-3501 - feat: add health check endpoint
- PR-3531 - fix: support NULL return values from CASE statements
- PR-3570 - feat: expose query status through EXPLAIN
- PR-3540 - fix(3524): improve pull query error message (MINOR)
- PR-3492 - feat: pass auth header to connect client for RBAC integration
- PR-3550 - fix: Revert named stores in expected topologies, disable naming stores from StreamJoined, re-enable join tests.
- PR-3529 - fix(3525): SET should only affect statements after it
- PR-3516 - fix: standardize KSQL up-casting
- PR-3467 - feat: change /metadata REST path to /v1/metadata
- PR-3479 - Unique(ish) query Id per pull query
- PR-3477 - feat: support quoted identifiers in column names
- PR-3518 - feat: add request logging
- PR-3499 - fix: fix Avro schema validation
- PR-3490 - feat: add configurable MetricsReporter to MetricCollectors
- PR-3466 - fix: changes required for compatibility with KIP-479
- PR-3494 - feat: build materializations from the physical plan
- PR-3486 - fix: make sure use of non-threadsafe UdfIndex is synchronized
- PR-3488 - fix: Use the correct Immutable interface
- PR-3464 - fix: improve print topic error message when topic does not exist
- PR-3474 - fix: race condition in KsStateStore
- PR-3471 - fix: Kafka Streams does not like negative time
- PR-3470 - fix: fix NPE when printing records with empty value (MINOR)
- PR-3473 - feat(static): switch partial datetime parser to use UTC by default
- PR-3461 - Feat(static): forward auth header when proxying static query calls to another KSQL node
- PR-3418 - feat: build ks app from an execution plan visitor
- PR-3449 - fix: better error message if tz invalid in datetime string
- PR-3439 - feat(static): support ROWKEY in the projection of static queries
- PR-3438 - feat(static): allow windowed queries without bounds on rowtime
- PR-3425 - feat: add KsqlUncaughtExceptionHandler and new KsqlRestConfig for enabling it
- PR-3402 - fix: fix test for KIP-307 final PR
- PR-3442 - fix(3441): stabilize the StaticQueryFunctionalTest
- PR-3417 - feat: use coherent naming scheme for generated java code
- PR-3435 - feat(static): support for partial datetimes on WindowStart bounds
- PR-3437 - fix: address upstream change in KafkaAvroDeserializer (revert previous fix)
- PR-3412 - fix: support UDAFs with different intermediate schema
- PR-3432 - fix: do not allow WITH clause to be created with invalid WINDOW_SIZE
- PR-3421 - fix: Remove dependencies on test-jars from ksql-functional-tests jar
- PR-3430 - feat(static): fail on ROWTIME in projection
- PR-3433 - fix: Rename Delimiter:parse(char c) to Delimiter.of(char c)
- PR-3424 - fix: rename Delimiter.of(String) to Delimiter.parse(String)
- PR-3427 - feat: Ignore local scratch QTT file called scratch.json from git
- PR-3428 - feat(static): unordered table elements and meta columns serialization
- PR-3423 - fix: typo in static query WHERE clause example
- PR-3422 - feat(static): allow logical schema to have fields in any order
- PR-3385 - feat: custom comparators for array, map and struct
- PR-3411 - feat: remove equalsIgnoreCase from all Name classes (MINOR)
- PR-3354 - feat: update query id generation to use command topic record offset
- PR-3404 - feat: Implement ROUND() UDF
- PR-3393 - feat: Implement user defined delimiter for value format
- PR-3391 - feat: move aggregations to plan builder
- PR-3407 - fix: error message when UDAF has STRUCT type with no schema
- PR-3341 - feat(cli): add the feature to turn of WRAP for CLI output
- PR-3397 - fix: better error message when users enter old style syntax for query
- PR-3400 - Move statement re-write logic into engine
- PR-3398 - fix(3356): struct rewritter missed EXPLAIN
- PR-3386 - feat: back out Connect auto-import feature
- PR-3361 - feat: move joins to plan builder
- PR-3302 - feat: Add support for average aggregate function
- PR-3360 - feat: move setup of the sink to plan builders
- PR-3313 - feat: add REST /metadata path to display KSQL server information (replaces /info)
- PR-3359 - feat: move groupBy into plan builders
- PR-3369 - feat(static): static select support
- PR-3362 - feat: move selectKey impl to plan builder
- PR-3381 - Fix: Consistency between inserted rowkey and key values when inserting into a table with a key
- PR-3368 - Avoid unnecessary repartition of stream on GROUP BY ROWKEY
- PR-3372 - fix: address upstream change in KafkaAvroDeserializer
- PR-3355 - Fix: NPE when coercing on INSERT + improve err msg + better coercion test coverage
- PR-3337 - fix: improve error message on query or print statement on /ksql (MINOR)
- PR-3345 - Better error message for missing join fields
- PR-3346 - feat: move filters to plan builder
- PR-3334 - feat: move toTable kstreams calls to plan builder
- PR-3343 - feat: add offset to QueuedCommand and flag to Command
- PR-3353 - feat: add logs/ to .gitignore (MINOR)
- PR-3340 - feat(static): initial drop of static query functionality
- PR-3351 - fix: switch AdminClient to be sandbox proxy
- PR-3339 - fix:Case is now preserved for an unquoted topic name
- PR-3328 - fix: KSQL should warn when forcing CSAS object names to upper cas…
- PR-3326 - fix: band-aid around RestQueryTranslationTest
- PR-3318 - fix: should be able to parse empty STRUCT schema (MINOR)
- PR-3320 - feat(static): add forEach() to KsqlStruct (MINOR)
- PR-3322 - feat(static): add custom jackson JSON serde for handling LogicalSchema
- PR-3314 - fix: revert default listeners value and update docs
- PR-3300 - feat(static): initial syntax for static queries
Version 5.3.2¶
Version 5.3.1¶
Detailed Changlog¶
- PR-3225 - Fix quoted reserved identifier 5.3
- PR-3197 - fix: validate CLI using REST / instead of REST /info
- PR-3176 - fix: default timestamp extractor override is not working
- PR-3111 - fix: add ksql-functional-tests to the ksql package
- PR-3147 - fix: filter null entries before creating KafkaConfigStore
- PR-3079 - feat: add config for enabling topic access validator
- PR-3069 - feat(rest server): add warnings when listing fails with kafka error
- PR-3104 - fix: COLLECT_LIST can now be applied to tables
- PR-2997 - feat: add extension for custom metrics (5.3.x)
- PR-2996 - feat: add config for custom metrics tags (5.3.x)
Version 5.3.0¶
KSQL 5.3.0 includes new features, including:
- Drop the requirement that
CREATE TABLEstatements must have aKEYset in theirWITHclause. This is now an optional optimization to avoid unnecessary repartition steps. See Github issue #2745 for more info. - Improved handling of
KEYfields. TheKEYfield is an optional copy of the Kafka record’s key held within the record’s value. Users can supply the name of the field that holds the copy of the key within theWITHclause. The improved handling may eliminate unnecessary repartition steps in certain queries. Please note that preexisting persistent queries, e.g. those created viaCREATE TABLE AS SELECT ...orCREATE STREAM AS SELECT ...orINSERT INTO ..., will continue to have the unnecessary repartition step. This is required to avoid the potential for data loss should this step be dropped. See Github issue #2636 for more info. INSERT INTO ... VALUESis now supported, with standard SQL syntax to insert rows to existing KSQL streams/tables. To disable this functionality, setksql.insert.into.values.enabledtofalsein the server properties.CREATE STREAMandCREATE TABLEwill now allow you to create the topic if it is missing. To do this, specify thePARTITIONSand optionallyREPLICASin theWITHclause.CREATE STREAM AS SELECT ...andCREATE TABLE AS SELECT ...statements now use the source topic partitions and replica counts for the sink topic if the config properties are not set in theWITHclause. In case of JOIN, the left hand side topic is used. This deprecatesksql.sink.partitionsandksql.sink.replicasconfig properties.- A new config variable,
ksql.internal.topic.replicas, was introduced to set the replica count for the internal topics created by KSQL Server. The internal topics include the command topic and the config topic. - A new KSQL testing tool was added. The tool is a command line utility that enables testing KSQL statements without requiring any infrastructure, like Apache Kafka and KSQL clusters.
KSQL 5.3.0 includes bug fixes, including:
The
ROWTIMEof the row generated when aJOINencounters late data was previous theROWTIMEof the late event, where as now it is the max ofROWTIMEof the rows involved in the join. This provides more deterministic join semantics.Return values of UDF and UDAFs are now correctly marked as optional, where previously there was potential for non-optional fields, which would result in serialization issues in the presence of
nullvalues.This is a forward compatible change in Avro, i.e. after upgrading, KSQL will be able to read old values using the new schema. However, it is important to ensure downstream consumers of the data are using the updated schema before upgrading KSQL, as otherwise deserialization may fail. The updated schema is best obtained from running the query in another KSQL cluster, running version 5.3.
See Github issue #2769 for more info.
Fixed issues with using
ASkeyword when aliasing sources. See #2732 for more info.
Detailed Changlog¶
- PR-3061 - fix: Update version checker to use new dependencies
- PR-3045 - fix: Create the streams state directory if it does not exist.
- PR-3036 - fix: Reapply fix to only pass source topic input
- PR-3027 - fix: SqlFormatter now handles windows
- PR-2999 - Add explicit kafka core dependency for support metrics
- PR-2948 - Using the correct props for the version checker in the headless mode.
- PR-2937 - MINOR: Use function reference when assigning to java.func.Supplier
- PR-2916 - Remove the hack to standarize expected topologies.
- PR-2923 - Return correctly parametrized processing log type
- PR-2919 - Mention ksql.streams.bootstrap.servers setting in scaling section
- PR-2889 - Add support for checking external preconditions while still serving
- PR-2898 - Minor: close websocket session on error
- PR-2892 - Don’t accept empty output.
- PR-2888 - Do not enable TopicAccessValidator if Kafka has unsupported authorizedOperations
- PR-2893 - fix dead store in TestOptionsParserTest
- PR-2881 - Testing tool split input
- PR-2309 - Support -cp and -hotfix for packaging
- PR-2882 - Backport insert into values bug fix
- PR-2877 - Update support-metrics-common artifact version (#2874)
- PR-2870 - Close ServiceContext when Session is closed
- PR-2843 - Testing tool improvements/bug fixes
- PR-2852 - Verify access authorization to Websockets endpoints
- PR-2860 - Pipe serviceContext down to EngineContext (Fixes #2853)
Version 5.2.0¶
KSQL 5.2 includes new features, including:
- Support for HTTPS.
- Support for CASE expression: KSQL now supports CASE conditional expression in Searched form where KSQL evaluates each condition from left to right. It returns the result for the first condition that evaluates to true. If no condition evaluates to true, the result for the ELSE clause will be returned. If there is no ELSE clause, null is returned.
- A new family of UDFs for improved handling of URIs (e.g. extracting information/decoding information), see UDF table for all URL functions
LIMITkeyword support forPRINT(#1316)- Support for read-after-write consistency: new commands don’t execute until previous commands have finished executing. This feature is enabled by default in the CLI (#2280) and can be implemented by the user for the REST API (Coordinate Multiple Requests).
- A log of record processing events to help users debug their KSQL queries. The log can be configured to log to Kafka to be consumed as a KSQL stream. See KSQL processing log for more details.
- Aggregation functionality has been extended. KSQL now supports:
GROUP BYmore than just simple columns, including fields within structs, arithmetic results, functions, string concatenations and literals.- literals in the projection, (a.k.a the
SELECTclause). - Multiple
HAVINGclauses, including the use of aggregate functions and literals.
- Automatic compatibility management for queries in headless mode across versions. Starting with 5.2, KSQL will automatically take care of ensuring query compatibility when upgrading. This means you won’t need to worry about setting properties correctly during upgrade, as has been required for previous upgrades. Refer to the architecture documentation for details. Note that it is still up to the user to set properties correctly before upgrading to 5.2. The upgrade doc has details about the properties required to safely upgrade to 5.2.
KSQL 5.2 includes bug fixes, including:
- Improved support for multi-line requests in interactive mode deployments. Prior to version 5.2 KSQL parsed the full request before attempting to execute any statements. Requests that contained later statements that were dependent the execution of prior statements may have failed. In version 5.2 and later, this is no longer an issue.
- Improved support for non-interactive, “headless” mode deployments. Prior to version 5.2 KSQL parsed the full script before attempting to execute any statements. The full parse would often fail when later statements relied on the execution of earlier statements. In version 5.2 and later, this is no longer an issue.
KSQL 5.2 deprecates some features, including:
- The use of the
RUN SCRIPTstatement via the REST API is now deprecated and will be removed in the next major release. (Github issue 2179). The feature circumnavigates certain correctness checks and is unnecessary, given the script content can be supplied in the main body of the request. If you are using theRUN SCRIPTfunctionality from the KSQL CLI you will not be affected, as this will continue to be supported. If you are using theRUN SCRIPTfunctionality directly against the REST API your requests will work with the 5.2 server, but will be rejected after the next major version release. Instead, include the contents of the script in the main body of your request.
Detailed Changlog¶
- PR-2594 - Minor: clean up references to CURRENT_TOPOLOGY_VERSION
- PR-2585 - Decrease Kafka Controller socket timeout in tests.
- PR-2587 - 5.2 fix for UDFs not being passed their config.
- PR-2584 - KSQL-2278: Ensure all SR props are passed where needed
- PR-2537 - Fix bug in Avro schema cleanup during drop source
- PR-2526 - Fix bug in Avro schema cleanup during cluster termination
- PR-2532 - Adds a name to map schemas (#2504)
- PR-2500 - SSL Support
Version 5.1.0¶
KSQL 5.1 includes new features, including:
WindowStart()andWindowEnd()UDFsStringToDate()andDateToString()UDFs
Detailed Changlog¶
- PR-2265 - MINOR: Fix bug encountered when restoring RUN SCRIPT
- PR-2240 - Bring version checker improvements to 5.1.x
- PR-2143 - Fix some bugs in recovery logic
- PR-2150 - Currently we don’t support AS for aliasing stream/table.
- PR-2149 - Using ksql topic name instead of Kafka topic name in topic map in metastore.
- PR-2090 - Add WindowStart and WindowEnd UDFs (#1993)
- PR-2075 - Disable optimizations for 5.1.x
- PR-2051 - Preserve originals when merging configs
- PR-2066 - Minor: Fix bug involving filters with NOT keyword.
- PR-2056 - Added stringtodate and datetostring UDFs for 5.1.x
- PR-2048 - Minor: Fix bug involving LIKE patterns without wildcards.
- PR-2045 - List UDAFs for 5.1.x
- PR-2023 - MINOR: Cause ‘ksql help’ and ‘ksql -help’ to behave the same as ‘ksql -h’ and ‘ksql –help’
- PR-1979 - Metrics refactor + fix a couple issues
- PR-2018 - Display stats timestamps in unambiguous format.
- PR-1981 - ST-1153: Switch to use cp-base-new and bash-config to hide passwords by default
- PR-1955 - Hide ssl configs and refactor KsqlResourceTest
Version 5.0.0¶
KSQL 5.0 includes a number of new features, including:
- User-Defined Functions (UDF).
- Support for nested data via the new STRUCT data type.
- Support for writing to existing streams using the new INSERT INTO statement.
- Stream-Stream Joins.
- Table-Table Joins.
- Table Aggregations.
- A revamped REST API that will be kept compatible from this release onwards.
- Confluent Platform Docker images for KSQL Server and CLI
- Java 10 support.
Detailed Changlog¶
- PR-1610 - Specify a namespace for avro schemas
- PR-1588 - Add example avro schemas to the ksql packaging build
- PR-1529 - Add a schema inference test
- PR-1511 - Fix NPE when printing null values
- PR-1547 - Fix the bug for null check in comparison expression code gen,
- PR-1549 - Ensure that the list of interceptors is modifiable before mutating it
- PR-1531 - Recommend the use of STRUCT instead of EXTRACTJSONFIELD
- PR-1526 - Include monitoring interceptor config in streams configs
- PR-1524 - Java 10 support
- PR-1479 - Write configs to command topic and read back before building queries
- PR-1512 - Data-gen now can create struct columns.
- PR-1517 - Fix spacing in struct schema string
- PR-1507 - Standardize on a single version for the avro-random-genator
- PR-1481 - Consistent schema format between websocket endpoint and ksql endpoint
- PR-1503 - Convert invalid avro field names before serializing
- PR-1501 - sort udf names
- PR-1500 - don’t bomb on non-existing plugin dir
- PR-1495 - Pass schema registry client configs onto the schema registry client
- PR-1482 - String masking functions
- PR-1434 - Schema Translation Test + Nested Support in Query Validation Test
- PR-1486 - list and describe updated to include aggregate functions
- PR-1478 - Use ‘:’ for struct field reference.
- PR-1467 - Support java based UDAFS
- PR-1390 - Upgrade handling step 1: clean up KSQL configs
- PR-1453 - Struct end to end
- PR-1465 - support null literal in UDFs #1462
- PR-1458 - add jar path to description of functions
- PR-1457 - Simple Security manager for UDFS
- PR-1449 - create module for UDF annotations to minimize dependencies
- PR-1470 - Use a string for formatted time field in clickstream schema
- PR-1448 - Add config for ext dir
- PR-1463 - Handle rename of KafkaConsumerService to ConsumerService
- PR-1455 - Rename SPAN to WITHIN everywhere
- PR-1435 - Support List/Show Functions (UDFs) and Describe Function <name>
- PR-1417 - Add new join types to KSQL
- PR-1430 - Updated AST builder to handle dot notation.
- PR-1442 - Simplified Security
- PR-1445 - Prefixed output topics
- PR-1426 - Throw exception when calling FunctionRegistry#getUdfFactory(name) if function doesn’t exist
- PR-1429 - UDF tidy ups
- PR-1423 - Collect metrics for UDFs
- PR-1416 - Validate that function names are valid java identifiers
- PR-1410 - Added query rewrite feature using AST.
- PR-1366 - Create and load Java based UDFs from an ext directory
- PR-1362 - Return schema description API entities as JSON objects
- PR-1400 - Add struct support for avro by using Connect Converter API
- PR-1406 - #1396 Fix regression issue with string concat using ‘+’ operator.
- PR-1407 - Avoid NPE if not function exists that handles required types.
- PR-1401 - Switch to optional schema fields since we can have null values.
- PR-1409 - Issue #1111: Document schemaRegistryURL arg for ksql-datagen in usage…
- PR-1383 - Using Connect data format in KSQL.
- PR-1140 - Do schema inference before writing to output topic
- PR-1370 - Wrap api query id strings in a class
- PR-1368 - Using ExpressionTypeManager to detect the types for function look up.
- PR-1379 - Disable server identification verification for SecureIntegrationTest
- PR-1369 - Using the kafka topic name in delete topic message.
- PR-1363 - Print an error if a message fails to produce with datagen
- PR-1356 - New geo distance function
- PR-1359 - handle array subsrcipts when generating function args
- PR-1353 - Support functions with same name but different arguments
- PR-1285 - Refactor WebSocket endpoints and add support for print topic
- PR-1350 - Child first class loader for UDFs
- PR-1337 - Fix the way LIMIT clauses are handled
- PR-1321 - Make FunctionRegistry an interface and make MetaStore implement it
- PR-1340 - Post fix udf instance names to ensure they are unique.
- PR-1335 - Don’t ignore leading spaces when saving history
- PR-1331 - Follow-on updates for KSQL_OPTS
- PR-1329 - Fix run class to work with Java 10 and use ExplicitGCInvokesConcurrent
- PR-1330 - Make KSQL_OPTS more prominent
- PR-1301 - Minor: Admin client leak
- PR-1315 - Consumed imports
- PR-1277 - Ksql 1217 optionally delete kafka topic with drop statement
- PR-1309 - Add crosslink from ‘starting KSQL server’ section to the headless mode instructions
- PR-1114 - Struct Data Type(Part-1): New struct type, DDL statements and Describe
- PR-1305 - MINOR: Add query validation test for stringtotimestamp with double single quote
- PR-1303 - The DESCRIBE ACL on the __consumer_offsets topic is not required.
- PR-1091 - Add parent reference to ast nodes
- PR-1249 - Include function names in error message for unsupported table aggrega…
- PR-1220 - Add external dependency on Avro Random Generator and remove its source code
- PR-576 - Insert into implementation
- PR-1256 - Better reporting of invalid serverAddress or connection issues in CLI
- PR-1197 - KSQL-883: Add KafkaClientSupplier to KSQL API
- PR-1278 - Add Preview Release information, update Latest News
- PR-1219 - Improved error message for Stream/Table and Query relation correctness.
- PR-1201 - Add versioning to the API (#1151)
- PR-1210 - Fix some more rest api inconsistencies
- PR-1136 - Table aggregations
- PR-1163 - Minor: Fix lifecycle of AdminClient and KafkaTopicClientImpl instances.
- PR-1208 - Add a Pull request template
- PR-1050 - Escape discovered avro field name if it is a ksql lexer token literal #(1043)
- PR-1198 - text is the new binary
- PR-1147 - SourceDescription cleanup + listing with descriptions
- PR-1179 - Shutdown cleaning should there be an error on start up.
- PR-1174 - Fix handling of table tombstones
- PR-1171 - Add back key-constraints
- PR-838 - Rename ‘Kafka output topic’ to just ‘Kafka topic’ in describe ext (#838)
- PR-1109 - Ksql 1054 better aggregation with complex expressions
- PR-652 - Added referential integrity enforcement for streams/tables and queries
- PR-756 - Add the ability to extract the record timestamp from a string field #646
- PR-1130 - Return proper errors from the HTTP server endpoints
- PR-944 - Ksql 660 schema registry clean up
- PR-1126 - Don’t depend on the internal ‘PlainSaslServer’ class from kafka
- PR-1103 - Return more stuff from the rest API
- PR-1120 - update error messages when failing to parse an avro schema
- PR-1108 - add websockets query endpoint
Schema Registry Changelog¶
Version 5.4.0¶
- PR-1244 - DG-134: Fix NPE when schema.registry.ssl.protocol not specified
- PR-1242 - DG-128: Allow DNS lookup policy to be configurable
- PR-1235 - DG-114: Fix tombstone behavior that causes CPU utilization to go up
- PR-1228 - DG-42: Upgrade to Avro 1.9.1; fixes #1122
- PR-1225 - Remove javax.validation::validation-api dependency from Schema Registry client
- PR-1026 - CC-3759 - Regenerate ID if in use during registration
- PR-1207 - Add /v1/metadata/id endpoint to display Schema Registry cluster information
- PR-1220 - Add optional query parameter to /schemas/ids/{id} to retrieve the current max ID of all schemas
- PR-1214 - MINOR: remove type parameter on Avro deserializer
- PR-1212 - CC-6428 - Use values to preserve defaults for SSL configuration
- PR-1210 - MMA-5160 - Add defaultToGlobal flag for config API
- PR-1144 - Add a configurable proxy to Schema Registry client
- PR-1208 - Retain RestService SSL method API
- PR-1167 - Mark API parameter for API Spec
- PR-969 - Avro Console Consumer print.timestamp property
- PR-957 - Ability to configure SSL truststore and keystore for Schema Registry clients
- PR-1205 - Use constant time lookup to check subject exists
- PR-1111 - Support reflection-based Avro serialization
- PR-1196 - API to retrieve all subjects associated with a schema id
- PR-1183 - MINOR: Change int to Integer for schemaRegistryPort
- PR-1180 - MINOR: handle null in RecordNameStrategy; fixes #1165 (TopicRecordNameStrategy creates invalid names on null values)
- PR-1179 - MINOR: add transitive levels to messages; fixes #1150 (Client reports misleading error when providing an unsupported compatibility level)
- PR-965 - Invalid compatibility message update to include three transitive states
- PR-1178 - Fix createParentPath issue where it was actually creating the child path
- PR-1135 - #941. Avoid NPE on parsing Avro schemas by throwing InvalidSchemaException (which is handled as BadRequest)
- PR-1172 - Add protected method for configuring forceOptional
- PR-1038 - Use try-with-resources to avoid NPE in RestService
- PR-1171 - MINOR: refactor configuration of RestService
- PR-1170 - MINOR: close admin client in tests
- PR-1006 - Fix issue “RecordNameStrategy throws SerializationException on null values”
- PR-1149 - CC-5036: Use swagger-maven-plugin to generate API specifications
- PR-1163 - CC-5283: Add temporary replacement for kafka.utils.ZkUtils
- PR-1148 - Use the new AbstractCoordinator interface
- PR-1140 - CC-4099: JsonErrorHandler for Schema Registry
- PR-1088 - CC-4345: Add per-subject locking and lastOffset tracking
- PR-1136 - Fix compilation error caused by signature change to TestUtils.createBrokerConfig
- PR-1127 - MINOR: fix abstract coordinator usage
Version 5.3.0¶
- PR-1009 - CC-3741: Remove use of IdentityHashMap
- PR-1020 - CC-3803: Corrected support for handling defaults of logical schema types
- PR-1018 - DOCS-319: Add reference to SR security plugin
- PR-1064 - DEVX-768: 5.2 C3 feature focus: new Schema Registry capabilities
- PR-1039 - DOCS-900: use compatibilityLevel for GET response examples
- PR-1067 - DOCS-1437: Change sr-short to sr for proper substitution
- PR-1073 - DOCS-1440: Remove CP install from SR lefthand TOC
- PR-1065 - DOCS-1333: Add Schema Migration docs
- PR-1055 - DOCS-1094: Schema Registry ACLs
- PR-1060 - MINOR: Backport of: Fix checkstyle to pass on 8.5 after upgrade from 6.19 (#857)
- PR-1074 - MINOR: Fix exception string formatting in Avro converter
- PR-1022 - MINOR: improve err message during duplicate ID detection
- PR-1021 - MINOR: add defensive check for tombstoning
Version 5.2.0¶
- PR-1062 - CC-4091: Enable reregistration of schema with different ID
- PR-1056 - CC-4062: Fix NPE caused after _schemas topic is compacted
- PR-1054 - CC-4056: Pass id/version when secondary forwards register requests
- PR-1045 - CC-4015 : Clean up tombstones from internal caches
- PR-1043 - Support explicit names for non-string map schemas.
- PR-1035 - CC-3956: Revert “Merge pull request #687 from sunbit-dev/canonical-schemas
- PR-687 - Fix #28: Normalize schemas to canonical form
- PR-1031 - CC-3930: Only send version/id if not null
- PR-1023 - MINOR: swap deleted w/non-deleted before clearing; fix javadoc
- PR-953 - CC-2669 Allow id and version to be passed to register; add modes
- PR-992 - CC-3635 - Add config to specify location of static resources
- PR-994 - Ability to pass additional topic configs to the schemas topic
- PR-948 - Make CachedSchemaRegistryClient thread-safe
- PR-930 - CC-3003 : Ability to inject custom headers to CachedSchemaRegistryClient
- PR-939 - MINOR: ensure store is initialized for getAllKeys
Version 5.1.0¶
- PR-933 - DOCS-989 - Fix legacy SR doc issues
- PR-936 - DOCS-929 - Fix confusing anchor links
- PR-938 - Duplicate Heading Update (#937)
- PR-921 - Pass ClientDnsLookup to NetworkClient and parseAndValidateAddresses
- PR-873 - Refactor SR to pull lookup datastructure to its own interface and Id Generation to its own interface
- PR-888 - CC-2674: Point to the right checkstyle suppression file
- PR-846 - CC-2367 : Tombstone deleted schemakey if the same schema is registered again in the subject
- PR-853 - CC-2441 : Add Cyclic schema support
- PR-861 - MINOR: Add Timer class. Change from Kafka (commit fc5f6b0e46ff81302b3e445fed0cdf454c942792)
- PR-857 - Fix checkstyle to pass on 8.5 after upgrade from 6.19
Version 5.0.0¶
- PR-837 - CC-2185: Deprecate schema.registry. prefixes on commonly modified configs and allow unprefixed versions
- PR-834 - DOCS-381 - Rename quick start and improve navigation
- PR-813 - Update maven-plugin-plugin to fix out of bounds error in build
- PR-811 - Fix SchemaRegistryCoordinator for upstream changes
- PR-810 - Remove KafkaJsonEncoder since Scala producers have been removed from Kafka
- PR-809 - Remove KafkaAvroEncoder since the Scala producers have been removed from Kafka
- PR-808 - Use java.util.Base64 instead of javax.xml.bind.DatatypeConverter
- PR-807 - Fix usage of internal Jersey utilities that no longer exist
- PR-806 - Inherit maven compiler plugin definition from common
- PR-804 - Ability to plugin an implementation of BasicAuthCredentialProvider
- PR-803 - CC-1958: Verify configs if createTopic fails because topic already exists
- PR-792 - Improve GC options
- PR-784 - Docs for Subject Name Strategy and Basic Auth
- PR-774 - Revert “Remove ZooKeeper from title”
- PR-773 - Remove ZooKeeper from title
- PR-756 - CLIENTS-986 Forward Auth headers
- PR-748 - confluent.maven.repo: use ‘https’
- PR-731 - Issue #704 - Make Serde constructors public
Version 4.1.2¶
Version 4.1.0¶
- PR-757 - Add WantedBy dependency to ensure startup order
- PR-741 - CC-1511 : Fix for subject strategy when used with new SubjectNameStrategy.
- PR-742 - Added delete to the client.
- PR-738 - CLIENTS-896: Fix path to patch/series
- PR-735 - Change to use updated Kafka internal API for JaasContext
- PR-718 - Log properly when Exception raised on SchemaRegistryRestApplication
- PR-723 - Merge packaging branches into master and add systemd init scripts
- PR-545 - #544: Add null check for schema doc while adding the property “connect.enum.doc” for schema conversion of enum types
- PR-680 - New (key|value).multi.type option for Avro serialization
- PR-710 - Support Multiple SchemaRegistryResourceExtension
- PR-702 - Set metrics tag for SR metrics
- PR-694 - check kafkaTopicReader and producer not null before closing
- PR-686 - issue#629: endpoint to return schema literal
- PR-696 - Fix sasl inherit
- PR-683 - Systemd
- PR-681 - Add getAllVersions to SchemaRegistryClient
- PR-673 - CLIENTS-729 : ability to inject BasicAuth credentials from producer/consumer SASL
- PR-676 - CLIENTS-774 : ability to forward authorization header
- PR-664 - CLIENTS-708 & CLIENTS-720 Fix enum type in fromConnectData
- PR-571 - Support basic auth credentials embedded in the schema.registry.url
- PR-657 - Add maxPollTimeout parameter to ConsumerNetworkClient constructor
Version 4.0.1¶
This release upgrades Jackson 2.5.4 to 2.9.4, and fixes handling of Avro enums.
Version 4.0.0¶
- PR-686 <https://github.com/confluentinc/schema-registry/pull/686> - Added REST endpoint to return the unescaped Avro schema for a specific subject and version.
Upgrade Notes¶
In 4.0.0, initial creation or validation of the topic used to store schemas has been reimplemented to use native Kafka protocol requests instead of accessing ZooKeeper directly. This means that you are no longer required to have direct access to the ZooKeeper cluster backing your Kafka cluster. However, note that this also requires appropriate permissions to create topics (on first execution of Schema Registry) or describe topics and configs (on subsequent executions to validate the topic is configured correctly). If you have authentication and authorization enabled on your Kafka cluster, you must ensure your principal has the correct permissions before upgrading Schema Registry cluster. Your principal must have the following permissions:
Create Schemas Topic
| Operation | Resource | Reason |
|---|---|---|
| Describe | Topic: kafkastore.topic |
Check existence of topic |
| Create | Cluster | Create the schemas topic, set compaction policy |
Validate Schemas Topic
| Operation | Resource | Reason |
|---|---|---|
| Describe | Topic: kafkastore.topic |
Check existence of topic |
| DescribeConfigs | Topic: kafkastore.topic |
Validate correct compaction policy on topic |
Version 3.3.0¶
- Upgrade avro to 1.8.2
- PR-561 - Use TO_AVRO_LOGICAL_CONVERTERS to convert default values that are Kafka connect logical data types to internal format which correspond to schema type. Logic copied from AvroData fromConnectData
- PR-549 - Replace usage of deprecated ZkUtils.DefaultAcls()
- Allow for some retries when validating that all nodes have the same primary
- Relocate Avro serdes under a new avro package
- Increment Magic Byte for SchemaKey and add compatibility tests
- Add Delete Schema support
- Added avro-serde for Kafka Streams. Pulled from the example project.
- #506 - The AvroMessageFormatter passes byte[] to an Avro encoder, but Avro only likes ByteBuffer. So we need to ByteBuffer.wrap() instead.
- Added optional kafkastore.group.id config to override the one automatically created by Schema Registry
- PR-476 - Adapt to KAFKA-4636 changes: Per listener security settings overrides (KIP-103)
Version 3.2.1¶
- PR-503 - CLIENTS-244: Update 3.2.0 changelog
- PR-499 - making sure Schema Registry doesn’t start with uncompacted topic
- PR-497 - CLIENTS-103: Fix ArrayIndexOutOfBoundsException in SchemaRegistryPerformance by counting the registration attempt even if it failed.
- PR-493 - Fixes for CLIENTS-257
- PR-494 - MINOR: Add compact schemas topic doc
- PR-458 - CLIENTS-104: Add a few retries during startup to allow for slow metadata propagation after creating the _schemas topic.
Version 3.2.0¶
- PR-428 - Maven Checkstyle
- PR-425 - Logical Type support
- PR-440 - Documentation changes to reflect pull request 415
- PR-451 - Generalize schema incompatibility message.
- PR-457 - Update ClusterTestHarness to use o.a.k.common.utils.Time.
- PR-458 - CLIENTS-104: Add a few retries during startup to allow for slow metadata propagation after creating the _schemas topic.
- PR-464 - Improve request URL building in client
- PR-465 - Add topic to error string to make debugging easier.
- PR-448 - Fixes the following avro issues (complex union, document preservation, output schema != input schema)
- PR-468 - CC-443: AvroData’s caches should be synchronized for thread-safety
- PR-473 - Fix build to work post KIP-103.
- PR-474 - Fix broker endpoint extraction to correctly translate to the non-ListenerName version that clients use to initiate broker connections.
- PR-472 - Handle primitive types when specific.avro.reader is true
- PR-477 - Docchangefor3.2
- PR-488 - Don’t re-invoke SchemaBuilder.version() and SchemaBuilder.name() if the value has already been set.
- PR-494 - MINOR: Add compact schemas topic doc
Version 3.1.1¶
No changes
Version 3.1.0¶
Schema Registry¶
- PR-429 - Moving licenses and notices to a new format, generated by an internal script.
- PR-415 - Option to apply fully transitive schema compatibility checking
- PR-412 - Require bash since we use some bashisms and fix a copyright.
- PR-396 - ZooKeeper and Kafka SASL support.
- PR-384 - Update the link to Google Java code Style
- PR-372 - Update KafkaStore to use moved TopicExistsException class.
- PR-373 - Added get all subjects.
- PR-364 - Increase testing timeouts from 5000ms to 15000ms
- PR-346 - configured log4j to write to log file
Version 3.0.0¶
Schema Registry¶
- PR-212 - change the documentation on port to have a high priority and list it higher up in the docs
- PR-298 - Bump version to 3.0.0-SNAPSHOT and Kafka dependency to 0.10.0.0-SNAPSHOT
- PR-300 - Using the new 0.9 Kafka consumer.
- PR-302 - Fix build to handle rack aware changes in Kafka.
- PR-305 - Update to match changed methods in CoreUtils
- PR-317 - Change the ‘host.name’ importance to high
- PR-319 - KafkaStore SSL support.
- PR-320 - API reference uses ‘integer’ Avro type which isn’t supported. ‘int’ is supported.
- PR-329 - https support.
Serializers, Formatters, and Converters¶
- PR-264 - Add null checks to json serializer/deserializer
- PR-274 - Add support for Avro projections in decoders
- PR-275 - Fixed references to confluent common version
- PR-276 - Unit tests and bugfix for NPE when using nested optional fields
- PR-278 - Test cases for optional nested structs
- PR-280 - Fix fromConnectData for optional complex types
- PR-290 - Issue #284 Cannot set max.schemas.per.subject due to cast exception
- PR-297 - Allows any CharSequence implementation to be considered a string
- PR-318 - Minor cleanup
- PR-323 - Fix #142 - handle parsing non-json responses in the RestService
- PR-332 - Add status storage topic to Connect Avro sample config.
Version 2.0.0¶
Schema Registry¶
- PR-141 - Incorrect path to log4j.properties file for simple zip file layout
- PR-143 - schema-registry-start does not work with -daemon argument
- PR-152 - Added point about Google code style to Readme.
- PR-163 - Expose more information when registry fails to start
- PR-165 - Add compatibility support for SchemaRegistryClient
- PR-167 - Update the versionCache with a new schemaVersionMap for a subject
- PR-169 - Use correct URL to update compatibility setting of a subject
- PR-180 - GH-177: Remove unneeded content-type headers for GET ops in quickstart and README
- PR-184 - Support multiple registry URLs in client
- PR-186 - Rename LocalSchemaRegistryClient to make it clear it is only intended to be used as a mock in tests. Fixes #185.
- PR-187 - Correct example response for POST /subjects/(string: subject)/versions
- PR-188 - Address GH-168; enable unit testing of CachedSchemaRegistryClient
- PR-195 - Fixed typo in Exception message
- PR-196 - Require Java 7
- PR-197 - Enable test code sharing
- PR-198 - Update jersey, jackson and junit versions to match rest-utils
- PR-202 - Correct minor docs error in example response
- PR-203 - Issue 194 rename main
- PR-207 - Issue #170: PUT /config/(string: subject) should return 4xx for unknown subjects
- PR-210 - Issue #208: Several RestApiTest test cases don’t test proper exception behavior
- PR-219 - Update Kafka version to 0.8.3-SNAPSHOT so we can start developing using upcoming 0.8.3 features.
- PR-237 - Update uses of ZkUtils to match changes made in KAFKA-2639.
- PR-240 - Fixes Typo in the docs. ‘actsas’ -> ‘acts as’
- PR-242 - Fixed Schema Registry build against kafka trunk
- PR-243 - Use x.y.z versioning scheme (i.e. 2.0.0-SNAPSHOT)
- PR-245 - Fix mvn assembly setup
- PR-252 - Use Kafka compiled with Scala 2.11
- PR-257 - Updated classpath in schema-registry-run-class to reflect changes in pom.xml
- PR-258 - CC-53: Add worker configs for Avro Kafka Connect that integrates with Schema Registry.
Serializers, Formatters, and Converters¶
- PR-146 - Added decoder to return a specific Avro record from bytes.
- PR-162 - Add implementation of new Deserializer interface.
- PR-192 - Add new module for Kafka JSON serialization stuff
- PR-193 - Add more JSON codec support
- PR-200 - Switch serializer config classes to use AbstractConfig from confluent-common instead of from Kafka.
- PR-222 - Add AvroConverter in new copycat-avro-converter jar to convert Copycat and Avro data.
- PR-234 - Add AvroConverter support for Decimal, Date, Time, and Timestamp logical types.
- PR-235 - Add caching of schema conversions in AvroData and AvroConverter.
- PR-247 - Update for Copycat -> Kafka Connect renaming.
- PR-251 - Add tests of conversion of null values from Kafka Connect and fix handling of null for int8 and int16 types.
- PR-254 - Ensure AvroConverter passes through null values without adding schemas and that deserialized null values are converted to SchemaAndValue.NULL.
- PR-255 - CC-44: Include version when deserializing Kafka Connect data.
- PR-256 - Encode null values for schemaless array entries and map keys and values as an Anything record.