Upgrade ksqlDB for Confluent Platform

Upgrading to ksqlDB 7.7 from ksqlDB 7.6

For a complete list of changes, see the ksqlDB pull requests.

Upgrading to ksqlDB 7.6 from ksqlDB 7.5

For a complete list of changes, see the ksqlDB pull requests.

Upgrading to ksqlDB 7.5 from ksqlDB 7.4

For a complete list of changes, see the ksqlDB changelog.

Upgrading to ksqlDB 7.4 from ksqlDB 7.3

  • For ksqlDB RBAC, the CREATE STREAM and CREATE TABLE statements now require the ResourceOwner role for source topics, instead of DeveloperRead.
  • All DDL statements that require permissions on a source topic must have the ResourceOwner role. For more information, see Grant a user access to resources.

Upgrading to ksqlDB 7.3 from ksqlDB 7.2

For a complete list of changes, see the ksqlDB changelog.

Upgrading to ksqlDB 7.2 from ksqlDB 7.1

Version 7.2 fixes an issue that occurred during upgrades from version 6.1 in which the upgrade may fail if DROP statements exist in the command topic for objects that are referenced by other objects.

For a complete list of changes, see the ksqlDB changelog.

Breaking Changes

  • When creating connectors through ksqlDB, ksqlDB no longer sets the key.converter config to StringConverter automatically because ksqlDB has supported key types other than strings for a number of releases.

Upgrading to ksqlDB 7.1 from ksqlDB 7.0

For a complete list of changes, see the ksqlDB changelog.

Important

If you are using Protobuf-wrapped primitive type structs in ksqlDB, for example google.protobuf.StringValue and similar types, do not upgrade to Confluent Platform 7.1.0 or 7.1.1. A fix is provided in 7.1.2.

Upgrading to ksqlDB 7.0 from ksqlDB 6.2

For a complete list of changes, see the ksqlDB changelog.

Upgrading to ksqlDB 6.2 from ksqlDB 6.1

For a complete list of changes, see the ksqlDB changelog.

Note

Queries written for ksqlDB 6.1 continue to run without changes under ksqlDB 6.2. New queries issued on 6.2 may have different behavior, even if the text of the query statement is the same.

Breaking Changes

  • Queries with GROUP BY clauses that contain multiple grouping expressions now result in multiple key columns, one for each grouping expression, rather than a single key column that is the string-concatenation of the grouping expressions. Note that this new behavior (and breaking change) apply only to new queries; existing queries will continue to run uninterrupted with the previous behavior, even across ksqlDB server upgrades.
  • Stream-table key-to-key joins on mismatched formats now repartition the table (right hand side) instead of the stream. Old enqueued commands are not affected, so this change should remain invisible to the end-user.
  • Upgrading to version 6.2 and later versions may fail if a DROP statement exists in the command topic for an object that is referenced by other objects. This issue was introduced in PR 6143.

Upgrading to ksqlDB 6.1 from ksqlDB 6.0

For a complete list of changes, see the ksqlDB changelog.

Note

Queries written for ksqlDB 6.0 continue to run without changes under ksqlDB 6.1. New queries issued on 6.1 may have different behavior, even if the text of the query statement is the same.

Breaking Changes

  • ksqlDB now creates windowed tables with cleanup policy compact,delete, rather than compact. Also, topics that back streams are always created with cleanup policy delete, rather than the broker default (by default, delete).

  • ksqlDB no longer emits unnecessary tombstones when a HAVING clause filters out a row from the source that is not in the output table. For example, given:

    -- source stream:
CREATE STREAM FOO (ID INT KEY, VAL INT) WITH (...);
-- aggregate into a table:
CREATE TABLE BAR AS
SELECT ID, SUM(VAL) AS SUM
FROM FOO
GROUP BY ID
HAVING SUM(VAL) > 0;
-- insert some values into the stream:
INSERT INTO FOO VALUES(1, -5);
INSERT INTO FOO VALUES(1, 6);
INSERT INTO FOO VALUES(1, -2);
INSERT INTO FOO VALUES(1, -1);

    Where previously the contents of the sink topic BAR would have contained records:

    Key Value Notes
    1 null Spurious tombstone: the table does not contain a row with key 1, so no tombstone is required.
    1 {sum=1} Row added as HAVING criteria now met
    1 null Row deleted as HAVING criteria now not met
    1 null Spurious tombstone: the table does not contain a row with key 1, so no tombstone is required.

    The topic will now contain:

    Key Value
    1 {sum=1}
    1 null

Downgrade considerations

If you try to downgrade ksqlDB to an earlier version, you may see the following internal RocksDB error:

Caused by: org.rocksdb.RocksDBException: Unknown Footer version. Maybe
this file was created with newer version of RocksDB?

This error indicates that a change in RocksDB version can prevent rolling back ksqlDB to earlier versions.

As a solution, ensure that the RocksDB version is consistent across your environment. Also, you may need to update the existing data to ensure compatibility with the current RocksDB version. This may involve steps like exporting the data before the version change and importing it after the change. Also, you may be able to delete the state store so ksqlDB recreates the state store from the changelog topic.