<a id="upgrading-ksql"></a>

# Upgrade ksqlDB for Confluent Platform

## Upgrade to ksqlDB 8.3.x

The security manager and the `ksql.udf.enable.security.manager` configuration property have been
removed due to support for Java 21.
For more information, see [Security manager](reference/user-defined-functions.md#security-manager).

## Upgrade to ksqlDB 8.2.x

For a complete list of changes, see the
[ksqlDB pull requests](https://github.com/confluentinc/ksql/pulls).

## Upgrade to ksqlDB 8.0.x

### Upgrade directly to 8.0.1 or later when using the “merge repartition topics” optimization

ksqlDB version 7.9 uses the `KStream.transformValues()` method. In ksqlDB
version 8.0.0, there is a migration from `KStream.transformValues()` to
`KStream.processValues()`, because the `transformValues()` method was
removed in Apache Kafka® 4.0.

#### IMPORTANT
If the “merge repartition topics” optimization is enabled in your
configuration, which is the default, upgrading to ksqlDB 8.0.0 may be
unsafe, due to a known bug in `KStream.processValues()`. For more
information, see [KAFKA-19668](https://issues.apache.org/jira/browse/KAFKA-19668).

If you are using this optimization, do not upgrade to ksqlDB 8.0.0.
Instead, upgrade directly to ksqlDB 8.0.1, which includes the fix for this issue.

### Migrate to Log4j2

Starting in Confluent Platform 8.0.x, the Kafka Log4j appender is deprecated, which means
Log4j properties that use the Kafka Log4j appender won’t work in ksqlDB
8.0.x and later.

To fix this, you need to update the `log4j.properties` file to use the new
`log4j2.yaml` file.

For a seamless upgrade, migrate your Log4j configuration *before* you upgrade
ksqlDB to 8.0.x, which means the new configs should be ready and prepared
to use on ksqlDB 8.0.x  directly. Log4j2 configs do not work with older
ksqlDB versions.

If you are migrating to Confluent Platform 8.0.x from earlier versions, you may want to use
existing Log4j configs, but this is not recommended. If you choose to do this,
you must update your Log4j properties in such a way that ksqlDB server can
start.

In this case, you must make the following changes:

- Ensure that the Log4j properties file has no references to the Kafka Log4j
  appender. If you want to use the Processing Log, you can still use it, but
  you must use another appender, like file appender. Not using the Kafka
  appender means processing-error messages are not pushed to the Processing Log
  topic.
- Ensure that there are no references to loggers whose classes are not in the
  class path. For example, in the
  [properties file in the ksqlDB repo for version 7.9.x](https://github.com/confluentinc/ksql/blob/7.9.x/config/log4j-rolling.properties),
  you must remove the `connect` logger and `kafka` client logger.

The following example Log4j configuration is backward compatible and works
with ksqlDB 8.0.x and later. When Log4j configs are used beyond ksqlDB
8.0.x, a warning message is displayed, indicating that some functionality
may not work, and you should use Log4j2 configs instead.

<details id="target-details">
<summary id="target-summary" style="display: list-item; cursor:pointer; color:#337ab7;">
  Example backward-compatible Log4j1 configuration
</summary>
```properties
log4j.rootLogger=INFO, main

# appenders
log4j.appender.main=org.apache.log4j.RollingFileAppender
log4j.appender.main.File=${ksql.log.dir}/ksql.log
log4j.appender.main.layout=org.apache.log4j.PatternLayout
log4j.appender.main.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.main.MaxFileSize=10MB
log4j.appender.main.MaxBackupIndex=5
log4j.appender.main.append=true

log4j.appender.streams=org.apache.log4j.RollingFileAppender
log4j.appender.streams.File=${ksql.log.dir}/ksql-streams.log
log4j.appender.streams.layout=org.apache.log4j.PatternLayout
log4j.appender.streams.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.appender.kafka=org.apache.log4j.RollingFileAppender
log4j.appender.kafka.File=${ksql.log.dir}/ksql-kafka.log
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout
log4j.appender.kafka.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.kafka.MaxFileSize=10MB
log4j.appender.kafka.MaxBackupIndex=5
log4j.appender.kafka.append=true

# loggers

log4j.logger.org.reflections=ERROR, main

# Uncomment the following line to stop KSQL from logging out each request it receives:
#log4j.logger.io.confluent.ksql.rest.server.resources.KsqlResource=WARN

# And this one to avoid the logs being spammed with KsqlConfig values.
# Though these can be useful for debugging / investigations.
#log4j.logger.io.confluent.ksql.util.KsqlConfig=WARN

## Kafka Streams logs:
log4j.logger.org.apache.kafka.streams=INFO, streams
log4j.additivity.org.apache.kafka.streams=false

## Other Kafka logs:
log4j.logger.kafka=WARN, kafka
log4j.additivity.kafka=false

log4j.logger.org.apache.zookeeper=WARN, kafka
log4j.additivity.org.apache.zookeeper=false

log4j.logger.org.apache.kafka=WARN, kafka
log4j.additivity.org.apache.kafka=false

log4j.logger.org.I0Itec.zkclient=WARN, kafka
log4j.additivity.org.I0Itec.zkclient=false

# To achieve high throughput on pull queries, avoid logging every request from Jetty
log4j.logger.io.confluent.rest-utils.requests=WARN
```

</details>
<br/>

For a working example, see the
[log4j-rolling.properties file](https://github.com/confluentinc/ksql/blob/8.0.x/config/log4j-rolling.properties)
in the ksqlDB repo.

### Migrate to Log4j2 KafkaAppender

In ksqlDB 8.0.x and later, the Log4j2 `KafkaAppender` is used for
processing logs. To get DEBUG or INFO output from ksqlDB Server, configure a
Kafka appender for the server logs. Assign the following configuration settings
in the ksqlDB Server config file. For example:

```yaml
Appenders:
  Kafka:
    name: kafka_appender
    topic: KSQL_LOG
    syncSend: true
    ignoreExceptions: false
    JsonLayout:
      Property:
        - name: schemas.enable
          value: false
    Property:
      - name: bootstrap.servers
        value: localhost:9092
Loggers:
  Logger:
    name: processing
    level: info
    AppenderRef:
      - ref: kafka_appender
```

For a complete list of changes, see the
[ksqlDB pull requests](https://github.com/confluentinc/ksql/pulls?q=is%3Apr+is%3Aclosed).

### Move OAuth callback handlers

In Kafka 4.0 and Confluent Platform 8.0.x, the OAuth callback handler classes were moved
from the deprecated `org.apache.kafka.common.security.oauthbearer.secured`
package to `org.apache.kafka.common.security.oauthbearer`.

If you configured OAuth callback handlers in your ksqlDB configuration,
update the class names in your `ksql-server.properties` file before upgrading to Confluent Platform 8.0.x:

- Change `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler`
  to `org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler`.
- Change `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler`
  to `org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler`.
- The old command topic configuration migrates automatically at runtime, and an upgrade
  does not impact resources created with the old configuration.

<a id="manual-jmx-reporter"></a>

### Manually configure the JMX reporter

Starting in Confluent Platform 8.0.x, the JMX reporter is no longer enabled by default.
If you want to instrument JMX metrics, you should configure the JMX reporter manually.

```none
ksql.streams.metric.reporters=org.apache.kafka.common.metrics.JmxReporter
```

## Upgrading to ksqlDB 7.4 from ksqlDB 7.3

- For ksqlDB RBAC, the CREATE STREAM and CREATE TABLE statements now
  require the `ResourceOwner` role for source topics, instead of
  `DeveloperRead`.
- All DDL statements that require permissions on a source topic must have the
  `ResourceOwner` role. For more information, see [Grant a user access to resources](../security/authorization/rbac/ksql-rbac.md#config-security-ksql-grant-access).

## Upgrading to ksqlDB 7.3 from ksqlDB 7.2

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0282-2022-08-02).

## Upgrading to ksqlDB 7.2 from ksqlDB 7.1

Version 7.2 [fixes](https://github.com/confluentinc/ksql/pull/8803) an issue
that occurred during upgrades from version 6.1 in which the upgrade may fail if
DROP statements exist in the command topic for objects that are referenced by
other objects.

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0260-2022-04-28).

### Breaking changes

- When creating connectors through ksqlDB, ksqlDB no longer sets the
  `key.converter` config to `StringConverter` automatically because
  ksqlDB has supported key types other than strings for a number of
  releases.

## Upgrading to ksqlDB 7.1 from ksqlDB 7.0

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0231-2021-12-14).

#### IMPORTANT
If you are using Protobuf-wrapped primitive type structs in ksqlDB,
for example `google.protobuf.StringValue` and similar types, do not
upgrade to Confluent Platform 7.1.0 or 7.1.1. A fix is provided in 7.1.2.

## Upgrading to ksqlDB 7.0 from ksqlDB 6.2

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0210-2021-09-15).

## Upgrading to ksqlDB 6.2 from ksqlDB 6.1

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0170-2021-04-26).

#### NOTE
Queries written for ksqlDB 6.1 continue to run without changes under
ksqlDB 6.2. New queries issued on 6.2 may have different behavior, even if
the text of the query statement is the same.

### Breaking changes

- Queries with GROUP BY clauses that contain multiple grouping expressions now
  result in multiple key columns, one for each grouping expression, rather than
  a single key column that is the string-concatenation of the grouping expressions.
  Note that this new behavior (and breaking change) apply only to new queries;
  existing queries will continue to run uninterrupted with the previous behavior,
  even across ksqlDB server upgrades.
- Stream-table key-to-key joins on mismatched formats now repartition the  table
  (right hand side) instead of the stream. Old enqueued commands are not affected,
  so this change should remain invisible to the end-user.
- Upgrading to version 6.2 and later versions may fail if a DROP statement
  exists in the command topic for an object that is referenced by other objects.
  This issue was introduced in [PR 6143](https://github.com/confluentinc/ksql/pull/6143).

## Upgrading to ksqlDB 6.1 from ksqlDB 6.0

For a complete list of changes, see the
[ksqlDB changelog](https://github.com/confluentinc/ksql/blob/master/CHANGELOG.md#0140-2020-10-28).

#### NOTE
Queries written for ksqlDB 6.0 continue to run without changes under
ksqlDB 6.1. New queries issued on 6.1 may have different behavior, even if
the text of the query statement is the same.

### Breaking changes

* ksqlDB now creates windowed tables with cleanup policy `compact,delete`,
  rather than `compact`. Also, topics that back streams are always created with
  cleanup policy `delete`, rather than the broker default (by default, `delete`).
* ksqlDB no longer emits unnecessary tombstones when a `HAVING` clause filters out a row
  from the source that is not in the output table. For example, given:
  ```sql
  -- source stream:
  CREATE STREAM FOO (ID INT KEY, VAL INT) WITH (...);
  -- aggregate into a table:
  CREATE TABLE BAR AS
  SELECT ID, SUM(VAL) AS SUM
  FROM FOO
  GROUP BY ID
  HAVING SUM(VAL) > 0;
  -- insert some values into the stream:
  INSERT INTO FOO VALUES(1, -5);
  INSERT INTO FOO VALUES(1, 6);
  INSERT INTO FOO VALUES(1, -2);
  INSERT INTO FOO VALUES(1, -1);
  ```

  Where previously the contents of the sink topic `BAR` would have contained records:

  | Key   | Value     | Notes                                                                                           |
  |-------|-----------|-------------------------------------------------------------------------------------------------|
  | `1`   | `null`    | Spurious tombstone: the table does not contain a row with key `1`, so no tombstone is required. |
  | `1`   | `{sum=1}` | Row added as HAVING criteria now met                                                            |
  | `1`   | `null`    | Row deleted as HAVING criteria now not met                                                      |
  | `1`   | `null`    | Spurious tombstone: the table does not contain a row with key `1`, so no tombstone is required. |

  The topic will now contain:

  | Key   | Value     |
  |-------|-----------|
  | `1`   | `{sum=1}` |
  | `1`   | `null`    |

## Downgrade considerations

If you try to downgrade ksqlDB to an earlier version, you may see the
following internal RocksDB error:

```none
Caused by: org.rocksdb.RocksDBException: Unknown Footer version. Maybe
this file was created with newer version of RocksDB?
```

This error indicates that a change in RocksDB version can prevent rolling back
ksqlDB to earlier versions.

As a solution, ensure that the RocksDB version is consistent across your
environment. Also, you may need to update the existing data to ensure
compatibility with the current RocksDB version. This may involve steps like
exporting the data before the version change and importing it after the change.
Also, you may be able to delete the state store so ksqlDB recreates the state
store from the changelog topic.