Migrate from Confluent Server to Confluent Kafka

This page explains how to migrate a Confluent Platform cluster from the confluent-server package to the confluent-kafka package. This procedure applies to Confluent Platform versions 8.1.0 and later.

The confluent-server package is a superset of confluent-kafka that includes additional proprietary features. The migration process involves removing the metadata associated with these features.

To do this, you’ll use the kafka-metadata-recovery offboard command. This tool creates a clean metadata snapshot that is compatible with confluent-kafka. When the cluster restarts, it loads this new snapshot to complete the migration.

Prerequisites

This migration procedure applies to Confluent Platform versions 8.1.0 and later. To check your Confluent Platform version, run the following command.

confluent local services kafka version

Migration procedure

You must perform the following steps on each broker node in your cluster.

  1. Shut down the cluster. Ensure all broker processes are stopped across the entire cluster before you proceed. This prevents state changes during the migration.

  2. On a broker node, create a temporary working directory for the migration. This allows you to work with a copy of the metadata, preventing accidental modification of live data.

    1. Define a variable for the temporary working directory:

      OFFBOARD_WORKING_DIR="/tmp/offboard-wd"

    2. Set this variable to the path of your active __cluster_metadata-0 directory:

      METADATA_DIRECTORY="/var/lib/kafka/data/__cluster_metadata-0"

    3. Create the working directory and copy the current metadata into it:

      mkdir -p "$OFFBOARD_WORKING_DIR"
cp -r "$METADATA_DIRECTORY" "$OFFBOARD_WORKING_DIR/"

  3. Use the kafka-metadata-shell tool to create a fresh snapshot from the copied metadata. This command writes a new <offset-epoch>.checkpoint file inside the copied metadata directory:

    echo "write-snapshot -o $OFFBOARD_WORKING_DIR/__cluster_metadata-0" | kafka-metadata-shell -d "$OFFBOARD_WORKING_DIR/__cluster_metadata-0"

  4. Run the kafka-metadata-recovery offboard command to convert the snapshot to a format compatible with open-source Kafka.

    1. Find the latest checkpoint file created in the previous step by the kafka-metadata-shell tool:

      CP_METADATA_SNAPSHOT_FILE=$(ls -1t "$OFFBOARD_WORKING_DIR"/__cluster_metadata-0/*.checkpoint | head -n 1)

    2. Run the offboard tool to generate the converted files:

      kafka-metadata-recovery offboard \
  --checkpoint "$CP_METADATA_SNAPSHOT_FILE" \
  --out-directory "$OFFBOARD_WORKING_DIR"

    A successful run prints the location of the new files:

    AK compatible Checkpoint file location:   /tmp/offboard-wd/0-1.checkpoint
Bootstrap file location:  /tmp/offboard-wd/bootstrap.checkpoint
kraft.version:    1
Apache metadata.version:  4.1-IV1

  5. Replace the original metadata files with the new, converted files.

    Note

    This step overwrites existing metadata files. Before proceeding, ensure you have a complete backup of your original $METADATA_DIRECTORY.

    1. Identify the new checkpoint file that was created in the working directory. The following command finds the checkpoint file while excluding the bootstrap.checkpoint file:

      OFFBOARDED_SNAPSHOT_FILE=$(find "$OFFBOARD_WORKING_DIR" -name '*.checkpoint' ! -name 'bootstrap.checkpoint')

    1. Copy the new bootstrap file to the parent of the original metadata directory:

      cp "$OFFBOARD_WORKING_DIR/bootstrap.checkpoint" "$(dirname "$METADATA_DIRECTORY")/"

    1. Copy the new data checkpoint file into the original metadata directory:

      cp "$OFFBOARDED_SNAPSHOT_FILE" "$METADATA_DIRECTORY/"

  6. Complete the following steps to finish the migration:

    • Repeat steps 2 through 5 on every other broker node in the cluster.
    • After all broker nodes are updated, restart the cluster using the confluent-kafka package. The cluster now boots using the converted metadata.

Related content