Migrate from Confluent Server to Confluent Kafka

To migrate a Confluent Platform cluster from the confluent-server package to the confluent-kafka package, shut down the cluster, convert the cluster metadata on each broker using the kafka-metadata-shell and kafka-metadata-recovery tools, then restart the cluster with the confluent-kafka package. The conversion strips out metadata for confluent-server proprietary features so the cluster can run on the open source confluent-kafka package.

Prerequisites

This migration procedure applies to Confluent Platform versions 8.1.0 and later. To check your Confluent Platform version, run the following command.

confluent local services kafka version

Migration procedure

Perform the following steps on each broker node in your cluster.

  1. Shut down the cluster. Ensure all broker processes are stopped across the entire cluster before you proceed. This prevents state changes during the migration.

  2. On a broker node, create a temporary working directory for the migration. This allows you to work with a copy of the metadata, preventing accidental modification of live data.

    1. Define a variable for the temporary working directory:

      OFFBOARD_WORKING_DIR="/tmp/offboard-wd"

    2. Set this variable to the path of your active __cluster_metadata-0 directory:

      METADATA_DIRECTORY="/var/lib/kafka/data/__cluster_metadata-0"

    3. Create the working directory and copy the current metadata into it:

      mkdir -p "$OFFBOARD_WORKING_DIR"
cp -r "$METADATA_DIRECTORY" "$OFFBOARD_WORKING_DIR/"

  3. Use the kafka-metadata-shell tool to create a fresh snapshot from the copied metadata. This command writes a new <offset-epoch>.checkpoint file inside the copied metadata directory:

    echo "write-snapshot -o $OFFBOARD_WORKING_DIR/__cluster_metadata-0" | kafka-metadata-shell -d "$OFFBOARD_WORKING_DIR/__cluster_metadata-0"

  4. Run the kafka-metadata-recovery offboard command to convert the snapshot to a format compatible with open source Kafka.

    1. Find the latest checkpoint file created in the previous step by the kafka-metadata-shell tool:

      CP_METADATA_SNAPSHOT_FILE=$(ls -1t "$OFFBOARD_WORKING_DIR"/__cluster_metadata-0/*.checkpoint | head -n 1)

    2. Run the offboard tool to generate the converted files:

      kafka-metadata-recovery offboard \
  --checkpoint "$CP_METADATA_SNAPSHOT_FILE" \
  --out-directory "$OFFBOARD_WORKING_DIR"

    A successful run prints the location of the new files:

    AK compatible Checkpoint file location:   /tmp/offboard-wd/0-1.checkpoint
Bootstrap file location:  /tmp/offboard-wd/bootstrap.checkpoint
kraft.version:    1
Apache metadata.version:  4.1-IV1

  5. Replace the original metadata files with the new, converted files.

    Note

    This step overwrites existing metadata files. Before proceeding, ensure you have a complete backup of your original $METADATA_DIRECTORY.

    1. Identify the new checkpoint file that was created in the working directory. The following command finds the checkpoint file while excluding the bootstrap.checkpoint file:

      OFFBOARDED_SNAPSHOT_FILE=$(find "$OFFBOARD_WORKING_DIR" -name '*.checkpoint' ! -name 'bootstrap.checkpoint')

    2. Copy the new bootstrap file to the parent of the original metadata directory:

      cp "$OFFBOARD_WORKING_DIR/bootstrap.checkpoint" "$(dirname "$METADATA_DIRECTORY")/"

    3. Copy the new data checkpoint file into the original metadata directory:

      cp "$OFFBOARDED_SNAPSHOT_FILE" "$METADATA_DIRECTORY/"

  6. Complete the following steps to finish the migration:

    1. Repeat steps two through five on every other broker node in the cluster.

    2. After all broker nodes are updated, restart the cluster using the confluent-kafka package. The cluster now boots using the converted metadata.

Related content