Migrate Kafka Clusters with the kcp Tool

kcp is a command-line interface (CLI) tool that you can use to migrate from existing Apache Kafka® cluster deployments to Confluent Cloud. kcp provides a set of features that help you analyze, plan, and execute migrations efficiently.

This topic provides an overview of kcp, its key features, and how to get started with the tool.

Note

Currently, kcp provides information on migrations from Amazon Web Services (AWS) Managed Streaming for Kafka (MSK) clusters to Confluent Cloud.

Cluster availability

To get started, determine the availability of your existing Kafka clusters.

  • If your clusters are available on the internet: You can run kcp commands directly from your local computer.
  • If your clusters are within a private network: You must use an existing bastion host or jump server that resides in the same VPC as your existing MSK clusters to run kcp commands. If you don’t have a bastion host, you can run kcp with the kcp create-asset flag to create one as part of the migration assets.

Installation

To install kcp, you download the latest release from the Releases page of the kcp repository on Github. Detailed installation steps follow.

Prerequisites:
  • Linux and macOS (Darwin) with ARM64 and AMD64 architectures.
  • Access to the internet.

To install the tool:

  1. Download the latest binary for your system. For example, the following command shows how you can download kcp version 0.4.1 for Linux AMD64:

    wget https://github.com/confluentinc/kcp/releases/download/v0.4.1/kcp_linux_amd64.tar.gz

  2. Extract the archive by running a command similar to the following:

    tar -xzf kcp_linux_amd64.tar.gz

  3. Change to the kcp directory, and set permissions so the binary is executable by running chmod with the +x option and the binary name. For example:

    cd kcp
chmod +x kcp

You can then run the commands directly, or move the binary to a directory in your system’s PATH for easier access.

Authentication

To evaluate MSK clusters, you must authenticate kcp with AWS credentials. kcp supports multiple authentication methods:

  • Environment variables: Export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN.
  • AWS credentials file: Configure with aws configure (requires AWS CLI).
  • AWS SSO/Identity Center: Use aws sso login (requires AWS CLI).
  • IAM Roles: Assume roles or use instance profiles.
  • Other tools: Any tool that sets AWS credentials in the standard locations such as Granted.

kcp tool usage

The kcp tool streamlines the migration process through these main functions:

  • Environment initialization: kcp helps initialize the local environment with required configuration variables, helping to simplify running subsequent commands.
  • Scanning and discovery: kcp scans and identifies resources within your existing Kafka deployments. This includes detailed information on clusters, brokers, topics, and consumer groups.
  • Report generation: kcp creates reports that provide migration planning and cost analysis. These reports detail cluster configurations, metrics, and other critical information.
  • Asset generation: kcp generates the necessary migration assets and infrastructure configurations, which can be provisioned using tools like Terraform.

The next few sections provide some detail on how to use the kcp tool.

This tool is updated frequently, and as a result, the documentation on GitHub is the most up-to-date. Access this documentation in the docs directory for the version you have installed. For example, for kcp version 0.4.1, see kcp Documentation.

kcp command reference

The kcp tool is structured around a few central commands, each with a specific purpose.

kcp help

Use the help command to get information about kcp command usage.

kcp --help

You should see output similar to the following:

A comprehensive CLI tool for planning and executing kafka cluster migrations to confluent cloud.

Usage:
kcp [command]

Available Commands:
completion   Generate the autocompletion script for the specified shell
create-asset Generate infrastructure and migration assets
discover     Multi-region, multi cluster discovery scan of AWS MSK
help         Help about any command
init         Generate a README.md and set of env vars to export
report       Generate reports on migration planning
scan         Scan AWS resources for migration planning
update       Update the kcp binary to the latest version
version      Show version information

Flags:
-h, --help   help for kcp

Use "kcp [command] --help" for more information about a command.

kcp discover

kcp discover performs a full discovery of all MSK clusters in an AWS account across specified regions. It collects associated resources, costs, and metrics. This command requires AWS credentials with permissions to list and describe MSK clusters. Expand the dropdown to see the full list of permissions required.

Permissions required for kcp discover
{
   "Version": "2012-10-17",
   "Statement": [
   {
      "Sid": "MSKScanPermissions",
      "Effect": "Allow",
      "Action": [
      "kafka:ListClustersV2",
      "kafka:ListReplicators",
      "kafka:ListVpcConnections",
      "kafka:GetCompatibleKafkaVersions",
      "kafka:GetBootstrapBrokers",
      "kafka:ListConfigurations",
      "kafka:DescribeClusterV2",
      "kafka:ListKafkaVersions",
      "kafka:ListNodes",
      "kafka:ListClusterOperationsV2",
      "kafka:ListScramSecrets",
      "kafka:ListClientVpcConnections",
      "kafka:GetClusterPolicy",
      "kafka:DescribeConfigurationRevision",
      "kafka:DescribeReplicator",
      "kafkaconnect:ListConnectors",
      "kafkaconnect:DescribeConnector"
      ],
      "Resource": "*"
   },
   {
      "Sid": "CostMetricsScanPermissions",
      "Effect": "Allow",
      "Action": [
      "cloudwatch:GetMetricData",
      "ce:GetCostAndUsage",
      "cloudwatch:GetMetricStatistics",
      "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
   },
   {
      "Sid": "MSKNetworkingScanPermission",
      "Effect": "Allow",
      "Action": ["ec2:DescribeSubnets"],
      "Resource": "*"
   }
 ]
}

Example of running the discover command:

kcp discover --region us-east-1,eu-west-3

The output is a JSON file containing detailed information about the discovered clusters, including their status, authentication methods, VPC connections, and available Kafka versions.

kcp scan

This command performs various types of scans on MSK clusters. It can perform scans on individual clusters, multiple clusters, or all clusters within an entire region. It can also scan Kafka broker logs to identify potential clients.

kcp report

This command generates reports based on the data collected during the discover and scan phases. The reports are essential for planning your migration, with data on cluster configuration, broker information, topic metadata, and consumer group details.

kcp create-asset

The create-asset command generates infrastructure as code for your migration. It includes sub-commands for various assets:

  • bastion-host: Provisions an EC2 bastion host with pre-configured migration tools.
  • migrate-acls: Generates scripts for migrating Access Control Lists.
  • migration-infra: Creates a complete migration infrastructure.
  • migration-scripts: Generates scripts to assist with the migration process.
  • reverse-proxy: Provisions a reverse proxy EC2 instance.

Related content