Topic Operations

Topics are the fundamental unit of organization in Kafka. Adding, modifying and deleting topics are operations you will perform on a regular basis. This topic describes how to use the Kafka command line tools to perform these operations.

Add a topic

You have the option of either adding topics manually or having them be created automatically when data is first published to a non-existent topic. If topics are auto-created then you may want to tune the default topic configurations used for auto-created topics.

Topics are added and modified using the kafka-topics.sh tool found in the \bin directory.

Confluent Tip

The Confluent CLI with the kafka-topic subcommand makes it easier create, modify and delete topics using the command line. For more information, see confluent kafka topic.

./bin/kafka-topics.sh --bootstrap-server broker_host:port --create --topic my_topic_name \
    --partitions 20 --replication-factor 3 --config x=y

• The replication factor specifies the number of servers a message will be written to. You should use a replication factor of 2 or 3 so that you can transparently bounce machines without interrupting data consumption. For example, if you specify a replication factor of 3 then up to 2 servers can fail before you will lose access to the data.
• The partition count controls how many event logs the topic are sharded into. Partition count has several impacts. First, each partition must fit entirely on a single server. So, if you have 20 partitions, the full data set, and read and write load, will be handled by no more than 20 servers (not counting replicas). Finally, the partition count impacts the maximum parallelism of your consumers. To learn more about partition count, see Kafka Replication and Committed Messages and Choose and Change the Partition Count.

When you specify topic configurations using the kafka-topics tool, these values override the broker’s default configuration settings, such as data retention time. The complete set of per-topic configurations is documented in topic level configurations.

Modify a topic

To modify the configuration settings for a topic, you can use the kafka-configs.sh tool, which can also be found in the /bin directory.

./bin/kafka-configs.sh --bootstrap-server broker_host:port --entity-type topics --entity-default --alter --add-config delete.retention.ms=172800000

You can also use this command to increase the nubmer of partitions of a topic. Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn’t change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is, if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.

Delete a topic

To delete a topic from a cluster, you can run the following command:

./bin/kafka-topics.sh --bootstrap-server broker_host:port --delete --topic my_topic_name

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.