# Kafka Topic Operations Apache Kafka® is an open-source distributed streaming system used for stream processing, real-time data pipelines, and data integration at scale. Originally created to handle real-time data feeds at LinkedIn in 2011, Kafka quickly evolved from a messaging queue to a full-fledged event streaming platform, capable of handling over one million messages per second, or trillions of messages per day. A [topic](../../_glossary.md#term-topic) is the fundamental unit of organization in Kafka. Adding, modifying and deleting topics are operations you will perform on a regular basis. You can either add topics manually with the [kafka-topics.sh](kafka-tools.md#kafka-topics-usage) tool or set them up be created automatically when data is first published to a non-existent topic. You can also use this tool to describe the current state of a topic, or delete a topic. Use the [kafka-configs.sh](kafka-tools.md#kafka-configs-tool) tool to modify settings for a topic that already exists. This topic provides more information on how to use these tools. ## Add a topic You can use the [kafka-topics.sh](kafka-tools.md#kafka-topics-usage) tool to create new topics and and specify topic configurations that override the broker’s default configuration settings, such as data retention time. Following are two key configuration properties. - `replication-factor` specifies the number of servers a message will be written to. You should use a replication factor of 2 or 3 so that you can transparently bounce machines without interrupting data consumption. For example, if you specify a replication factor of 3 then up to 2 servers can fail before you will lose access to the data. - `partitions` controls how many event logs the topic are sharded into. Partition count has several impacts. First, each partition must fit entirely on a single server. So, if you have 20 partitions, the full data set, and read and write load, will be handled by no more than 20 servers (not counting replicas). Finally, the partition count impacts the maximum parallelism of your consumers. To learn more about partition count, see [Kafka Replication and Committed Messages](../design/replication.md#replication) and [Choose and Change the Partition Count in Kafka](partition-determination.md#partition-determination). The complete set of per-topic configurations is documented in [topic level configurations](/platform/current/installation/configuration/topic-configs.html). The following command shows how to create a topic, specifying the partitions, replication factor, and an additional configuration value. ```shell bin/kafka-topics.sh --bootstrap-server --create --topic \ --partitions 20 --replication-factor 3 --config = ``` ## Describe a topic You can describe a topic using the [kafka-topics.sh](kafka-tools.md#kafka-topics-usage) tool. The following command shows how to describe a topic: ```shell bin/kafka-topics.sh --bootstrap-server --describe --topic ``` The tool output lists the configuration properties and values for the topic. ## Change the retention value for a topic If topics are auto-created you may want to tune the default topic configurations applied to them. To modify most configuration settings for a topic, you can use the [kafka-configs.sh](kafka-tools.md#kafka-configs-tool) tool, which can be found in the `/bin` directory. You can change how the data in a topic is retained by changing the following topic properties: - `retention.ms` - Sets retention by time - `retention.bytes` - Sets retention by size The following example shows how to change topic retention. This command sets the retention size to 500 MB, and the retention time to -1, meaning infinite. ```shell bin/kafka-configs.sh --bootstrap-server --entity-type topics --entity-name --alter --add-config retention.ms=-1,retention.bytes=524288000 ``` ## Increase partitions for a topic To increase partitions for a topic, you can use the `kafka-topics` tool. Remember that partitions can be used to semantically partition data, and adding partitions doesn’t change the partitioning of existing data. This means that when you add partitions, you may disturb consumers that rely on a particular partition. For example, if data is partitioned by `hash(key) % number_of_partitions`, partitioning will be shuffled when partitions are added. ```shell bin/kafka-topics.sh --bootstrap-server --topic --alter --partitions ``` ## Delete a topic Use the [kafka-topics.sh](kafka-tools.md#kafka-topics-usage) tool to delete a topic from a cluster. The following command shows how to delete a topic. ```shell bin/kafka-topics.sh --bootstrap-server --delete --topic ``` ## Related content - [Kafka Command-Line Interface (CLI) Tools](kafka-tools.md#kafka-cli-tools) - [Kafka Producer Design](../design/producer-design.md#producer-design) - [Kafka Consumer Design: Consumers, Consumer Groups, and Offsets](../design/consumer-design.md#consumer-design) - [Kafka Message Delivery Guarantees](../design/delivery-semantics.md#delivery-semantics) - [Kafka Log Compaction](../design/log_compaction.md#log-compaction) - [Kafka Replication and Committed Messages](../design/replication.md#replication) - [Kafka Quotas](../design/quotas.md#quotas) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).