Manage Kafka Topics for Confluent Platform Using Confluent for Kubernetes Blueprints

Confluent for Kubernetes (CFK) Blueprints allows you to declaratively create and manage Kafka topics as KafkaTopic custom resources (CRs) in Kubernetes.

Each KafkaTopic CR is mapped to a topic and kept in sync with the corresponding Kafka topic. This allows you to have a separate workflow where you create topics as part of Confluent deployment, and your client applications only need to produce and consume from the topics.

CFK Blueprints supports creating a topic on the multiple Kafka clusters.

Create a Kafka topic

1. Create a KafkaTopic CR for a new topic:

   apiVersion: apps.cpc.platform.confluent.io/v1beta1
   kind: KafkaTopic
   spec:
     name:                --- [1]
     kafkaClusterRef:     --- [2]
       - name:            --- [3]
         namespace:       --- [4]
     parameters:
       configs:           --- [5]
       partitions:        --- [6]
       replicationFactor: --- [7]
   

   • [1] The topic name. If not configured, the KafkaTopic CR name is used as the topic name.

   • [2] A list of Kafka clusters to create this topic in.

     The create or update operation fails if any one of the Kafka cluster fails to create a topic for multi Kafka cluster deployment.

   • [3] Required. The name of the KafkaCluster CR.

   • [4] The namespace of the KafkaCluster CR.

   • [5] A map of key and value pairs that are used to pass the topic configuration settings. For the list of available topic configuration settings, see Kafka Topic Configurations.

   • [6] The number of partitions for the topic. The default value is 1.

   • [7] The number of copies of data over multiple brokers. The default value is 1.

2. Apply the change:

   kubectl apply -f <topic-CR-yaml-file>
   

Get a Kafka topic

To get a topic, use the kubectl get command:

kubectl get kafkatopic.apps <existing-topic-name>

Update a Kafka topic

To update a topic, change the topic CR configuration, and apply the changes using the kubectl edit or kubectl apply command.

kubectl edit kafkatopic.app <existing-topic-name>

kubectl apply -f <topic-yaml-file>

The following example adds a cleanup policy to the above topic-a topic:

apiVersion: apps.cpc.platform.confluent.io/v1beta1
kind: KafkaTopic
spec:
  name: topic-a
    kafkaClusterRefs:
      - name: kafka-dev
  parameters:
    replicationFactor: 3
    partitions: 12
    configs:
      cleanup.policy: "compact"

Warning

spec.replicationFactor and spec.partitions cannot be updated.

Delete a Kafka topic

To delete a topic, use the kubectl delete command.

For example:

Kubectl delete kafkatopic.apps <existing-topic-name>