kcat (formerly kafkacat) Utility for Confluent Platform

kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. You can use kcat to produce, consume, and list topic and partition information for Kafka. Described as “netcat for Kafka”, it is a swiss-army knife of tools for inspecting and creating data in Kafka.

It is similar to Kafka Console Producer (kafka-console-producer) and Kafka Console Consumer (kafka-console-consumer), but even more powerful.

Important

kcat is an open-source utility, available at https://github.com/edenhill/kcat. It is not supported by Confluent and is not included in Confluent Platform.

Consumer Mode

In consumer mode, kcat reads messages from a topic and partition and prints them to standard output (stdout). You must specify a Kafka broker (-b) and topic (-t). You can optionally specify delimiter (-D). The default delimiter is newline.

You can supply kcat with a broker (-b) and a topic (-t) and view see its contents:

 kcat -b localhost:9092 -t mysql_users
% Auto-selecting Consumer mode (use -P or -C to override)
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
{"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
[...]

kcat automatically selects its mode depending on the terminal or pipe type.

  • If data is being piped to kcat it will automatically select producer (-P) mode.
  • If data is being piped from kcat (e.g. standard terminal output) it will automatically select consumer (-C) mode.

You can explicitly specify mode by using the consumer (-C) or producer (-P) flag. You can also specify how many messages you want to see with the lowercase mode identifier and a number (e.g. -c<num>). For example, to consume a single message:

 kcat -b localhost:9092 -t mysql_users -C -c1
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

You can view the message key by using the -K argument with a delimiter. For example, to view the message key with a tab delimiter:

 kcat -b localhost:9092 -t mysql_users -C -c1 -K\t
1   {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

The -f flag takes arguments specifying both the format of the output and the fields to include. Here’s a simple example of pretty-printing the key and value pairs for each message:

 kcat -b localhost:9092 -t mysql_users -C -c1 -f 'Key: %k\nValue: %s\n'
Key: 1
Value: {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

Note that the -K: was replaced because the key parameter is specified in the -f format string now.

A more advanced use of -f would be to show even more metadata - offsets, timestamps, and even data lengths:

 kcat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'

Key (1 bytes): 1
Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Timestamp: 1520618381093        Partition: 0    Offset: 0
--

Key (1 bytes): 2
Value (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
Timestamp: 1520618381093        Partition: 0    Offset: 1
--

Producer Mode

In producer mode, kcat reads messages from standard input (stdin). You must specify a Kafka broker (-b) and topic (-t). You can optionally specify a delimiter (-D). The default delimiter is newline.

You can easily send data to a topic using kcat. Run it with the -P command and enter the data you want, and then press Ctrl-D to finish:

 kcat -b localhost:9092 -t new_topic -P
test

Replay it (replace -P with -C) to verify:

 kcat -b localhost:9092 -t new_topic -C
test

You can send data to kcat by adding data from a file. The following example treats each line of file /tmp/msgs as an individual message by using the -l flag. Without the -l flag, the entire file is treated as its own message. This is useful for sending binary data. This example also uses the -T flag to also echo the input to stdout.

 kcat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs
These are
three messages
sent through kcat

You can specify the key for messages, using the same -K parameter plus delimiter character that was used for the previous consumer example:

 kcat -b localhost:9092 -t keyed_topic -P -K:
1:foo
2:bar

 kcat -b localhost:9092 -t keyed_topic -C -f 'Key: %k\nValue: %s\n'
Key: 1
Value: foo
Key: 2
Value: bar

You can set the partition:

 kcat -b localhost:9092 -t partitioned_topic -P -K: -p 1
1:foo
 kcat -b localhost:9092 -t partitioned_topic -P -K: -p 2
2:bar
 kcat -b localhost:9092 -t partitioned_topic -P -K: -p 3
3:wibble

Replay, using the format and -f field as above:

 kcat -b localhost:9092 -t partitioned_topic -C -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
% Reached end of topic partitioned_topic [0] at offset 0

Key (1 bytes): 1
Value (3 bytes): foo
Timestamp: 1520620113485        Partition: 1    Offset: 0
--

Key (1 bytes): 2
Value (3 bytes): bar
Timestamp: 1520620121165        Partition: 2    Offset: 0
--

Key (1 bytes): 3
Value (6 bytes): wibble
Timestamp: 1520620129112        Partition: 3    Offset: 0
--

Metadata Listing Mode

In metadata list mode (-L), kcat displays the current state of the Kafka cluster and its topics, partitions, replicas and in-sync replicas (ISR).

kcat -b localhost:9092 -L

Add the JSON (-J) option to have the output emitted as JSON. This can be useful when passing this data to other applications for further processing.

kcat -b mybroker -L -J

For more information and examples, see the kcat GitHub repository.

Configure kcat to authenticate to Confluent Cloud

To configure kcat to talk to Confluent Cloud, provide the Confluent Cloud bootstrap server, the API key and secret, and the security protocol details. For example:

kcat -b pkc-12345.us-west4.gcp.confluent.cloud:9092 \
-X security.protocol=sasl_ssl -X sasl.mechanisms=PLAIN \
-X sasl.username=<api-key-key>  -X sasl.password=<api-key-secret>  \
-L

To learn more, see Configure Confluent Cloud Clients in the Confluent Cloud documentation.

kcat example code

For Hello World examples of kcat, see kcat: Command Example for Kafka. All examples include a producer and consumer that can connect to any Kafka cluster running on-premises or in Confluent Cloud.