Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

kafkacat Utility

kafkacat is a command line utility that you can use to test and debug Apache Kafka® deployments. You can use kafkacat to produce, consume, and list topic and partition information for Kafka. Described as “netcat for Kafka”, it is a swiss-army knife of tools for inspecting and creating data in Kafka.

It is similar to Kafka Console Producer (kafka-console-producer) and Kafka Console Consumer (kafka-console-consumer), but even more powerful.

Important

kafkacat is an open-source utility, available at https://github.com/edenhill/kafkacat. It is not supported by Confluent and is not included in Confluent Platform.

Consumer Mode

In consumer mode, kafkacat reads messages from a topic and partition and prints them to standard output (stdout). You must specify an Kafka broker (-b) and topic (-t). You can optionally specify delimiter (-D). The default delimiter is newline.

You can supply kafkacat with a broker (-b) and a topic (-t) and view see its contents:

 kafkacat -b localhost:9092 -t mysql_users
% Auto-selecting Consumer mode (use -P or -C to override)
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
{"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
[...]

kafkacat automatically selects its mode depending on the terminal or pipe type.

• If data is being piped to kafkacat it will automatically select producer (-P) mode.
• If data is being piped from kafkacat (e.g. standard terminal output) it will automatically select consumer (-C) mode.

You can explicitly specify mode by using the consumer (-C) or producer (-P) flag. You can also specify how many messages you want to see with the lowercase mode identifier and a number (e.g. -c<num>). For example, to consume a single message:

 kafkacat -b localhost:9092 -t mysql_users -C -c1
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

You can view the message key by using the -K argument with a delimiter. For example, to view the message key with a tab delimiter:

 kafkacat -b localhost:9092 -t mysql_users -C -c1 -K\t
1   {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

The -f flag takes arguments specifying both the format of the output and the fields to include. Here’s a simple example of pretty-printing the key and value pairs for each message:

 kafkacat -b localhost:9092 -t mysql_users -C -c1 -f 'Key: %k\nValue: %s\n'
Key: 1
Value: {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

Note that the -K: was replaced because the key parameter is specified in the -f format string now.

A more advanced use of -f would be to show even more metadata - offsets, timestamps, and even data lengths:

 kafkacat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'

Key (1 bytes): 1
Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Timestamp: 1520618381093        Partition: 0    Offset: 0
--

Key (1 bytes): 2
Value (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
Timestamp: 1520618381093        Partition: 0    Offset: 1
--

Producer Mode

In producer mode, kafkacat reads messages from standard input (stdin). You must specify an Kafka broker (-b) and topic (-t). You can optionally specify a delimiter (-D). The default delimiter is newline.

You can easily send data to a topic using kafkacat. Run it with the -P command and enter the data you want, and then press Ctrl-D to finish:

 kafkacat -b localhost:9092 -t new_topic -P
test

Replay it (replace -P with -C) to verify:

 kafkacat -b localhost:9092 -t new_topic -C
test

You can send data to kafkacat by adding data from a file. The following example treats each line of file /tmp/msgs as an individual message by using the -l flag. Without the -l flag, the entire file is treated as its own message. This is useful for sending binary data. This example also uses the -T flag to also echo the input to stdout.

 kafkacat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs
These are
three messages
sent through kafkacat

You can specify the key for messages, using the same -K parameter plus delimiter character that was used for the previous consumer example:

 kafkacat -b localhost:9092 -t keyed_topic -P -K:
1:foo
2:bar

 kafkacat -b localhost:9092 -t keyed_topic -C -f 'Key: %k\nValue: %s\n'
Key: 1
Value: foo
Key: 2
Value: bar

You can set the partition:

 kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 1
1:foo
 kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 2
2:bar
 kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 3
3:wibble

Replay, using the format and -f field as above:

 kafkacat -b localhost:9092 -t partitioned_topic -C -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
% Reached end of topic partitioned_topic [0] at offset 0

Key (1 bytes): 1
Value (3 bytes): foo
Timestamp: 1520620113485        Partition: 1    Offset: 0
--

Key (1 bytes): 2
Value (3 bytes): bar
Timestamp: 1520620121165        Partition: 2    Offset: 0
--

Key (1 bytes): 3
Value (6 bytes): wibble
Timestamp: 1520620129112        Partition: 3    Offset: 0
--

Metadata Listing Mode

In metadata list mode (-L), kafkacat displays the current state of the Kafka cluster and its topics, partitions, replicas and in-sync replicas (ISR).

kafkacat -b localhost:9092 -L

Add the JSON (-J) option to have the output emitted as JSON. This can be useful when passing this data to other applications for further processing.

kafkacat -b mybroker -L -J

For more information and examples, see the kafkacat GitHub repository.