Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

kafkacat Utility

kafkacat is a command line utility that you can use to test and debug Apache Kafka deployments. You can use kafkacat to produce, consume, and list topic and partition information for Kafka. Described as “netcat for Kafka”, it is a swiss-army knife of tools for inspecting and creating data in Kafka.

It is similar to Kafka Console Producer (kafka-console-producer) and Kafka Console Consumer (kafka-console-consumer), but even more powerful.

Important

kafkacat is an open-source utility, available at https://github.com/edenhill/kafkacat. It is not supported by Confluent and is not included in Confluent Platform.

Consumer Mode

In consumer mode, kafkacat reads messages from a topic and partition and prints them to standard output (stdout). You must specify a Kafka broker (-b) and topic (-t). You can optionally specify delimiter (-D). The default delimiter is newline.

You can supply kafkacat with a broker (-b) and a topic (-t) and view see its contents:

$ kafkacat -b localhost:9092 -t mysql_users
% Auto-selecting Consumer mode (use -P or -C to override)
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
{"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
[...]

kafkacat automatically selects its mode depending on the terminal or pipe type.

• If data is being piped to kafkacat it will automatically select producer (-P) mode.
• If data is being piped from kafkacat (e.g. standard terminal output) it will automatically select consumer (-C) mode.

You can explicitly specify mode by using the consumer (-C) or producer (-P) flag. You can also specify how many messages you want to see with the lowercase mode identifier and a number (e.g. -c<num>). For example, to consume a single message:

$ kafkacat -b localhost:9092 -t mysql_users -C -c1
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

You can view the message key by using the -K argument with a delimiter. For example, to view the message key with a tab delimiter:

$ kafkacat -b localhost:9092 -t mysql_users -C -c1 -K\t
1   {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

The -f flag takes arguments specifying both the format of the output and the fields to include. Here’s a simple example of pretty-printing the key and value pairs for each message:

$ kafkacat -b localhost:9092 -t mysql_users -C -c1 -f 'Key: %k\nValue: %s\n'
Key: 1
Value: {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}

Note that the -K: was replaced because the key parameter is specified in the -f format string now.

A more advanced use of -f would be to show even more metadata - offsets, timestamps, and even data lengths:

$ kafkacat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'

Key (1 bytes): 1
Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Timestamp: 1520618381093        Partition: 0    Offset: 0
--

Key (1 bytes): 2
Value (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
Timestamp: 1520618381093        Partition: 0    Offset: 1
--

Producer Mode

In producer mode, kafkacat reads messages from standard input (stdin). You must specify a Kafka broker (-b) and topic (-t). You can optionally specify a delimiter (-D). The default delimiter is newline.

You can easily send data to a topic using kafkacat. Run it with the -P command and enter the data you want, and then press Ctrl-D to finish:

$ kafkacat -b localhost:9092 -t new_topic -P
test

Replay it (replace -P with -C) to verify:

$ kafkacat -b localhost:9092 -t new_topic -C
test

You can send data to kafkacat by adding data from a file. The following example treats each line of file /tmp/msgs as an individual message by using the -l flag. Without the -l flag, the entire file is treated as its own message. This is useful for sending binary data. This example also uses the -T flag to also echo the input to stdout.

$ kafkacat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs
These are
three messages
sent through kafkacat

You can specify the key for messages, using the same -K parameter plus delimiter character that was used for the previous consumer example:

$ kafkacat -b localhost:9092 -t keyed_topic -P -K:
1:foo
2:bar

$ kafkacat -b localhost:9092 -t keyed_topic -C -f 'Key: %k\nValue: %s\n'
Key: 1
Value: foo
Key: 2
Value: bar

You can set the partition:

$ kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 1
1:foo
$ kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 2
2:bar
$ kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 3
3:wibble

Replay, using the format and -f field as above:

$ kafkacat -b localhost:9092 -t partitioned_topic -C -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
% Reached end of topic partitioned_topic [0] at offset 0

Key (1 bytes): 1
Value (3 bytes): foo
Timestamp: 1520620113485        Partition: 1    Offset: 0
--

Key (1 bytes): 2
Value (3 bytes): bar
Timestamp: 1520620121165        Partition: 2    Offset: 0
--

Key (1 bytes): 3
Value (6 bytes): wibble
Timestamp: 1520620129112        Partition: 3    Offset: 0
--

Metadata Listing Mode

In metadata list mode (-L), kafkacat displays the current state of the Kafka cluster and its topics, partitions, replicas and in-sync replicas (ISR).

$ kafkacat -b localhost:9092 -L

Add the JSON (-J) option to have the output emitted as JSON. This can be useful when passing this data to other applications for further processing.

$ kafkacat -b mybroker -L -J

For more information and examples, see the kafkacat GitHub repository.