kcat (formerly kafkacat) Utility for Confluent Platform¶
kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. You can use kcat to produce, consume, and list topic and partition information for Kafka. Described as “netcat for Kafka”, it is a swiss-army knife of tools for inspecting and creating data in Kafka.
It is similar to Kafka Console Producer (kafka-console-producer) and Kafka Console Consumer (kafka-console-consumer),
but even more powerful.
Important
kcat is an open-source utility, available at https://github.com/edenhill/kcat. It is not supported by Confluent and is not included in Confluent Platform.
Consumer Mode¶
In consumer mode, kcat reads messages from a topic and partition and prints them to standard output (stdout). You must
specify a Kafka broker (-b) and topic (-t). You can optionally specify delimiter (-D). The default delimiter
is newline.
You can supply kcat with a broker (-b) and a topic (-t) and view see its contents:
kcat -b localhost:9092 -t mysql_users
% Auto-selecting Consumer mode (use -P or -C to override)
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
{"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
[...]
kcat automatically selects its mode depending on the terminal or pipe type.
- If data is being piped to kcat it will automatically select producer (
-P) mode. - If data is being piped from kcat (e.g. standard terminal output) it will automatically select consumer (
-C) mode.
You can explicitly specify mode by using the consumer (-C) or producer (-P) flag. You can also specify how many
messages you want to see with the lowercase mode identifier and a number (e.g. -c<num>). For example, to consume a single
message:
kcat -b localhost:9092 -t mysql_users -C -c1
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
You can view the message key by using the -K argument with a delimiter. For example, to view the message key with a
tab delimiter:
kcat -b localhost:9092 -t mysql_users -C -c1 -K\t
1 {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
The -f flag takes arguments specifying both the format of the output and the fields to include. Here’s a simple example
of pretty-printing the key and value pairs for each message:
kcat -b localhost:9092 -t mysql_users -C -c1 -f 'Key: %k\nValue: %s\n'
Key: 1
Value: {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Note that the -K: was replaced because the key parameter is specified in the -f format string now.
A more advanced use of -f would be to show even more metadata - offsets, timestamps, and even data lengths:
kcat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
Key (1 bytes): 1
Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Timestamp: 1520618381093 Partition: 0 Offset: 0
--
Key (1 bytes): 2
Value (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"}
Timestamp: 1520618381093 Partition: 0 Offset: 1
--
Producer Mode¶
In producer mode, kcat reads messages from standard input (stdin). You must specify a Kafka broker (-b) and topic
(-t). You can optionally specify a delimiter (-D). The default delimiter is newline.
You can easily send data to a topic using kcat. Run it with the -P command and enter the data you want, and
then press Ctrl-D to finish:
kcat -b localhost:9092 -t new_topic -P
test
Replay it (replace -P with -C) to verify:
kcat -b localhost:9092 -t new_topic -C
test
You can send data to kcat by adding data from a file. The following example treats each line of file /tmp/msgs as an individual
message by using the -l flag. Without the -l flag, the entire file is treated as its own message. This is
useful for sending binary data. This example also uses the -T flag to also echo the input to stdout.
kcat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs
These are
three messages
sent through kcat
You can specify the key for messages, using the same -K parameter plus delimiter character that was used for the
previous consumer example:
kcat -b localhost:9092 -t keyed_topic -P -K:
1:foo
2:bar
kcat -b localhost:9092 -t keyed_topic -C -f 'Key: %k\nValue: %s\n'
Key: 1
Value: foo
Key: 2
Value: bar
You can set the partition:
kcat -b localhost:9092 -t partitioned_topic -P -K: -p 1
1:foo
kcat -b localhost:9092 -t partitioned_topic -P -K: -p 2
2:bar
kcat -b localhost:9092 -t partitioned_topic -P -K: -p 3
3:wibble
Replay, using the format and -f field as above:
kcat -b localhost:9092 -t partitioned_topic -C -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
% Reached end of topic partitioned_topic [0] at offset 0
Key (1 bytes): 1
Value (3 bytes): foo
Timestamp: 1520620113485 Partition: 1 Offset: 0
--
Key (1 bytes): 2
Value (3 bytes): bar
Timestamp: 1520620121165 Partition: 2 Offset: 0
--
Key (1 bytes): 3
Value (6 bytes): wibble
Timestamp: 1520620129112 Partition: 3 Offset: 0
--
Metadata Listing Mode¶
In metadata list mode (-L), kcat displays the current state of the Kafka cluster and its topics, partitions, replicas
and in-sync replicas (ISR).
kcat -b localhost:9092 -L
Add the JSON (-J) option to have the output emitted as JSON. This can be useful when passing this data to other applications
for further processing.
kcat -b mybroker -L -J
For more information and examples, see the kcat GitHub repository.
Configure kcat to authenticate to Confluent Cloud¶
To configure kcat to talk to Confluent Cloud, provide the Confluent Cloud bootstrap server, the API key and secret, and the security protocol details. For example:
kcat -b pkc-12345.us-west4.gcp.confluent.cloud:9092 \
-X security.protocol=sasl_ssl -X sasl.mechanisms=PLAIN \
-X sasl.username=<api-key-key> -X sasl.password=<api-key-secret> \
-L
To learn more, see Configure Confluent Cloud Clients in the Confluent Cloud documentation.
kcat example code¶
For Hello World examples of kcat, see kcat: Command Example for Kafka. All examples include a producer and consumer that can connect to any Kafka cluster running on-premises or in Confluent Cloud.