.. _kafkacat-usage: |kcat| Utility ============== |kcat| is a command line utility that you can use to test and debug |ak-tm| deployments. You can use |kcat| to produce, consume, and list topic and partition information for |ak|. Described as "netcat for |ak|", it is a swiss-army knife of tools for inspecting and creating data in |ak|. It is similar to |ak| Console Producer (``kafka-console-producer``) and |ak| Console Consumer (``kafka-console-consumer``), but even more powerful. .. important:: |kcat| is an open-source utility, available at https://github.com/edenhill/kafkacat. It is not supported by Confluent and is not included in |cp|. Consumer Mode ------------- In consumer mode, |kcat| reads messages from a topic and partition and prints them to standard output (stdout). You must specify a |ak| broker (``-b``) and topic (``-t``). You can optionally specify delimiter (``-D``). The default delimiter is newline. You can supply |kcat| with a broker (``-b``) and a topic (``-t``) and view see its contents: .. code:: shell kafkacat -b localhost:9092 -t mysql_users % Auto-selecting Consumer mode (use -P or -C to override) {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"} {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"} [...] .. tip: |kcat| doesn't currently support Avro. |kcat| automatically selects its mode depending on the terminal or pipe type. - If data is being piped to |kcat| it will automatically select producer (``-P``) mode. - If data is being piped from |kcat| (e.g. standard terminal output) it will automatically select consumer (``-C``) mode. You can explicitly specify mode by using the consumer (``-C``) or producer (``-P``) flag. You can also specify how many messages you want to see with the lowercase mode identifier and a number (e.g. ``-c``). For example, to consume a single message: .. code:: shell kafkacat -b localhost:9092 -t mysql_users -C -c1 {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"} You can view the message key by using the ``-K`` argument with a delimiter. For example, to view the message key with a tab delimiter: .. code:: shell kafkacat -b localhost:9092 -t mysql_users -C -c1 -K\t 1 {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"} The ``-f`` flag takes arguments specifying both the format of the output and the fields to include. Here's a simple example of pretty-printing the key and value pairs for each message: .. code:: shell kafkacat -b localhost:9092 -t mysql_users -C -c1 -f 'Key: %k\nValue: %s\n' Key: 1 Value: {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"} Note that the ``-K:`` was replaced because the key parameter is specified in the ``-f`` format string now. A more advanced use of ``-f`` would be to show even more metadata - offsets, timestamps, and even data lengths: .. code:: shell kafkacat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n' Key (1 bytes): 1 Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"} Timestamp: 1520618381093 Partition: 0 Offset: 0 -- Key (1 bytes): 2 Value (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"} Timestamp: 1520618381093 Partition: 0 Offset: 1 -- Producer Mode ------------- In producer mode, |kcat| reads messages from standard input (stdin). You must specify a |ak| broker (``-b``) and topic (``-t``). You can optionally specify a delimiter (``-D``). The default delimiter is newline. You can easily send data to a topic using |kcat|. Run it with the ``-P`` command and enter the data you want, and then press ``Ctrl-D`` to finish: .. code:: shell kafkacat -b localhost:9092 -t new_topic -P test Replay it (replace ``-P`` with ``-C``) to verify: .. code:: shell kafkacat -b localhost:9092 -t new_topic -C test You can send data to |kcat| by adding data from a file. The following example treats each line of file ``/tmp/msgs`` as an individual message by using the ``-l`` flag. Without the ``-l`` flag, the entire file is treated as its own message. This is useful for sending binary data. This example also uses the ``-T`` flag to also echo the input to ``stdout``. .. code:: shell kafkacat -b localhost:9092 -t -T -P -l /tmp/msgs These are three messages sent through kafkacat You can specify the key for messages, using the same ``-K`` parameter plus delimiter character that was used for the previous consumer example: .. code:: shell kafkacat -b localhost:9092 -t keyed_topic -P -K: 1:foo 2:bar kafkacat -b localhost:9092 -t keyed_topic -C -f 'Key: %k\nValue: %s\n' Key: 1 Value: foo Key: 2 Value: bar You can set the partition: .. code:: shell kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 1 1:foo kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 2 2:bar kafkacat -b localhost:9092 -t partitioned_topic -P -K: -p 3 3:wibble Replay, using the format and ``-f`` field as above: .. code:: shell kafkacat -b localhost:9092 -t partitioned_topic -C -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n' % Reached end of topic partitioned_topic [0] at offset 0 Key (1 bytes): 1 Value (3 bytes): foo Timestamp: 1520620113485 Partition: 1 Offset: 0 -- Key (1 bytes): 2 Value (3 bytes): bar Timestamp: 1520620121165 Partition: 2 Offset: 0 -- Key (1 bytes): 3 Value (6 bytes): wibble Timestamp: 1520620129112 Partition: 3 Offset: 0 -- Metadata Listing Mode --------------------- In metadata list mode (``-L``), |kcat| displays the current state of the |ak| cluster and its topics, partitions, replicas and in-sync replicas (ISR). .. code:: bash kafkacat -b localhost:9092 -L Add the JSON (``-J``) option to have the output emitted as JSON. This can be useful when passing this data to other applications for further processing. .. code:: bash kafkacat -b mybroker -L -J For more information and examples, see the `kafkacat GitHub repository `_.