Kafka Producer for Confluent Platform

An Apache Kafka® Producer is a client application that publishes (writes) events to a Kafka cluster. This section gives an overview of the Kafka producer and an introduction to the configuration settings for tuning.

A producer partitioner maps each message to a topic partition. The partitioners shipped with Kafka send all messages with the same non-empty key to the same partition.

The producer sends each produce request to the leader of that partition. Unlike a consumer, a producer needs no group coordination.

Important

Setting the partition field explicitly in a ProducerRecord overrides the default partitioner behavior.

The partitioner assigns partitions as follows:

Key provided: The partitioner hashes the key with the murmur2 algorithm and divides it by the number of partitions. The same key always maps to the same partition.
No key: The partitioner assigns partitions with awareness of batching, a strategy called sticky partitioning. Records go to the same partition as the prior record until the batch is full or sent. Partitions for newly created batches are assigned randomly.

For more information, see the Kafka Improvement Proposal (KIP) KIP-480: Sticky Partitioner and the related Confluent blog post.

The partition leader is the broker that handles all writes for a partition. In-sync replicas (ISR) are the replicas that are fully caught up with the leader. Each partition in the Kafka cluster has a leader and a set of replicas among the brokers. All writes to the partition must go through the partition leader. The replicas are kept in sync by fetching from the leader. When the leader shuts down or fails, the next leader is chosen from among the in-sync replicas. Depending on how the producer is configured, each produce request to the partition leader can be held until the replicas have successfully acknowledged the write. This gives the producer some control over message durability at some cost to overall throughput.

A message is committed when all in-sync replicas have acknowledged the write. Consumers can read only committed messages, regardless of the producer’s acknowledgement settings. This ensures that a broker failure cannot lose a message after a consumer has read it.

Messages that only the leader acknowledged (acks=1) can be lost if the partition leader fails before replicas copy them. This is often a reasonable compromise because it ensures durability in most cases without significantly reducing throughput.

Producer tuning focuses on two goals: high throughput through batching and compression, and message delivery guarantees through acknowledgements and retries. The following sections describe the settings that control each goal.

Kafka producer configuration

The key Kafka producer settings are bootstrap.servers, acks, retries, batch.size, linger.ms, compression.type, and buffer.memory. For the full list, see Kafka Producer Configurations.

Core producer settings

These settings are the same for Java, C/C++, Python, Go, and .NET clients.

bootstrap.servers: Set this property so that the producer can find the Kafka cluster.
client.id: Optional, but you should set this property on each instance because it helps you correlate requests on the broker with the client instance which made it, which can be helpful in debugging and troubleshooting scenarios. You can also use client IDs to enforce client quotas. For more information, see client quotas.

Configure message durability with acks

The acks setting controls message durability. The default is all.

Value	Guarantee	Throughput
`all` (default)	The partition leader and all in-sync replicas acknowledge the write.	Lowest
`1`	Only the partition leader acknowledges the write.	Medium
`0`	No acknowledgement. The broker sends no response, so you cannot confirm the write or determine the message offset.	Highest

For the C/C++, Python, Go, and .NET clients, acks is a per-topic setting. Apply it globally with default_topic_conf (C/C++) or default.topic.config (Python, Go, .NET).

Message ordering settings

In general, messages are written to the broker in the same order that they are received by the producer client. However, this can change depending on some settings.

retries: This setting enables message retries when set to a value larger than 0 (which is the default). If retries are enabled, message reordering can occur because the retry might occur after a later write succeeded.
max.in.flight.requests.per.connection: Set to 1 to enable retries without reordering. This limits the producer to one in-flight request per broker connection. Without retries enabled, the broker preserves the order of writes it receives, but individual send failures can leave gaps.

Configure batching and compression

Kafka producers attempt to collect sent messages into batches to improve throughput.

batch.size: Use with the Java client to control the maximum size in bytes of each message batch.
linger.ms: Use this setting to give more time for batches to fill by delaying the producer sending.
compression.type: Enable compression. Compression covers full message batches, so larger batches typically give a higher compression ratio.
batch.num.messages: Use with the C/C++, Python, Go, and .NET clients to set a limit on the number of messages contained in each batch.

The snappy codec requires write access to the /tmp directory. If /tmp is mounted noexec, set a writable directory for snappy with:

-Dorg.xerial.snappy.tempdir=/path/to/newtmp

Limit memory for unsent messages

buffer.memory: Limits the total memory that is available to the Java client for collecting unsent messages. When this limit is hit, the producer blocks on additional sends for as long as max.block.ms before it raises an exception.
request.timeout.ms: Set a timeout to avoid keeping records queued indefinitely. If this timeout expires before a message can be successfully sent, the producer removes it from the queue and throws an exception. The C/C++, Python, Go, and .NET clients have similar settings.

Kafka producer examples

Example Kafka producer code is available in Java, Python, Go, .NET, JavaScript Client, C/C++, REST, and Spring Boot.

For a tutorial on how to build a Kafka producer that can write records to Confluent Cloud or Confluent Platform, see How to build your first Apache KafkaProducer application.
For producer examples, see Build Consumer, use the language selector to choose a language, and click Build Producer to see example producer code.