Kafka Producer for Confluent Platform
An Apache Kafka® Producer is a client application that publishes (writes) events to a Kafka cluster. This section gives an overview of the Kafka producer and an introduction to the configuration settings for tuning.
A producer partitioner maps each message to a topic partition. The partitioners shipped with Kafka send all messages with the same non-empty key to the same partition.
The producer sends each produce request to the leader of that partition. Unlike a consumer, a producer needs no group coordination.
Important
Setting the partition field explicitly in a ProducerRecord
overrides the default partitioner behavior.
The partitioner assigns partitions as follows:
Key provided: The partitioner hashes the key with the murmur2 algorithm and divides it by the number of partitions. The same key always maps to the same partition.
No key: The partitioner assigns partitions with awareness of batching, a strategy called sticky partitioning. Records go to the same partition as the prior record until the batch is full or sent. Partitions for newly created batches are assigned randomly.
For more information, see the Kafka Improvement Proposal (KIP) KIP-480: Sticky Partitioner and the related Confluent blog post.
The partition leader is the broker that handles all writes for a partition. In-sync replicas (ISR) are the replicas that are fully caught up with the leader. Each partition in the Kafka cluster has a leader and a set of replicas among the brokers. All writes to the partition must go through the partition leader. The replicas are kept in sync by fetching from the leader. When the leader shuts down or fails, the next leader is chosen from among the in-sync replicas. Depending on how the producer is configured, each produce request to the partition leader can be held until the replicas have successfully acknowledged the write. This gives the producer some control over message durability at some cost to overall throughput.
A message is committed when all in-sync replicas have acknowledged the write. Consumers can read only committed messages, regardless of the producer’s acknowledgement settings. This ensures that a broker failure cannot lose a message after a consumer has read it.
Messages that only the leader acknowledged (acks=1) can be lost if the
partition leader fails before replicas copy them. This is often a reasonable
compromise because it ensures durability in most cases without significantly
reducing throughput.
Producer tuning focuses on two goals: high throughput through batching and compression, and message delivery guarantees through acknowledgements and retries. The following sections describe the settings that control each goal.
Kafka producer configuration
The key Kafka producer settings are bootstrap.servers, acks,
retries, batch.size, linger.ms, compression.type, and
buffer.memory. For the full list, see
Kafka Producer Configurations.
Core producer settings
These settings are the same for Java, C/C++, Python, Go, and .NET clients.
bootstrap.servers: Set this property so that the producer can find the Kafka cluster.client.id: Optional, but you should set this property on each instance because it helps you correlate requests on the broker with the client instance which made it, which can be helpful in debugging and troubleshooting scenarios. You can also use client IDs to enforce client quotas. For more information, see client quotas.
Configure message durability with acks
The acks setting controls message durability. The default is all.
Value |
Guarantee |
Throughput |
|---|---|---|
|
The partition leader and all in-sync replicas acknowledge the write. |
Lowest |
|
Only the partition leader acknowledges the write. |
Medium |
|
No acknowledgement. The broker sends no response, so you cannot confirm the write or determine the message offset. |
Highest |
For the C/C++, Python, Go, and .NET clients, acks is a per-topic setting.
Apply it globally with default_topic_conf (C/C++) or
default.topic.config (Python, Go, .NET).
Message ordering settings
In general, messages are written to the broker in the same order that they are received by the producer client. However, this can change depending on some settings.
retries: This setting enables message retries when set to a value larger than0(which is the default). If retries are enabled, message reordering can occur because the retry might occur after a later write succeeded.max.in.flight.requests.per.connection: Set to1to enable retries without reordering. This limits the producer to one in-flight request per broker connection. Without retries enabled, the broker preserves the order of writes it receives, but individual send failures can leave gaps.
Configure batching and compression
Kafka producers attempt to collect sent messages into batches to improve throughput.
batch.size: Use with the Java client to control the maximum size in bytes of each message batch.linger.ms: Use this setting to give more time for batches to fill by delaying the producer sending.compression.type: Enable compression. Compression covers full message batches, so larger batches typically give a higher compression ratio.batch.num.messages: Use with the C/C++, Python, Go, and .NET clients to set a limit on the number of messages contained in each batch.
The snappy codec requires write access to the /tmp directory. If /tmp
is mounted noexec, set a writable directory for snappy with:
-Dorg.xerial.snappy.tempdir=/path/to/newtmp
buffer.memory.allocation.strategy: Controls how the producer reserves memory for record batches. Use with the Java Client in Kafka 4.4 and later. Accepted values arefull(default) andincremental. Theincrementalstrategy is opt-in, and you can use it only whencompression.typeisnone.With the default
fullstrategy, the producer reserves the entirebatch.sizefor every partition that it actively writes to. Setbuffer.memoryto at least the active partition count multiplied bybatch.size. The active partition count shifts at runtime with key distribution and partition count. For example, an application that sends to 1,000 partitions with a 4 MiB batch size requires 4 GiB of buffer memory under thefullstrategy. Each partition receives a fraction of that amount in any linger window.With
incremental, introduced in KIP-1332, memory use scales with throughput andlinger.ms. Because it doesn’t depend on the number of active partitions, sizing is more predictable. Useincrementalfor workloads that are uncompressed, use high-cardinality keys, and span many low-volume partitions.For more information, see KIP-1332: Dynamic memory allocation for the Kafka producer.
Limit memory for unsent messages
buffer.memory: Limits the total memory that is available to the Java client for collecting unsent messages. When this limit is hit, the producer blocks on additional sends for as long asmax.block.msbefore it raises an exception.request.timeout.ms: Set a timeout to avoid keeping records queued indefinitely. If this timeout expires before a message can be successfully sent, the producer removes it from the queue and throws an exception. The C/C++, Python, Go, and .NET clients have similar settings.
Transactional error handling
A transactional producer calls initTransactions, beginTransaction,
send, sendOffsetsToTransaction, commitTransaction, and
abortTransaction. Kafka 4.1 and later clients group the exceptions that these
methods throw into categories. Your application can decide how to recover
based on the exception type instead of on individual error codes. The categories
are the same for Java and librdkafka clients.
Exception category |
Handled by |
What your application does |
|---|---|---|
|
Client |
Nothing. The client retries the operation internally, and the exception does not reach your application. |
|
Client |
Nothing. The client refreshes its metadata, retries the operation internally, and the exception does not reach your application. |
|
Your application |
Call |
|
Your application |
Close the producer, create a new one, and reinitialize the application. The producer cannot continue. |
|
Your application |
Correct the client configuration. The producer does not have to restart, but your application can choose to close it. |
|
Your application |
Decide how to recover. Treating this category the same way as
|
Structure your transaction loop with an inner catch block for
TransactionAbortableException, which you recover from by aborting. Add an outer
catch blocks for the categories that require you to restart or reconfigure
the producer. Catch the more specific categories before KafkaException,
because they extend it:
while (isRunning) {
try {
try {
producer.beginTransaction();
// Poll records, process them, and send the results.
producer.sendOffsetsToTransaction(offsets, groupMetadata);
producer.commitTransaction();
} catch (TransactionAbortableException e) {
producer.abortTransaction();
resetToLastCommittedPositions(consumer);
}
} catch (InvalidConfigurationException e) {
closeAll();
throw e;
} catch (ApplicationRecoverableException e) {
closeAll();
initializeApplication();
} catch (KafkaException e) {
closeAll();
initializeApplication();
}
}
Important
On the transactional send path, retriable exceptions surface as
TransactionAbortableException. If your application retries the send after
catching TimeoutException, update it to abort the transaction instead.
For a complete example, see TransactionalClientDemo.java. For the full list of exceptions in each category, see KIP-1050: Consistent error handling for Transactions.
Kafka producer examples
Example Kafka producer code is available in Java, Python, Go, .NET, JavaScript Client, C/C++, REST, and Spring Boot.
For a tutorial on how to build a Kafka producer that can write records to Confluent Cloud or Confluent Platform, see How to build your first Apache KafkaProducer application.
For producer examples, see Build Consumer, use the language selector to choose a language, and click Build Producer to see example producer code.