Kafka Producer for Confluent Platform

An Apache Kafka® Producer is a client application that publishes (writes) events to a Kafka cluster. This section gives an overview of the Kafka producer and an introduction to the configuration settings for tuning.

A producer partitioner maps each message to a topic partition. The partitioners shipped with Kafka send all messages with the same non-empty key to the same partition.

The producer sends each produce request to the leader of that partition. Unlike a consumer, a producer needs no group coordination.

Important

Setting the partition field explicitly in a ProducerRecord overrides the default partitioner behavior.

The partitioner assigns partitions as follows:

  • Key provided: The partitioner hashes the key with the murmur2 algorithm and divides it by the number of partitions. The same key always maps to the same partition.

  • No key: The partitioner assigns partitions with awareness of batching, a strategy called sticky partitioning. Records go to the same partition as the prior record until the batch is full or sent. Partitions for newly created batches are assigned randomly.

For more information, see the Kafka Improvement Proposal (KIP) KIP-480: Sticky Partitioner and the related Confluent blog post.

The partition leader is the broker that handles all writes for a partition. In-sync replicas (ISR) are the replicas that are fully caught up with the leader. Each partition in the Kafka cluster has a leader and a set of replicas among the brokers. All writes to the partition must go through the partition leader. The replicas are kept in sync by fetching from the leader. When the leader shuts down or fails, the next leader is chosen from among the in-sync replicas. Depending on how the producer is configured, each produce request to the partition leader can be held until the replicas have successfully acknowledged the write. This gives the producer some control over message durability at some cost to overall throughput.

A message is committed when all in-sync replicas have acknowledged the write. Consumers can read only committed messages, regardless of the producer’s acknowledgement settings. This ensures that a broker failure cannot lose a message after a consumer has read it.

Messages that only the leader acknowledged (acks=1) can be lost if the partition leader fails before replicas copy them. This is often a reasonable compromise because it ensures durability in most cases without significantly reducing throughput.

Producer tuning focuses on two goals: high throughput through batching and compression, and message delivery guarantees through acknowledgements and retries. The following sections describe the settings that control each goal.

Kafka producer configuration

The key Kafka producer settings are bootstrap.servers, acks, retries, batch.size, linger.ms, compression.type, and buffer.memory. For the full list, see Kafka Producer Configurations.

Core producer settings

These settings are the same for Java, C/C++, Python, Go, and .NET clients.

  • bootstrap.servers: Set this property so that the producer can find the Kafka cluster.

  • client.id: Optional, but you should set this property on each instance because it helps you correlate requests on the broker with the client instance which made it, which can be helpful in debugging and troubleshooting scenarios. You can also use client IDs to enforce client quotas. For more information, see client quotas.

Configure message durability with acks

The acks setting controls message durability. The default is all.

Value

Guarantee

Throughput

all (default)

The partition leader and all in-sync replicas acknowledge the write.

Lowest

1

Only the partition leader acknowledges the write.

Medium

0

No acknowledgement. The broker sends no response, so you cannot confirm the write or determine the message offset.

Highest

For the C/C++, Python, Go, and .NET clients, acks is a per-topic setting. Apply it globally with default_topic_conf (C/C++) or default.topic.config (Python, Go, .NET).

Message ordering settings

In general, messages are written to the broker in the same order that they are received by the producer client. However, this can change depending on some settings.

  • retries: This setting enables message retries when set to a value larger than 0 (which is the default). If retries are enabled, message reordering can occur because the retry might occur after a later write succeeded.

  • max.in.flight.requests.per.connection: Set to 1 to enable retries without reordering. This limits the producer to one in-flight request per broker connection. Without retries enabled, the broker preserves the order of writes it receives, but individual send failures can leave gaps.

Configure batching and compression

Kafka producers attempt to collect sent messages into batches to improve throughput.

  • batch.size: Use with the Java client to control the maximum size in bytes of each message batch.

  • linger.ms: Use this setting to give more time for batches to fill by delaying the producer sending.

  • compression.type: Enable compression. Compression covers full message batches, so larger batches typically give a higher compression ratio.

  • batch.num.messages: Use with the C/C++, Python, Go, and .NET clients to set a limit on the number of messages contained in each batch.

The snappy codec requires write access to the /tmp directory. If /tmp is mounted noexec, set a writable directory for snappy with:

-Dorg.xerial.snappy.tempdir=/path/to/newtmp
  • buffer.memory.allocation.strategy: Controls how the producer reserves memory for record batches. Use with the Java Client in Kafka 4.4 and later. Accepted values are full (default) and incremental. The incremental strategy is opt-in, and you can use it only when compression.type is none.

    With the default full strategy, the producer reserves the entire batch.size for every partition that it actively writes to. Set buffer.memory to at least the active partition count multiplied by batch.size. The active partition count shifts at runtime with key distribution and partition count. For example, an application that sends to 1,000 partitions with a 4 MiB batch size requires 4 GiB of buffer memory under the full strategy. Each partition receives a fraction of that amount in any linger window.

    With incremental, introduced in KIP-1332, memory use scales with throughput and linger.ms. Because it doesn’t depend on the number of active partitions, sizing is more predictable. Use incremental for workloads that are uncompressed, use high-cardinality keys, and span many low-volume partitions.

    For more information, see KIP-1332: Dynamic memory allocation for the Kafka producer.

Limit memory for unsent messages

  • buffer.memory: Limits the total memory that is available to the Java client for collecting unsent messages. When this limit is hit, the producer blocks on additional sends for as long as max.block.ms before it raises an exception.

  • request.timeout.ms: Set a timeout to avoid keeping records queued indefinitely. If this timeout expires before a message can be successfully sent, the producer removes it from the queue and throws an exception. The C/C++, Python, Go, and .NET clients have similar settings.

Transactional error handling

A transactional producer calls initTransactions, beginTransaction, send, sendOffsetsToTransaction, commitTransaction, and abortTransaction. Kafka 4.1 and later clients group the exceptions that these methods throw into categories. Your application can decide how to recover based on the exception type instead of on individual error codes. The categories are the same for Java and librdkafka clients.

Exception category

Handled by

What your application does

RetriableException

Client

Nothing. The client retries the operation internally, and the exception does not reach your application.

RefreshRetriableException

Client

Nothing. The client refreshes its metadata, retries the operation internally, and the exception does not reach your application.

TransactionAbortableException

Your application

Call abortTransaction, reset the consumer to the last committed offsets, and reprocess the records. The producer stays usable.

ApplicationRecoverableException

Your application

Close the producer, create a new one, and reinitialize the application. The producer cannot continue.

InvalidConfigurationException

Your application

Correct the client configuration. The producer does not have to restart, but your application can choose to close it.

KafkaException

Your application

Decide how to recover. Treating this category the same way as ApplicationRecoverableException is a reasonable default.

Structure your transaction loop with an inner catch block for TransactionAbortableException, which you recover from by aborting. Add an outer catch blocks for the categories that require you to restart or reconfigure the producer. Catch the more specific categories before KafkaException, because they extend it:

while (isRunning) {
    try {
        try {
            producer.beginTransaction();
            // Poll records, process them, and send the results.
            producer.sendOffsetsToTransaction(offsets, groupMetadata);
            producer.commitTransaction();
        } catch (TransactionAbortableException e) {
            producer.abortTransaction();
            resetToLastCommittedPositions(consumer);
        }
    } catch (InvalidConfigurationException e) {
        closeAll();
        throw e;
    } catch (ApplicationRecoverableException e) {
        closeAll();
        initializeApplication();
    } catch (KafkaException e) {
        closeAll();
        initializeApplication();
    }
}

Important

On the transactional send path, retriable exceptions surface as TransactionAbortableException. If your application retries the send after catching TimeoutException, update it to abort the transaction instead.

For a complete example, see TransactionalClientDemo.java. For the full list of exceptions in each category, see KIP-1050: Consistent error handling for Transactions.

Kafka producer examples

Example Kafka producer code is available in Java, Python, Go, .NET, JavaScript Client, C/C++, REST, and Spring Boot.