Java Client for Apache Kafka
The Apache Kafka® Java Client is the Java library for producing records to and consuming records from Kafka topics. It’s the foundational library for building distributed applications that process event streams, with built-in fault tolerance and scalability.
Ready to get started?
Sign up for Confluent Cloud, the fully managed cloud-native service for Apache Kafka® and get started for free using the Cloud quick start.
Download Confluent Platform, the self managed, enterprise-grade distribution of Apache Kafka and get started using the Confluent Platform quick start.
For a step-by-step guide on building a Java client application for Kafka, see Getting Started with Apache Kafka and Java.
Java client components
The next sections describe the Java client components.
Producer components
A producer sends records to Kafka topics. A Java producer has the following key components:
ProducerRecord. Represents a record, or message, to send to Kafka. It requires a topic name and a value, and optionally a key and a partition.
KafkaProducer. Sends records to their topics in Kafka.
Serializer. Converts records into bytes to send to Kafka. Kafka includes serializers for common types, and you can also write custom serializers.
Consumer components
A consumer reads data from Kafka topics. A Java consumer has the following key components:
ConsumerRecord. Represents a record, or message, retrieved from Kafka.
KafkaConsumer. Polls records from Kafka.
Deserializer. Converts bytes fetched from Kafka back into objects. As with serializers, Kafka includes deserializers for common types, and you can also create custom deserializers.
ConsumerGroup. A set of consumers that share the same group ID. They jointly consume messages from one or more topics, and each consumer in the group consumes a unique set of partitions.
Install the Java client
To install the Java client for Kafka, add the kafka-clients dependency to your
pom.xml. The Confluent Maven repository includes compiled versions of Kafka
and all JARs in the Confluent Platform packages, so you can reference the Kafka client
and other libraries in your Maven project.
Add the Confluent Maven repository
To add the Confluent Maven repository, include the following in your pom.xml file:
<repositories>
<repository>
<id>confluent</id>
<url>https://packages.confluent.io/maven/</url>
</repository>
<!-- further repository entries here -->
</repositories>
Reference the client
To use the client, reference it in the pom.xml file.
For example, to reference the Kafka version that Confluent Platform 7.0.1 includes, use the
following in your pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>7.0.1-ccs</version>
</dependency>
<!-- further dependency entries here -->
</dependencies>
Note
Apache Kafka and Confluent Platform version names
Confluent always contributes patches back to the Kafka open source project.
The versions, and version names, included in Confluent Platform can differ from the Apache
artifacts when Confluent Platform and Kafka releases don’t align. In that case, Confluent
keeps the groupId and artifactId identical. It appends the -ccs
suffix to the Confluent Platform version identifier to distinguish these artifacts from the
Apache artifacts.
Reference other Confluent libraries
You can reference artifacts for all Java libraries included with Confluent Platform. For
example, to use the Avro serializer, include the following in your
pom.xml:
<dependencies>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<!-- For Confluent Platform 7.0.1 -->
<version>7.0.1</version>
</dependency>
<!-- further dependency entries here -->
</dependencies>
Tip
You can also specify the kafka-protobuf-serializer or
kafka-jsonschema-serializer artifacts. For more information, see
Schema Formats, Serializers, and Deserializers.
Quick start
For a step-by-step tutorial with code samples for the producer and consumer, see Getting Started with Apache Kafka and Java.
For more examples, including how to produce and consume Avro data with Schema Registry, see Java client examples.
Produce messages with the Java client
To produce messages with the Java client, create a KafkaProducer with a
Properties object, then call send() for each record.
Initialize a producer
Construct the Java producer with a standard Properties object.
Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("bootstrap.servers", "host1:9092,host2:9092");
config.put("acks", "all");
config.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
config.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(config);
If a configuration is invalid, the KafkaProducer constructor throws a
KafkaException.
Write messages asynchronously
The send() method writes a record asynchronously. It returns a future that
you can poll for the result of the send:
final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
Future<RecordMetadata> future = producer.send(record);
To run code after the write completes, provide a callback. In Java, you
create the callback as a Callback object:
final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
producer.send(record, new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if (e != null)
log.debug("Send failed for record {}", record, e);
}
});
Avoid expensive work in the callback, because the producer’s IO thread runs it.
Write messages synchronously
To write synchronously, call get() on the future that send() returns.
The call blocks until the write completes:
Future<RecordMetadata> future = producer.send(record);
RecordMetadata metadata = future.get();
Consume messages with the Java client
To consume messages with the Java client, create a KafkaConsumer with a
Properties object, subscribe to topics, and call poll() in a loop.
Initialize a consumer
Construct the Java consumer with a standard Properties object.
Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("group.id", "foo");
config.put("bootstrap.servers", "host1:9092,host2:9092");
config.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
config.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(config);
If a configuration is invalid, the KafkaConsumer constructor throws a
KafkaException.
How the Java consumer works
The Java consumer runs an event loop that the poll() method drives. This
design follows the UNIX select and poll system calls. A basic
consumption loop takes the following form:
while (running) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
process(records); // application-specific processing
consumer.commitSync();
}
Calls to poll() drive the consumer’s main work, including:
Joining the consumer group and handling partition rebalances.
Sending periodic offset commits when you enable autocommit.
Sending and receiving fetch requests for assigned partitions.
Keep the consumer in the group
The consumer must call poll() regularly to stay in the group. If the event
loop ends, or if record processing delays the next poll() call past
max.poll.interval.ms, the consumer leaves the group and another member
takes over its partitions. The group also removes a consumer that crashes or
can’t send heartbeats within session.timeout.ms.
With the default classic group protocol, a background heartbeat thread
sends heartbeats while your application processes records. Slow processing
alone doesn’t expire the session timeout. The max.poll.interval.ms check
catches a consumer that keeps sending heartbeats but makes no progress.
This behavior is by design. As long as the group assigns partitions to the consumer, no other member of the group can consume from the same partitions. The client ensures that the consumer is making progress and hasn’t become a zombie.
This behavior protects your application from a large class of failures. You must tune two settings so the consumer stays within the poll interval during normal record processing:
Raise
max.poll.interval.msto give the consumer more time to process a batch of records. A higher value can delay a group rebalance, because the consumer joins a rebalance only inside the call topoll().Lower
max.poll.recordsto limit the number of records returned from eachpoll()call. This keeps the work on each iteration bounded, so worst-case behavior is predictable.
Note
The session.timeout.ms setting applies only to the classic group
protocol. With group.protocol=consumer, the broker controls the session
timeout through group.consumer.session.timeout.ms.
Handle commit failures
If the consumer exceeds max.poll.interval.ms, the call to commit offsets for
the processed records typically throws a CommitFailedException. If you use
autocommit, you might not notice the failure, because the consumer silently
ignores autocommit failures. You see them only if they occur often enough to
affect lag metrics. You can catch
this exception and either ignore it or perform any needed rollback logic:
while (running) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
process(records); // application-specific processing
try {
consumer.commitSync();
} catch (CommitFailedException e) {
// application-specific rollback of processed records
}
}
Poll for messages in a loop
The consumer API centers on the poll() method, which retrieves records from
the brokers. The subscribe() method controls which topics poll()
fetches. Typically, you call subscribe() first to set up the topics of
interest, then loop on poll() until the application shuts down.
The consumer intentionally avoids a specific threading model. Except for
wakeup(), it isn’t safe for multi-threaded access, so call it from a single
thread. In the following example, a
Runnable wraps the poll loop, which makes it easy to use with an
ExecutorService:
public abstract class BasicConsumeLoop implements Runnable {
private final KafkaConsumer<K, V> consumer;
private final List<String> topics;
private final AtomicBoolean shutdown;
private final CountDownLatch shutdownLatch;
public BasicConsumeLoop(Properties config, List<String> topics) {
this.consumer = new KafkaConsumer<>(config);
this.topics = topics;
this.shutdown = new AtomicBoolean(false);
this.shutdownLatch = new CountDownLatch(1);
}
public abstract void process(ConsumerRecord<K, V> record);
public void run() {
try {
consumer.subscribe(topics);
while (!shutdown.get()) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(500));
records.forEach(record -> process(record));
}
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
public void shutdown() throws InterruptedException {
shutdown.set(true);
shutdownLatch.await();
}
}
The poll timeout is hard-coded to 500 milliseconds. If the consumer receives no
records before the timeout expires, poll() returns an empty record set. If
your message processing involves any setup overhead, add a shortcut check for
this case.
Shut down the consumer
The previous example adds a flag that the loop checks on each iteration. After
you trigger shutdown, the consumer waits at most 500 milliseconds, plus the
message processing time. The delay occurs because the trigger might happen
while the consumer is in poll(). To avoid this delay, see
Shut down the consumer with wakeup.
Always call close() after you finish using the consumer. This call closes
active sockets, cleans up internal state, and immediately triggers a group
rebalance that reassigns the consumer’s partitions to another member of the
group. If you don’t close the consumer, the group reassigns its partitions only
after max.poll.interval.ms expires, or after session.timeout.ms expires
if the process exits. The poll loop example adds a latch so the
consumer has time to finish closing before shutdown.
Shut down the consumer with wakeup
An alternative poll loop pattern uses Duration.ofMillis(Long.MAX_VALUE) for
the poll() timeout. To break out of the loop, call the consumer’s
wakeup() method from a separate thread. The call throws a
WakeupException from the thread blocking in poll(). If the thread isn’t
currently blocking, wakeup() wakes the next poll() invocation:
public abstract class ConsumeLoop implements Runnable {
private final KafkaConsumer<K, V> consumer;
private final List<String> topics;
private final CountDownLatch shutdownLatch;
public ConsumeLoop(KafkaConsumer<K, V> consumer, List<String> topics) {
this.consumer = consumer;
this.topics = topics;
this.shutdownLatch = new CountDownLatch(1);
}
public abstract void process(ConsumerRecord<K, V> record);
public void run() {
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
records.forEach(record -> process(record));
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
public void shutdown() throws InterruptedException {
consumer.wakeup();
shutdownLatch.await();
}
}
Commit offsets synchronously
The simplest and most reliable way to manually commit offsets is a synchronous
commit with commitSync(). This method blocks until the commit completes
successfully:
private void doCommitSync() {
try {
consumer.commitSync();
} catch (WakeupException e) {
// we're shutting down, but finish the commit first and then
// rethrow the exception so that the main loop can exit
doCommitSync();
throw e;
} catch (CommitFailedException e) {
// the commit failed with an unrecoverable error. if there is any
// internal state which depended on the commit, you can clean it
// up here. otherwise it's reasonable to ignore the error and go on
log.debug("Commit failed", e);
}
}
public void run() {
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
records.forEach(record -> process(record));
doCommitSync();
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
This example wraps the call to commitSync() in a try/catch block. The
wakeup() method might trigger while the commit is pending. The recursive
call is safe because wakeup() triggers only once.
The consumer throws CommitFailedException when the commit can’t complete
because the group has rebalanced. This is the main thing to be careful of when
using the Java client. If message processing takes longer than
max.poll.interval.ms, the consumer leaves the group and the commit fails.
To handle this, choose one of two options:
Give the consumer more time. Raise
max.poll.interval.msso the handler has enough time to finish processing messages, or lowermax.poll.recordsto limit the batch size.Process messages in a separate thread. You must manage flow control so the threads can keep up. For example, pushing messages into a blocking queue is probably not enough unless the processing rate keeps up with the delivery rate. In that case, you might not need a separate thread. Pushing messages into a blocking queue can also make the problem worse if the poll loop blocks on a call to
offer()while the background thread handles an even larger batch of messages. The Java API offers apause()method to help in these situations.
Set max.poll.interval.ms large enough that commit failures from rebalances
are rare. The drawback is that a higher value can delay a group rebalance,
because the consumer joins a rebalance only inside the call to poll().
Choose a delivery guarantee
Committing after processing a message gives you at-least-once delivery. Committing before processing gives you at-most-once delivery, but you must handle commit failures carefully. The synchronous commit example gives at-least-once delivery because the commit follows message processing.
To get at-most-once delivery, commit before you process the records. To handle
commit failures, change doCommitSync to return whether the commit
succeeded, and process the records only when it did. The synchronous commit no
longer needs to catch the WakeupException:
private boolean doCommitSync() {
try {
consumer.commitSync();
return true;
} catch (CommitFailedException e) {
// the commit failed with an unrecoverable error. if there is any
// internal state which depended on the commit, you can clean it
// up here. otherwise it's reasonable to ignore the error and go on
log.debug("Commit failed", e);
return false;
}
}
public void run() {
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
if (doCommitSync())
records.forEach(record -> process(record));
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
Correct offset management is crucial because it affects delivery semantics.
Commit offsets asynchronously
To commit asynchronously, call commitAsync() after you process each batch.
The consumer sends the request and returns immediately:
public void run() {
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
records.forEach(record -> process(record));
consumer.commitAsync();
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
The API gives you a callback that it invokes when the commit succeeds or fails:
public void run() {
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
records.forEach(record -> process(record));
consumer.commitAsync(new OffsetCommitCallback() {
public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
if (exception != null)
log.debug("Commit failed for offsets {}", offsets, exception);
}
});
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
To commit synchronously on rebalances and on close, use the variant of
subscribe() that accepts a ConsumerRebalanceListener. The listener has
three methods to hook into rebalance behavior: onPartitionsRevoked,
onPartitionsAssigned, and onPartitionsLost. The onPartitionsLost
method is a default method that calls onPartitionsRevoked unless you
override it. The following example commits synchronously in
onPartitionsRevoked and when the consumer closes. It overrides
onPartitionsLost as a no-op, because the consumer no longer owns those
partitions and can’t commit offsets for them:
private void doCommitSync() {
try {
consumer.commitSync();
} catch (WakeupException e) {
// we're shutting down, but finish the commit first and then
// rethrow the exception so that the main loop can exit
doCommitSync();
throw e;
} catch (CommitFailedException e) {
// the commit failed with an unrecoverable error. if there is any
// internal state which depended on the commit, you can clean it
// up here. otherwise it's reasonable to ignore the error and go on
log.debug("Commit failed", e);
}
}
public void run() {
try {
consumer.subscribe(topics, new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
doCommitSync();
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {}
@Override
public void onPartitionsLost(Collection<TopicPartition> partitions) {
// the partitions are already owned by other consumers, so don't commit
}
});
while (true) {
ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
records.forEach(record -> process(record));
consumer.commitAsync();
}
} catch (WakeupException e) {
// ignore, we're closing
} catch (Exception e) {
log.error("Unexpected error", e);
} finally {
try {
doCommitSync();
} finally {
consumer.close();
shutdownLatch.countDown();
}
}
}
Java client API documentation
For the full Java client API, see the Java client API documentation.
Limitations
Confluent Cloud doesn’t support the following methods: