Java Client for Apache Kafka

The Apache Kafka® Java Client is the Java library for producing records to and consuming records from Kafka topics. It’s the foundational library for building distributed applications that process event streams, with built-in fault tolerance and scalability.

Ready to get started?

For a step-by-step guide on building a Java client application for Kafka, see Getting Started with Apache Kafka and Java.

Java client components

The next sections describe the Java client components.

Producer components

A producer sends records to Kafka topics. A Java producer has the following key components:

  • ProducerRecord. Represents a record, or message, to send to Kafka. It requires a topic name and a value, and optionally a key and a partition.

  • KafkaProducer. Sends records to their topics in Kafka.

  • Serializer. Converts records into bytes to send to Kafka. Kafka includes serializers for common types, and you can also write custom serializers.

Consumer components

A consumer reads data from Kafka topics. A Java consumer has the following key components:

  • ConsumerRecord. Represents a record, or message, retrieved from Kafka.

  • KafkaConsumer. Polls records from Kafka.

  • Deserializer. Converts bytes fetched from Kafka back into objects. As with serializers, Kafka includes deserializers for common types, and you can also create custom deserializers.

  • ConsumerGroup. A set of consumers that share the same group ID. They jointly consume messages from one or more topics, and each consumer in the group consumes a unique set of partitions.

Install the Java client

To install the Java client for Kafka, add the kafka-clients dependency to your pom.xml. The Confluent Maven repository includes compiled versions of Kafka and all JARs in the Confluent Platform packages, so you can reference the Kafka client and other libraries in your Maven project.

Add the Confluent Maven repository

To add the Confluent Maven repository, include the following in your pom.xml file:

<repositories>

  <repository>
    <id>confluent</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>

  <!-- further repository entries here -->

</repositories>

Reference the client

To use the client, reference it in the pom.xml file. For example, to reference the Kafka version that Confluent Platform 7.0.1 includes, use the following in your pom.xml:

<dependencies>

  <dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>7.0.1-ccs</version>
  </dependency>

  <!-- further dependency entries here -->

</dependencies>

Note

Apache Kafka and Confluent Platform version names

Confluent always contributes patches back to the Kafka open source project. The versions, and version names, included in Confluent Platform can differ from the Apache artifacts when Confluent Platform and Kafka releases don’t align. In that case, Confluent keeps the groupId and artifactId identical. It appends the -ccs suffix to the Confluent Platform version identifier to distinguish these artifacts from the Apache artifacts.

Reference other Confluent libraries

You can reference artifacts for all Java libraries included with Confluent Platform. For example, to use the Avro serializer, include the following in your pom.xml:

<dependencies>

  <dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <!-- For Confluent Platform 7.0.1 -->
    <version>7.0.1</version>
  </dependency>

  <!-- further dependency entries here -->

</dependencies>

Tip

You can also specify the kafka-protobuf-serializer or kafka-jsonschema-serializer artifacts. For more information, see Schema Formats, Serializers, and Deserializers.

Quick start

For a step-by-step tutorial with code samples for the producer and consumer, see Getting Started with Apache Kafka and Java.

For more examples, including how to produce and consume Avro data with Schema Registry, see Java client examples.

Produce messages with the Java client

To produce messages with the Java client, create a KafkaProducer with a Properties object, then call send() for each record.

Initialize a producer

Construct the Java producer with a standard Properties object.

Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("bootstrap.servers", "host1:9092,host2:9092");
config.put("acks", "all");
config.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
config.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(config);

If a configuration is invalid, the KafkaProducer constructor throws a KafkaException.

Write messages asynchronously

The send() method writes a record asynchronously. It returns a future that you can poll for the result of the send:

final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
Future<RecordMetadata> future = producer.send(record);

To run code after the write completes, provide a callback. In Java, you create the callback as a Callback object:

final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
producer.send(record, new Callback() {
  public void onCompletion(RecordMetadata metadata, Exception e) {
    if (e != null)
      log.debug("Send failed for record {}", record, e);
  }
});

Avoid expensive work in the callback, because the producer’s IO thread runs it.

Write messages synchronously

To write synchronously, call get() on the future that send() returns. The call blocks until the write completes:

Future<RecordMetadata> future = producer.send(record);
RecordMetadata metadata = future.get();

Consume messages with the Java client

To consume messages with the Java client, create a KafkaConsumer with a Properties object, subscribe to topics, and call poll() in a loop.

Initialize a consumer

Construct the Java consumer with a standard Properties object.

Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("group.id", "foo");
config.put("bootstrap.servers", "host1:9092,host2:9092");
config.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
config.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(config);

If a configuration is invalid, the KafkaConsumer constructor throws a KafkaException.

How the Java consumer works

The Java consumer runs an event loop that the poll() method drives. This design follows the UNIX select and poll system calls. A basic consumption loop takes the following form:

while (running) {
  ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
  process(records); // application-specific processing
  consumer.commitSync();
}

Calls to poll() drive the consumer’s main work, including:

  • Joining the consumer group and handling partition rebalances.

  • Sending periodic offset commits when you enable autocommit.

  • Sending and receiving fetch requests for assigned partitions.

Keep the consumer in the group

The consumer must call poll() regularly to stay in the group. If the event loop ends, or if record processing delays the next poll() call past max.poll.interval.ms, the consumer leaves the group and another member takes over its partitions. The group also removes a consumer that crashes or can’t send heartbeats within session.timeout.ms.

With the default classic group protocol, a background heartbeat thread sends heartbeats while your application processes records. Slow processing alone doesn’t expire the session timeout. The max.poll.interval.ms check catches a consumer that keeps sending heartbeats but makes no progress.

This behavior is by design. As long as the group assigns partitions to the consumer, no other member of the group can consume from the same partitions. The client ensures that the consumer is making progress and hasn’t become a zombie.

This behavior protects your application from a large class of failures. You must tune two settings so the consumer stays within the poll interval during normal record processing:

  • Raise max.poll.interval.ms to give the consumer more time to process a batch of records. A higher value can delay a group rebalance, because the consumer joins a rebalance only inside the call to poll().

  • Lower max.poll.records to limit the number of records returned from each poll() call. This keeps the work on each iteration bounded, so worst-case behavior is predictable.

Note

The session.timeout.ms setting applies only to the classic group protocol. With group.protocol=consumer, the broker controls the session timeout through group.consumer.session.timeout.ms.

Handle commit failures

If the consumer exceeds max.poll.interval.ms, the call to commit offsets for the processed records typically throws a CommitFailedException. If you use autocommit, you might not notice the failure, because the consumer silently ignores autocommit failures. You see them only if they occur often enough to affect lag metrics. You can catch this exception and either ignore it or perform any needed rollback logic:

while (running) {
  ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
  process(records); // application-specific processing
  try {
    consumer.commitSync();
  } catch (CommitFailedException e) {
    // application-specific rollback of processed records
  }
}

Poll for messages in a loop

The consumer API centers on the poll() method, which retrieves records from the brokers. The subscribe() method controls which topics poll() fetches. Typically, you call subscribe() first to set up the topics of interest, then loop on poll() until the application shuts down.

The consumer intentionally avoids a specific threading model. Except for wakeup(), it isn’t safe for multi-threaded access, so call it from a single thread. In the following example, a Runnable wraps the poll loop, which makes it easy to use with an ExecutorService:

public abstract class BasicConsumeLoop implements Runnable {
  private final KafkaConsumer<K, V> consumer;
  private final List<String> topics;
  private final AtomicBoolean shutdown;
  private final CountDownLatch shutdownLatch;

  public BasicConsumeLoop(Properties config, List<String> topics) {
    this.consumer = new KafkaConsumer<>(config);
    this.topics = topics;
    this.shutdown = new AtomicBoolean(false);
    this.shutdownLatch = new CountDownLatch(1);
  }

  public abstract void process(ConsumerRecord<K, V> record);

  public void run() {
    try {
      consumer.subscribe(topics);

      while (!shutdown.get()) {
        ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(500));
        records.forEach(record -> process(record));
      }
    } finally {
      consumer.close();
      shutdownLatch.countDown();
    }
  }

  public void shutdown() throws InterruptedException {
    shutdown.set(true);
    shutdownLatch.await();
  }
}

The poll timeout is hard-coded to 500 milliseconds. If the consumer receives no records before the timeout expires, poll() returns an empty record set. If your message processing involves any setup overhead, add a shortcut check for this case.

Shut down the consumer

The previous example adds a flag that the loop checks on each iteration. After you trigger shutdown, the consumer waits at most 500 milliseconds, plus the message processing time. The delay occurs because the trigger might happen while the consumer is in poll(). To avoid this delay, see Shut down the consumer with wakeup.

Always call close() after you finish using the consumer. This call closes active sockets, cleans up internal state, and immediately triggers a group rebalance that reassigns the consumer’s partitions to another member of the group. If you don’t close the consumer, the group reassigns its partitions only after max.poll.interval.ms expires, or after session.timeout.ms expires if the process exits. The poll loop example adds a latch so the consumer has time to finish closing before shutdown.

Shut down the consumer with wakeup

An alternative poll loop pattern uses Duration.ofMillis(Long.MAX_VALUE) for the poll() timeout. To break out of the loop, call the consumer’s wakeup() method from a separate thread. The call throws a WakeupException from the thread blocking in poll(). If the thread isn’t currently blocking, wakeup() wakes the next poll() invocation:

public abstract class ConsumeLoop implements Runnable {
  private final KafkaConsumer<K, V> consumer;
  private final List<String> topics;
  private final CountDownLatch shutdownLatch;

  public ConsumeLoop(KafkaConsumer<K, V> consumer, List<String> topics) {
    this.consumer = consumer;
    this.topics = topics;
    this.shutdownLatch = new CountDownLatch(1);
  }

  public abstract void process(ConsumerRecord<K, V> record);

  public void run() {
    try {
      consumer.subscribe(topics);

      while (true) {
        ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
        records.forEach(record -> process(record));
      }
    } catch (WakeupException e) {
      // ignore, we're closing
    } catch (Exception e) {
      log.error("Unexpected error", e);
    } finally {
      consumer.close();
      shutdownLatch.countDown();
    }
  }

  public void shutdown() throws InterruptedException {
    consumer.wakeup();
    shutdownLatch.await();
  }
}

Commit offsets synchronously

The simplest and most reliable way to manually commit offsets is a synchronous commit with commitSync(). This method blocks until the commit completes successfully:

private void doCommitSync() {
  try {
    consumer.commitSync();
  } catch (WakeupException e) {
    // we're shutting down, but finish the commit first and then
    // rethrow the exception so that the main loop can exit
    doCommitSync();
    throw e;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
  }
}

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
      records.forEach(record -> process(record));
      doCommitSync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}

This example wraps the call to commitSync() in a try/catch block. The wakeup() method might trigger while the commit is pending. The recursive call is safe because wakeup() triggers only once.

The consumer throws CommitFailedException when the commit can’t complete because the group has rebalanced. This is the main thing to be careful of when using the Java client. If message processing takes longer than max.poll.interval.ms, the consumer leaves the group and the commit fails. To handle this, choose one of two options:

  • Give the consumer more time. Raise max.poll.interval.ms so the handler has enough time to finish processing messages, or lower max.poll.records to limit the batch size.

  • Process messages in a separate thread. You must manage flow control so the threads can keep up. For example, pushing messages into a blocking queue is probably not enough unless the processing rate keeps up with the delivery rate. In that case, you might not need a separate thread. Pushing messages into a blocking queue can also make the problem worse if the poll loop blocks on a call to offer() while the background thread handles an even larger batch of messages. The Java API offers a pause() method to help in these situations.

Set max.poll.interval.ms large enough that commit failures from rebalances are rare. The drawback is that a higher value can delay a group rebalance, because the consumer joins a rebalance only inside the call to poll().

Choose a delivery guarantee

Committing after processing a message gives you at-least-once delivery. Committing before processing gives you at-most-once delivery, but you must handle commit failures carefully. The synchronous commit example gives at-least-once delivery because the commit follows message processing.

To get at-most-once delivery, commit before you process the records. To handle commit failures, change doCommitSync to return whether the commit succeeded, and process the records only when it did. The synchronous commit no longer needs to catch the WakeupException:

private boolean doCommitSync() {
  try {
    consumer.commitSync();
    return true;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
    return false;
  }
}

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
      if (doCommitSync())
        records.forEach(record -> process(record));
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}

Correct offset management is crucial because it affects delivery semantics.

Commit offsets asynchronously

To commit asynchronously, call commitAsync() after you process each batch. The consumer sends the request and returns immediately:

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
      records.forEach(record -> process(record));
      consumer.commitAsync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}

The API gives you a callback that it invokes when the commit succeeds or fails:

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
      records.forEach(record -> process(record));
      consumer.commitAsync(new OffsetCommitCallback() {
        public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
          if (exception != null)
              log.debug("Commit failed for offsets {}", offsets, exception);
        }
      });
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}

To commit synchronously on rebalances and on close, use the variant of subscribe() that accepts a ConsumerRebalanceListener. The listener has three methods to hook into rebalance behavior: onPartitionsRevoked, onPartitionsAssigned, and onPartitionsLost. The onPartitionsLost method is a default method that calls onPartitionsRevoked unless you override it. The following example commits synchronously in onPartitionsRevoked and when the consumer closes. It overrides onPartitionsLost as a no-op, because the consumer no longer owns those partitions and can’t commit offsets for them:

private void doCommitSync() {
  try {
    consumer.commitSync();
  } catch (WakeupException e) {
    // we're shutting down, but finish the commit first and then
    // rethrow the exception so that the main loop can exit
    doCommitSync();
    throw e;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
  }
}

public void run() {
  try {
    consumer.subscribe(topics, new ConsumerRebalanceListener() {
      @Override
      public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        doCommitSync();
      }

      @Override
      public void onPartitionsAssigned(Collection<TopicPartition> partitions) {}

      @Override
      public void onPartitionsLost(Collection<TopicPartition> partitions) {
        // the partitions are already owned by other consumers, so don't commit
      }
    });

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));
      records.forEach(record -> process(record));
      consumer.commitAsync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    try {
      doCommitSync();
    } finally {
      consumer.close();
      shutdownLatch.countDown();
    }
  }
}

Java client API documentation

For the full Java client API, see the Java client API documentation.

Limitations

Confluent Cloud doesn’t support the following methods: