org.apache.kafka.connect.sink

Class SinkTask

java.lang.Object

org.apache.kafka.connect.sink.SinkTask

All Implemented Interfaces:

Task

public abstract class SinkTask
extends Object
implements Task

SinkTask is a Task that takes records loaded from Kafka and sends them to another system. Each task instance is assigned a set of partitions by the Connect framework and will handle all records received from those partitions. As records are fetched from Kafka, they will be passed to the sink task using the put(Collection) API, which should either write them to the downstream system or batch them for later writing. Periodically, Connect will call flush(Map) to ensure that batched records are actually pushed to the downstream system.

Below we describe the lifecycle of a SinkTask.

1. Initialization: SinkTasks are first initialized using initialize(SinkTaskContext) to prepare the task's context and start(Map) to accept configuration and start any services needed for processing.
2. Partition Assignment: After initialization, Connect will assign the task a set of partitions using open(Collection). These partitions are owned exclusively by this task until they have been closed with close(Collection).
3. Record Processing: Once partitions have been opened for writing, Connect will begin forwarding records from Kafka using the put(Collection) API. Periodically, Connect will ask the task to flush records using flush(Map) as described above.
4. Partition Rebalancing: Occasionally, Connect will need to change the assignment of this task. When this happens, the currently assigned partitions will be closed with close(Collection) and the new assignment will be opened using open(Collection).
5. Shutdown: When the task needs to be shutdown, Connect will close active partitions (if there are any) and stop the task using stop()

Field Summary

Fields

Modifier and Type	Field and Description
static String	TOPICS_CONFIG The configuration key that provides the list of topics that are inputs for this SinkTask.
static String	TOPICS_REGEX_CONFIG The configuration key that provides a regex specifying which topics to include as inputs for this SinkTask.

Constructor Summary

Constructors

Constructor and Description
SinkTask()

Method Summary

Modifier and Type	Method and Description
void	close(Collection<org.apache.kafka.common.TopicPartition> partitions) The SinkTask uses this method to close writers for partitions that are no longer assigned to the SinkTask.
void	flush(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> currentOffsets) Flush all records that have been put(Collection) for the specified topic-partitions.
void	initialize(SinkTaskContext context) Initialize the context of this task.
void	onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions) Deprecated. Use open(Collection) for partition initialization.
void	onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions) Deprecated. Use close(Collection) instead for partition cleanup.
void	open(Collection<org.apache.kafka.common.TopicPartition> partitions) The SinkTask uses this method to create writers for newly assigned partitions in case of partition rebalance.
Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>	preCommit(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> currentOffsets) Pre-commit hook invoked prior to an offset commit.
abstract void	put(Collection<SinkRecord> records) Put the records in the sink.
abstract void	start(Map<String,String> props) Start the Task.
abstract void	stop() Perform any cleanup to stop this task.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.connect.connector.Task

version

Field Detail

TOPICS_CONFIG

public static final String TOPICS_CONFIG

The configuration key that provides the list of topics that are inputs for this SinkTask.

See Also:

Constant Field Values

TOPICS_REGEX_CONFIG

public static final String TOPICS_REGEX_CONFIG

The configuration key that provides a regex specifying which topics to include as inputs for this SinkTask.

See Also:

Constant Field Values

Constructor Detail

SinkTask

public SinkTask()

Method Detail

initialize

public void initialize(SinkTaskContext context)

Initialize the context of this task. Note that the partition assignment will be empty until Connect has opened the partitions for writing with open(Collection).

Parameters:

context - The sink task's context

start

public abstract void start(Map<String,String> props)

Start the Task. This should handle any configuration parsing and one-time setup of the task.

Specified by:

start in interface Task

Parameters:

props - initial configuration

put

public abstract void put(Collection<SinkRecord> records)

Put the records in the sink. This should either write them to the downstream system or batch them for later writing. If this method returns before the records are written to the downstream system, the task must implement flush(Map) or preCommit(Map) to ensure that offsets are only committed for records that have been written to the downstream system (hence avoiding data loss during failures).

If this operation fails, the SinkTask may throw a RetriableException to indicate that the framework should attempt to retry the same call again. Other exceptions will cause the task to be stopped immediately. SinkTaskContext.timeout(long) can be used to set the maximum time before the batch will be retried.

Parameters:

records - the collection of records to send

flush

public void flush(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> currentOffsets)

Flush all records that have been put(Collection) for the specified topic-partitions.

Parameters:

currentOffsets - the current offset state as of the last call to put(Collection), provided for convenience but could also be determined by tracking all offsets included in the SinkRecords passed to put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>). Note that the topic, partition and offset here correspond to the original Kafka topic partition and offset, before any transformations have been applied. These can be tracked by the task through the SinkRecord.originalTopic(), SinkRecord.originalKafkaPartition() and SinkRecord.originalKafkaOffset() methods.

preCommit

public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> preCommit(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> currentOffsets)

Pre-commit hook invoked prior to an offset commit.

The default implementation simply invokes flush(Map) and is thus able to assume all currentOffsets are safe to commit.

Parameters:

currentOffsets - the current offset state as of the last call to put(Collection), provided for convenience but could also be determined by tracking all offsets included in the SinkRecords passed to put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>). Note that the topic, partition and offset here correspond to the original Kafka topic partition and offset, before any transformations have been applied. These can be tracked by the task through the SinkRecord.originalTopic(), SinkRecord.originalKafkaPartition() and SinkRecord.originalKafkaOffset() methods.

Returns:

an empty map if Connect-managed offset commit is not desired, otherwise a map of offsets by topic-partition that are safe to commit. Note that the returned topic-partition to offsets map should use the original Kafka topic partitions and offsets instead of the transformed values.

open

public void open(Collection<org.apache.kafka.common.TopicPartition> partitions)

The SinkTask uses this method to create writers for newly assigned partitions in case of partition rebalance. This method will be called after partition re-assignment completes and before the SinkTask starts fetching data. Any errors raised from this method will cause the task to stop.

Note that the topic partitions here correspond to the original Kafka topic partitions, before any transformations have been applied.

Parameters:

partitions - The list of partitions that are now assigned to the task (may include partitions previously assigned to the task)

onPartitionsAssigned

@Deprecated
public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)

Deprecated. Use open(Collection) for partition initialization.

close

public void close(Collection<org.apache.kafka.common.TopicPartition> partitions)

The SinkTask uses this method to close writers for partitions that are no longer assigned to the SinkTask. This method will be called before a rebalance operation starts and after the SinkTask stops fetching data. After being closed, Connect will not write any records to the task until a new set of partitions has been opened. Any errors raised from this method will cause the task to stop.

Note that the topic partitions here correspond to the original Kafka topic partitions, before any transformations have been applied.

Parameters:

partitions - The list of partitions that should be closed

onPartitionsRevoked

@Deprecated
public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)

Deprecated. Use close(Collection) instead for partition cleanup.

stop

public abstract void stop()

Perform any cleanup to stop this task. In SinkTasks, this method is invoked only once outstanding calls to other methods have completed (e.g., put(Collection) has returned) and a final flush(Map) and offset commit has completed. Implementations of this method should only need to perform final cleanup operations, such as closing network connections to the sink system.

Specified by:

stop in interface Task