org.apache.kafka.connect.sink

Class SinkRecord

java.lang.Object

org.apache.kafka.connect.connector.ConnectRecord<SinkRecord>

org.apache.kafka.connect.sink.SinkRecord

public class SinkRecord
extends ConnectRecord<SinkRecord>

SinkRecord is a ConnectRecord that has been read from Kafka and includes the original Kafka record's topic, partition and offset (before any transformations have been applied) in addition to the standard fields. This information should be used by the SinkTask to coordinate offset commits.

It also includes the TimestampType, which may be TimestampType.NO_TIMESTAMP_TYPE, and the relevant timestamp, which may be null.

Constructor Summary

Constructors

Constructor and Description
SinkRecord(String topic, int partition, Schema keySchema, Object key, Schema valueSchema, Object value, long kafkaOffset)
SinkRecord(String topic, int partition, Schema keySchema, Object key, Schema valueSchema, Object value, long kafkaOffset, Long timestamp, org.apache.kafka.common.record.TimestampType timestampType)
SinkRecord(String topic, int partition, Schema keySchema, Object key, Schema valueSchema, Object value, long kafkaOffset, Long timestamp, org.apache.kafka.common.record.TimestampType timestampType, Iterable<Header> headers)
SinkRecord(String topic, int partition, Schema keySchema, Object key, Schema valueSchema, Object value, long kafkaOffset, Long timestamp, org.apache.kafka.common.record.TimestampType timestampType, Iterable<Header> headers, String originalTopic, Integer originalKafkaPartition, long originalKafkaOffset) This constructor is intended for use by the Connect runtime only and plugins (sink connectors or transformations) should not use this directly outside testing code.

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object o)
int	hashCode()
long	kafkaOffset()
SinkRecord	newRecord(String topic, Integer kafkaPartition, Schema keySchema, Object key, Schema valueSchema, Object value, Long timestamp) Create a new record of the same type as itself, with the specified parameter values.
SinkRecord	newRecord(String topic, Integer kafkaPartition, Schema keySchema, Object key, Schema valueSchema, Object value, Long timestamp, Iterable<Header> headers) Create a new record of the same type as itself, with the specified parameter values.
long	originalKafkaOffset() Get the original offset for this sink record, before any transformations were applied.
Integer	originalKafkaPartition() Get the original topic partition for this sink record, before any transformations were applied.
String	originalTopic() Get the original topic for this sink record, before any transformations were applied.
org.apache.kafka.common.record.TimestampType	timestampType()
String	toString()

Methods inherited from class org.apache.kafka.connect.connector.ConnectRecord

headers, kafkaPartition, key, keySchema, timestamp, topic, value, valueSchema

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

SinkRecord

public SinkRecord(String topic,
                  int partition,
                  Schema keySchema,
                  Object key,
                  Schema valueSchema,
                  Object value,
                  long kafkaOffset)

SinkRecord

public SinkRecord(String topic,
                  int partition,
                  Schema keySchema,
                  Object key,
                  Schema valueSchema,
                  Object value,
                  long kafkaOffset,
                  Long timestamp,
                  org.apache.kafka.common.record.TimestampType timestampType)

SinkRecord

public SinkRecord(String topic,
                  int partition,
                  Schema keySchema,
                  Object key,
                  Schema valueSchema,
                  Object value,
                  long kafkaOffset,
                  Long timestamp,
                  org.apache.kafka.common.record.TimestampType timestampType,
                  Iterable<Header> headers)

SinkRecord

public SinkRecord(String topic,
                  int partition,
                  Schema keySchema,
                  Object key,
                  Schema valueSchema,
                  Object value,
                  long kafkaOffset,
                  Long timestamp,
                  org.apache.kafka.common.record.TimestampType timestampType,
                  Iterable<Header> headers,
                  String originalTopic,
                  Integer originalKafkaPartition,
                  long originalKafkaOffset)

This constructor is intended for use by the Connect runtime only and plugins (sink connectors or transformations) should not use this directly outside testing code.

Method Detail

kafkaOffset

public long kafkaOffset()

timestampType

public org.apache.kafka.common.record.TimestampType timestampType()

originalTopic

public String originalTopic()

Get the original topic for this sink record, before any transformations were applied. In order to be compatible with transformations that mutate topic names, this method should be used by sink tasks instead of ConnectRecord.topic() for any internal offset tracking purposes (for instance, reporting offsets to the Connect runtime via SinkTask.preCommit(Map)).

This method was added in Apache Kafka 3.6. Sink connectors that use this method but want to maintain backward compatibility in order to be able to be deployed on older Connect runtimes should guard the call to this method with a try-catch block, since calling this method will result in a NoSuchMethodError when the sink connector is deployed to Connect runtimes older than Kafka 3.6. For example:

 String originalTopic;
 try {
     originalTopic = record.originalTopic();
 } catch (NoSuchMethodError e) {
     log.warn("This connector is not compatible with SMTs that mutate topic names, topic partitions or offset values on this version of Kafka Connect");
     originalTopic = record.topic();
 }
 
 

Note that sink connectors that do their own offset tracking will be incompatible with SMTs that mutate topic names when deployed to older Connect runtimes that do not support this method.

Returns:

the topic for this record before any transformations were applied

Since:

3.6

originalKafkaPartition

public Integer originalKafkaPartition()

Get the original topic partition for this sink record, before any transformations were applied. In order to be compatible with transformations that mutate topic partitions, this method should be used by sink tasks instead of ConnectRecord.kafkaPartition() for any internal offset tracking purposes (for instance, reporting offsets to the Connect runtime via SinkTask.preCommit(Map)).

This method was added in Apache Kafka 3.6. Sink connectors that use this method but want to maintain backward compatibility in order to be able to be deployed on older Connect runtimes should guard the call to this method with a try-catch block, since calling this method will result in a NoSuchMethodError when the sink connector is deployed to Connect runtimes older than Kafka 3.6. For example:

 String originalKafkaPartition;
 try {
     originalKafkaPartition = record.originalKafkaPartition();
 } catch (NoSuchMethodError e) {
     log.warn("This connector is not compatible with SMTs that mutate topic names, topic partitions or offset values on this version of Kafka Connect");
     originalKafkaPartition = record.kafkaPartition();
 }
 
 

Note that sink connectors that do their own offset tracking will be incompatible with SMTs that mutate topic partitions when deployed to older Connect runtimes that do not support this method.

Returns:

the topic partition for this record before any transformations were applied

Since:

3.6

originalKafkaOffset

public long originalKafkaOffset()

Get the original offset for this sink record, before any transformations were applied. In order to be compatible with transformations that mutate offset values, this method should be used by sink tasks instead of kafkaOffset() for any internal offset tracking purposes (for instance, reporting offsets to the Connect runtime via SinkTask.preCommit(Map)).

This method was added in Apache Kafka 3.6. Sink connectors that use this method but want to maintain backward compatibility in order to be able to be deployed on older Connect runtimes should guard the call to this method with a try-catch block, since calling this method will result in a NoSuchMethodError when the sink connector is deployed to Connect runtimes older than Kafka 3.6. For example:

 String originalKafkaOffset;
 try {
     originalKafkaOffset = record.originalKafkaOffset();
 } catch (NoSuchMethodError e) {
     log.warn("This connector is not compatible with SMTs that mutate topic names, topic partitions or offset values on this version of Kafka Connect");
     originalKafkaOffset = record.kafkaOffset();
 }
 
 

Note that sink connectors that do their own offset tracking will be incompatible with SMTs that mutate offset values when deployed to older Connect runtimes that do not support this method.

Returns:

the offset for this record before any transformations were applied

Since:

3.6

newRecord

public SinkRecord newRecord(String topic,
                            Integer kafkaPartition,
                            Schema keySchema,
                            Object key,
                            Schema valueSchema,
                            Object value,
                            Long timestamp)

Description copied from class: ConnectRecord

Create a new record of the same type as itself, with the specified parameter values. All other fields in this record will be copied over to the new record. Since the headers are mutable, the resulting record will have a copy of this record's headers.

Specified by:

newRecord in class ConnectRecord<SinkRecord>

Parameters:

topic - the name of the topic; may be null

kafkaPartition - the partition number for the Kafka topic; may be null

keySchema - the schema for the key; may be null

key - the key; may be null

valueSchema - the schema for the value; may be null

value - the value; may be null

timestamp - the timestamp; may be null

Returns:

the new record

newRecord

public SinkRecord newRecord(String topic,
                            Integer kafkaPartition,
                            Schema keySchema,
                            Object key,
                            Schema valueSchema,
                            Object value,
                            Long timestamp,
                            Iterable<Header> headers)

Description copied from class: ConnectRecord

Create a new record of the same type as itself, with the specified parameter values. All other fields in this record will be copied over to the new record.

Specified by:

newRecord in class ConnectRecord<SinkRecord>

Parameters:

topic - the name of the topic; may be null

kafkaPartition - the partition number for the Kafka topic; may be null

keySchema - the schema for the key; may be null

key - the key; may be null

valueSchema - the schema for the value; may be null

value - the value; may be null

timestamp - the timestamp; may be null

headers - the headers; may be null or empty

Returns:

the new record

equals

public boolean equals(Object o)

Overrides:

equals in class ConnectRecord<SinkRecord>

hashCode

public int hashCode()

Overrides:

hashCode in class ConnectRecord<SinkRecord>

toString

public String toString()

Overrides:

toString in class ConnectRecord<SinkRecord>