Kafka Connect Single Message Transform Reference for Confluent Platform¶
Single Message Transformations (SMTs) are applied to messages as they flow through Connect. SMTs transform inbound messages after a source connector has produced them, but before they are written to Kafka. SMTs transform outbound messages before they are sent to a sink connector. The following SMTs are available for use with Kafka Connect.
Tip
For a tutorial and a deep dive into this topic, see How to Use Single Message Transforms in Kafka Connect.
If none of the available SMTs provide the necessary transformation, you can create your own for Confluent Platform. For more information, see Create Custom Kafka Connect Single Message Transforms for Confluent Platform.
Caution
This document contains SMT information that applies to all connectors, self-managed and fully-managed. However, some information linked to from this page may go to platform-specific docs that do not apply to fully-managed connectors. For instance, RegexRouter and Custom SMTs are not currently available with managed connector SMTs.
| Transform | Description |
|---|---|
| Cast | Cast fields or the entire key or value to a specific type (for example, to force an integer field to a smaller width). |
| Drop | Drop either a key or a value from a record and set it to null. |
| DropHeaders | Not currently available for managed connectors. Drop one or more headers from each record. |
| EventRouter | Only available for managed Debezium connectors. Route Debezium outbox events using a regex configuration option. |
| ExtractField | Extract the specified field from a Struct when schema present, or a Map in the case of schemaless data. Any null values are passed through unmodified. |
| ExtractTopic | Replace the record topic with a new topic derived from its key or value. |
| Filter (Apache Kafka) | Drop all records. Designed to be used in conjunction with a Predicate. |
| Filter (Confluent) | Include or drop records that match a configurable filter.condition. |
| Flatten | Flatten a nested data structure. This generates names for each field by concatenating the field names at each level with a configurable delimiter character. |
| GzipDecompress | Not currently available for managed connectors. Gzip-decompress the entire byteArray key or value input. |
| HeaderFrom | Not currently available for managed connectors. Moves or copies fields in a record key or value into the record’s header. |
| HoistField | Wrap data using the specified field name in a Struct when schema present, or a Map in the case of schemaless data. |
| InsertField | Insert field using attributes from the record metadata or a configured static value. |
| InsertHeader | Not currently available for managed connectors. Insert a literal value as a record header. |
| MaskField | Mask specified fields with a valid null value for the field type. |
| MessageTimeStampRouter | Update the record’s topic field as a function of the original topic value and the record’s timestamp field. |
| RegexRouter | Not currently available for managed connectors. Update the record topic using the configured regular expression and replacement string. |
| ReplaceField (Apache Kafka) | Filter or rename fields. |
| ReplaceField (Confluent) | Filter or rename nested fields. |
| SetSchemaMetadata | Set the schema name, version, or both on the record’s key or value schema. |
| TimestampConverter | Convert timestamps between different formats such as Unix epoch, strings, and Connect Date and Timestamp types. |
| TimestampRouter | Update the record’s topic field as a function of the original topic value and the record timestamp. |
| TombstoneHandler | Manage tombstone records. A tombstone record is defined as a record with the entire value field being null, whether or not it has ValueSchema. |
| TopicRegexRouter | Only available for managed Source connectors. Update the record topic using the configured regular expression and replacement string. |
| ValueToKey | Replace the record key with a new key formed from a subset of fields in the record value. |