The following provides usage information for the Apache Kafka® SMT org.apache.kafka.connect.transforms.ValueToKey.


Replace the record key with a new key formed from a subset of fields in the record value.


These examples show how to use ValueToKey by itself and in conjunction with a second SMT.

Transform Fields to Message Key

This configuration snippet shows how to use ValueToKey to transform the UserId, city, and state fields into a message key.

"transforms": "ValueToKey",

Before: {"userId": 12, "address": "1942 Wilhelm Boulevard", "city": "Topeka", "state": "KS", "country": "US"}

After: {"userId": 12, "city": "Topeka", "state": "KS"}

Chained Transformation

You can use SMTs together to perform a more complex transformation.

The following examples show how the ValueToKey and ExtractField SMTs are chained together to set the key for data coming from a JDBC Source Connector. During the transform, ValueToKey copies the message c1 field into the message key and then ExtractField extracts just the integer portion of that field.


The following shows what the message looked like before the transform.

"./bin/kafka-avro-console-consumer \
                              --bootstrap-server localhost:9092 \
                              --property schema.registry.url=http://localhost:8081 \
                              --property print.key=true \
                              --from-beginning \
                              --topic mysql-foobar

null {"c1":{"int":1},"c2":{"string":"foo"},"create_ts":1501796305000,"update_ts":1501796305000}
null {"c1":{"int":2},"c2":{"string":"foo"},"create_ts":1501796665000,"update_ts":1501796665000}

After the connector configuration is applied, new rows are inserted (piped) into the MySQL table:

"echo "insert into foobar (c1,c2) values (100,'bar');"|mysql --user=username --password=pw demo

The following is displayed in the Avro console consumer. Note that the key (the first value on the line) matches the value of c1, which was defined with the transforms.

100 {"c1":{"int":100},"c2":{"string":"bar"},"create_ts":1501799535000,"update_ts":1501799535000}


Name Description Type Default Valid Values Importance
fields Field names on the record value to extract as the record key. list   non-empty list high


Transformations can be configured with predicates so that the transformation is applied only to records which satisfy a condition. You can use predicates in a transformation chain and, when combined with the Apache Kafka® Filter, predicates can conditionally filter out specific records.

Predicates are specified in the connector configuration. The following properties are used:

  • predicates: A set of aliases for predicates applied to one or more transformations.
  • predicates.$alias.type: Fully qualified class name for the predicate.
  • predicates.$alias.$predicateSpecificConfig: Configuration properties for the predicate.

All transformations have the implicit config properties predicate and negate. A predicular predicate is associated with a transformation by setting the transformation’s predicate configuration to the predicate’s alias. The predicate’s value can be reversed using the negate configuration property.

Kafka Connect includes the following predicates:

  • org.apache.kafka.connect.predicates.TopicNameMatches: Matches records in a topic with a name matching a particular Java regular expression.
  • org.apache.kafka.connect.predicates.HasHeaderKey: Matches records which have a header with the given key.
  • org.apache.kafka.connect.predicates.RecordIsTombstone: Matches tombstone records (that is, records with a null value).

Predicate Examples

Example 1:

You have a source connector that produces records to many different topics and you want to do the following:

  • Filter out the records in the foo topic entirely.
  • Apply the ExtractField transformation with the field name other_field to records in all topics, except the topic bar.

To do this, you need to first filter out the records destined for the topic foo. The Filter transformation removes records from further processing.

Next, you use the TopicNameMatches predicate to apply the transformation only to records in topics which match a certain regular expression. The only configuration property for TopicNameMatches is a Java regular expression used as a pattern for matching against the topic name. The following example shows this configuration:



Using this configuration, ExtractField is then applied only when the topic name of the record is not bar. The reason you can’t use TopicNameMatches directly is because it would apply the transformation to matching topic names, not topic names which do not match. The transformation’s implicit negate configuration properties inverts the set of records which a predicate matches. This configuration addition is shown below:





Example 2:

The following configuration shows how to use a predicate in a transformation chain with the ExtractField transformation and the negate=true configuration property:


The transform t2 is only applied when the predicate has-my-prefix is false (using the negate=true parameter). The predicate is configured by the keys with prefix predicates.has-my-prefix. The predicate class is org.apache.kafka.connect.predicates.TopicNameMatch and it’s pattern parameter has the value my-prefix-.* . With this configuration, the transformation is applied only to records where the topic name does not start with my-prefix-.


The benefit of defining the predicate separately from the transform is it makes it easier to apply the same predicate to multiple transforms. For example, you can have one set of transforms use one predicate and another set of transforms use the same predicate for negation.

Predicate Properties

Name Description Type Default Valid Values Importance
TopicNameMatches A predicate which is true for records with a topic name that matches the configured regular expression. string   non-empty string, valid regex medium
HasHeaderKey A predicate which is true for records with at least one header with the configured name. string   non-empty string medium
RecordIsTombstone A predicate which is true for records which are tombstones (that is, records with a null value).       medium