Kafka Connect ReplaceField (Confluent) SMT Usage Reference for Confluent Platform

The ReplaceField SMT (io.confluent.connect.transforms.ReplaceField) filters or renames fields, including nested fields, within a record’s Struct or Map.

Description

Filter or rename fields within a Struct or Map. Supports nested records.

Use the concrete transformation type designed for the record key (io.confluent.connect.transforms.ReplaceField$Key) or value (io.confluent.connect.transforms.ReplaceField$Value).

For the Apache Kafka® version, see ReplaceField (Apache Kafka).

Properties

Name	Description	Type	Default	Valid values	Importance
exclude	Fields to exclude from the resulting Struct or Map. This takes precedence over the include list.	list	“”	Comma-separated list of fields to exclude.	medium
renames	Field rename mappings. For example, a nested field renaming list is foo.a3.c1:bar,abc.c2.d2:def. Here, foo.a3.c1 represents the field c1 to be renamed to bar value. Similarly, abc.c2.d2 represents the field d2 to be renamed to def value.	list	“”	Comma-separated list of colon-delimited renaming pairs, for example, foo:bar,abc:xyz.	medium
include	Fields to include. If specified, only the named fields will be included in the resulting Struct or Map.	list	“”	Comma-separated list of fields to include.	medium
replace.null.with.default	Specifies whether to replace null fields with their defined default values. When set to true, the default value is applied. Otherwise, the field remains null.	boolean	true	true or false	medium
delimiter	Special characters act as a delimiter. For example, a period (.) in foo.a3.c1.	string	“.”	Any special character except a colon (:). For example, $ or -.	medium

Examples

These examples show how to configure and use ReplaceField to drop or rename a field.

Including all fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, and rename specific nested fields within the foo structure to xyz, bar, luck, and happy respectively.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy"

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{etc=etc, bar={a1=k1, a3={b2={happy=f4}, luck={c1=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}, abc=42}

Including one-level field

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, include only the foo.a3 field, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a3"

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}, luck={c1=f4}}}}

Including multi-level fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, include only the foo.a3.b2.c1 and foo.a2 fields, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a3.b2.c1,foo.a2"

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}}

Excluding multi-level fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont, foo.a1, foo.a2.b2, foo.a3.b1.c1 fields, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont,foo.a1,foo.a2.b2,foo.a3.b1.c1",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": ""

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{etc=etc, bar={a3={b2={happy=f4}, luck={}}, xyz={b1={c1=f4}}}, abc=42}

Excluding none

The configuration snippet shows how to use ReplaceField transformation to only rename specific fields within the foo to xyz, bar, luck, and happy without excluding or including any fields.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": ""

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{bar={a1=k1, a3={b2={happy=f4}, luck={c1=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}, abc=42, etc=etc, dont=whatever}

Fields in both including and excluding lists

The configuration snippet shows how to use ReplaceField transformation to rename certain fields, while also specifying which fields to include or exclude. In the following example, only included fields should be subject to renaming and transformed in the final output.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "foo.a1,foo.a2",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a2,foo.a3.b2.c1"

This transforms the original message as shown:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}}}}

Predicates

Configure transformations with predicates to ensure they process only records that satisfy a particular condition. You can also use predicates in a transformation chain with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Platform to conditionally filter specific records. For more information, see Predicates.