Kafka Connect ReplaceField (Confluent) SMT Usage Reference for Confluent Cloud or Confluent Platform

The following provides usage information for the Confluent SMT io.confluent.connect.transforms.ReplaceField.

Description

Filter or rename fields within a Struct or Map.

Use the concrete transformation type designed for the record key (io.confluent.connect.transforms.ReplaceField$Key) or value (io.confluent.connect.transforms.ReplaceField$Value).

Examples

These examples show how to configure and use ReplaceField to drop or rename a field.

Including all fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, and rename specific nested fields within the foo structure to xyz, bar, luck, and happy respectively.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy"

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{etc=etc, bar={a1=k1, a3={b2={happy=f4}, luck={c1=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}, abc=42}

Including one-level field

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, include only the foo.a3 field, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a3"

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}, luck={c1=f4}}}}

Including multi-level fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont field, include only the foo.a3.b2.c1 and foo.a2 fields, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a3.b2.c1,foo.a2"

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}}

Excluding multi-level fields

The configuration snippet shows how to use ReplaceField transformation to exclude the dont, foo.a1, foo.a2.b2, foo.a3.b1.c1 fields, and rename specific fields within the foo to xyz, bar, luck, and happy.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "dont,foo.a1,foo.a2.b2,foo.a3.b1.c1",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": ""

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{etc=etc, bar={a3={b2={happy=f4}, luck={}}, xyz={b1={c1=f4}}}, abc=42}

Excluding none

The configuration snippet shows how to use ReplaceField transformation to only rename specific fields within the foo to xyz, bar, luck, and happy without excluding or including any fields.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": ""

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{bar={a1=k1, a3={b2={happy=f4}, luck={c1=f4}}, xyz={b2={c1=f4}, b1={c1=f4}}}, abc=42, etc=etc, dont=whatever}

Fields in both including and excluding lists

The configuration snippet shows how to use ReplaceField transformation to rename certain fields, while also specifying which fields to include or exclude. In the following example, only included fields should be subject to renaming and transformed in the final output.

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "io.confluent.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.exclude": "foo.a1,foo.a2",
"transforms.ReplaceField.renames": "foo.a2:xyz,foo:bar,foo.a3.b1:luck,foo.a3.b2.c1:happy",
"transforms.ReplaceField.include": "foo.a2,foo.a3.b2.c1"

This transforms the original message as seen below:

Before:

{abc=42, etc=etc, foo={a1=k1, a2={b2={c1=f4}, b1={c1=f4}}, a3={b2={c1=f4}, b1={c1=f4}}}, dont=whatever}

After:

{foo={a3={b2={happy=f4}}}}

Tip

For additional examples, see Replace Field for managed connectors.

Properties

Name	Description	Type	Default	Valid Values	Importance
`exclude`	Fields to exclude from the resulting Struct or Map. This takes precedence over the include list.	list	“”	Comma-separated list of fields to exclude.	medium
`renames`	Field rename mappings. For example, a nested field renaming list is `foo.a3.c1:bar,abc.c2.d2:def`. Here, `foo.a3.c1` represents the field `c1` to be renamed to `bar` value. Similarly, `abc.c2.d2` represents the field `d2` to be renamed to `def` value.	list	“”	Comma-separated list of colon-delimited renaming pairs, e.g. `foo:bar,abc:xyz`	medium
`include`	Fields to include. If specified, only the named fields will be included in the resulting Struct or Map.	list	“”	Comma-separated list of fields to include.	medium
`replace.null.with.default`	Fields to replace null with the default value. If set to `true`, it replaces null with the default value. Otherwise, leave them as null.	boolean	“”	true or false	medium
`delimiter`	Special characters act as a delimiter. For example, a period (.) in `foo.a3.c1`.	string	“.”	Special characters. For example, `$, -` and cannot be a colon (:).	medium

Predicates

Transformations can be configured with predicates so that the transformation is applied only to records which satisfy a condition. You can use predicates in a transformation chain and, when combined with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Cloud or Confluent Platform, predicates can conditionally filter out specific records. For details and examples, see Predicates.