Kafka Connect Flatten SMT Usage Reference for Confluent Platform

The Flatten SMT (org.apache.kafka.connect.transforms.Flatten) flattens a nested Apache Kafka® record key or value into a flat map by concatenating field names with a configurable delimiter.

Description

Flatten a nested data structure, generating names for each field by concatenating the field names at each level with a configurable delimiter character. Applies to a Struct when a schema is present, or a Map for schemaless data.

The default delimiter is the period (.) character.

Use the specific transformation type designed for the record key (org.apache.kafka.connect.transforms.Flatten$Key) or value (org.apache.kafka.connect.transforms.Flatten$Value).

Note

The Flatten SMT does not flatten arrays. Arrays are preserved as-is.
When using an Avro schema, set delimiter to _ because Avro field names allow only alphanumeric characters and underscore. For the full configuration, see Flatten with the underscore delimiter (for Avro schemas).

Properties

Name	Description	Type	Default	Valid Values	Importance
`delimiter`	Delimiter to insert between field names from the input record when generating field names for the output record.	string	`.`	Any string (commonly `.` or `_`)	medium

Flatten with the period delimiter (for JSON schemas)

The following configuration snippet shows how to use Flatten to concatenate field names with the period . delimiter character.

"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "."

Before:

{
  "content": {
    "id": 42,
    "name": {
      "first": "David",
      "middle": null,
      "last": "Wong"
    }
  }
}

After:

{
  "content.id": 42,
  "content.name.first": "David",
  "content.name.middle": null,
  "content.name.last": "Wong"
}

Flatten with the underscore delimiter (for Avro schemas)

Set delimiter to _ when using Avro schemas. The Avro schema specification only allows alphanumeric characters and underscore in field names. The following configuration snippet shows how to use Flatten to concatenate field names with the underscore _ delimiter.

"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "_"

Before:

{
  "content": {
    "id": 42,
    "name": {
      "first": "David",
      "middle": null,
      "last": "Wong"
    }
  }
}

After:

{
  "content_id": 42,
  "content_name_first": "David",
  "content_name_middle": null,
  "content_name_last": "Wong"
 }

Predicates

Configure transformations with predicates to ensure they only process records satisfying a particular condition. You can also use predicates in a transformation chain along with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Platform to conditionally filter specific records. For more information, refer to Predicates.