.. _filter:

.. meta::
   :description: This document provides usage information for the Confluent Filter SMT io.confluent.connect.transforms.Filter.

============
Filter
============

The following provides usage information for the Confluent SMT
``io.confluent.connect.transforms.Filter``.

Description
-----------

Include or drop records that match the ``filter.condition`` predicate.

The ``filter.condition`` is a predicate specifying `JSON path <https://github.com/json-path/JsonPath>`__ that is applied to each record processed, and when this predicate successfully matches the record is either included (when ``filter.type=include``) or excluded (when ``filter.type=exclude``).

The ``missing.or.null.behavior`` property defines how the transform behaves when a record does not have the field(s) used in the filter condition predicate. By default, the behavior is set to fail. However, the behavior can also be set to include or exclude the record that is missing the predicate's field(s).

Use the transformation type designed for the record key
(``io.confluent.connect.transforms.Filter$Key``) or value
(``io.confluent.connect.transforms.Filter$Value``).

Installation
------------

This transformation is developed by Confluent and does not ship by default with |ak| or |cp|. You can install this transformation using the :ref:`Confluent Hub Client <confluent_hub_client>`:

.. sourcecode:: properties

    confluent-hub install confluentinc/connect-transforms:latest

Examples
--------

The configuration snippets below shows how to use and configure the ``Filter`` SMT.

The ``filter.condition`` is a predicate specifying JSON path. For more
information on JSON path format, see `JSON path <https://github.com/json-path/JsonPath>`__. The ``filter.condition`` predicate specified in the examples below is satisfied when the nested field ``'nestedKey'`` in field ``'key'`` is equal to either ``'value1'`` or ``'value2'``.

``filterExample1`` below defines the configuration for ``including`` records satisfying the filter condition. The filter condition is applied to record ``value``. The ``fail`` behavior is configured to throw an exception and **fail the connector task** when the record does not have the field(s) used in the filter condition.

.. sourcecode:: properties

    transforms=filterExample1
    # Use the 'io.confluent.connect.transforms.Filter$Value' as the source (record 'value') on which
    # 'filter.condition' shall be applied.
    transforms.filterExample1.type=io.confluent.connect.transforms.Filter$Value
    transforms.filterExample1.filter.condition=$.key[?(@.nestedKey == "value1" || @nestedKey == "value2")]
    # Use 'include' to pass through all records that match the predicate.
    transforms.filterExample1.filter.type=include
    # Use the 'fail' behavior to throw an exception and fail the connector task when the record does
    # not have the field(s) used in the 'filter.condition'.
    transforms.filterExample1.missing.or.null.behavior=fail

``filterExample2`` below defines the configuration for ``excluding`` records satisfying the filter condition. The filter condition is applied to record ``key``. The ``include`` behavior is configured to **pass the record through** when the record does not have the field(s) used in the filter condition.

.. sourcecode:: properties

    transforms=filterExample2
    # Use the 'io.confluent.connect.transforms.Filter$Key' as the source (record 'key') on which
    # 'filter.condition' shall be applied.
    transforms.filterExample2.type=io.confluent.connect.transforms.Filter$Key
    transforms.filterExample2.filter.condition=$.key[?(@.nestedKey == "value1" || @nestedKey == "value2")]
    # Use 'exclude' to drop all records that match the predicate.
    transforms.filterExample2.filter.type=exclude
    # Use the 'include' behavior to pass the record through when the record does not have the field(s)
    # used in the 'filter.condition'.
    transforms.filterExample2.missing.or.null.behavior=include

Properties
----------

.. list-table::
   :header-rows: 1
   :widths: 15 30 10 10 10 10

   *  - Name
      - Description
      - Type
      - Default
      - Valid Values
      - Importance
   *  - ``filter.condition``
      - Specifies the criteria used to match records to be included or excluded
        by this transformation. Use JSON Path predicate notation defined in:
        https://github.com/json-path/JsonPath.
      - string
      -
      -
      - high
   *  - ``filter.type``
      - Specifies the action to perform with records that match the
        ``filter.condition`` predicate. Use ``include`` to pass through all
        records that match the predicate and drop all records that do not
        satisfy the predicate, or use ``exclude`` to drop all records that match
        the predicate.
      - string
      -
      - [include, exclude]
      - high
   *  - ``missing.or.null.behavior``
      - Specifies the behavior when the record does not have the field(s) used
        in the ``filter.condition``. Use ``fail`` to throw an exception and fail
        the connector task, ``include`` to pass the record through, or
        ``exclude`` to drop the record.
      - string
      - ``fail``
      - [fail, include, exclude]
      - medium