.. _spooldir_json_source_connector:

=====================
JSON Source Connector
=====================

This connector is used to `stream JSON files <https://en.wikipedia.org/wiki/JSON_Streaming>`_  from a directory while also converting the data based on the schema supplied in the configuration.

To use this connector, use a connector configuration that specifies the name of this connector class in the ``connector.class`` configuration property:

.. codewithvars:: properties

    connector.class=com.github.jcustenborder.kafka.connect.spooldir.SpoolDirJsonSourceConnector

The other connector-specific configuration properties are described below.

.. _spooldir_connector_json_example:

JSON Source Connector Example
-----------------------------

This example follows the same steps as the Quick Start. Review the Quick Start for help running the |cp| and installing the Spool Dir connectors.

#.  Generate a JSON dataset using the command below:

    .. codewithvars:: bash

        curl "https://api.mockaroo.com/api/17c84440?count=500&key=25fd9c80" > "json-spooldir-source.json"

#.  Create a ``spooldir.properties`` file with the following contents:

    .. codewithvars:: properties

        name=JsonSpoolDir
        tasks.max=1
        connector.class=com.github.jcustenborder.kafka.connect.spooldir.SpoolDirJsonSourceConnector
        input.path=/path/to/data
        input.file.pattern=json-spooldir-source.json
        error.path=/path/to/error
        finished.path=/path/to/finished
        halt.on.error=false
        topic=spooldir-json-topic

#.  Load the SpoolDir JSON Source Connector.

    .. include:: ../../../includes/confluent-local-consume-limit.rst

    .. codewithvars:: bash

        |confluent_load| spooldir|dash| -d spooldir.properties

    .. important:: Don't use the :ref:`cli` in production environments.

.. _spooldir_connector_json_schemaless_example:

JSON Schemaless Source Connector Example
----------------------------------------

This example follows the same steps as the Quick Start. Review the Quick Start for help running the |cp| and installing the Spool Dir connectors.

#.  Generate a JSON dataset using the command below:

    .. codewithvars:: bash

        curl "https://api.mockaroo.com/api/17c84440?count=500&key=25fd9c80" > "json-spooldir-source.json"

#.  Create a ``spooldir.properties`` file with the following contents:

    .. codewithvars:: properties

        name=SchemaLessJsonSpoolDir
        tasks.max=1
        connector.class=com.github.jcustenborder.kafka.connect.spooldir.SpoolDirSchemaLessJsonSourceConnector
        input.path=/path/to/data
        input.file.pattern=json-spooldir-source.json
        error.path=/path/to/error
        finished.path=/path/to/finished
        halt.on.error=false
        topic=spooldir-schemaless-json-topic
        value.converter=org.apache.kafka.connect.storage.StringConverter

#.  Load the SpoolDir Schemaless JSON Source Connector.

    .. include:: ../../../includes/confluent-local-consume-limit.rst

    .. codewithvars:: bash

        |confluent_load| spooldir|dash| -d spooldir.properties

    .. important:: Don't use the :ref:`cli` in production environments.


.. _spooldir_json_source_connector_config:

Configuration Properties
------------------------

.. include:: includes/common_config.rst
.. include:: includes/common_file_system_config.rst
.. include:: includes/common_schema_config.rst
.. include:: includes/common_timestamp_parser_config.rst