.. _spooldir_connector:

|kconnect-long| Spool Dir Connectors
=====================================

The |kconnect-long| Spool Dir Connectors provide the capability to watch a directory for files and read the data as new files are written to the input directory. Once a file has been read, it will be placed into the configured ``finished.path`` directory. Each record in the input file will be converted based on the user-supplied schema or an auto-generated schema.

The following connectors are included with the |kconnect| Spool Dir Connector package:

* :ref:`spooldir_csv_source_connector`
* :ref:`spooldir_json_source_connector`
* :ref:`spooldir_schemaless_json_source_connector`
* :ref:`spooldir_line_delimited_source_connector`
* :ref:`spooldir_elf_source_connector`

Install Spool Dir Connectors
----------------------------

.. include:: ../includes/connector-install.rst

.. include:: ../includes/connector-install-hub.rst

.. codewithvars:: bash

   confluent-hub install jcustenborder/kafka-connect-spooldir:latest

.. include:: ../includes/connector-install-version.rst

.. codewithvars:: bash

   confluent-hub install jcustenborder/kafka-connect-spooldir:1.0.31


--------------------------
Install Connector Manually
--------------------------

`Download and extract the ZIP file <https://www.confluent.io/connector/kafka-connect-spooldir/#download>`_ for your
connector and then follow the manual connector installation :ref:`instructions <connect_install_connectors>`.

.. _spooldir_connector_license:

License
-------

The Spool Dir connector is an open source connector and does not require a Confluent Enterprise License.

.. _spooldir_connector_quickstart:

Quick Start
------------

The following steps show the ``SpoolDirCsvSourceConnector`` loading a mock CSV file to an |ak| topic named ``spooldir-testing-topic``. The other connectors are similar but load from different file types.

#.  Install the connector through the :ref:`confluent_hub_client`.

    .. codewithvars:: bash

        # run from your Confluent Platform installation directory
        confluent-hub install jcustenborder/kafka-connect-spooldir:latest

    .. tip:: By default, it will install the plugin into ``share/confluent-hub-components`` and add the directory to the plugin path. If this is the first connector you have installed, you may need to restart the connect server for the plugin path change to take effect.

#.  Start the Confluent Platform.

    .. codewithvars:: bash

        |confluent_start|

#.  Create a data directory and generate test data.

    .. codewithvars:: bash

        mkdir data && curl "https://api.mockaroo.com/api/58605010?count=1000&key=25fd9c80" > "data/csv-spooldir-source.csv"

#.  Set up directories for files with errors and files that finished successfully.

    .. codewithvars:: bash

        mkdir error && mkdir finished

#.  Create a ``spooldir.json`` file with the following contents:

    .. codewithvars:: json

        {
          "name": "CsvSpoolDir",
          "config": {
            "tasks.max": "1",
            "connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector",
            "input.path": "/path/to/data",
            "input.file.pattern": "csv-spooldir-source.csv",
            "error.path": "/path/to/error",
            "finished.path": "/path/to/finished",
            "halt.on.error": "false",
            "topic": "spooldir-testing-topic",
            "csv.first.row.as.header": "true",
            "schema.generation.enabled": "true"
          }
        }

#.  Load the SpoolDir CSV Source Connector.

    .. include:: ../../includes/confluent-local-consume-limit.rst

    .. codewithvars:: bash

       |confluent_load| spooldir|dash| -d spooldir.json

    .. important:: Don't use the :ref:`cli` in production environments.

#.  Confirm that the connector is in a ``RUNNING`` state.

    .. codewithvars:: bash

        |confluent_status| spooldir

#.  Confirm that the messages are being sent to |ak|.

    .. codewithvars:: bash

        kafka-avro-console-consumer \
            --bootstrap-server localhost:9092 \
            --property schema.registry.url=http://localhost:8081 \
            --topic spooldir-testing-topic \
            --from-beginning | jq '.'

#.  Confirm that the source CSV file has been moved to the ``finished`` directory.

.. _spooldir_connector_examples:

Examples
--------

* :ref:`spooldir_connector_csv_with_schema_example`
* :ref:`spooldir_connector_tsv_example`
* :ref:`spooldir_connector_json_schemaless_example`
* :ref:`spooldir_connector_json_example`
* :ref:`spooldir_connector_json_line_delimited_example`


Additional Documentation
------------------------

.. toctree::
    :maxdepth: 1
    :titlesonly:
    :glob:

    connectors/*