# Spool Dir Connectors for Confluent Platform The Kafka Connect Spool Dir Source connectors watch a directory for files and read the data as new files are written to the input directory. Once a file has been read, it will be placed into the configured `finished.path` directory. Each record in the input file will be converted based on the user-supplied schema or an auto-generated schema. The following connectors are included with the Connect Spool Dir Source Connector package: #### IMPORTANT Effective July 6, 2025, only self-managed connector versions that meet or exceed the minimum version listed on the [Supported Connector Versions](https://docs.confluent.io/platform/7.8/connect/supported-connector-version.html#supported-connector-versions-till-cp-7-8) page receive support from Confluent. Older, unsupported connector versions have been removed from Confluent Marketplace and are no longer available for download. * [CSV Source Connector for Confluent Platform](connectors/csv_source_connector.md#spooldir-csv-source-connector) * [JSON Source Connector for Confluent Platform](connectors/json_source_connector.md#spooldir-json-source-connector) * [Schemaless JSON Source Connector for Confluent Platform](connectors/schemaless_json_source_connector.md#spooldir-schemaless-json-source-connector) * [Line-Delimited Source Connector for Confluent Platform](connectors/line_delimited_source_connector.md#spooldir-line-delimited-source-connector) * [Extended Log File Format Source Connector for Confluent Platform](connectors/elf_source_connector.md#spooldir-elf-source-connector) * [Binary File Source Connector for Confluent Platform](connectors/binary_source_connector.md#spooldir-binary-source-connector) ## Features Each of the Spool Dir Source connectors include the following features: - [At least once delivery](#spooldir-source-at-least-once-delivery) - [Multiple tasks](#spooldir-source-multiple-tasks) - [Client-side encryption](#spooldir-source-csfle-sm) ### At least once delivery Each connector in the Spool Dir Source connector package guarantees that records are delivered at least once to the Kafka topic. If the connector restarts, there may be some duplicate records in the Kafka topic. ### Multiple tasks Each connector in the Spool Dir Source connector package supports running one or more tasks. You can specify the number of tasks in the `tasks.max` configuration parameter. Multiple tasks may improve performance when moving a large amount of data. #### WARNING The Spool Dir Source connector may fail when running many tasks. This might occur if you use a regex in the `input.file.pattern` property that causes the connector to include `.processing` files–for example, `"input.file.pattern"="SAMPLE.*"`–in this way, the connector won’t exclude the files currently being processed and will output duplicate records and fail. To work around this issue, you must provide an extension in the input file format–for example, `"input.file.pattern"="SAMPLE.*.txt"`. This modifies the regular expression to match only `.txt` files and allows the connector to process the correct files. ### Client-side encryption This connector supports Client-Side Field Level Encryption (CSFLE) and Client-Side Payload Encryption (CSPE). For more information, see [Manage Client-Side Encryption](https://docs.confluent.io/platform/current/connect/manage-csfle.html). ## Install the Spool Dir Source Connector Package You can install this connector by using the [confluent connect plugin install](https://docs.confluent.io/confluent-cli/current/command-reference/connect/plugin/confluent_connect_plugin_install.html) command, or by manually downloading the ZIP file. ### Prerequisites - You must install the connector on every machine where Connect will run. - An installation of the latest (`latest`) connector version. To install the `latest` connector version, navigate to your Confluent Platform installation directory and run the following command: ```bash confluent connect plugin installjcustenborder/kafka-connect-spooldir:latest ``` You can install a specific version by replacing `latest` with a version number as shown in the following example: ```bash confluent connect plugin installjcustenborder/kafka-connect-spooldir:1.0.31 ``` ### Install the connector manually [Download and extract the ZIP file](https://www.confluent.io/hub/confluentinc/kafka-connect-spooldir) for your connector and then follow the manual connector installation [instructions](/kafka-connectors/self-managed/userguide.html#connect-installing-plugins). ## License The Spool Dir connector is an open source connector and does not require a Confluent Enterprise License. ## Configuration Properties For a complete list of configuration properties, see the specific connector documentation. For an example of how to get Kafka Connect connected to [Confluent Cloud](/cloud/current/index.html), see [Connect Self-Managed Kafka Connect to Confluent Cloud](/cloud/current/cp-component/connect-cloud-config.html#distributed-cluster). ## Quick Start The following steps show the `SpoolDirCsvSourceConnector` loading a mock CSV file to a Kafka topic named `spooldir-testing-topic`. The other connectors are similar but load from different file types. Prerequisites : - [Confluent Platform](/platform/current/installation/index.html) - [Confluent CLI](https://docs.confluent.io/confluent-cli/current/installing.html) (requires separate installation) 1. Install the connector through the [Confluent Hub Client](/kafka-connectors/self-managed/confluent-hub/client.html). ```bash # run from your Confluent Platform installation directory confluent connect plugin installjcustenborder/kafka-connect-spooldir:latest ``` 2. Start Confluent Platform using the Confluent CLI [confluent local](https://docs.confluent.io/confluent-cli/current/command-reference/local/index.html) commands. ```bash confluent local start ``` 3. Create a data directory and generate test data. ```bash mkdir data && curl "https://api.mockaroo.com/api/58605010?count=1000&key=25fd9c80" > "data/csv-spooldir-source.csv" ``` 4. Set up directories for files with errors and files that finished successfully. ```bash mkdir error && mkdir finished ``` 5. Create a `spooldir.json` file with the following contents: ```json { "name": "CsvSpoolDir", "config": { "tasks.max": "1", "connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector", "input.path": "/path/to/data", "input.file.pattern": "csv-spooldir-source.csv", "error.path": "/path/to/error", "finished.path": "/path/to/finished", "halt.on.error": "false", "topic": "spooldir-testing-topic", "csv.first.row.as.header": "true", "schema.generation.enabled": "true" } } ``` 6. Load the SpoolDir CSV Source connector. ```bash confluent local load spooldir --config spooldir.json ``` #### IMPORTANT Don’t use the [confluent local](https://docs.confluent.io/confluent-cli/current/command-reference/local/index.html) commands in production environments. 7. Confirm that the connector is in a `RUNNING` state. ```bash confluent local status spooldir ``` 8. Confirm that the messages are being sent to Kafka. ```bash kafka-avro-console-consumer \ --bootstrap-server localhost:9092 \ --property schema.registry.url=http://localhost:8081 \ --topic spooldir-testing-topic \ --from-beginning | jq '.' ``` 9. Confirm that the source CSV file has been moved to the `finished` directory. ## Examples * [CSV with Schema Example](connectors/csv_source_connector.md#spooldir-connector-csv-with-schema-example) * [TSV Input File Example](connectors/csv_source_connector.md#spooldir-connector-tsv-example) * [JSON Schemaless Source Connector Example](connectors/json_source_connector.md#spooldir-connector-json-schemaless-example) * [JSON Source Connector Example](connectors/json_source_connector.md#spooldir-connector-json-example) * [FIX Encoded Lines Example](connectors/line_delimited_source_connector.md#spooldir-connector-json-line-delimited-example)