Schemaless JSON Source Connector for Confluent Platform¶

Kafka Connect Schemaless JSON Source connector is used to stream JSON files from an SFTP directory. It will not try to convert the JSON records to a schema. The recommended converter to use is the StringConverter.

value.converter=org.apache.kafka.connect.storage.StringConverter

To use this connector, use a connector configuration that specifies the name of this connector class in the connector.class configuration property as shown below:

connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector

JSON Schemaless Source Connector Example¶

This example follows the same steps as the Quick Start. Review the Quick Start for help running the Confluent Platform and installing the SFTP connector package.

Prerequisites

Confluent Platform
Confluent CLI (requires separate installation)

Generate a JSON dataset using the command below:

echo $'{"id":1,"first_name":"Roscoe","last_name":"Brentnall","email":"rbrentnall0@mediafire.com","gender":"Male","ip_address":"202.84.142.254","last_login":"2018-02-12T06:26:23Z","account_balance":1450.68,"country":"CZ","favorite_color":"#4eaefa"}\n{"id":2,"first_name":"Gregoire","last_name":"Fentem","email":"gfentem1@nsw.gov.au","gender":"Male","ip_address":"221.159.106.63","last_login":"2015-03-27T00:29:56Z","account_balance":1392.37,"country":"ID","favorite_color":"#e8f686"}' > "json-sftp-source.json"

Create an sftp.properties file with the following contents:

name=SchemaLessJsonSftp
tasks.max=1
connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector
input.path=/path/to/data
error.path=/path/to/error
finished.path=/path/to/finished
input.file.pattern=json-sftp-source.json
behavior.on.error=IGNORE
sftp.username=username
sftp.password=password
sftp.host=localhost
sftp.port=22
kafka.topic=sftp-schemaless-json-topic
value.converter=org.apache.kafka.connect.storage.StringConverter

Load the SFTP Schemaless JSON Source connector using the Confluent CLI confluent local services connect connector load command.

Caution

You must include a double dash (--) between the connector name and your flag. For more information, see this post.
```
confluent local services connect connector load SchemaLessJsonSftp --config sftp.properties
```
Important

The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

Configuration Properties¶

For connector-specific configuration properties, see SFTP Source Connector Configuration Properties.