Schemaless JSON Source Connector for Confluent Platform

Kafka Connect Schemaless JSON Source connector is used to stream JSON files from an SFTP directory. It will not try to convert the JSON records to a schema. The recommended converter to use is the StringConverter.

value.converter=org.apache.kafka.connect.storage.StringConverter

To use this connector, use a connector configuration that specifies the name of this connector class in the connector.class configuration property as shown below:

connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector

JSON Schemaless Source Connector Example

This example follows the same steps as the Quick Start. Review the Quick Start for help running the Confluent Platform and installing the SFTP connector package.

Prerequisites

  1. Generate a JSON dataset using the command below:

    echo $'{"id":1,"first_name":"Roscoe","last_name":"Brentnall","email":"rbrentnall0@mediafire.com","gender":"Male","ip_address":"202.84.142.254","last_login":"2018-02-12T06:26:23Z","account_balance":1450.68,"country":"CZ","favorite_color":"#4eaefa"}\n{"id":2,"first_name":"Gregoire","last_name":"Fentem","email":"gfentem1@nsw.gov.au","gender":"Male","ip_address":"221.159.106.63","last_login":"2015-03-27T00:29:56Z","account_balance":1392.37,"country":"ID","favorite_color":"#e8f686"}' > "json-sftp-source.json"

  2. Create an sftp.properties file with the following contents:

    name=SchemaLessJsonSftp
tasks.max=1
connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector
input.path=/path/to/data
error.path=/path/to/error
finished.path=/path/to/finished
input.file.pattern=json-sftp-source.json
behavior.on.error=IGNORE
sftp.username=username
sftp.password=password
sftp.host=localhost
sftp.port=22
kafka.topic=sftp-schemaless-json-topic
value.converter=org.apache.kafka.connect.storage.StringConverter

  3. Load the SFTP Schemaless JSON Source connector using the Confluent CLI confluent local services connect connector load command.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    confluent local services connect connector load SchemaLessJsonSftp --config sftp.properties

    Important

    • The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

Configuration Properties

For connector-specific configuration properties, see SFTP Source Connector Configuration Properties