Schemaless JSON Source Connector for Confluent Platform

Kafka Connect Schemaless JSON Source connector is used to stream JSON files from an SFTP directory. It will not try to convert the JSON records to a schema. The recommended converter to use is the StringConverter.

value.converter=org.apache.kafka.connect.storage.StringConverter

To use this connector, use a connector configuration that specifies the name of this connector class in the connector.class configuration property as shown below:

connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector

JSON Schemaless Source Connector Example

This example follows the same steps as the Quick Start. Review the Quick Start for help running the Confluent Platform and installing the SFTP connector package.

Prerequisites
  1. Generate a JSON dataset using the command below:

    echo $'{"id":1,"first_name":"Roscoe","last_name":"Brentnall","email":"rbrentnall0@mediafire.com","gender":"Male","ip_address":"202.84.142.254","last_login":"2018-02-12T06:26:23Z","account_balance":1450.68,"country":"CZ","favorite_color":"#4eaefa"}\n{"id":2,"first_name":"Gregoire","last_name":"Fentem","email":"gfentem1@nsw.gov.au","gender":"Male","ip_address":"221.159.106.63","last_login":"2015-03-27T00:29:56Z","account_balance":1392.37,"country":"ID","favorite_color":"#e8f686"}' > "json-sftp-source.json"
    
  2. Create an sftp.properties file with the following contents:

    name=SchemaLessJsonSftp
    tasks.max=1
    connector.class=io.confluent.connect.sftp.SftpSchemaLessJsonSourceConnector
    input.path=/path/to/data
    error.path=/path/to/error
    finished.path=/path/to/finished
    input.file.pattern=json-sftp-source.json
    behavior.on.error=IGNORE
    sftp.username=username
    sftp.password=password
    sftp.host=localhost
    sftp.port=22
    kafka.topic=sftp-schemaless-json-topic
    value.converter=org.apache.kafka.connect.storage.StringConverter
    
  3. Load the SFTP Schemaless JSON Source connector using the Confluent CLI confluent local services connect connector load command.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    confluent local services connect connector load SchemaLessJsonSftp --config sftp.properties
    

    Important

    The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

Configuration Properties

For connector-specific configuration properties, see SFTP Source Connector Configuration Properties