Binary File Source Connector for Confluent Platform

The Kafka Connect Binary File Source connector monitors the SFTP directory specified in input.path for files and reads them as binaries, forwarding the complete contents of each file as a binary record value. The recommended converter to use is the ByteArrayConverter.

value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

To use this connector, specify the name of the connector class in the connector.class configuration property.

connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector

Binary File Source Connector Example

This example follows the same steps as the Quick Start. Review the Quick Start for help running the Confluent Platform and installing the SFTP connector package.

Prerequisites
  1. Generate an HTML file using the command below:

    echo $'"<html>\n <head>\n </head>\n <body>\n   <h1>Hello World<h1>\n </body>\n</html>"' > "hello_world.html"
    
  2. Create an sftp.properties file with the following contents:

    name=BinaryFileSftp
    tasks.max=1
    connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector
    input.path=/path/to/data
    error.path=/path/to/error
    finished.path=/path/to/finished
    input.file.pattern=hello_world.html
    behavior.on.error=IGNORE
    sftp.username=username
    sftp.password=password
    sftp.host=localhost
    sftp.port=22
    kafka.topic=sftp-binary-file-topic
    value.converter=org.apache.kafka.connect.storage.ByteArrayConverter
    
  3. Load the SFTP Binary File Source connector using the Confluent CLI confluent local services connect connector load command.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    confluent local services connect connector load BinaryFileSftp --config sftp.properties
    

Binary File Source Connector Example (Avro)

The following example loads a binary file into a specific field in an Avro-formatted message using the Hoist SMT.

  1. Generate an HTML file using the command below:

    echo $'"<html>\n <head>\n </head>\n <body>\n   <h1>Hello World<h1>\n </body>\n</html>"' > "hello_world.html"
    
  2. Create an sftp.properties file with the following contents:

    name=BinaryFileSftp
    tasks.max=1
    connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector
    input.path=/path/to/data
    error.path=/path/to/error
    finished.path=/path/to/finished
    input.file.pattern=hello_world.html
    behavior.on.error=IGNORE
    sftp.username=username
    sftp.password=password
    sftp.host=localhost
    sftp.port=22
    kafka.topic=sftp-binary-file-avro-topic
    value.converter=io.confluent.connect.avro.AvroConverter
    transforms=HoistField
    transforms.HoistField.type=org.apache.kafka.connect.transforms.HoistField$Value
    transforms.HoistField.field=contents
    
  3. Load the SFTP Binary File Source connector using the Confluent CLI confluent local services connect connector load command.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    confluent local services connect connector load BinaryFileSftp --config sftp.properties
    

    Important

    • The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

Configuration Properties

For connector-specific configuration properties, see SFTP Source Connector Configuration Properties.