Binary File Source Connector for Confluent Platform¶

The Kafka Connect Binary File Source connector monitors the SFTP directory specified in input.path for files and reads them as binaries, forwarding the complete contents of each file as a binary record value. The recommended converter to use is the ByteArrayConverter.

value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

To use this connector, specify the name of the connector class in the connector.class configuration property.

connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector

Binary File Source Connector Example¶

This example follows the same steps as the Quick Start. Review the Quick Start for help running the Confluent Platform and installing the SFTP connector package.

Prerequisites

Confluent Platform
Confluent CLI (requires separate installation)

Generate an HTML file using the command below:

echo $'"<html>\n <head>\n </head>\n <body>\n   <h1>Hello World<h1>\n </body>\n</html>"' > "hello_world.html"

Create an sftp.properties file with the following contents:

name=BinaryFileSftp
tasks.max=1
connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector
input.path=/path/to/data
error.path=/path/to/error
finished.path=/path/to/finished
input.file.pattern=hello_world.html
behavior.on.error=IGNORE
sftp.username=username
sftp.password=password
sftp.host=localhost
sftp.port=22
kafka.topic=sftp-binary-file-topic
value.converter=org.apache.kafka.connect.storage.ByteArrayConverter

Load the SFTP Binary File Source connector using the Confluent CLI confluent local services connect connector load command.

Caution

You must include a double dash (--) between the connector name and your flag. For more information, see this post.
```
confluent local services connect connector load BinaryFileSftp --config sftp.properties
```

Binary File Source Connector Example (Avro)¶

The following example loads a binary file into a specific field in an Avro-formatted message using the Hoist SMT.

Generate an HTML file using the command below:

echo $'"<html>\n <head>\n </head>\n <body>\n   <h1>Hello World<h1>\n </body>\n</html>"' > "hello_world.html"

Create an sftp.properties file with the following contents:

name=BinaryFileSftp
tasks.max=1
connector.class=io.confluent.connect.sftp.SftpBinaryFileSourceConnector
input.path=/path/to/data
error.path=/path/to/error
finished.path=/path/to/finished
input.file.pattern=hello_world.html
behavior.on.error=IGNORE
sftp.username=username
sftp.password=password
sftp.host=localhost
sftp.port=22
kafka.topic=sftp-binary-file-avro-topic
value.converter=io.confluent.connect.avro.AvroConverter
transforms=HoistField
transforms.HoistField.type=org.apache.kafka.connect.transforms.HoistField$Value
transforms.HoistField.field=contents

Load the SFTP Binary File Source connector using the Confluent CLI confluent local services connect connector load command.

Caution

You must include a double dash (--) between the connector name and your flag. For more information, see this post.
```
confluent local services connect connector load BinaryFileSftp --config sftp.properties
```
Important

The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

Configuration Properties¶

For connector-specific configuration properties, see SFTP Source Connector Configuration Properties.