Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

HDFS 2 Connector Configuration Options

To use this connector, specify the name of the connector class in the connector.class configuration property.

connector.class=io.confluent.connect.hdfs.HdfsSinkConnector

Connector-specific configuration properties are described below.

HDFS

hdfs.url

The HDFS connection URL. This configuration has the format of hdfs://hostname:port and specifies the HDFS to export data to. This property is deprecated and will be removed in future releases. Use store.url instead.

  • Type: string
  • Default: null
  • Importance: high
hadoop.conf.dir

The Hadoop configuration directory.

  • Type: string
  • Default: “”
  • Importance: high
hadoop.home

The Hadoop home directory.

  • Type: string
  • Default: “”
  • Importance: high
logs.dir

Top level directory to store the write ahead logs.

  • Type: string
  • Default: logs
  • Importance: high

Security

hdfs.authentication.kerberos

Configuration indicating whether HDFS is using Kerberos for authentication.

  • Type: boolean
  • Default: false
  • Importance: high
  • Dependents: connect.hdfs.principal, connect.hdfs.keytab, hdfs.namenode.principal, kerberos.ticket.renew.period.ms
connect.hdfs.principal

The principal to use when HDFS is using Kerberos to for authentication.

  • Type: string
  • Default: “”
  • Importance: high
connect.hdfs.keytab

The path to the keytab file for the HDFS connector principal. This keytab file should only be readable by the connector user.

  • Type: string
  • Default: “”
  • Importance: high
hdfs.namenode.principal

The principal for HDFS Namenode.

  • Type: string
  • Default: “”
  • Importance: high
kerberos.ticket.renew.period.ms

The period in milliseconds to renew the Kerberos ticket.

  • Type: long
  • Default: 3600000
  • Importance: low

Connector

format.class

The format class to use when writing data to the store. Format classes implement the io.confluent.connect.storage.format.Format interface.

  • Type: class
  • Default: io.confluent.connect.hdfs.avro.AvroFormat
  • Importance: high

These classes are available by default:

  • io.confluent.connect.hdfs.avro.AvroFormat
  • io.confluent.connect.hdfs.json.JsonFormat
  • io.confluent.connect.hdfs.parquet.ParquetFormat
  • io.confluent.connect.hdfs.string.StringFormat
flush.size

Number of records written to store before invoking file commits.

  • Type: int
  • Importance: high
rotate.interval.ms

The time interval in milliseconds to invoke file commits. This configuration ensures that file commits are invoked every configured interval. This configuration is useful when data ingestion rate is low and the connector didn’t write enough messages to commit files. The default value -1 means that this feature is disabled.

  • Type: long
  • Default: -1
  • Importance: high
rotate.schedule.interval.ms

The time interval in milliseconds to periodically invoke file commits. This configuration ensures that file commits are invoked every configured interval. Time of commit will be adjusted to 00:00 of selected timezone. Commit will be performed at scheduled time regardless previous commit time or number of messages. This configuration is useful when you have to commit your data based on current server time, like at the beginning of every hour. The default value -1 means that this feature is disabled.

  • Type: long
  • Default: -1
  • Importance: medium
schema.cache.size

The size of the schema cache used in the Avro converter.

  • Type: int
  • Default: 1000
  • Importance: low
enhanced.avro.schema.support

Enable enhanced Avro schema support in AvroConverter: Enum symbol preservation and Package Name awareness.

  • Type: boolean
  • Default: false
  • Importance: low
connect.meta.data

Allow the Connect converter to add its metadata to the output schema.

  • Type: boolean
  • Default: true
  • Importance: low
retry.backoff.ms

The retry backoff in milliseconds. This config is used to notify Connect to retry delivering a message batch or performing recovery in case of transient exceptions.

  • Type: long
  • Default: 5000
  • Importance: low
shutdown.timeout.ms

Clean shutdown timeout. This makes sure that asynchronous Hive metastore updates are completed during connector shutdown.

  • Type: long
  • Default: 3000
  • Importance: medium
filename.offset.zero.pad.width

Width to zero-pad offsets in store’s filenames if offsets are too short in order to provide fixed-width filenames that can be ordered by simple lexicographic sorting.

  • Type: int
  • Default: 10
  • Valid Values: [0,…]
  • Importance: low
avro.codec

The Avro compression codec to be used for output files. Available values: null, deflate, snappy and bzip2 (Codec source is org.apache.avro.file.CodecFactory).

  • Type: string
  • Default: null
  • Valid Values: [null, deflate, snappy, bzip2]
  • Importance: low

Storage

storage.class

The underlying storage layer.

  • Type: class
  • Default: io.confluent.connect.hdfs.storage.HdfsStorage
  • Importance: high
topics.dir

Top level directory to store the data ingested from Apache Kafka®.

  • Type: string
  • Default: topics
  • Importance: high
store.url

Store’s connection URL, if applicable. For example: hdfs://hostname:port.

  • Type: string
  • Default: null
  • Importance: high
directory.delim

Directory delimiter pattern

  • Type: string
  • Default: /
  • Importance: medium
file.delim

File delimiter pattern

  • Type: string
  • Default: +
  • Importance: medium

Partitioner

partitioner.class

The partitioner to use when writing data to the store. The following partitioners are available:

  • DefaultPartitioner preserves the Kafka partitions.
  • DailyPartitioner partitions data according to date.
  • HourlyPartitioner partitions data according to hour.
  • FieldPartitioner partitions the data to different directories according to the value of the partitioning field specified in partition.field.name.
  • TimeBasedPartitioner partitions data according to ingestion time.
  • Type: class
  • Default: io.confluent.connect.storage.partitioner.DefaultPartitioner
  • Importance: high
  • Dependents: partition.field.name, partition.duration.ms, path.format, locale, timezone
partition.field.name

The name of the partitioning field when FieldPartitioner is used. You can enter multiple partitioning field names using comma-separated names.

  • Type: list
  • Default: “”
  • Importance: medium
partition.duration.ms

The duration of a partition milliseconds used by TimeBasedPartitioner. The default value -1 means that we are not using TimeBasedPartitioner.

  • Type: long
  • Default: -1
  • Importance: medium
path.format

This configuration is used to set the format of the data directories when partitioning with TimeBasedPartitioner. The format set in this configuration converts the Unix timestamp to proper directories strings. For example, if you set path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH, the data directories will have the format /year=2015/month=12/day=07/hour=15/.

  • Type: string
  • Default: “”
  • Importance: medium
locale

The locale to use when partitioning with TimeBasedPartitioner. Used to format dates and times. For example, use en-US for US English, en-GB for UK English, or fr-FR for French (in France). These may vary by Java version. See the available locales.

  • Type: string
  • Default: “”
  • Importance: medium
timezone

The timezone to use when partitioning with TimeBasedPartitioner. Used to format and compute dates and times. Use standard short names for timezones such as UTC or (without daylight savings) PST, EST, and ECT, or longer standard names such as America/Los_Angeles, America/New_York, and Europe/Paris. These may vary by Java version. See the available timezones within each locale, such as those within the US English locale.

  • Type: string
  • Default: “”
  • Importance: medium
timestamp.extractor

The extractor that gets the timestamp for records when partitioning with TimeBasedPartitioner. It can be set to Wallclock, Record or RecordField in order to use one of the built-in timestamp extractors or be given the fully-qualified class name of a user-defined class that extends the TimestampExtractor interface.

  • Type: string
  • Default: Wallclock
  • Importance: medium
timestamp.field

The record field to be used as timestamp by the timestamp extractor.

  • Type: string
  • Default: timestamp
  • Importance: medium

Hive

hive.integration

Configuration indicating whether to integrate with Hive when running the connector.

  • Type: boolean
  • Default: false
  • Importance: high
  • Dependents: hive.metastore.uris, hive.conf.dir, hive.home, hive.database
hive.metastore.uris

The Hive metastore URIs, can be IP address or fully-qualified domain name and port of the metastore host.

  • Type: string
  • Default: “”
  • Importance: high
hive.conf.dir

Hive configuration directory

  • Type: string
  • Default: “”
  • Importance: high
hive.home

Hive home directory.

  • Type: string
  • Default: “”
  • Importance: high
hive.database

The database to use when the connector creates tables in Hive.

  • Type: string
  • Default: default
  • Importance: high

Schema

schema.compatibility

The schema compatibility rule to use when the connector is observing schema changes. The supported configurations are NONE, BACKWARD, FORWARD and FULL.

  • Type: string
  • Default: NONE
  • Importance: high