Kafka Connectors¶

Self-managed Connectors for Confluent Platform

You can use self-managed Apache Kafka® connectors to move data in and out of Kafka. The self-managed connectors are for use with Confluent Platform. For more information on fully-managed connectors, see Confluent Cloud.

Popular connectors

JDBC Source and Sink

The Kafka Connect JDBC Source connector imports data from any relational database with a JDBC driver into an Kafka topic. The Kafka Connect JDBC Sink connector exports data from Kafka topics to any relational database with a JDBC driver.

JMS Source

The Kafka Connect JMS Source connector is used to move messages from any JMS-compliant broker into Kafka.

Elasticsearch Service Sink

The Kafka Connect Elasticsearch Service Sink connector moves data from Kafka to Elasticsearch. It writes data from a topic in Kafka to an index in Elasticsearch.

Amazon S3 Sink

The Kafka Connect Amazon S3 Sink connector exports data from Kafka topics to S3 objects in either Avro, JSON, or Bytes formats.

HDFS 2 Sink

The Kafka Connect HDFS 2 Sink connector allows you to export data from Apache Kafka® topics to HDFS 2.x files in a variety of formats. The connector integrates with Hive to make data immediately available for querying with HiveQL.

Replicator

Replicator allows you to easily and reliably replicate topics from one Kafka cluster to another.

Managing connectors

Supported connectors Confluent supports many self-managed connectors that import and export data from some of the most commonly used data systems. Practically all connectors are available from Confluent Hub. Preview connectors Confluent introduces preview connectors to gain early feedback from users. Preview connectors are only suitable for evaluation and non-production purposes.

Installing connectors Install the connectors by using the Confluent Hub client (recommended) or manually install by downloading the plugin file. Configuring connectors Connector configurations are key-value mappings. In distributed mode, they are included in the JSON payload sent over the REST API request that creates (or modifies) the connector.

Licensing connectors With a Developer License, you can use Confluent Platform commercial connectors on an unlimited basis in Connect clusters that use a single-broker Apache Kafka® cluster. A 30-day trial period is available when using a multi-broker cluster. Monitoring connectors You can manage and monitor Connect, connectors, and clients using JMX and the REST interface.

Adding connectors or software The Kafka Connect Base image contains Kafka Connect and all of its dependencies. To add new connectors to this image, you need to build a new Docker image that has the new connectors installed. Upgrading a connector plugin Upgrading a connector is similar to upgrading any other Apache Kafka® client application. Refer to the documentation for individual connector plugins if you have a need for rolling upgrades.

Manually installing Community connectors If a connector is not available on Confluent Hub, you can use the JARs to directly install the connectors into your Apache Kafka® installation. Kafka Connect Kafka Connect, an open source component of Apache Kafka®, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

All connectors

ActiveMQ Sink¶

ActiveMQ Sink

The Kafka Connect ActiveMQ Sink Connector is used to move messages from Kafka to an ActiveMQ cluster.

ActiveMQ Source¶

ActiveMQ Source

The Kafka Connect ActiveMQ Source Connector is used to read messages from an ActiveMQ cluster and write them to an Kafka topic.

Amazon CloudWatch Logs Source¶

Amazon CloudWatch Logs Source

The Kafka Connect Amazon CloudWatch Logs Source connector is used to import data from Amazon CloudWatch Logs, and write them into a Kafka topic.

Amazon CloudWatch Metrics Sink¶

Amazon CloudWatch Metrics Sink

The Kafka Connect Amazon CloudWatch Metrics Sink connector is used to export data to Amazon CloudWatch Metrics from a Kafka topic.

Amazon Kinesis Source¶

Amazon Kinesis Source

The Kafka Connect Kinesis Source connector is used to pull data from Amazon Kinesis and persist the data to an Kafka topic.

Amazon S3 Sink¶

Amazon S3 Sink

The Kafka Connect Amazon S3 Sink connector exports data from Kafka topics to S3 objects in either Avro, JSON, or Bytes formats.

Amazon S3 Source¶

Amazon S3 Source

The Kafka Connect Amazon S3 Source connector reads data exported to S3 by the Connect Amazon S3 Sink connector and publishes it back to an Kafka topic.

Amazon SQS Source¶

Amazon SQS Source

The Kafka Connect Simple Queue Service (SQS) Source connector moves messages from Amazon SQS Queues into Kafka. It supports both Standard and FIFO queues.

AMPS Source¶

AMPS Source

The Kafka Connect Advanced Message Processing System (AMPS) Source connector allows you to export data from AMPS to Kafka. The connector subscribes to messages from an AMPS topic and writes this data to a Kafka topic.

Apache HBase Sink¶

Apache HBase Sink

The Kafka Connect Apache HBase Sink Connector moves data from Kafka to Apache HBase. It writes data from a topic in Kafka to a table in the specified HBase instance.

Appdynamics Metrics Sink¶

Appdynamics Metrics Sink

The Kafka Connect AppDynamics Metrics Sink connector is used to export metrics from Kafka topics to AppDynamics using the AppDynamics Machine Agent.

Amazon DynamoDB Sink¶

Amazon DynamoDB Sink

The Kafka Connect Amazon DynamoDB Sink Connector is used to export messages from Kafka to Amazon DynamoDB, allowing you to export your Kafka data into your DynamoDB key-value and document database.

AWS Lambda Sink¶

AWS Lambda Sink

The Kafka Connect AWS Lambda Sink connector pulls records from one or more Kafka topics, converts them to JSON, and executes an AWS Lambda function.

Amazon Redshift Sink¶

Amazon Redshift Sink

The Kafka Connect Amazon Redshift Sink connector allows you to export data from Kafka topics to Amazon Redshift. The connector polls data from Kafka and writes this data to an Amazon Redshift database.

Azure Blob Storage Sink¶

Azure Blob Storage Sink

The Kafka Connect Azure Blob Storage connector exports data from Kafka topics to Azure Blob Storage objects in either Avro, JSON, Bytes or Parquet formats.

Azure Cognitive Search Sink¶

Azure Cognitive Search Sink

The Kafka Connect Azure Cognitive Search Sink connector moves data from Kafka to Azure Cognitive Search. It writes each event from a topic in Kafka to an index in Azure Cognitive Search.

Azure Blob Storage Source¶

Azure Blob Storage Source

The Kafka Connect Azure Blob Storage Source connector provides the capability to read data exported to Azure Blob Storage by the Azure Blob Storage Sink connector and publish it back to a Kafka topic.

Azure Data Lake Storage Gen1 Sink¶

Azure Data Lake Storage Gen1 Sink

The Kafka Connect Azure Data Lake Storage Gen1 Sink connector can export data from Kafka topics to Azure Data Lake Storage Gen1 files in either Avro or JSON formats.

Azure Data Lake Storage Gen2 Sink¶

Azure Data Lake Storage Gen2 Sink

The Kafka Connect Azure Data Lake Storage Gen2 Sink connector can export data from Kafka topics to Azure Data Lake Storage Gen2 files in Avro, JSON, Parquet or ByteArray formats.

Azure Event Hubs Source¶

Azure Event Hubs Source

The Kafka Connect Azure Event Hubs Source Connector is used to poll data from Azure Event Hubs and persist the data to an Kafka topic.

Azure Functions Sink¶

Azure Functions Sink

The Kafka Connect Azure Functions Sink Connector integrates Kafka with Azure Functions.

Azure Service Bus Source¶

Azure Service Bus Source

The Kafka Connect Azure Service Bus connector is a multi-tenant cloud messaging service you can use to send information between applications and services.

Azure Synapse Analytics Sink¶

Azure Synapse Analytics Sink

The Kafka Connect Azure Synapse Analytics Sink connector allows you to export data from Kafka topics to an Azure Synapse Analytics. The connector polls data from Kafka to writes to Synapse Analytics.

Cassandra Sink¶

Cassandra Sink

The Kafka Connect Cassandra Sink connector is a high-speed mechanism for writing data to Apache Cassandra.

Databricks Delta Lake Sink¶

Databricks Delta Lake Sink

The Kafka Connect Databricks Delta Lake Sink connector is used to periodically poll data from Kafka, copy the data into an Amazon S3 staging bucket, and then commit the records to a Databricks Delta Lake instance.

Data Diode (Source and Sink)¶

Data Diode (Source and Sink)

The Kafka Connect Data Diode Source and Sink connectors are used in tandem to replicate one or more Kafka topics from a source Kafka cluster to a destination Kafka cluster over UDP protocol.

Datadog Metrics Sink¶

Datadog Metrics Sink

The Kafka Connect Datadog Metrics Sink connector is used to export data from Kafka topics to Datadog using the Timeseries API - Post.

Datagen Source¶

Datagen Source (Development Testing)

The Kafka Connect Datagen Source connector generates mock source data for development and testing. This connector is not suitable for production use.

Elasticsearch Service Sink¶

Elasticsearch Service Sink

The Kafka Connect Elasticsearch Service Sink connector moves data from Kafka to Elasticsearch. It writes data from a topic in Kafka to an index in Elasticsearch.

FileStream Connectors¶

FileStream

The Kafka Connect FileStream Connector examples are intended to show how a simple connector runs for those first getting started with Kafka Connect as either a user or developer. It is not recommended for production use.

FTPS Source¶

FTPS Source

The Kafka Connect FTPS Source Connector provides the capability to watch a directory on an FTPS server for files and read the data as new files are written to the FTPS input directory.

FTPS Sink¶

FTPS Sink

The Kafka Connect FTPS Sink connector provides the capability to export data from Kafka topics to files in an FTPS servers directory.

GitHub Source¶

GitHub Source

The Kafka Connect GitHub Source Connector is used to write meta data (detect changes in real time or consume the history) from GitHub to Kafka topics.

Google BigQuery Sink¶

Google BigQuery Sink

The Kafka Connect Google BigQuery Sink Connector is used to stream data into BigQuery tables. When streaming data from Kafka topics, the sink connector can automatically create BigQuery tables.

Google Cloud BigTable Sink¶

Google Cloud BigTable Sink

The Kafka Connect BigTable Sink Connector moves data from Kafka to Google Cloud BigTable. It writes data from a topic in Kafka to a table in the specified BigTable instance.

Google Cloud Dataproc Sink¶

Google Cloud Dataproc Sink

The Kafka Connect Google Cloud Dataproc Sink Connector integrates Kafka with managed HDFS instances in Google Cloud Dataproc.

Google Cloud Functions Sink¶

Google Cloud Functions Sink

The Kafka Connect Google Cloud Functions Sink Connector integrates Kafka with Google Cloud Functions. The connector consumes records from Kafka topics and executes a Google Cloud Function.

Google Cloud Pub/Sub Source¶

Google Cloud Pub/Sub Source

The Kafka Connect Google Cloud Pub/Sub source connector reads messages from a Pub/Sub topic and writes them to an Kafka topic.

Google Cloud Spanner Sink¶

Google Cloud Spanner Sink

The Kafka Connect Google Cloud Spanner Sink connector moves data from Kafka to a Google Cloud Spanner database. It writes data from a topic in Kafka to a table in the specified Spanner database.

Google Cloud Storage Sink¶

Google Cloud Storage Sink

The Kafka Connect Google Cloud (GCS) Sink and Source connectors allow you to export data from Kafka topics to GCS storage objects in various formats and import data to Kafka from GCS storage.

Google Cloud Storage Source¶

Google Cloud Storage Source

The Kafka Connect Google Cloud (GCS) Sink and Source connectors allow you to export data from Kafka topics to GCS storage objects in various formats and import data to Kafka from GCS storage.

Google Firebase Source¶

Google Firebase Source

The Kafka Connect Google Firebase Source connector enables users to read data from a Google Firebase Realtime Database and persist the data in Kafka topics.

Google Firebase Sink¶

Google Firebase Sink

The Kafka Connect Google Firebase Sink connector enables users to read data from multiple Kafka topics and write them to Google Firebase Realtime Database.

HDFS 2 Sink¶

HDFS 2 Sink

The Kafka Connect HDFS 2 Sink connector allows you to export data from Kafka topics to HDFS 2.x files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.

HDFS 2 Source¶

HDFS 2 Source

The Kafka Connect HDFS 2 Source connector provides the capability to read data exported to HDFS 2 by the Kafka Connect HDFS 2 Sink connector and publish it back to an Kafka topic.

HDFS 3 Sink¶

HDFS 3 Sink

The Kafka Connect HDFS 3 connector allows you to export data from Kafka topics to HDFS 3.x files in a variety of formats. The connector integrates with Hive to make data immediately available for querying with HiveQL.

HDFS 3 Source¶

HDFS 3 Source

The Kafka Connect HDFS 3 Source connector provides the capability to read data exported to HDFS 3 by the Kafka Connect HDFS 3 Sink connector and publish it back to an Kafka topic.

HTTP Sink¶

HTTP Sink

The Kafka Connect HTTP Sink connector integrates Kafka with an API through HTTP or HTTPS.

IBM MQ Sink¶

IBM MQ Sink

The Kafka Connect IBM MQ Sink connector is used to move messages from Kafka to an IBM MQ cluster.

IBM MQ Source¶

IBM MQ Source

The Kafka Connect IBM MQ Source connector is used to read messages from an IBM MQ cluster and write them to an Kafka topic.

InfluxDB Source¶

InfluxDB Source

The Kafka Connect InfluxDB Source connector allows you to import data from an InfluxDB host into an Kafka topic. The Kafka Connect InfluxDB Sink connector writes data from an Kafka topic to an InfluxDB host.

InfluxDB Sink¶

InfluxDB Sink

The Kafka Connect InfluxDB Sink connector writes data from an Kafka topic to an InfluxDB host.

JDBC Source¶

JDBC Source

The JDBC Source connector imports data from any relational database with a JDBC driver into an Kafka topic. The Kafka Connect JDBC Sink connector exports data from Kafka topics to any relational database with a JDBC driver.

JDBC Sink¶

JDBC Sink

The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver.

Jira Source¶

Jira

The Kafka Connect Jira Source Connector is used to move data from Jira to an Kafka topic.

JMS Sink¶

JMS Sink

The Kafka Connect JMS Sink connector is used to move messages from Kafka to any JMS-compliant broker.

JMS Source¶

JMS Source

The Kafka Connect JMS Source connector is used to move messages from any JMS-compliant broker into Kafka.

Kudu Source¶

Kudu Source

The Kafka Connect Kudu Source connector allows you to import data to an Kafka topic from a Kudo columnar relational database using the Impala JDBC driver.

Kudu Sink¶

Kudu Sink

The Kafka Connect Kudu Sink connector exports data from an Kafka topic to a Kudo columnar relational database using an Impala JDBC driver.

MapR DB Sink¶

MapR DB Sink

The Kafka Connect MapR DB Sink connector provides a way to export data from an Kafka topic and write data to a MapR DB cluster.

Marketo Source¶

Marketo Source

The Kafka Connect Marketo Source connector copies data into Kafka from various Marketo entities and activity entities using the Marketo REST API.

Microsoft SQL Server Source (Deprecated)¶

Microsoft SQL Server Source (Deprecated)

The Kafka Connect Microsoft SQL Server Connector monitors source databases for changes and writes the changes in real-time to Kafka.

MongoDB Source (Debezium)¶

MongoDB Source (Debezium)

The Debezium MongoDB Source Connector can monitor a MongoDB replica set or a MongoDB sharded cluster for document changes in databases and collections, recording those changes as events in Kafka topics.

MQTT Source¶

MQTT Source

The Kafka Connect MQTT Source connector is used to integrate with existing MQTT servers. The Kafka Connect MQTT Source Connector connects to a MQTT broker and subscribes to the specified topics.

MQTT Sink¶

MQTT Sink

The Kafka Connect MQTT connector is used to integrate with existing MQTT servers. The Kafka Connect MQTT Sink Connector connects to an MQTT broker and publishes data to an MQTT topic.

MySQL Source (Debezium)¶

MySQL Source (Debezium)

The Debezium MySQL Source Connector can obtain a snapshot of the existing data and record all of the row-level changes in the databases on a MySQL server or cluster.

Netezza Sink¶

Netezza Sink

The Kafka Connect Netezza Sink connector exports data from Kafka topics to Netezza. The connector polls data from Kafka to write to Netezza based on a topic subscription.

HEAVY-AI (formerly OmniSci) Sink¶

HEAVY-AI Sink

The Kafka Connect HEAVY-AI Sink connector allows you to export data from an Kafka topic to HEAVY-AI. The connector polls data from Kafka and writes to HEAVY-AI based on a topic subscription.

Oracle CDC Source¶

Oracle CDC Source

The Kafka Connect Oracle CDC Source connector captures each change to rows in a database and represents each of those as change event records in Kafka topics. The connector uses Oracle LogMiner to read the database redo log.

Pagerduty Sink¶

Pagerduty Sink

The Kafka Connect PagerDuty Sink connector is used to read records from an Kafka topic and create Pagerduty incidents.

PostgresSQL Source (Debezium)¶

PostgresSQL Source (Debezium)

The Debezium PostgreSQL Source Connector can obtain a snapshot of the existing data in a PostgreSQL database and then monitor and record all subsequent row-level changes to that data.

Prometheus Metrics Sink¶

Prometheus Metrics Sink

The Kafka Connect Prometheus Metrics Sink connector exports data from multiple Kafka topics and makes the data available to an endpoint which is scraped by a Prometheus server.

RabbitMQ Source¶

RabbitMQ Source

The Kafka Connect RabbitMQ Source connector integrates with RabbitMQ servers, using the AMQP protocol. The RabbitMQ Source connector reads data from a RabbitMQ queue or topic and persists the data in an Kafka topic.

RabbitMQ Sink¶

RabbitMQ Sink

The Kafka Connect RabbitMQ Sink connector integrates with RabbitMQ servers, using the AMQP protocol. The RabbitMQ Sink connector reads data from one or more Kafka topics and sends the data to a RabbitMQ exchange.

Redis Sink¶

Redis Sink

The Kafka Connect Redis Sink connector is used to export data from Kafka topics to Redis.

Replicator¶

Replicator

Replicator allows you to easily and reliably replicate topics from one Kafka cluster to another.

Salesforce (Source and Sink)¶

Salesforce (Source and Sink)

The Salesforce Source and Sink connector package provides connectors that integrate Salesforce.com with Kafka.

ServiceNow Source¶

ServiceNow Source

The Kafka Connect ServiceNow Source connector is used to move creations and updates in a ServiceNow Table to Kafka in real-time.

ServiceNow Sink¶

ServiceNow Sink

The Kafka Connect ServiceNow Sink connector is used to export Kafka records to a ServiceNow table.

SFTP Source¶

SFTP Source

The connectors in the Kafka Connect SFTP Source connector package provide the capability to watch an SFTP directory for files and read the data as new files are written to the SFTP input directory.

SFTP Sink¶

SFTP Sink

The SFTP Sink Connector periodically polls data from Kafka and in turn writes it to the SFTP files.

SNMP Trap Source¶

SNMP Trap Source

The Kafka Connect SNMP Trap Source connector receives data (SNMP traps) from devices through SNMP and convert the trap messages into Kafka records.

Solace Source¶

Solace Source

The Kafka Connect Solace Source connector is used to move messages from a Solace PubSub+ cluster to Kafka.

Solace Sink¶

Solace Sink

The Kafka Connect Solace Sink connector is used to move messages from Kafka to a Solace PubSub+ cluster.

Splunk Sink¶

Splunk Sink

The Kafka Connect Splunk Sink connector moves messages from Kafka to Splunk.

Splunk Source¶

Splunk Source

The Kafka Connect Splunk Source connector integrates Splunk with Kafka. The connector receives data from applications that would normally send data to a Splunk HTTP Event Collector (HEC).

Splunk S2S Source¶

Splunk S2S Source

The Splunk S2S Source Connector provides a way to integrate Splunk with Kafka. The connector receives data from the Splunk universal forwarder (UF).

Spool Dir¶

Spool Dir

The connectors in the Kafka Connect Spool Dir connector package monitor a directory for new files and read the data as new files are written to the input directory.

SQL Server Source (Debezium)¶

SQL Server Source (Debezium)

The Debezium SQL Server Source Connector can obtain a snapshot of the existing data in a SQL Server database and then monitor and record all subsequent row-level changes to that data.

Syslog Source¶

Syslog Source

The Kafka Connect Syslog Source connector to consume data from network devices. Supported formats are RFC 3164, RFC 5424, and Common Event Format (CEF).

Teradata Source¶

Teradata Source

The Kafka Connect Teradata Source connector allows you to import data from Teradata into Kafka topics.

Teradata Sink¶

Teradata Sink

The Kafka Connect Teradata Sink connector allows you to export data from Kafka topics to Teradata.

TIBCO Sink¶

TIBCO Sink

The Kafka Connect TIBCO Sink connector is used to move messages from Kafka to TIBCO Enterprise Messaging Service (EMS).

TIBCO Source¶

TIBCO Source

The Kafka Connect TIBCO Source connector is used to move messages from TIBCO Enterprise Messaging Service (EMS) to Kafka.

Vertica Sink¶

Vertica Sink

The Kafka Connect Vertica Sink connector exports data from Kafka topics to Vertica. The Vertica Sink connector periodically polls records from Kafka and adds them to a Vertica table.

VMware Tanzu GemFire Sink¶

VMware Tanzu GemFire Sink

The Kafka Connect VMware Tanzu GemFire Sink connector exports data from Kafka to Tanzu GemFire. The Tanzu GemFire Sink connector periodically polls data from Kafka and adds it to Tanzu GemFire.

Weblogic JMS Source¶

Weblogic JMS Source

The Kafka Connect Weblogic JMS Source connector is used to read messages from an Oracle Weblogic JMS Server and write them into an Kafka topic.

Zendesk Source¶

Zendesk Source

The Kafka Connect Zendesk Source connector copies data into Kafka from various Zendesk support tables using the Zendesk Support API.