Connect External Systems to Confluent Cloud¶
Kafka Connect, an open source component of Apache Kafka®, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. Using Kafka Connect, you can deploy connector implementations for common data sources and sinks to move data into and out of Kafka.
- Source Connector
- A source connector ingests entire databases and streams table updates to Kafka topics. It can also collect metrics from all of your application servers and store these in Kafka topics, making the data available for stream processing with low latency.
- Sink Connector
- A sink connector delivers data from Kafka topics into secondary indexes such as Elasticsearch, or batch systems such as Hadoop for offline analysis.
Kafka Connect is focused on streaming data to and from Kafka, making it simpler for you to write high quality, reliable, and high performance connector plugins. It also enables the framework to make guarantees that are difficult to achieve using other frameworks. Kafka Connect is an integral component of an ETL pipeline, when combined with Kafka and a stream processing framework.
Kafka Connect can be deployed either as a standalone process that runs jobs on a single machine (for example, log collection), or as a distributed, scalable, fault-tolerant service supporting an entire organization. Kafka Connect provides a low barrier to entry and low operational overhead. You can start small with a standalone environment for development and testing, and then scale up to a full production environment to support a large organization’s data pipeline.
The main benefits of using Kafka Connect are:
Data Centric Pipeline – Connect uses meaningful data abstractions to pull or push data to Kafka.
Flexibility and Scalability – Connect runs with streaming and batch-oriented systems on a single node (standalone) or scaled to an organization-wide service (distributed).
Reusability and Extensibility – Connect leverages existing connectors or extends them to tailor to your needs and provides lower time to production.
For a deeper dive into the benefits of using Kafka Connect, listen to Why Kafka Connect? featuring Robin Moffatt.
The following Confluent Cloud connectors are supported by Confluent:
Preview connectors are not currently supported and are not recommended for production use. For specific connector limitations, see Cloud connector limitations.
The following Confluent Cloud connectors are available for preview:
- Amazon Kinesis Source Connector
- Azure DataLake Gen2 Storage Sink Connector
- Amazon Redshift Sink Connector
- Azure Event Hubs Source Connector
- Google BigQuery Sink Connector
- Google Cloud Dataproc Sink Connector
- Google Cloud Spanner Sink Connector
- Google Pub/Sub Source Connector
- Microsoft SQL Server Source Connector
- MySQL Source Cloud Connector
- Oracle Database Source Cloud Connector
- PostgresSQL Source Cloud Connector