Connect to External Systems¶
Confluent Cloud offers pre-built, fully managed, Apache Kafka® Connectors that make it easy to instantly connect to popular data sources and sinks. With a simple UI-based configuration and elastic scaling with no infrastructure to manage, Confluent Cloud Connectors make moving data in and out of Kafka an effortless task, giving you more time to focus on app development.
- Source connector
- A source connector, such as the Microsoft SQL Server Source connector, ingests entire databases and streams table updates to Kafka topics. It can also collect metrics from all of your application servers and store these in Kafka topics, making the data available for stream processing with low latency.
- Sink connector
- A sink connector delivers data from Kafka topics into secondary indexes, such as Google BigQuery or batch systems like Amazon S3, for offline analysis.
Note
See Limitations for Connect Confluent Cloud limitations.
Supported connectors¶
The following Confluent Cloud connectors are supported by Confluent:
- ActiveMQ Source
- Amazon CloudWatch Logs Source
- Amazon CloudWatch Metrics Sink
- Amazon DynamoDB Sink
- Amazon Kinesis Source
- Amazon Redshift Sink
- Amazon SQS Source
- Amazon S3 Sink
- Amazon S3 Source
- AWS Lambda Sink
- Azure Blob Storage Sink
- Azure Cognitive Search Sink
- Azure Cosmos DB Sink
- Azure Cosmos DB Source
- Azure Data Lake Storage Gen2 Sink
- Azure Event Hubs Source
- Azure Functions Sink
- Azure Service Bus Source
- Azure Synapse Analytics Sink
- Databricks Delta Lake Sink
- Datadog Metrics Sink
- Datagen Source (development and testing)
- Elasticsearch Service Sink
- GitHub Source
- Google BigQuery Sink
- Google Cloud BigTable Sink
- Google Cloud Functions Sink
- Google Cloud Spanner Sink
- Google Cloud Storage Sink
- Google Cloud Storage Source
- Google Pub/Sub Source
- HTTP Sink
- IBM MQ Source
- InfluxDB 2 Sink
- InfluxDB 2 Source
- Jira Source
- Microsoft SQL Server CDC Source (Debezium)
- Microsoft SQL Server Sink (JDBC)
- Microsoft SQL Server Source (JDBC)
- MongoDB Atlas Sink
- MongoDB Atlas Source
- MQTT Sink
- MQTT Source
- MySQL CDC Source (Debezium)
- MySQL Sink (JDBC)
- MySQL Source (JDBC)
- Oracle CDC Source
- Oracle Database Sink
- Oracle Database Source
- PagerDuty Sink
- PostgreSQL CDC Source (Debezium)
- PostgreSQL Sink (JDBC)
- PostgreSQL Source (JDBC)
- RabbitMQ Sink
- RabbitMQ Source Connector
- Redis Sink
- Salesforce Bulk API Source
- Salesforce CDC Source
- Salesforce Platform Event Sink
- Salesforce Platform Event Source
- Salesforce PushTopic Source
- Salesforce SObject Sink
- ServiceNow Sink
- ServiceNow Source
- SFTP Sink
- SFTP Source
- Snowflake Sink
- Solace Sink
- Splunk Sink
- Zendesk Source
Preview connectors¶
Caution
Preview connectors are not currently supported and are not recommended for production use.
The following Confluent Cloud connectors are available for preview:
Cloud platforms support¶
The following table shows the cloud platforms supported by each connector.
| Cloud Connector | AWS | Azure | GCP |
|---|---|---|---|
| Amazon CloudWatch Logs Source | Yes | Yes | Yes |
| Amazon CloudWatch Metrics Sink | Yes | No | No |
| Amazon DynamoDB Sink | Yes | No | No |
| Amazon Kinesis Source | Yes | Yes | Yes |
| Amazon Redshift Sink | Yes | No | No |
| Amazon SQS Source | Yes | Yes | Yes |
| Amazon S3 Sink | Yes | No | No |
| Amazon S3 Source | Yes | No | No |
| AWS Lambda Sink | Yes | No | No |
| Azure Blob Storage Sink | No | Yes | No |
| Azure Cognitive Search Sink | No | Yes | No |
| Azure Cosmos DB Sink | No | Yes | No |
| Azure Cosmos DB Source | No | Yes | No |
| Azure Data Lake Storage Gen2 Sink | No | Yes | No |
| Azure Event Hubs Source | Yes | Yes | Yes |
| Azure Functions Sink | No | Yes | No |
| Azure Service Bus Source | Yes | Yes | Yes |
| Azure Synapse Analytics Sink | No | Yes | No |
| Databricks Delta Lake Sink | Yes | No | No |
| Datadog Metrics Sink | Yes | Yes | Yes |
| Datagen Source | Yes | Yes | Yes |
| Elasticsearch Service Sink | Yes | Yes | Yes |
| GitHub Source | Yes | Yes | Yes |
| Google BigQuery Sink | No | No | Yes |
| Google Cloud BigTable Sink | No | No | Yes |
| Google Cloud Dataproc Sink | No | No | Yes |
| Google Cloud Functions Sink | No | No | Yes |
| Google Cloud Spanner Sink | No | No | Yes |
| Google Cloud Storage Sink | No | No | Yes |
| Google Pub/Sub Source | Yes | Yes | Yes |
| HTTP Sink | Yes | Yes | Yes |
| IBM MQ Source | Yes | Yes | Yes |
| InfluxDB 2 Sink | Yes | Yes | Yes |
| InfluxDB 2 Source | Yes | Yes | Yes |
| Jira Source | Yes | Yes | Yes |
| Microsoft SQL Server Sink | Yes | Yes | Yes |
| Microsoft SQL Server Source CDC (Debezium) | Yes | Yes | Yes |
| Microsoft SQL Server Source | Yes | Yes | Yes |
| MongoDB Atlas Sink | Yes | Yes | Yes |
| MongoDB Atlas Source | Yes | Yes | Yes |
| MQTT Sink | Yes | Yes | Yes |
| MQTT Source | Yes | Yes | Yes |
| MySQL Source CDC (Debezium) | Yes | Yes | Yes |
| MySQL Sink | Yes | Yes | Yes |
| MySQL Source | Yes | Yes | Yes |
| Oracle CDC Source | Yes | Yes | Yes |
| Oracle Database Sink | Yes | Yes | Yes |
| Oracle Database Source | Yes | Yes | Yes |
| PagerDuty Sink | Yes | Yes | Yes |
| PostgreSQL CDC Source (Debezium) | Yes | Yes | Yes |
| PostgreSQL Sink | Yes | Yes | Yes |
| PostgreSQL Source | Yes | Yes | Yes |
| RabbitMQ Sink Connector | Yes | Yes | Yes |
| RabbitMQ Source | Yes | Yes | Yes |
| Redis Sink | Yes | Yes | Yes |
| Salesforce Bulk API Source | Yes | Yes | Yes |
| Salesforce CDC Source | Yes | Yes | Yes |
| Salesforce Platform Event Sink | Yes | Yes | Yes |
| Salesforce Platform Event Source | Yes | Yes | Yes |
| Salesforce PushTopic Source | Yes | Yes | Yes |
| Salesforce SObject Sink | Yes | Yes | Yes |
| ServiceNow Sink | Yes | Yes | Yes |
| ServiceNow Source | Yes | Yes | Yes |
| SFTP Sink | Yes | Yes | Yes |
| SFTP Source | Yes | Yes | Yes |
| Snowflake Sink | Yes | Yes | Yes |
| Solace Sink | Yes | Yes | Yes |
| Splunk Sink | Yes | Yes | Yes |
| Zendesk Source | Yes | Yes | Yes |
Networking and DNS Considerations¶
Consider the following cluster network types when determining the public Internet access configuration for resources that fully-managed connectors must access. For Confluent Cloud networking details, see the Cloud Networking docs.
Important
To use static egress IP addresses, you must be running a multi-tenant or Dedicated cluster on AWS and on the public Internet (that is, without VPC peering, Transit Gateway, or AWS PrivateLink).
Cluster types and connectivity¶
The following tabs provide network connectivity details for managed connectors. Note that a Connect node runs in the same VPC/VNet as the cluster the Connect node was provisioned with. This is true for all cluster types (Basic, Standard, and Dedicated). For Confluent Cloud networking details, see the Cloud Networking docs.
The following information applies to a managed Sink or Source connector connecting to an external system using a public IP address.
| Cluster network type | Public IP address connectivity | IP range used by the connector |
|---|---|---|
| Public Endpoint (AWS) | Yes | Fixed set of static egress IP addresses (see static egress IP addresses) |
| Public Endpoint (Azure and Google Cloud) | Yes | Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located |
| VPC Peering and Transit Gateway | Yes | Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located |
| Private Link | Yes | Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located |
The following information applies to a managed Sink or Source connector connecting to an external system using a private IP address.
| Cluster network type | Private IP address connectivity | IP range used by the connector |
|---|---|---|
| VPC Peering and Transit Gateway | Yes | Source IP address used is from the /16 CIDR range configured by the customer for the Confluent Cloud Cluster |
| Private Link | No | N/A |
| Public Endpoint | No | N/A |
Note
See the following cloud provider documentation for additional information:
DNS considerations¶
Fully qualified domain names: Some services require fully qualified domain names (FQDNs) to access the service. In order for a managed connector to access such a service, the service must use public DNS records pointing to the IP address (public or private). Private DNS zones are not supported in Confluent Cloud.
Private service endpoints: Cloud service providers offer the ability to set up private endpoints with custom or vanity DNS names for native cloud provider services. Private endpoints are only supported if the provider supports resolving the endpoints using public DNS.
Confluent CLI¶
The CLI steps for managed connectors show examples using version 2 of the Confluent CLI. For more information see, Migrate to Confluent CLI v2.
API for fully-managed connectors¶
For information and examples to use with the Confluent Cloud API for fully-managed connectors, see the Confluent Cloud API for Connect documentation.
Cloud Console connector controls¶
You can use the GUI buttons to start, stop, pause, and delete a connector. Select and display one of your listed connectors to view the controls.
Connector data previews¶
For information about connector data previews, see Connector Data Previews.
Single message transforms¶
For information about using single message transforms (SMTs), see Single Message Transforms for Managed Connectors.
View connector events¶
For information about viewing Confluent Cloud connector events, see View Connector Events.
Important
Viewing connector events is restricted to the OrganizationAdmin RBAC role. Viewing events is not available for other roles.
Service accounts¶
For information about setting up service accounts, see Service Accounts.
RBAC for managed connectors¶
For information about RBAC and managed connectors, see RBAC for Managed Connectors.
Dead letter queue¶
For information about accessing and using the Confluent Cloud Dead Letter Queue, see Dead Letter Queue.
Connector limitations¶
To view a list of connector limitations, see Limitations.
About preview features¶
For a description of the Confluent Cloud Connect preview program, see Confluent Cloud Connect Preview.