Connect to External Systems in Confluent Cloud

Confluent Cloud offers pre-built, fully managed, Apache Kafka® Connectors that make it easy to instantly connect to popular data sources and sinks. With a simple UI-based configuration and elastic scaling with no infrastructure to manage, Confluent Cloud Connectors make moving data in and out of Kafka an effortless task, giving you more time to focus on app development.

Note

  • If you want to bring your custom connector to Confluent Cloud, see Custom Connectors for Confluent Cloud.
  • Connect with Confluent is a program where partners work with Confluent to set up a Partner Integration. From this integration, your customers can start producing and consuming with a few clicks in your UI. For more information, see Connect with Confluent.
Source connector
A source connector, such as the Microsoft SQL Server Source connector, ingests entire databases and streams table updates to Kafka topics. It can also collect metrics from all of your application servers and store these in Kafka topics, making the data available for stream processing with low latency.
Sink connector
A sink connector delivers data from Kafka topics into secondary indexes, such as Google BigQuery or batch systems like Amazon S3, for offline analysis.

Note

Supported connectors

The following Confluent Cloud connectors are supported by Confluent:

Preview connectors

Caution

Preview features are not currently supported and are not recommended for production use. A preview feature is a Confluent Cloud component that is being introduced to gain early feedback. Preview connectors and features can be used for evaluation and non-production testing purposes or to provide feedback to Confluent. Comments, questions, and suggestions related to preview features are encouraged and can be submitted to ccloud-connect-preview@confluent.io.

The following Confluent Cloud connectors are available for preview:

Cloud platforms support

The following table shows the cloud platforms supported by each connector.

Cloud Connector AWS Azure GCP
Amazon CloudWatch Logs Source Yes Yes Yes
Amazon CloudWatch Metrics Sink Yes No No
Amazon DynamoDB Sink Yes No No
Amazon Kinesis Source Yes Yes Yes
Amazon Redshift Sink Yes No No
Amazon SQS Source Yes Yes Yes
Amazon S3 Sink Yes No No
Amazon S3 Source Yes Yes Yes
AWS Lambda Sink Yes No No
Azure Blob Storage Sink No Yes No
Azure Blob Storage Source Yes Yes Yes
Azure Cognitive Search Sink No Yes No
Azure Cosmos DB Sink No Yes No
Azure Cosmos DB Source No Yes No
Azure Data Lake Storage Gen2 Sink No Yes No
Azure Event Hubs Source Yes Yes Yes
Azure Functions Sink No Yes No
Azure Service Bus Source Yes Yes Yes
Azure Synapse Analytics Sink No Yes No
Databricks Delta Lake Sink Yes No No
Datadog Metrics Sink Yes Yes Yes
Datagen Source Yes Yes Yes
Elasticsearch Service Sink Yes Yes Yes
GitHub Source Yes Yes Yes
Google BigQuery Sink No No Yes
Google Cloud BigTable Sink No No Yes
Google Cloud Dataproc Sink No No Yes
Google Cloud Functions Sink No No Yes
Google Cloud Spanner Sink No No Yes
Google Cloud Storage Sink No No Yes
Google Cloud Storage Source Yes Yes Yes
Google Cloud Pub/Sub Source Yes Yes Yes
HTTP Sink Yes Yes Yes
HTTP Source Yes Yes Yes
IBM MQ Source Yes Yes Yes
InfluxDB 2 Sink Yes Yes Yes
InfluxDB 2 Source Yes Yes Yes
Jira Source Yes Yes Yes
Microsoft SQL Server Sink Yes Yes Yes
Microsoft SQL Server Source CDC (Debezium) Yes Yes Yes
Microsoft SQL Server Source Yes Yes Yes
MongoDB Atlas Sink Yes Yes Yes
MongoDB Atlas Source Yes Yes Yes
MQTT Sink Yes Yes Yes
MQTT Source Yes Yes Yes
MySQL Source CDC (Debezium) Yes Yes Yes
MySQL Sink Yes Yes Yes
MySQL Source Yes Yes Yes
New Relic Metrics Sink Yes Yes Yes
Oracle CDC Source Yes Yes Yes
Oracle Database Sink Yes Yes Yes
Oracle Database Source Yes Yes Yes
PagerDuty Sink Yes Yes Yes
PostgreSQL CDC Source (Debezium) Yes Yes Yes
PostgreSQL Sink Yes Yes Yes
PostgreSQL Source Yes Yes Yes
RabbitMQ Sink Connector Yes Yes Yes
RabbitMQ Source Yes Yes Yes
Redis Sink Yes Yes Yes
Salesforce Bulk API Source Yes Yes Yes
Salesforce Bulk API 2.0 Sink Yes Yes Yes
Salesforce Bulk API 2.0 Source Yes Yes Yes
Salesforce CDC Source Yes Yes Yes
Salesforce Platform Event Sink Yes Yes Yes
Salesforce Platform Event Source Yes Yes Yes
Salesforce PushTopic Source Yes Yes Yes
Salesforce SObject Sink Yes Yes Yes
ServiceNow Sink Yes Yes Yes
ServiceNow Source Yes Yes Yes
SFTP Sink Yes Yes Yes
SFTP Source Yes Yes Yes
Snowflake Sink Yes Yes Yes
Solace Sink Yes Yes Yes
Splunk Sink Yes Yes Yes
Zendesk Source Yes Yes Yes

Networking, DNS, and service endpoints

Consider the following when determining the public Internet access configuration for resources that fully-managed connectors must access. For Confluent Cloud networking details, see the Cloud Networking docs.

Networking

The following tabs provide network connectivity IP address details. Note that a Connect node runs in the same VPC/VNet as the cluster the Connect node was provisioned with. This is true for all cluster types (Basic, Standard, Enterprise (Limited Availability) and Dedicated). For Confluent Cloud networking details, see the Cloud Networking docs.

Note

The following information applies to a managed Sink or Source connector connecting to an external system using a public IP address.

Cluster network type Public IP address connectivity IP range used by the connector
Public Endpoint Yes Fixed set of egress static IP addresses (see egress static IP addresses)
VPC Peering and Transit Gateway Yes Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located
Private Link Yes Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located

DNS considerations

Fully qualified domain names: Some services require fully qualified domain names (FQDNs) to access the service. In order for a managed connector to access such a service, the service must use public DNS records pointing to the IP address (public or private). Private DNS zones are not supported in Confluent Cloud.

Private service endpoints: Cloud service providers offer the ability to set up private endpoints with custom or vanity DNS names for native cloud provider services. Private endpoints are only supported if the provider supports resolving the endpoints using public DNS.

Service and gateway endpoints

Azure service endpoints and AWS gateway endpoints provide secure and direct private connectivity to Azure and AWS services over the cloud provider network backbone using an optimized route. These endpoints are located in the Confluent Cloud VPC/Vnet.

Managed connector network traffic is routed over cloud service provider (CSP) secure public endpoints for the following services:

AWS
Azure
  • Azure Blob Storage
  • Azure Cosmos DB
  • Azure Event Hubs
  • Azure Service Bus
  • Microsoft SQL Server

Private hosted services and peered VPCs

Under certain scenarios you must use a proxy configuration to allow a connector to attach to a database, storage, or other service running on a private host. A specific scenario for using a proxy is when Kafka clusters are in a peered VPC configuration and a connector needs to attach to a non-peered VPC. This is because peering network connectivity is non-transitive so the connector cannot attach to endpoints in the non-peered VPC.

The following shows an example of this scenario on GCP where a proxy configuration is required.

There are the following three VPCs running on GCP:

  • VPC A: The Confluent Cloud VPC.
  • VPC B: The customer VPC.
  • VPC C: The Cloud SQL VPC.

In this configuration, there is no transitive peering from VPC A to the private Cloud SQL database running on VPC C. For the connector to be able to attach to the database, a proxy client is added to VPC B so the connector can attach to a proxy server added to VPC C. For more information about setting up this GCP proxy configuration, see How the Cloud SQL Auth proxy works).

Note

AWS Transit Gateway (TGW) clusters do not have this transitive issue unless there is a third VPC to connect to. For example, Confluent Cloud VPC A > TGW > Customer VPC B > Customer VPC C (or a cloud service VPC C as shown in the GCP example scenario).

For Confluent Cloud networking details, see the Cloud Networking docs.

Confluent Cloud API for Connect

For information and examples to use with the Confluent Cloud API for fully-managed connectors, see the Confluent Cloud API for Connect documentation.

Custom connectors

For information about bringing a custom connector for Confluent Cloud, see Custom Connectors for Confluent Cloud.

Cloud Console connector controls

You can use the GUI buttons to start, stop, pause, and delete a connector. Select and display one of your listed connectors to view the controls.

Connector Start, Stop, Pause, and Delete buttons

Connector data previews

For information about connector data previews, see Confluent Cloud Connector Data Previews.

Single message transforms

For information about using single message transforms (SMTs), see Configure Single Message Transforms for Kafka Connectors in Confluent Cloud.

View connector events

For information about viewing Confluent Cloud connector events, see View Confluent Cloud Connector Events.

Important

Viewing connector events is restricted to the OrganizationAdmin RBAC role. Viewing events is not available for other roles.

Service accounts

For information about setting up service accounts, see Confluent Cloud Connector Service Accounts.

RBAC for managed connectors

For information about RBAC and managed connectors, see RBAC for Managed Connectors in Confluent Cloud.

Dead letter queue

For information about accessing and using the Confluent Cloud Dead Letter Queue, see Confluent Cloud Dead Letter Queue.

Connector limitations

To view a list of connector limitations, see Limits for Confluent Cloud Connectors.