# Manage Provider Integration for Fully Managed Connectors in Confluent Cloud
Fully managed connectors in Confluent Cloud can use Confluent Provider Integration to authenticate with your
cloud service provider (CSP) resources using each provider’s native identity mechanism, for example,
AWS IAM roles, Google Cloud service account impersonation, and Microsoft Entra ID applications.
With this feature, connectors use temporary credentials or impersonation to provide secure,
least-privileged access to CSP resources for data ingestion and transfer. This eliminates the need for
long-term access keys and ensures connector access is tightly scoped and governed by your CSP’s identity
and authorization policies.
When configuring a connector, you use provider integration to **act-as** a trusted identity in your CSP to
access required cloud resources. This enables the connector to access only the required resources (for example, S3 or GCS
buckets, Cosmos DB, DynamoDB, or other permitted services) allowed by your resource access policies. Access to Confluent Cloud
resources (like connectors and topics) remains managed by Confluent Cloud access controls, such as access control
lists (ACLs) or role-based access control (RBAC).
This guide describes how to configure a provider integration for fully managed source
and sink connectors.
- [AWS Provider Integration](#connector-aws-pi)
- [Google Cloud Provider Integration](#connector-gcp-pi)
- [Azure Provider Integration](#connector-az-pi)
For more information about provider integration in Confluent Cloud,
see [Integrate with Cloud Service Providers in Confluent Cloud](../integrations/provider-integrations/overview.md#provider-integrations-overview). The guide assumes that you already have a
working Confluent Cloud environment and a CSP account.
## Limitations
By default, the resource quota for number of integrations per **Organization** is set to 100 across all CSPs. Contact
Confluent [Support](https://support.confluent.io/) if you want to increase the quota limit.
## Manage an AWS Provider Integration
### Supported connectors
The following connectors support IAM role-based authorization using AWS Provider Integration:
### Source connectors
- [Amazon CloudWatch Logs Source connector](cc-amazon-cloudwatch-logs-source.md#cc-cloudwatch-source-setup-connection)
- [Amazon DynamoDB CDC Source connector](cc-amazon-dynamodb-cdc-source/cc-amazon-dynamodb-cdc-source.md#cc-amazon-dynamodb-cdc-source-setup-connection)
- [Amazon Kinesis Source connector](cc-kinesis-source.md#cc-kinesis-source-setup-connection)
- [Amazon S3 Source connector](cc-s3-source.md#cc-s3-source-setup-connection)
- [Amazon SQS Source connector](cc-amazon-sqs-source.md#cc-amazon-sqs-source-setup-connection)
- [MySQL CDC Source V2 connector](cc-mysql-source-cdc-v2-debezium/cc-mysql-source-cdc-v2-debezium.md#cc-mysql-cdc-source-v2-debezium-aws-iam-auth)
- [MySQL Source connector](cc-mysql-source.md#cc-mysql-source-aws-iam-auth)
- [PostgreSQL CDC Source V2 connector](cc-postgresql-cdc-source-v2-debezium/cc-postgresql-cdc-source-v2-debezium.md#cc-postgresql-cdc-source-v2-debezium-aws-iam-auth)
- [PostgreSQL Source connector](cc-postgresql-source.md#cc-postgresql-source-aws-iam-auth)
- [Redis Kafka Source connector](cc-redis-kafka-source.md#cc-redis-kafka-source-setup-connection)
### Sink connectors
- [Amazon DocumentDB Sink connector](cc-amazon-document-db-sink/cc-document-db-sink.md#cc-document-db-sink-setup-connection)
- [Amazon DynamoDB Sink connector](cc-amazon-dynamo-db-sink.md#cc-amazon-dynamodb-sink-setup-connection)
- [AWS Lambda Sink connector](cc-aws-lambda-sink.md#cc-aws-lambda-sink-setup-connection)
- [Amazon Redshift Sink connector](cc-amazon-redshift-sink.md#cc-amazon-redshift-sink-setup-connection)
- [Amazon S3 Sink connector](cc-s3-sink/cc-s3-sink.md#cc-s3-sink-setup-connection)
- [MySQL Sink connector](cc-mysql-sink.md#cc-mysql-sink-aws-iam-auth)
- [PostgreSQL Sink connector](cc-postgresql-sink.md#cc-postgresql-sink-aws-iam-auth)
- [Redis Kafka Sink connector](cc-redis-kafka-db-sink.md#cc-redis-kafka-sink-setup-connection)
### Create an AWS Provider Integration
You must configure a provider integration in Confluent Cloud before
you modify an existing connector or create a new one with provider integration enabled.
To configure an AWS Provider Integration, you begin by selecting a role type (new or existing)
and creating an IAM permission policy in AWS. Next, you create a new role in AWS and map
it within Confluent Cloud. Finally, you update the trust policy in AWS with the
Confluent IAM role configurations to complete the setup.
For detailed setup, see [Create an AWS Provider Integration](../integrations/provider-integrations/create-provider-integration-aws.md#create-provider-integration-aws-steps).
### Create a connector with an AWS IAM role
Now that you’ve successfully configured a Confluent IAM role to assume the role in your AWS account,
you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.
Make a note of the following:
- You must have a `ResourceOwner` or `Assigner` role scoped to the `Provider-Integration` resource type in your environment to
create a connector using the provider integration resource.
- Use [Read a provider integration](../api-docs/pi-api.md#ccloud-pi-api-read) API to fetch `provider.integration.id` in your Confluent environment.
### Cloud Console
**Authenticate using AWS IAM role**
At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step:
1. Under **AWS credentials**, select IAM Roles in the **Authentication method** field.
2. Select your provider integration name in the **Provider integration name** field.
**Validate AWS IAM role authorization**
After the connector is running, verify the messages and the resources for successful IAM role
authorization through Confluent Provider Integration.
### View topic messages
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`.
3. Click the connector you just created above.
4. View the messages produced to a topic.
### View provider integration resources
1. Go to the **Environments** page at
[https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which
you just created a provider integration.
2. Click the **Provider integrations** tab.
3. Under the **Resources** column, view the resources.
4. Click on the resource link to view the associated connector that you just created above.
### Confluent CLI
**Authenticate using AWS IAM role**
Create a JSON file that contains the connector configuration properties with provider integration.
Add `authentication.method` and `provider.integration.id` in the JSON file.
Set `authentication.method` as `IAM Roles` in this case.
```none
{
"name":"Test-S3-Sink-Connector",
"connector.class":"S3_SINK",
"kafka.auth.mode":"KAFKA_API_KEY",
"kafka.api.key":"",
"kafka.api.secret":"",
"authentication.method":"IAM Roles",
"provider.integration.id":"dlz-f3a90de",
"s3.region":"",
"s3.bucket.name":"",
"input.data.format":"JSON",
"output.data.format":"JSON",
"time.interval":"DAILY",
"tasks.max":"1",
"topics": ", "
}
```
**Load the properties file**
Enter the following command to load the JSON and start the connector:
```none
confluent connect cluster create --config-file .json
```
For example:
```none
confluent connect cluster create --config-file s3-sink-config.json
```
Example output:
```none
Created connector confluent-s3-sink lcc-ix4dl
```
**Validate AWS IAM role authorization**
After successful provisioning, verify messages are populating your Kafka topic and view resources
related to the provider integration.
### Verify messages
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`.
3. Click the connector you just created above.
4. View the messages produced to a topic.
### View provider integration resources
Enter the following command to view the provider integration resources:
```none
confluent provider-integration describe
```
For example:
```none
confluent provider-integration describe cspi-12345
```
Example output:
```none
+-------------------+--------------------------------------------------------+
| ID | cspi-12345 |
| Name | s3-provider-integration |
| Provider | aws |
| Environment | env-abcdef |
| IAM Role ARN | arn:aws:iam::000000000000:role/confluent-aws-role |
| External ID | 123456-2312-21bc-n2xx-oo1a7b7890 |
| Customer Role ARN | arn:aws:iam::000000000000:role/my-test-aws-role |
+-------------------+--------------------------------------------------------+
```
### Confluent REST API
**Authenticate using AWS IAM role**
Add `authentication.method` and `provider.integration.id` in
the [Create a connector](connect-api-section.md#ccloud-connect-api-create-connector) API request. Set `authentication.method`
as `IAM Roles` in this case.
```bash
curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic ' \
--header 'content-type: application/json' \
--data '{
"name":"Test-S3-Sink-Connector",
"config":{
"connector.class":"S3_SINK",
"name":"Test-S3-Sink-Connector",
"kafka.auth.mode":"KAFKA_API_KEY",
"kafka.api.key":"",
"kafka.api.secret":"",
"s3.region":"",
"s3.bucket.name":"",
"output.data.format":"JSON",
"time.interval":"DAILY",
"tasks.max":"1",
"topics": ", ",
"authentication.method":"IAM Roles",
"provider.integration.id":"dlz-f3a90de"
},
}'
```
**Validate AWS IAM role authorization**
Your new connector is now processing the messages after successful provisioning.
Use Confluent Cloud Console to view the messages.
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`. Click the connector you just created in the Step 4.
3. View the messages produced to a topic, for example, from the AWS S3 bucket.
This validates the successful IAM role authorization through Confluent Provider Integration feature.
## Manage a Google Cloud Provider Integration
### Supported connectors
The following connectors support authorization through Google Cloud service account impersonation:
### Source connectors
- [Google Cloud Pub/Sub Source connector](cc-google-pubsub-source.md#cc-google-pubsub-source-setup-connection)
- [Google Cloud Storage (GCS) Source connector](cc-gcs-source.md#cc-gcs-source-setup-connection)
- [MySQL Source connector](cc-mysql-source.md#cc-mysql-source-setup-connection)
- [PostgreSQL CDC Source V2 (Debezium) connector](cc-postgresql-cdc-source-v2-debezium/cc-postgresql-cdc-source-v2-debezium.md#cc-postgresql-source-cdc-v2-debezium-setup-connection)
- [PostgreSQL Source connector](cc-postgresql-source.md#cc-postgresql-source-setup-connection)
### Sink connectors
- [Google BigQuery Sink V2 connector](cc-gcp-bigquery-storage-sink.md#cc-gcp-bigquery-storage-sink-setup-connection)
- [Google Cloud Storage Sink connector](cc-gcs-sink.md#cc-gcs-storage-sink-sink-setup-connection)
- [MySQL Sink connector](cc-mysql-sink.md#cc-mysql-sink-setup-connection)
- [PostgreSQL Sink connector](cc-postgresql-sink.md#cc-postgresql-sink-setup-connection)
### Create a Google Cloud Provider Integration
You must create a Google Cloud Provider Integration in Confluent Cloud before
you modify an existing connector or create a new one that uses provider integration.
To create a Google Cloud Provider Integration:
: - Generate a Google Cloud service account in Confluent Cloud.
- Create a [new role](https://console.cloud.google.com/iam-admin/roles) in your Google Cloud account and assign required resource access permission to run the connector
- Create a [new service account](https://console.cloud.google.com/iam-admin/serviceaccounts) in your Google Cloud account and assign it the role you just created.
- Grant `Service Account Token Creator` access to Confluent’s Google Cloud service account on your Google Cloud service account.
- Map your Google Cloud service account to establish a trust relationship with Confluent’s Google Cloud service account.
- (Optional) Verify that the trust is established between Confluent Cloud and your Google Cloud account using the **Validate** option.
For more information about Google Cloud Provider Integration setup,
see [Integrate with Google Cloud in Confluent Cloud](../integrations/provider-integrations/create-provider-integration-gc.md#create-provider-integration-gc).
### Create a connector with a Google Cloud Provider Integration
Now that you’ve successfully created a Google Cloud Provider Integration in Confluent Cloud,
you can now use it to create a connector. For example, you can create a PostgreSQL CDC Source V2 (Debezium) connector or a
Google Cloud Storage Sink connector, using Google Cloud service account impersonation as the authentication method.
Make a note of the following:
- You must have a `ProviderIntegrationResourceOwner` or `ProviderIntegrationAssigner` role to
create a connector using the provider integration resource.
- Use [Read a provider integration](../api-docs/pi-api.md#ccloud-pi-api-read) API to fetch `provider.integration.id` in your Confluent environment.
### Cloud Console
**Authenticate using Google Cloud service account**
At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step:
1. Under **GCP credentials**, select **Google service account impersonation** in the **Authentication method** field.
2. Select your provider integration name in the **Provider integration name** field.
**Verify Google Cloud service account authorization**
After the connector is running, verify the messages and the resources for successful service account
authorization through Confluent Provider Integration.
### View topic messages
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`.
3. Click the connector you just created above.
4. View the messages produced to a topic.
### View provider integration resources
1. Go to the **Environments** page at
[https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which
you just created a provider integration.
2. Click the **Provider integrations** tab.
3. Under the **Resources** column, view the resources.
4. Click on the resource link to view the associated connector that you just created above.
### Confluent REST API
**Authenticate using Google Cloud service account**
Add `authentication.method` and `provider.integration.id` in
the [Create a connector](connect-api-section.md#ccloud-connect-api-create-connector) API request. Set `authentication.method`
as `Google service account impersonation` in this case.
```bash
curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic ' \
--header 'content-type: application/json' \
--data '{
"name": "GcsSinkConnector_1",
"config": {
"topics": "topic_0",
"input.data.format": "JSON",
"connector.class": "GcsSink",
"name": "GcsSinkConnector_1",
"kafka.auth.mode": "SERVICE_ACCOUNT",
"kafka.service.account.id": "sa-devcjqj79q",
"provider.integration.id": "cspi-12345",
"authentication.method": "Google service account impersonation",
"gcs.bucket.name": "gcs-system-tests-connect",
"time.interval": "HOURLY",
"tasks.max": "1"
}
}
```
**Verify Google Cloud service account authorization**
Your new connector is now processing the messages after successful provisioning.
Use Confluent Cloud Console to view the messages.
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`. Click the connector you just created in the Step 4.
3. View the messages produced to a topic, for example, from the GCS bucket.
This validates the successful Google Cloud service account-based authorization through Confluent Provider Integration.
## Manage an Microsoft Azure Provider Integration
### Supported connectors
The following connectors support Microsoft Entra ID-based authorization using Microsoft Azure Provider Integration:
### Source connectors
- [Azure Blob Storage Source connector](cc-azure-blob-source.md#cc-azure-blob-source-setup-connection)
- [Azure Cosmos DB Source V2 connector](cc-azure-cosmos-source-v2.md#cc-cosmosdb-source-v2-setup-connection)
- [Azure Event Hubs Source connector](cc-azure-event-hubs-source.md#cc-azure-event-hubs-source-setup-connection)
- [Microsoft SQL Server CDC Source V2 (Debezium) connector](cc-microsoft-sql-server-source-cdc-v2-debezium/cc-microsoft-sql-server-source-cdc-v2-debezium.md#cc-microsoft-sql-server-cdc-source-v2-debezium-setup-connection)
- [Microsoft SQL Server Source connector](cc-microsoft-sql-server-source.md#cc-microsoft-sql-server-source-setup-connection)
- [MySQL Source connector](cc-mysql-source.md#cc-mysql-source-setup-connection)
- [PostgreSQL Source connector](cc-postgresql-source.md#cc-postgresql-source-setup-connection)
### Sink connectors
- [Azure Blob Storage Sink connector](cc-azure-blob-sink/cc-azure-blob-sink.md#cc-azure-blob-sink-setup-connection)
- [Azure Cosmos DB Sink V2 connector](cc-azure-cosmos-sink-v2.md#cc-azure-cosmos-v2-sink-setup-connection)
- [Azure Data Lake Storage Gen2 Sink connector](cc-azure-datalakeGen2-storage-sink.md#cc-azure-datalakegen2-sink-setup-connection)
- [Microsoft SQL Server Sink connector](cc-microsoft-sql-server-sink.md#cc-microsoft-sql-server-sink-setup-connection)
- [MySQL Sink connector](cc-mysql-sink.md#cc-mysql-sink-setup-connection)
- [PostgreSQL Sink connector](cc-postgresql-sink.md#cc-postgresql-sink-setup-connection)
### Create a Microsoft Azure Provider Integration
You must create a Microsoft Azure Provider Integration in Confluent Cloud before
you modify an existing connector or create a new one that uses provider integration.
To create a Azure Provider Integration:
: - Generate a multi-tenant Entra ID app in Confluent’s Azure account.
- Create a new service principal in your [Microsoft Entra ID console](https://entra.microsoft.com/) using Confluent’s Entra ID application
and assign required resource access permission to run the connector.
- Map your Azure tenant ID to establish a trust relationship with Confluent’s Entra ID application.
- (Optional) Verify that the trust is established between Confluent Cloud and your Azure account using **Validate** option.
For more information about Microsoft Azure Provider Integration setup,
see [Integrate with Azure in Confluent Cloud](../integrations/provider-integrations/create-provider-integration-azure.md#create-provider-integration-azure).
### Create a connector with a Microsoft Azure Provider Integration
Now that you’ve successfully created an Microsoft Azure Provider Integration in Confluent Cloud, you can use it to create a connector.
For example, you can create an Azure Cosmos DB Source V2 connector or an Azure Data Lake Storage Gen2 Sink connector,
using the service principal with Microsoft Entra ID application as the authentication method.
Make a note of the following:
- You must have a `ProviderIntegrationResourceOwner` or `ProviderIntegrationAssigner` role to
create a connector using the provider integration resource.
- Use [Read a provider integration](../api-docs/pi-api.md#ccloud-pi-api-read) API to fetch `provider.integration.id` in your Confluent environment.
### Cloud Console
**Authenticate using Microsoft Entra ID**
At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step:
1. Under **Azure credentials**, select **Microsoft Entra ID application** in the **Authentication method** field.
2. Select your provider integration name in the **Provider integration** field.
**Verify Microsoft Entra ID-based authentication**
After the connector is running, verify the messages and the resources for successful Entra ID-based
authorization through Confluent Provider Integration.
### View topic messages
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`.
3. Click the connector you just created above.
4. View the messages produced to a topic.
### View provider integration resources
1. Go to the **Environments** page at
[https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which
you just created a provider integration.
2. Click the **Provider integrations** tab.
3. Under the **Resources** column, view the resources.
4. Click on the resource link to view the associated connector that you just created above.
### Confluent REST API
**Authenticate using Microsoft Entra ID**
Add `authentication.method` and `provider.integration.id` in
the [Create a connector](connect-api-section.md#ccloud-connect-api-create-connector) API request. Set `authentication.method`
as `Microsoft Entra ID application` in this case.
```bash
curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic ' \
--header 'content-type: application/json' \
--data '{
"name": "CosmosDbSinkV2Connector_0",
"config": {
"provider.integration.id": "cspi-12345",
"authentication.method": "Microsoft Entra ID application",
"connector.class": "CosmosDbSinkV2",
"name": "CosmosDbSinkV2Connector_0",
"input.data.format": "JSON",
"kafka.auth.mode": "SERVICE_ACCOUNT",
"kafka.service.account.id": "sa-dev1234q",
"topics": "topic_0",
"azure.cosmos.account.endpoint": "https://azurepi.documents.azure.com:443/",
"azure.cosmos.sink.containers.topicMap": "topic_0#test-container",
"azure.cosmos.sink.database.name": "test-db",
"tasks.max": "1",
"region.ignore": "true"
},
}'
```
**Verify Microsoft Entra ID-based authentication**
Your new connector is now processing the messages after successful provisioning.
Use Confluent Cloud Console to view the messages.
1. Sign in to your Confluent Cloud account.
2. In the left navigation menu, click `Connectors`. Click the connector you just created in the Step 4.
3. View the messages produced to a topic, for example, from the GCS bucket.
This validates the successful Entra ID-based authorization through Confluent Provider Integration.
## Delete resources
When you are finished with the setup, delete the resources you created to avoid
unexpected charges to your account.
### Cloud Console
### Delete a connector
1. Sign in to your Confluent Cloud account.
2. Click the connector you created and choose the **Settings** tab.
3. Click **Delete connector**, enter the connector name, and click **Confirm**.
### Delete a provider integration
1. Go to the **Environments** page at
[https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which
you just created a provider integration.
2. Click the **Provider integration** tab.
3. Click **Delete**, enter the provider integration name, and click **Confirm**.
#### NOTE
The DELETE request will fail if any Confluent workloads are using this provider integration.
### Confluent CLI
### Delete a connector
Enter the following command to delete the connector:
```none
confluent connect cluster delete --config-file .json
```
### Delete a provider integration
Enter the following command to remove the provider integration in your environment:
```none
confluent provider-integration delete
```
For example:
```none
confluent provider-integration delete cspi-12345
```
#### NOTE
The delete request will fail if any Confluent workloads are using this provider integration.
### Confluent REST API
### Delete a connector
1. Sign in to your Confluent Cloud account.
2. Click the connector you created and choose the **Settings** tab.
3. Click **Delete connector**, enter the connector name, and click **Confirm**.
### Delete a provider integration
Use [Delete a provider integration](../api-docs/pi-api.md#ccloud-pi-api-delete) API to remove the provider integration in your environment.
Successful completion delete an integration configuration.
#### NOTE
The DELETE request will fail if any Confluent workloads are using this provider integration.
## Related content
- [Create an AWS Provider Integration](../integrations/provider-integrations/create-provider-integration-aws.md#create-provider-integration-aws)
- [Create a Google Cloud Provider Integration](../integrations/provider-integrations/create-provider-integration-gc.md#create-provider-integration-gc)
- [Create an Azure Provider Integration](../integrations/provider-integrations/create-provider-integration-azure.md#create-provider-integration-azure)
- Learn about [Provider Integration APIs](../api-docs/pi-api.md#ccloud-pi-api).
- Know more about [Provider Integration CLI](https://docs.confluent.io/confluent-cli/current/command-reference/provider-integration/index.html).
- Learn about [Provider Integration resource](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_provider_integration)
in the Terraform registry. See a [Terraform example](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_connector#example-managed-amazon-s3-sink-connectorhttpsdocsconfluentiocloudcurrentconnectorscc-s3-sinkhtml-that-uses-a-service-account-to-communicate-with-your-kafka-cluster-and-iam-roles-for-aws-authentication)
of setting up a fully managed S3 Sink connector with IAM role-based AWS authentication.
- For the second-generation Azure and Google Cloud provider integration
flow, use the
[confluent_provider_integration_setup](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_provider_integration_setup)
resource to create the integration, then the
[confluent_provider_integration_authorization](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_provider_integration_authorization)
resource to validate it.