Manage Provider Integration for Fully-Managed Connectors in Confluent Cloud¶

Fully-managed connectors in Confluent Cloud can use Confluent Provider Integration to authenticate with your cloud service provider (CSP) resources using each provider’s native identity mechanism, for example, AWS IAM roles, Google Cloud service account impersonation, and Microsoft Entra ID applications. With this feature, connectors use temporary credentials or impersonation to provide secure, least-privileged access to CSP resources for data ingestion and transfer. This eliminates the need for long-term access keys and ensures connector access is tightly scoped and governed by your CSP’s identity and authorization policies.

When configuring a connector, you use provider integration to act-as a trusted identity in your CSP to access required cloud resources. This enables the connector to access only the required resources (for example, S3 or GCS buckets, Cosmos DB, DynamoDB, or other permitted services) allowed by your resource access policies. Access to Confluent Cloud resources (like connectors and topics) remains managed by Confluent Cloud access controls, such as access control lists (ACLs) or role-based access control (RBAC).

This guide describes how to configure a provider integration for fully-managed source and sink connectors.

AWS Provider Integration
Google Cloud Provider Integration
Azure Provider Integration

For more information about provider integration in Confluent Cloud, see Integrate with Cloud Service Providers in Confluent Cloud. The guide assumes that you already have a working Confluent Cloud environment and a CSP account.

Limitations¶

By default, the resource quota for number of integrations per Organization is set to 100 across all CSPs. Contact Confluent Support if you want to increase the quota limit.

Manage an AWS Provider Integration¶

Supported connectors¶

The following connectors support IAM role-based authorization using AWS Provider Integration:

Create an AWS Provider Integration¶

You must configure a provider integration in Confluent Cloud before you modify an existing connector or create a new one with provider integration enabled.

To configure an AWS Provider Integration, you begin by selecting a role type (new or existing) and creating an IAM permission policy in AWS. Next, you create a new role in AWS and map it within Confluent Cloud. Finally, you update the trust policy in AWS with the Confluent IAM role configurations to complete the setup.

For detailed setup, see Steps to create an AWS provider integration.

Tip

Video walkthrough: This video shows how to create an AWS provider integration in Confluent Cloud that uses an AWS IAM role to let Confluent Cloud securely access your AWS resources, without having to manage static keys or passwords!

Create a connector with an AWS IAM role¶

Now that you’ve successfully configured a Confluent IAM role to assume the role in your AWS account, you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.

Make a note of the following:

You must have a ResourceOwner or Assigner role scoped to the Provider-Integration resource type in your environment to create a connector using the provider integration resource.
Use Read a provider integration API to fetch provider.integration.id in your Confluent environment.

Authenticate using AWS IAM role

At the Add Source/Sink connector screen, complete the following in the Authentication step:

Under AWS credentials, select IAM Roles in the Authentication method field.
Select your provider integration name in the Provider integration name field.

Validate AWS IAM role authorization

After the connector is running, verify the messages and the resources for successful IAM role authorization through Confluent Provider Integration.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors.
Click the connector you just created above.
View the messages produced to a topic.

Authenticate using AWS IAM role

Create a JSON file that contains the connector configuration properties with provider integration. Add authentication.method and provider.integration.id in the JSON file. Set authentication.method as IAM Roles in this case.

{
   "name":"Test-S3-Sink-Connector",
   "connector.class":"S3_SINK",
   "kafka.auth.mode":"KAFKA_API_KEY",
   "kafka.api.key":"<my-kafka-api-key>",
   "kafka.api.secret":"<my-kafka-api-secret>",
   "authentication.method":"IAM Roles",
   "provider.integration.id":"dlz-f3a90de",
   "s3.region":"<s3-region>",
   "s3.bucket.name":"<my-iam-s3-bucket>",
   "input.data.format":"JSON",
   "output.data.format":"JSON",
   "time.interval":"DAILY",
   "tasks.max":"1",
   "topics": "<topic-1>, <topic-2>"
}

Load the properties file

Enter the following command to load the JSON and start the connector:

confluent connect cluster create --config-file <file-name>.json

For example:

confluent connect cluster create --config-file s3-sink-config.json

Example output:

Created connector confluent-s3-sink lcc-ix4dl

Validate AWS IAM role authorization

After successful provisioning, verify messages are populating your Kafka topic and view resources related to the provider integration.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors.
Click the connector you just created above.
View the messages produced to a topic.

Enter the following command to view the provider integration resources:

confluent provider-integration describe <provider-integration-id>

For example:

confluent provider-integration describe cspi-12345

Example output:

+-------------------+--------------------------------------------------------+
| ID                | cspi-12345                                             |
| Name              | s3-provider-integration                                |
| Provider          | aws                                                    |
| Environment       | env-abcdef                                             |
| IAM Role ARN      | arn:aws:iam::000000000000:role/confluent-aws-role      |
| External ID       | 123456-2312-21bc-n2xx-oo1a7b7890                       |
| Customer Role ARN | arn:aws:iam::000000000000:role/my-test-aws-role        |
+-------------------+--------------------------------------------------------+

Authenticate using AWS IAM role

Add authentication.method and provider.integration.id in the Create a connector API request. Set authentication.method as IAM Roles in this case.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name":"Test-S3-Sink-Connector",
   "config":{
   "connector.class":"S3_SINK",
   "name":"Test-S3-Sink-Connector",
   "kafka.auth.mode":"KAFKA_API_KEY",
   "kafka.api.key":"<my-kafka-api-key>",
   "kafka.api.secret":"<my-kafka-api-secret>",
   "s3.region":"<s3-region>",
   "s3.bucket.name":"<my-iam-s3-bucket>",
   "output.data.format":"JSON",
   "time.interval":"DAILY",
   "tasks.max":"1",
   "topics": "<topic-1>, <topic-2>",
   "authentication.method":"IAM Roles",
   "provider.integration.id":"dlz-f3a90de"
   },
}'

Validate AWS IAM role authorization

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.
View the messages produced to a topic, for example, from the AWS S3 bucket.

This validates the successful IAM role authorization through Confluent Provider Integration feature.

Manage a Google Cloud Provider Integration¶

Supported connectors¶

The following connectors support authorization through Google Cloud service account impersonation:

PostgreSQL CDC Source V2 (Debezium) connector

Create a Google Cloud Provider Integration¶

You must create a Google Cloud Provider Integration in Confluent Cloud before you modify an existing connector or create a new one that uses provider integration.

To create a Google Cloud Provider Integration:

Generate a Google Cloud service account in Confluent Cloud.
Create a new role in your Google Cloud account and assign required resource access permission to run the connector
Create a new service account in your Google Cloud account and assign it the role you just created.
Grant Service Account Token Creator access to Confluent’s Google Cloud service account on your Google Cloud service account.
Map your Google Cloud service account to establish a trust relationship with Confluent’s Google Cloud service account.
(Optional) Verify that the trust is established between Confluent Cloud and your Google Cloud account using the Validate option.

For more information about Google Cloud Provider Integration setup, see Integrate with Google Cloud in Confluent Cloud.

Create a connector with a Google Cloud Provider Integration¶

Now that you’ve successfully created a Google Cloud Provider Integration in Confluent Cloud, you can now use it to create a connector. For example, you can create a PostgreSQL CDC Source V2 (Debezium) connector or a Google Cloud Storage Sink connector, using Google Cloud service account impersonation as the authentication method.

Make a note of the following:

You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
Use Read a provider integration API to fetch provider.integration.id in your Confluent environment.

Authenticate using Google Cloud service account

At the Add Source/Sink connector screen, complete the following in the Authentication step:

Under GCP credentials, select Google service account impersonation in the Authentication method field.
Select your provider integration name in the Provider integration name field.

Verify Google Cloud service account authorization

After the connector is running, verify the messages and the resources for successful service account authorization through Confluent Provider Integration.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors.
Click the connector you just created above.
View the messages produced to a topic.

Authenticate using Google Cloud service account

Add authentication.method and provider.integration.id in the Create a connector API request. Set authentication.method as Google service account impersonation in this case.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "GcsSinkConnector_1",
   "config": {
     "topics": "topic_0",
     "input.data.format": "JSON",
     "connector.class": "GcsSink",
     "name": "GcsSinkConnector_1",
     "kafka.auth.mode": "SERVICE_ACCOUNT",
     "kafka.service.account.id": "sa-devcjqj79q",
     "provider.integration.id": "cspi-12345",
     "authentication.method": "Google service account impersonation",
     "gcs.bucket.name": "gcs-system-tests-connect",
     "time.interval": "HOURLY",
     "tasks.max": "1"
   }
}

Verify Google Cloud service account authorization

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.
View the messages produced to a topic, for example, from the GCS bucket.

This validates the successful Google Cloud service account-based authorization through Confluent Provider Integration.

Manage an Microsoft Azure Provider Integration¶

Supported connectors¶

The following connectors support Microsoft Entra ID-based authorization using Microsoft Azure Provider Integration:

Create a Microsoft Azure Provider Integration¶

You must create a Microsoft Azure Provider Integration in Confluent Cloud before you modify an existing connector or create a new one that uses provider integration.

To create a Azure Provider Integration:

Generate a multi-tenant Entra ID app in Confluent’s Azure account.
Create a new service principal in your Microsoft Entra ID console using Confluent’s Entra ID application and assign required resource access permission to run the connector.
Map your Azure tenant ID to establish a trust relationship with Confluent’s Entra ID application.
(Optional) Verify that the trust is established between Confluent Cloud and your Azure account using Validate option.

For more information about Microsoft Azure Provider Integration setup, see Integrate with Azure in Confluent Cloud.

Create a connector with a Microsoft Azure Provider Integration¶

Now that you’ve successfully created an Microsoft Azure Provider Integration in Confluent Cloud, you can use it to create a connector. For example, you can create an Azure Cosmos DB Source V2 connector or an Azure Data Lake Storage Gen2 Sink connector, using the service principal with Microsoft Entra ID application as the authentication method.

Make a note of the following:

You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
Use Read a provider integration API to fetch provider.integration.id in your Confluent environment.

Authenticate using Microsoft Entra ID

At the Add Source/Sink connector screen, complete the following in the Authentication step:

Under Azure credentials, select Microsoft Entra ID application in the Authentication method field.
Select your provider integration name in the Provider integration field.

Verify Microsoft Entra ID-based authentication

After the connector is running, verify the messages and the resources for successful Entra ID-based authorization through Confluent Provider Integration.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors.
Click the connector you just created above.
View the messages produced to a topic.

Authenticate using Microsoft Entra ID

Add authentication.method and provider.integration.id in the Create a connector API request. Set authentication.method as Microsoft Entra ID application in this case.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "CosmosDbSinkV2Connector_0",
   "config": {
      "provider.integration.id": "cspi-12345",
      "authentication.method": "Microsoft Entra ID application",
      "connector.class": "CosmosDbSinkV2",
      "name": "CosmosDbSinkV2Connector_0",
      "input.data.format": "JSON",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-dev1234q",
      "topics": "topic_0",
      "azure.cosmos.account.endpoint": "https://azurepi.documents.azure.com:443/",
      "azure.cosmos.sink.containers.topicMap": "topic_0#test-container",
      "azure.cosmos.sink.database.name": "test-db",
      "tasks.max": "1",
      "region.ignore": "true"
   },
}'

Verify Microsoft Entra ID-based authentication

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

Sign in to your Confluent Cloud account.
In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.
View the messages produced to a topic, for example, from the GCS bucket.

This validates the successful Entra ID-based authorization through Confluent Provider Integration.

Delete resources¶

When you are finished with the setup, delete the resources you created to avoid unexpected charges to your account.

Sign in to your Confluent Cloud account.
Click the connector you created and choose the Settings tab.
Click Delete connector, enter the connector name, and click Confirm.

Enter the following command to delete the connector:

confluent connect cluster delete --config-file <file-name>.json

Enter the following command to remove the provider integration in your environment:

confluent provider-integration delete <provider-integration-id>

For example:

confluent provider-integration delete cspi-12345

Note

The delete request will fail if any Confluent workloads are using this provider integration.

Manage Provider Integration for Fully-Managed Connectors in Confluent Cloud¶

Limitations¶

Manage an AWS Provider Integration¶

Supported connectors¶

Create an AWS Provider Integration¶

Create a connector with an AWS IAM role¶

Manage a Google Cloud Provider Integration¶

Supported connectors¶

Create a Google Cloud Provider Integration¶

Create a connector with a Google Cloud Provider Integration¶

Manage an Microsoft Azure Provider Integration¶

Supported connectors¶

Create a Microsoft Azure Provider Integration¶

Create a connector with a Microsoft Azure Provider Integration¶

Delete resources¶

Related content¶