Quick Start for Confluent Cloud Provider Integration¶
Confluent Cloud Provider Integration offers Identity and Access Management (IAM) Role-based authorization that lets you adopt the temporary security credentials of an IAM role, which acts as a set of permission policies. Trusted entities, such as IAM users, applications, or cloud services can assume this role. Using this approach, you can create a secure access connection between source or sink resources on AWS and Confluent Cloud for data ingestion or transfer.
This quick start walks you through the Confluent Cloud Console, CLI and APIs to get you up and running with IAM role-based authorization using Confluent Provider Integration. The quick start assumes that you already have a working Confluent Cloud environment and an AWS account.
Supported connectors¶
The following connectors support IAM role-based authorization using Confluent Provider Integration:
Source connectors¶
The following fully-managed source connector supports provider integration setup:
Sink connectors¶
The following fully-managed sink connector supports provider integration setup:
Prerequisites¶
- Access to AWS Management Console.
- Access to Confluent Cloud.
- The OrganizationAdmin or EnvironmentAdmin role to setup provider integration. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin.
- In addition to the above role, the ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role can create a connector using the provider integration resource. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin. Use the following example CRN
"crn_pattern": "crn://confluent.cloud/organization=org-ID/environment=env-ID/provider-integration=cspi-123j1"
to identify the provider integration resources. - An environment ID for your organization.
- A Confluent Cloud API key to authenticate with the Provider Integration API. For information about how to create a Confluent Cloud API key, see Manage API Keys.
- cURL and jq installed to use the API request examples in this document.
Quota and limits¶
By default, the resource quota for number of integrations per Organization is set to 100. Contact Confluent Support if you want to increase the quota limit.
Using the Confluent Cloud Console¶
Configure provider integration¶
Sign in to your Confluent Cloud account and go through the following instructions to configure your first provider integration in Confluent Cloud.
Step 1: Select a role in AWS¶
You can create a new role or select an existing role to create a provider integration. In this step, you choose to create a new role to configure your first provider integration.
Note
If you select an existing IAM role, you can proceed directly to mapping the role in Confluent Cloud.
- Go to the Environments page at https://confluent.cloud/environments and select the environment in which you want to create your first provider integration.
- Click the Provider integrations tab.
- Click Add integration. The role screen appears.
- Select New role to create a new role in your AWS account.
- Click Continue.
Step 2: Create permission policy in AWS¶
In this step, you will create an IAM permission policy in your AWS account.
- Open the AWS console at https://console.aws.amazon.com/iam/ and create a new permission policy. For more information, see Create AWS IAM Policies.
- In Confluent Cloud Console, select a Confluent resource to view the permission policy needed for
AWS interaction. Copy the
Permission-policy.json
and paste it into the policy editor in the AWS console. Save your new IAM permission policy. - Click Continue.
Step 3: Create a new role in AWS and map in Confluent¶
In this step, you will create a new IAM role in your AWS account and map the role in Confluent.
Open the AWS console at https://console.aws.amazon.com/iam/ and create a new IAM role. For more information, see Create AWS IAM Roles.
In the Trusted entity type screen, select Custom trust policy. In Confluent Cloud Console, copy the
Trust-policy.json
and paste it into the policy editor in the AWS console. Save your new IAM role.Note
You will need to update the policy with the Confluent-specific configurations such as
Confluent ARN
andExternalID
after mapping the role in Confluent.Copy the
ARN
of the IAM role you just created for use in mapping the role in Confluent.
- In the AWS ARN field, paste the ARN copied during the creation of a new IAM role in AWS.
- In the Provider integration name field, specify a name for the integration.
- Click Continue.
Note
Copy the ARN
of the IAM role you just created for use in the provider integration setup in Confluent.
Step 4: Update trust policy in AWS¶
In this step, you will update the trust policy in the AWS account with Confluent IAM role configurations. This allows the Confluent IAM role to assume the role in your AWS account.
- In Confluent Cloud Console, copy the
Trust-policy.json
file that includesiam_role_arn
,external_id
andTagSession
for each IAM role association. - Open the AWS console at https://console.aws.amazon.com/iam/ and go the trust policy you created in the Step 3. For more information, see Edit trust policies.
- Paste the updated
Trust-policy.json
into the trust policy editor in the AWS console.
Create a connector with an IAM role¶
At the Add Source/Sink connector screen, complete the following in the Authentication step:
- Under AWS credentials, select IAM Roles in the Authentication method field.
- Select your provider integration name in the Provider integration name field.
Validate IAM assume role authorization in Confluent¶
After successful provisioning, your new connector is processing messages. Use the Confluent Cloud Console to view these messages and the resources related to the provider integration.
- Sign in to your Confluent Cloud account.
- In the left navigation menu, click
Connectors
. - Click the connector you just created above.
- View the messages produced to a topic.
- Go to the Environments page at https://confluent.cloud/environments and select the environment in which you just created a provider integration.
- Click the Provider integrations tab.
- Under the Resources column, view the resources.
- Click on the resource link to view the associated connector that you just created above.
This validates the successful IAM role authorization through Confluent Provider Integration.
Delete resources¶
When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.
- Sign in to your Confluent Cloud account.
- Click the connector you created and choose the Settings tab.
- Click Delete connector, enter the connector name, and click Confirm.
- Go to the Environments page at https://confluent.cloud/environments and select the environment in which you just created a provider integration.
- Click the Provider integration tab.
- Click Delete, enter the provider integration name, and click Confirm.
Note
The DELETE request will fail if any Confluent workloads are using this provider integration.
Using the Confluent CLI¶
Step 1: Create an IAM policy and role in AWS¶
Follow the AWS instructions to create an IAM policy and role in your AWS account.
Step 2: Create an IAM role mapping in Confluent¶
Enter the following command to set up a provider integration and create an IAM role mapping in Confluent:
confluent provider-integration create <name> [flags]
For example, to create a provider integration named s3-provider-integration, add
AWS ARN in --customer-role-arn
and environment ID in --environment
string.
confluent provider-integration create s3-provider-integration --cloud aws --customer-role-arn arn:aws:iam::000000000000:role/my-test-aws-role --environment env-abcdef
Example output:
+-------------------+--------------------------------------------------------+
| ID | cspi-12345 |
| Name | s3-provider-integration |
| Provider | aws |
| Environment | env-abcdef |
| IAM Role ARN | arn:aws:iam::990301199020:role/cspi-12345 |
| External ID | 123456-2312-21bc-n2xx-oo1a7b7890 |
| Customer Role ARN | arn:aws:iam::037803949979:role/test-iam-role9999999 |
+-------------------+--------------------------------------------------------+
Step 3: Update trust policy in AWS¶
Follow the AWS instructions to update the trust policy in the AWS account with Confluent IAM role configurations. This allows the Confluent IAM role to assume the role in your AWS account.
Step 4: Create a connector configuration file with provider integration properties in Confluent¶
Create a JSON file that contains the connector configuration properties with provider integration. Add authentication.method
and provider.integration.id
in
the JSON file. Set authentication.method
as IAM Roles
in this case.
{
"name":"Test-S3-Sink-Connector",
"connector.class":"S3_SINK",
"kafka.auth.mode":"KAFKA_API_KEY",
"kafka.api.key":"<my-kafka-api-key>",
"kafka.api.secret":"<my-kafka-api-secret>",
"authentication.method":"IAM Roles",
"provider.integration.id":"dlz-f3a90de",
"s3.region":"<s3-region>",
"s3.bucket.name":"<my-iam-s3-bucket>",
"input.data.format":"JSON",
"output.data.format":"JSON",
"time.interval":"DAILY",
"tasks.max":"1",
"topics": "<topic-1>, <topic-2>"
}
Step 5: Load the properties file and create the connector in Confluent¶
Enter the following command to load the JSON and start the connector:
confluent connect cluster create --config-file <file-name>.json
For example:
confluent connect cluster create --config-file s3-sink-config.json
Example output:
Created connector confluent-s3-sink lcc-ix4dl
Step 6: Validate IAM assume role authorization in Confluent¶
After successful provisioning, verify messages are populating your Kafka topic and view resources related to the provider integration.
- Sign in to your Confluent Cloud account.
- In the left navigation menu, click
Connectors
. - Click the connector you just created above.
- View the messages produced to a topic.
Enter the following command to view the provider integration resources:
confluent provider-integration describe <provider-integration-id>
For example:
confluent provider-integration describe cspi-12345
Example output:
+-------------------+--------------------------------------------------------+
| ID | cspi-12345 |
| Name | s3-provider-integration |
| Provider | aws |
| Environment | env-abcdef |
| IAM Role ARN | arn:aws:iam::990301199020:role/cspi-12345 |
| External ID | 123456-2312-21bc-n2xx-oo1a7b7890 |
| Customer Role ARN | arn:aws:iam::037803949979:role/test-iam-role9999999 |
+-------------------+--------------------------------------------------------+
Step 7: Delete resources¶
When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.
Enter the following command to delete the connector:
confluent connect cluster delete --config-file <file-name>.json
Enter the following command to remove the provider integration in your environment:
confluent provider-integration delete <provider-integration-id>
For example:
confluent provider-integration delete cspi-12345
Note
The delete request will fail if any Confluent workloads are using this provider integration.
Using the Confluent APIs¶
Step 1: Create an IAM policy and role in AWS¶
Follow the instructions to create an IAM policy and role in your AWS account.
Open the AWS console at https://console.aws.amazon.com/iam/.
Navigate to Policies, and then click Create policy.
In the Policy editor screen, select the JSON option.
Copy and paste the S3 or DynamoDB permission policy document in the editor screen.
Click Next to review the permissions.
In the Review and create page, enter a Policy name and a Description.
Click Create policy to save your new IAM policy.
Note
For more information about AWS IAM Policies, see Create AWS IAM Policies.
Now you can get started creating IAM roles in your AWS account.
Navigate to Roles, and then click Create role.
In the Trusted entity type screen, select Custom trust policy.
In the Custom trust policy editor, copy and paste the following trust policy. Note that the
Principal
andExternalID
value will be added after you set up the Provider Integration in Confluent via Confluent Cloud APIs.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Principal": { "AWS": "*" }, "Action": "sts:AssumeRole" } ] }
Click Next to review the permissions, and then click Add permissions. Select the permission policy you created earlier for the S3 bucket.
In the Name, review, and create screen, enter a Role name and a Description.
Click Create role to save your new IAM role.
Note
Copy the ARN
of the IAM role you just created for use in the provider integration setup in Confluent.
Step 2: Create an IAM role mapping in Confluent¶
Use Register a provider integration API to set up a provider integration and create an IAM role mapping in Confluent.
Add
environment ID
andcustomer_iam_role_arn
in the POST API request.curl --request POST \ --url https://api.confluent.cloud/pim/v1/integrations \ --header 'Authorization: Basic <base64-encoded-key-and-secret>' \ --header 'content-type: application/json' \ --data '{ "display_name":"s3_provider_integration", "provider":"AWS", "config":{ "customer_iam_role_arn":"arn:aws:iam::000000000000:role/my-test-aws-role", "kind":"AwsIntegrationConfig" }, "environment":{ "id":"env-00000" } }'
Make a note of the
iam_role_arn
,external_id
andTagSession
generated in the output for each IAM role association. These configurations are used by Confluent when assuming the IAM role in your AWS account.
Note
Use Read a provider integration API to fetch Confluent IAM role configurations in your environment.
Step 3: Update trust policy in AWS¶
Follow the steps to update the trust policy with Confluent IAM role configurations in the AWS account. This allows the Confluent IAM role to assume the role in your AWS account.
Open the AWS console at https://console.aws.amazon.com/iam/
Navigate to Roles, and then open the IAM role you created in the Step 1.
In Trust relationships tab, click Edit trust policy and update the following configurations:
- Change
Effect
to Allow. - Add
Confluent IAM ARN
from Step 2. - Add
ExternalId
from Step 2. - Add
TagSession
from Step 2.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::058264372625:role/cspi-devc4j3k8" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "19fa38ea-43bb-4307-89e1-2072a694a8ec" } } }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::000000000000:role/my-test-aws-role" }, "Action": "sts:TagSession" } ] }
- Change
Click Update policy to update the trust policy.
Step 4: Create a connector with an IAM role in Confluent¶
Now that you’ve successfully configured a Confluent IAM role that can assume the role in your AWS account, you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.
Make a note of the following:
- You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
- Use Read a provider integration API to fetch
provider.integration.id
in your Confluent environment.
For example, to create a S3 sink connector, add authentication.method
and provider.integration.id
in
the Create a connector API request. Set authentication.method
as IAM Roles
in this case.
curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
"name":"Test-S3-Sink-Connector",
"config":{
"connector.class":"S3_SINK",
"name":"Test-S3-Sink-Connector",
"kafka.auth.mode":"KAFKA_API_KEY",
"kafka.api.key":"<my-kafka-api-key>",
"kafka.api.secret":"<my-kafka-api-secret>",
"s3.region":"<s3-region>",
"s3.bucket.name":"<my-iam-s3-bucket>",
"output.data.format":"JSON",
"time.interval":"DAILY",
"tasks.max":"1",
"topics": "<topic-1>, <topic-2>",
"authentication.method":"IAM Roles",
"provider.integration.id":"dlz-f3a90de"
},
}'
Step 5: Validate IAM assume role authorization in Confluent¶
Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.
- Sign in to your Confluent Cloud account.
- In the left navigation menu, click
Connectors
. Click the connector you just created in the Step 4. - View the messages produced to a topic, for example, from the AWS S3 bucket.
This validates the successful IAM role authorization through Provider Integration feature.
Step 6: Delete resources¶
When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.
- Sign in to your Confluent Cloud account.
- Click the connector you created and choose the Settings tab.
- Click Delete connector, enter the connector name, and click Confirm.
Use Delete a provider integration API to remove the provider integration in your environment. Successful completion delete an integration configuration.
Note
The DELETE request will fail if any Confluent workloads are using this provider integration.