Quick Start for Confluent Cloud Provider Integration¶
Confluent Cloud Provider Integration offers Identity and Access Management (IAM) Role-based authorization that lets you adopt the temporary security credentials of an IAM role, which acts as a set of permission policies. Trusted entities, such as IAM users, applications, or cloud services can assume this role. Using this approach, you can create a secure access connection between source or sink resources on AWS and Confluent Cloud for data ingestion or transfer.
Note
This feature is currently available in Early Access for evaluation of AWS IAM roles. To provide feedback, contact Confluent Support.
This quick start walks you through the following steps to get you up and running with IAM role-based authorization using Confluent Provider Integration feature.
- Step 1: Create an IAM policy and role in AWS
- Step 2: Create an IAM role mapping in Confluent
- Step 3: Update trust policy in AWS
- Step 4: Create a connector with an IAM role in Confluent
- Step 5: Validate IAM assume role authorization in Confluent
- Prerequisites
- Access to AWS Management Console.
- Access to Confluent Cloud.
- The OrganizationAdmin or EnvironmentAdmin role to setup provider integration. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin.
- In addition to the above role, the ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role can create a connector using the provider integration resource.
If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin. Use the following example CRN
"crn_pattern": "crn://confluent.cloud/organization=org-ID/environment=env-ID/provider-integration=cspi-123j1"
to identify the provider integration resources. - An environment ID for your organization.
- A Confluent Cloud API key to authenticate with the Provider Integration API. For information about how to create a Confluent Cloud API key, see Manage API Keys.
- cURL and jq installed to use the API request examples in this document.
The quick start workflows assume you already have a working Confluent Cloud environment and an AWS account with an S3 bucket or a DynamoDB table created for IAM roles.
Step 1: Create an IAM policy and role in AWS¶
Follow the instructions to create an IAM policy and role in your AWS account.
Open the AWS console at https://console.aws.amazon.com/iam/.
Navigate to Policies, and then click Create policy.
In the Policy editor screen, select the JSON option.
Copy and paste the S3 or DynamoDB permission policy document in the editor screen.
Click Next to review the permissions.
In the Review and create page, enter a Policy name and a Description.
Click Create policy to save your new IAM policy.
Note
For more information about AWS IAM Policies, see Create AWS IAM Policies.
Now you can get started creating IAM roles in your AWS account.
Navigate to Roles, and then click Create role.
In the Trusted entity type screen, select Custom trust policy.
In the Custom trust policy editor, copy and paste the following trust policy. Note that the
Principal
andExternalID
value will be added after you set up the Provider Integration in Confluent via Confluent APIs.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Principal": { "AWS": "*" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "dummy-externalId" } } } ] }
Click Next to review the permissions, and then click Add permissions. Select the permission policy you created earlier for the S3 bucket.
In the Name, review, and create screen, enter a Role name and a Description.
Click Create role to save your new IAM role.
Note
Copy the ARN
of the IAM role you just created for use in the provider integration setup in Confluent.
Step 2: Create an IAM role mapping in Confluent¶
Use Register a provider integration API to setup a provider integration and create an IAM role mapping in Confluent.
Add
environment ID
andcustomer_iam_role_arn
in the the POST API request.curl --request POST \ --url https://api.confluent.cloud/pim/v1/integrations \ --header 'Authorization: Basic <base64-encoded-key-and-secret>' \ --header 'content-type: application/json' \ --data '{ "display_name":"s3_provider_integration", "provider":"AWS", "config":{ "customer_iam_role_arn":"arn:aws:iam::000000000000:role/my-test-aws-role", "kind":"AwsIntegrationConfig" }, "environment":{ "id":"env-00000" } }'
Make a note of the
iam_role_arn
andexternal_id
generated in the output for each IAM role association. These configurations are used by Confluent when assuming the IAM role in your AWS account.
Note
Use Read a provider integration API to fetch Confluent IAM role configurations in your environment.
Step 3: Update trust policy in AWS¶
Follow the steps to update the trust policy with Confluent IAM role configurations in the AWS account. This allows the Confluent IAM role to assume the role in your AWS account.
Open the AWS console at https://console.aws.amazon.com/iam/
Navigate to Roles, and then open the IAM role you created in the Step 1.
In Trust relationships tab, click Edit trust policy and update the following configurations:
- Change
Effect
to Allow. - Under
Principal
, add Confluent IAM ARN from Step 2. - Under
Condition
, add theExternalId
from Step 2.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::000000000000:role/my-test-aws-role" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "95c35493-41aa-44f8-9154-5a25cbbc1865" } } } ] }
- Change
Click Update policy to update the trust policy.
Step 4: Create a connector with an IAM role in Confluent¶
Now that you’ve successfully configured a Confluent IAM role that can assume the role in your AWS account, you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.
Make a note of the following:
- You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
- Use Read a provider integration API to fetch
provider.integration.id
in your Confluent environment.
For example, to create a S3 sink connector, add authentication.method
and provider.integration.id
in
the Create a connector API request. Set authentication.method
as IAM Roles
in this case.
curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
"name":"Test-S3-Sink-Connector",
"config":{
"connector.class":"S3_SINK",
"name":"S3_SINKConnector_Test",
"kafka.auth.mode":"KAFKA_API_KEY",
"kafka.api.key":"<my-kafka-api-key>",
"kafka.api.secret":"<my-kafka-api-secret>",
"s3.region":"<s3-region>",
"s3.bucket.name":"<my-iam-s3-bucket>",
"output.data.format":"JSON",
"time.interval":"DAILY",
"tasks.max":"1",
"authentication.method":"IAM Roles",
"provider.integration.id":"dlz-f3a90de"
},
}'
Step 5: Validate IAM assume role authorization in Confluent¶
Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.
- Sign in to your Confluent Cloud account.
- In the left navigation menu, click
Connectors
. Click the connector you just created in the Step 4. - View the messages produced to a topic, for example, from the AWS S3 bucket.
This validates the successful IAM role authorization through Provider Integration feature.
When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.
- Sign in to your Confluent Cloud account.
- Click the connector you created and choose the Settings tab.
- Click Delete connector, enter the connector name, and click Confirm.
Use Delete a provider integration API to remove the provider integration in your environment. Successful completion delete an integration configuration.
Note
The DELETE request will fail if any Confluent workloads are using this provider integration.