Quick Start for Confluent Cloud Provider Integration

Confluent Cloud Provider Integration offers Identity and Access Management (IAM) Role-based authorization that lets you adopt the temporary security credentials of an IAM role, which acts as a set of permission policies. Trusted entities, such as IAM users, applications, or cloud services can assume this role. Using this approach, you can create a secure access connection between source or sink resources on AWS and Confluent Cloud for data ingestion or transfer.

Note

This feature is currently available in Early Access for evaluation of AWS IAM roles. To provide feedback, contact Confluent Support.

This quick start walks you through the following steps to get you up and running with IAM role-based authorization using Confluent Provider Integration feature.

Prerequisites
  • Access to AWS Management Console.
  • Access to Confluent Cloud.
  • The OrganizationAdmin or EnvironmentAdmin role to setup provider integration. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin.
  • In addition to the above role, the ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role can create a connector using the provider integration resource. If you do not have the appropriate role, reach out to your OrganizationAdmin or EnvironmentAdmin. Use the following example CRN "crn_pattern": "crn://confluent.cloud/organization=org-ID/environment=env-ID/provider-integration=cspi-123j1" to identify the provider integration resources.
  • An environment ID for your organization.
  • A Confluent Cloud API key to authenticate with the Provider Integration API. For information about how to create a Confluent Cloud API key, see Manage API Keys.
  • cURL and jq installed to use the API request examples in this document.

The quick start workflows assume you already have a working Confluent Cloud environment and an AWS account with an S3 bucket or a DynamoDB table created for IAM roles.

Step 1: Create an IAM policy and role in AWS

Follow the instructions to create an IAM policy and role in your AWS account.

  1. Open the AWS console at https://console.aws.amazon.com/iam/.

  2. Navigate to Policies, and then click Create policy.

  3. In the Policy editor screen, select the JSON option.

  4. Copy and paste the S3 or DynamoDB permission policy document in the editor screen.

  5. Click Next to review the permissions.

  6. In the Review and create page, enter a Policy name and a Description.

  7. Click Create policy to save your new IAM policy.

    Note

    For more information about AWS IAM Policies, see Create AWS IAM Policies.

Now you can get started creating IAM roles in your AWS account.

Step 2: Create an IAM role mapping in Confluent

Use Register a provider integration API to setup a provider integration and create an IAM role mapping in Confluent.

  1. Add environment ID and customer_iam_role_arn in the the POST API request.

    curl --request POST \
 --url https://api.confluent.cloud/pim/v1/integrations \
 --header 'Authorization: Basic <base64-encoded-key-and-secret>' \
 --header 'content-type: application/json' \
 --data '{
   "display_name":"s3_provider_integration",
   "provider":"AWS",
   "config":{
         "customer_iam_role_arn":"arn:aws:iam::000000000000:role/my-test-aws-role",
         "kind":"AwsIntegrationConfig"
     },
   "environment":{
   "id":"env-00000"
  }
}'

  2. Make a note of the iam_role_arn and external_id generated in the output for each IAM role association. These configurations are used by Confluent when assuming the IAM role in your AWS account.

Note

Use Read a provider integration API to fetch Confluent IAM role configurations in your environment.

Step 3: Update trust policy in AWS

Follow the steps to update the trust policy with Confluent IAM role configurations in the AWS account. This allows the Confluent IAM role to assume the role in your AWS account.

  1. Open the AWS console at https://console.aws.amazon.com/iam/

  2. Navigate to Roles, and then open the IAM role you created in the Step 1.

  3. In Trust relationships tab, click Edit trust policy and update the following configurations:

    • Change Effect to Allow.
    • Under Principal, add Confluent IAM ARN from Step 2.
    • Under Condition, add the ExternalId from Step 2.
    {
     "Version": "2012-10-17",
     "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::000000000000:role/my-test-aws-role"
         },
      "Action": "sts:AssumeRole",
      "Condition": {
          "StringEquals": {
              "sts:ExternalId": "95c35493-41aa-44f8-9154-5a25cbbc1865"
            }
         }
      }
    ]
}

  4. Click Update policy to update the trust policy.

Step 4: Create a connector with an IAM role in Confluent

Now that you’ve successfully configured a Confluent IAM role that can assume the role in your AWS account, you can now create a connector, for example, a S3 sink connector or a DynamoDB source connector, with the IAM role.

Make a note of the following:

  • You must have a ProviderIntegrationResourceOwner or ProviderIntegrationAssigner role to create a connector using the provider integration resource.
  • Use Read a provider integration API to fetch provider.integration.id in your Confluent environment.

For example, to create a S3 sink connector, add authentication.method and provider.integration.id in the Create a connector API request. Set authentication.method as IAM Roles in this case.

curl --request POST \
 --url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
 --header 'Authorization: Basic <base64-encoded-key-and-secret>' \
 --header 'content-type: application/json' \
 --data '{
    "name":"Test-S3-Sink-Connector",
    "config":{
    "connector.class":"S3_SINK",
    "name":"S3_SINKConnector_Test",
    "kafka.auth.mode":"KAFKA_API_KEY",
    "kafka.api.key":"<my-kafka-api-key>",
    "kafka.api.secret":"<my-kafka-api-secret>",
    "s3.region":"<s3-region>",
    "s3.bucket.name":"<my-iam-s3-bucket>",
    "output.data.format":"JSON",
    "time.interval":"DAILY",
    "tasks.max":"1",
    "authentication.method":"IAM Roles",
    "provider.integration.id":"dlz-f3a90de"
    },
 }'

Step 5: Validate IAM assume role authorization in Confluent

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

  1. Sign in to your Confluent Cloud account.
  2. In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.
  3. View the messages produced to a topic, for example, from the AWS S3 bucket.

This validates the successful IAM role authorization through Provider Integration feature.

When you are finished with the Quick Start, delete the resources you created to avoid unexpected charges to your account.

  1. Sign in to your Confluent Cloud account.
  2. Click the connector you created and choose the Settings tab.
  3. Click Delete connector, enter the connector name, and click Confirm.

Related content