Amazon Web Services Secrets Manager integration

This topic explains how to integrate AWS Secrets Manager with Confluent Cloud to retrieve authentication-related secrets for fully-managed connectors.

Prerequisites

Before you start integrating AWS Secrets Manager with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources:

• AWS Console at https://console.aws.amazon.com

• AWS Secrets Manager

• Confluent Cloud Console at https://confluent.cloud

Supported regions

For public Dedicated, secret manager integration is supported in all AWS regions except ap-southeast-5.

For private link Dedicated, secret manager integration is supported in all AWS regions.

PrivateLink setup for connectors using AWS Secrets Manager (Optional)

If you want to create connectors within an AWS PrivateLink Confluent Cloud cluster, see Egress PrivateLink Endpoints Setup Guide: First-Party Services on AWS for Confluent Cloud.

Ensure your cluster’s VPC can access the AWS Secrets Manager endpoint so that the connector has access.

Skip this step if you don’t want to run your connector within an AWS PrivateLink Confluent Cloud cluster.

Step 1: Set up AWS Secrets Manager

Create secrets in AWS Secrets Manager and add your authentication-related secrets such as database credentials, access keys, and similar sensitive information. For more information, see Create an AWS Secrets Manager secret and Retrieve AWS Secrets Manager secrets

Step 2: Configure an AWS Provider Integration to access AWS Secrets Manager

You must configure an AWS Provider Integration in Confluent Cloud to establish secure access to your AWS Secrets Manager before you create a connector that uses secret manager integration.

For detailed instructions, see Create an AWS Provider Integration.

Tip

Note the provider integration’s id. You have to use it as secret.manager.provider.integration.id in your connector configuration.

The Confluent Cloud Console displays the id under the provider integration’s name.

Or, use Read a provider integration API to fetch the provider integration’s id.

Attach the secretsmanager:GetSecretValue permission to the IAM role associated with your AWS provider integration to access your AWS Secrets Manager using the AWS console or AWS CLI.

Example IAM policy:

{
   "Version": "2012-10-17",
   "Statement": [
      {
      "Effect": "Allow",
      "Action": [
         "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret-*"
      }
   ]
}

Step 3: Use AWS Secrets Manager integration with a connector

Now that you have successfully integrated AWS Secrets Manager with Confluent Cloud, you can use it within a connector. For example, you can add a MySQL CDC Source V2 connector, and use AWS Secrets Manager to fetch sensitive data such as passwords for authentication.

Authenticate using secrets from AWS Secrets Manager

Cloud Console

At the Add Source/Sink connector screen, complete the following in the Authentication step:

1. Choose an authentication method that uses static credentials like Password and Access Key.

2. Toggle the Use secret manager setting on to let Confluent Cloud fetch credentials from AWS Secrets Manager.

3. Select AWS_SECRET_MANAGER from the Secret manager dropdown.

4. From the Configurations from Secret manager multi-select dropdown, select the connector fields that Confluent Cloud should resolve from AWS Secrets Manager. For each selected field, enter the secret identifier for that field instead of the literal credential value. For example:

   • For a plaintext secret, if you select database.password then paste the ARN into the database.password field as arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf.

   • For a JSON secret, if you select both database.user and database.password, then paste the corresponding ARN references into each field.

     • For database.user, paste the input as arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::user.

     • For database.password, paste the input as arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::password.

   For more information, see Configuration value format.

5. Select your provider integration name from the Provider integration dropdown.

6. Complete your connector’s configuration. For detailed documentation on supported connectors, see Supported connectors.

Confluent CLI

1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON:

   • authentication.method

   • secret.manager.enabled

   • secret.manager

   • secret.manager.managed.configs

   • secret.manager.provider.integration.id

   • The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the database.password and the database.user fields.

     For more information on the AWS Secrets Manager’s secret identifier format, see Configuration value format.

   {
      "name": "MySqlCdcSource1",
      "config": {
         "connector.class": "MySqlCdcSourceV2",
         "name": "MySqlCdcSource1",
         "kafka.auth.mode": "SERVICE_ACCOUNT",
         "kafka.service.account.id": "sa-xxxxxx",
         "authentication.method": "Password",
         "secret.manager.enabled": "true",
         "secret.manager": "AWS_SECRET_MANAGER",
         "secret.manager.managed.configs": "database.password,database.user",
         "secret.manager.provider.integration.id": "cspi-devxxxxxx",
         "database.hostname": "203.0.113.0",
         "database.port": "3306",
         "database.user": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::user",
         "database.password": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::password",
         "database.include.list": "db123",
         "database.ssl.mode": "preferred",
         "output.data.format": "JSON",
         "output.key.format": "JSON",
         "table.include.list": "table123",
         "topic.prefix": "mysqlcdc",
         "tasks.max": "1"
      }
   }
   

2. Enter the following command to load the JSON and start the connector:

   confluent connect cluster create --config-file <file-name>.json
   

   For example, if your filename is mysqlcdc_with_secrets_manager.json, then run:

   confluent connect cluster create --config-file mysqlcdc_with_secrets_manager.json
   

   Example output:

   Created connector MySqlCdcSource1 lcc-ix4dl
   

Confluent REST API

Include the following fields in the configuration payload of your Create a connector API request.

• authentication.method

• secret.manager.enabled

• secret.manager

• secret.manager.managed.configs

• secret.manager.provider.integration.id

• The secret-name whose value you want Confluent Cloud to fetch. In this example, it is database.password and database.user field values. For more information on the AWS Secrets Manager’s secret identifier format, see Configuration value format.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "MySqlCdcSource1",
   "config": {
      "connector.class": "MySqlCdcSourceV2",
      "name": "MySqlCdcSource1",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-xxxxxx",
      "authentication.method": "Password",
      "secret.manager.enabled": "true",
      "secret.manager": "AWS_SECRET_MANAGER",
      "secret.manager.managed.configs": "database.password,database.user",
      "secret.manager.provider.integration.id": "cspi-devxxxxxx",
      "database.hostname": "203.0.113.0",
      "database.port": "3306",
      "database.user": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::user",
      "database.password": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::password",
      "database.include.list": "db123",
      "database.ssl.mode": "preferred",
      "output.data.format": "JSON",
      "output.key.format": "JSON",
      "table.include.list": "table123",
      "topic.prefix": "mysqlcdc",
      "tasks.max": "1"
   }
}'

Verify AWS Secrets Manager authentication

After the connector is provisioned and running, verify the successful authorization through the credentials fetched from AWS Secrets Manager.

View topic messages

1. Sign in to your Confluent Cloud account.

2. In the left navigation menu, click Connectors.

3. Click the connector you just created.

4. View the messages produced to or received from a topic.

View provider integration resources

1. Go to the Environments page at https://confluent.cloud/environments and select the environment in which you created your provider integration.

2. Click the Provider integrations tab.

3. Under the Resources column, view the resources.

4. Click the resource link to view the associated connector that you created.

Configuration value format

For the connector configurations that are managed through AWS Secrets Manager, you must specify secret identifiers as configuration values according to the following specified formats:

1. If your secret is stored as plaintext (non-JSON format), specify the configuration value as the secret’s full ARN.

   For example, to have Confluent Cloud fetch the database password, specify arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf.

2. If your secrets are stored as a JSON object within the AWS Secrets Manager secret, then you must specify each configuration value as <SECRET_ARN>::<config-key>.

   For example, if the secret at ARN arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf stores a JSON that looks like:

   {
      "user": "dbadmin",
      "password": "$trongPassw0rdExamp1e"
   }
   

   • To fetch the database user, specify the connector configuration value as arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::user.

   • To fetch the database password, specify the connector configuration value as arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::password.

Note

Using only the secret name (for example, mysql-db-secret) is not supported. You must provide the full ARN.

Troubleshooting

Connector fails to start with authentication error

• Verify your AWS Secrets Manager secret ARN is correct.

• Ensure the IAM role associated with your provider integration has the required secretsmanager:GetSecretValue permission on your AWS Secrets Manager.

• Check that the secret exists in your AWS Secrets Manager and contains a valid value.

• Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see Configuration value format.