# Amazon Web Services Secrets Manager integration Integrate AWS Secrets Manager with Confluent Cloud to let fully managed connectors retrieve authentication credentials such as database passwords and access keys from AWS Secrets Manager at runtime, instead of storing those values directly in the connector configuration. ## Prerequisites Before you start integrating AWS Secrets Manager with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources: - AWS Console at [https://console.aws.amazon.com](https://console.aws.amazon.com) - AWS Secrets Manager - Confluent Cloud Console at [https://confluent.cloud](https://confluent.cloud) ## Supported regions For Dedicated, secret manager integration is supported in all AWS regions. ## PrivateLink setup for connectors using AWS Secrets Manager (Optional) AWS Secrets Manager integration supports VPC Peering and PrivateLink. If your cluster uses VPC Peering, no extra networking setup is needed. If you want to create connectors within an AWS PrivateLink Confluent Cloud cluster, see [Egress PrivateLink Endpoints Setup Guide: First-Party Services on AWS for Confluent Cloud](../networking/aws-eap-1st-party.md#cc-aws-eap-1st-party). Ensure your cluster’s VPC can access the AWS Secrets Manager endpoint so that the connector has access. If you don’t want to run your connector within an AWS PrivateLink Confluent Cloud cluster, skip this step. ## Step 1: Set up AWS Secrets Manager Create secrets in AWS Secrets Manager and add your authentication-related secrets such as database credentials, access keys, and similar sensitive information. For more information, see [Create an AWS Secrets Manager secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) and [Retrieve AWS Secrets Manager secrets](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) ## Step 2: Configure an AWS Provider Integration to access AWS Secrets Manager You must configure an AWS Provider Integration in Confluent Cloud to establish secure access to your AWS Secrets Manager before you create a connector that uses secret manager integration. For detailed instructions, see [Create an AWS Provider Integration](../../integrations/provider-integrations/create-provider-integration-aws.md#create-provider-integration-aws-steps). Attach the `secretsmanager:GetSecretValue` permission to the IAM role associated with your AWS provider integration to access your AWS Secrets Manager using the AWS console or AWS CLI. Example IAM policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret-*" } ] } ``` ## Step 3: Use AWS Secrets Manager integration with a connector Now that you have successfully integrated AWS Secrets Manager with Confluent Cloud, you can use it within a connector. For example, you can add a MySQL CDC Source V2 connector, and use AWS Secrets Manager to fetch sensitive data such as passwords for authentication. ### Authenticate using secrets from AWS Secrets Manager ### Cloud Console At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step: 1. Choose an authentication method that uses static credentials like `Password` and `Access Key`. 2. Toggle the **Use secret manager** setting on to let Confluent Cloud fetch credentials from AWS Secrets Manager. 3. Select **AWS_SECRET_MANAGER** from the **Secret manager** dropdown. 4. From the **Configurations from Secret manager** multi-select dropdown, select the connector fields that Confluent Cloud should resolve from AWS Secrets Manager. For each selected field, enter the secret identifier for that field instead of the literal credential value. For example: - For a plaintext secret, if you select `database.password` then paste the ARN into the `database.password` field as `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf`. - For a JSON secret, if you select both `database.user` and `database.password`, then paste the corresponding ARN references into each field. - For `database.user`, paste the input as `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::user`. - For `database.password`, paste the input as `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::password`. For more information, see [Configuration value format](#aws-secrets-manager-secret-format). 5. Select your provider integration name from the **Provider integration** dropdown. 6. Complete your connector’s configuration. For detailed documentation on supported connectors, see [Supported connectors](overview.md#secret-manager-supported-connectors). ### Confluent CLI 1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON: - The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the `database.password` and the `database.user` fields. For more information on the AWS Secrets Manager’s secret identifier format, see [Configuration value format](#aws-secrets-manager-secret-format). ```json { "name": "MySqlCdcSource1", "config": { "connector.class": "MySqlCdcSourceV2", "name": "MySqlCdcSource1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "AWS_SECRET_MANAGER", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "3306", "database.user": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::user", "database.password": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::password", "database.include.list": "db123", "database.ssl.mode": "preferred", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "table123", "topic.prefix": "mysqlcdc", "tasks.max": "1" } } ``` 2. Enter the following command to load the JSON and start the connector: ```bash confluent connect cluster create --config-file .json ``` For example, if your filename is `mysqlcdc_with_secrets_manager.json`, then run: ```bash confluent connect cluster create --config-file mysqlcdc_with_secrets_manager.json ``` Example output: ```bash Created connector MySqlCdcSource1 lcc-ix4dl ``` ### Confluent REST API Include the following fields in the configuration payload of your [Create a connector](../connect-api-section.md#ccloud-connect-api-create-connector) API request. - The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The secret-name whose value you want Confluent Cloud to fetch. In this example, it is `database.password` and `database.user` field values. For more information on the AWS Secrets Manager’s secret identifier format, see [Configuration value format](#aws-secrets-manager-secret-format). ```bash curl --request POST \ --url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \ --header 'Authorization: Basic ' \ --header 'content-type: application/json' \ --data '{ "name": "MySqlCdcSource1", "config": { "connector.class": "MySqlCdcSourceV2", "name": "MySqlCdcSource1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "AWS_SECRET_MANAGER", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "3306", "database.user": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::user", "database.password": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret::password", "database.include.list": "db123", "database.ssl.mode": "preferred", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "table123", "topic.prefix": "mysqlcdc", "tasks.max": "1" } }' ``` ### Verify AWS Secrets Manager authentication After the connector is provisioned and running, verify the successful authorization through the credentials fetched from AWS Secrets Manager. ### View topic messages 1. Sign in to your Confluent Cloud account. 2. In the left navigation menu, click **Connectors**. 3. Click the connector you just created. 4. View the messages produced to or received from a topic. ### View provider integration resources 1. Go to the **Environments** page at [https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which you created your provider integration. 2. Click the **Provider integrations** tab. 3. Under the **Resources** column, view the resources. 4. Click the resource link to view the associated connector that you created. ### Configuration value format For the connector configurations that are managed through AWS Secrets Manager, you must specify secret identifiers as configuration values in one of the following formats: 1. If your secret is stored as plaintext (non-JSON format), specify the configuration value as the secret’s full ARN. For example, to have Confluent Cloud fetch the database password, specify `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf`. 2. If your secrets are stored as a JSON object within the AWS Secrets Manager secret, then you must specify each configuration value as `::`. For example, if the secret at ARN `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf` stores a JSON that looks like: ```json { "user": "dbadmin", "password": "$trongPassw0rdExamp1e" } ``` - To fetch the database user, specify the connector configuration value as `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::user`. - To fetch the database password, specify the connector configuration value as `arn:aws:secretsmanager:us-east-1:123456789012:secret:mysql-db-secret-AbCdEf::password`. #### NOTE Using only the secret name (for example, `mysql-db-secret`) is not supported. You must provide the full ARN. ## Troubleshooting **Connector fails to start with authentication error** - Verify your AWS Secrets Manager secret ARN is correct. - Ensure the IAM role associated with your provider integration has the required `secretsmanager:GetSecretValue` permission on your AWS Secrets Manager. - Check that the secret exists in your AWS Secrets Manager and contains a valid value. - Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see [Configuration value format](#aws-secrets-manager-secret-format).