# Google Cloud Secret Manager integration Integrate Google Cloud Secret Manager with Confluent Cloud to let fully managed connectors retrieve authentication credentials such as database passwords and access keys from Google Cloud Secret Manager at runtime, instead of storing those values directly in the connector configuration. ## Prerequisites Before you start integrating Google Cloud Secret Manager with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources: - Google Cloud console at [https://console.cloud.google.com](https://console.cloud.google.com) - Google Cloud Secret Manager - Confluent Cloud Console at [https://confluent.cloud](https://confluent.cloud) ## Supported regions For Dedicated, secret manager integration is supported in all Google Cloud regions. ## Private networking setup for connectors using Google Cloud Secret Manager (Optional) On Google Cloud, Confluent Cloud reaches Google Cloud Secret Manager over the `*.googleapis.com` endpoints. How those endpoints resolve depends on your cluster’s network type. The required setup differs from AWS and Azure, where you create one egress endpoint per service or per resource. - **VPC Peering clusters**: No setup is required. - **Private Service Connect (PSC) clusters**: Create an egress Private Service Connect endpoint that targets the `all-google-apis` bundle, and add a DNS record for `googleapis.com`. This single bundle fronts every `*.googleapis.com` endpoint, so one endpoint covers Google Cloud Secret Manager along with services such as BigQuery, Cloud Storage, and Pub/Sub. If you already created this endpoint for another Google Cloud connector, Google Cloud Secret Manager routing works without any extra setup. For instructions, see [Use Google Cloud Egress Private Service Connect Endpoints for Dedicated Clusters on Confluent Cloud](../../networking/gcp-egress-psc.md#cloud-networking-gcp-psc-egress) and [Egress Private Service Connect Endpoints Setup Guide: First-Party Services on Google Cloud for Confluent Cloud](../networking/gcp-eap-1st-party.md#cc-gcp-eap-1st-party). If your cluster does not use private networking, skip this step. ## Step 1: Set up Google Cloud Secret Manager Create secrets in Google Cloud Secret Manager and add your authentication-related secrets such as database credentials, access keys, and similar sensitive information. For more information, see [Create a secret](https://cloud.google.com/secret-manager/docs/create-secret) and [Access a secret version](https://cloud.google.com/secret-manager/docs/access-secret-version) in the Google Cloud documentation. ## Step 2: Configure a Google Cloud Provider Integration to access Google Cloud Secret Manager You must configure a Google Cloud Provider Integration in Confluent Cloud to establish secure access to your Google Cloud Secret Manager before you create a connector that uses secret manager integration. On Google Cloud, the provider integration uses Workload Identity Federation and service account impersonation, so Confluent Cloud accesses your secrets without any long-lived static credentials. For detailed instructions, see [Create a Google Cloud Provider Integration in Confluent Cloud](../../integrations/provider-integrations/create-provider-integration-gc.md#create-provider-integration-gc). Grant the **Secret Manager Secret Accessor** role (`roles/secretmanager.secretAccessor`) on your target secret to your Google Cloud service account associated with the provider integration. This role includes the `secretmanager.versions.access` permission that Confluent Cloud requires to read the secret. You can grant this role using the Google Cloud console or the `gcloud` CLI. For instructions, see [Manage access to secrets](https://cloud.google.com/secret-manager/docs/manage-access-to-secrets) in the Google Cloud documentation. (Optional) Use the **Validate** option on the provider integration to verify that trust is established between Confluent Cloud and your Google Cloud account. ![Validate option at the end of a Google Cloud provider integration row in the Confluent Cloud Console](connectors/secret-manager-integration/images/gcp-cspi-validate-button.png) #### NOTE On Google Cloud, policy updates take a couple of minutes to propagate across the infrastructure. If you select **Validate** immediately after you grant the role, validation can fail with an error similar to the one in the screenshot below. Wait a few minutes and validate again. ![Provider integration validation error in Confluent Cloud reporting that the required Google Cloud IAM role has not yet propagated](connectors/secret-manager-integration/images/gcp-provider-integration-validate-error.png) ## Step 3: Use Google Cloud Secret Manager integration with a connector Now that you have successfully integrated Google Cloud Secret Manager with Confluent Cloud, you can use it within a connector. For example, you can add a PostgreSQL CDC Source V2 connector, and use Google Cloud Secret Manager to fetch sensitive data such as passwords for authentication. ### Authenticate using secrets from Google Cloud Secret Manager ### Cloud Console At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step: 1. Choose an authentication method that uses static credentials, such as `Password`. 2. Toggle the **Use secret manager** setting on to let Confluent Cloud fetch credentials from Google Cloud Secret Manager. 3. Select **GCP_SECRET_MANAGER** from the **Secret manager** dropdown. 4. From the **Configurations from Secret manager** multi-select dropdown, select the connector fields that Confluent Cloud should resolve from Google Cloud Secret Manager. For each selected field, enter the secret identifier for that field instead of the literal credential value. For example: - For a plaintext secret, if you select `database.password` then paste the secret resource name into the `database.password` field as `projects//secrets/postgres-db-secret`. - For a JSON secret, if you select both `database.user` and `database.password`, then paste the corresponding references into each field. - For `database.user`, paste the input as `projects//secrets/postgres-db-secret::user`. - For `database.password`, paste the input as `projects//secrets/postgres-db-secret::password`. For more information, see [Configuration value format](#gc-secret-manager-secret-format). 5. Select your provider integration name from the **Provider integration** dropdown. 6. Complete your connector’s configuration. For detailed documentation on supported connectors, see [Supported connectors](overview.md#secret-manager-supported-connectors). ### Confluent CLI 1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON: - The authentication type that the connector supports. For the exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the `database.password` and the `database.user` fields. For more information on the Google Cloud Secret Manager’s secret identifier format, see [Configuration value format](#gc-secret-manager-secret-format). ```json { "name": "PostgresCdcSourceV2_1", "config": { "connector.class": "PostgresCdcSourceV2", "name": "PostgresCdcSourceV2_1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "GCP_SECRET_MANAGER", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "5432", "database.user": "projects/123456789012/secrets/postgres-db-secret::user", "database.password": "projects/123456789012/secrets/postgres-db-secret::password", "database.dbname": "postgres", "database.sslmode": "prefer", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "public.table123", "topic.prefix": "postgrescdc", "tasks.max": "1" } } ``` 2. Enter the following command to load the JSON and start the connector: ```bash confluent connect cluster create --config-file .json ``` For example, if your filename is `postgrescdc_with_secret_manager.json`, then run: ```bash confluent connect cluster create --config-file postgrescdc_with_secret_manager.json ``` Example output: ```bash Created connector PostgresCdcSourceV2_1 lcc-ix4dl ``` ### Confluent REST API Include the following fields in the configuration payload of your [Create a connector](../connect-api-section.md#ccloud-connect-api-create-connector) API request. - The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The secret identifier whose value you want Confluent Cloud to fetch. In this example, it is the `database.password` and `database.user` field values. For more information on the Google Cloud Secret Manager’s secret identifier format, see [Configuration value format](#gc-secret-manager-secret-format). ```bash curl --request POST \ --url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \ --header 'Authorization: Basic ' \ --header 'content-type: application/json' \ --data '{ "name": "PostgresCdcSourceV2_1", "config": { "connector.class": "PostgresCdcSourceV2", "name": "PostgresCdcSourceV2_1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "GCP_SECRET_MANAGER", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "5432", "database.user": "projects/123456789012/secrets/postgres-db-secret::user", "database.password": "projects/123456789012/secrets/postgres-db-secret::password", "database.dbname": "postgres", "database.sslmode": "prefer", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "public.table123", "topic.prefix": "postgrescdc", "tasks.max": "1" } }' ``` ### Verify Google Cloud Secret Manager authentication After the connector is provisioned and running, verify the successful authorization through the credentials fetched from Google Cloud Secret Manager. ### View topic messages 1. Sign in to your Confluent Cloud account. 2. In the left navigation menu, click **Connectors**. 3. Click the connector you just created. 4. View the messages produced to or received from a topic. ### View provider integration resources 1. Go to the **Environments** page at [https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which you created your provider integration. 2. Click the **Provider integrations** tab. 3. Under the **Resources** column, view the resources. 4. Click the resource link to view the associated connector that you created. ### Configuration value format For the connector configurations that you manage through Google Cloud Secret Manager, you must specify secret identifiers as configuration values in one of the following formats: 1. If your secret is stored as plaintext (non-JSON format), specify the configuration value as the secret’s resource name in the format `projects//secrets/`. For example, to have Confluent Cloud fetch the database password, specify `projects/123456789012/secrets/postgres-db-secret`. To pin a specific secret version, append `/versions/`, for example `projects/123456789012/secrets/postgres-db-secret/versions/3`. If you omit the version, Confluent Cloud reads the `latest` version. 2. If your secrets are stored as a JSON object within the Google Cloud Secret Manager secret, then you must specify each configuration value as `projects//secrets/::`. For example, if the secret `projects/123456789012/secrets/postgres-db-secret` stores a JSON that looks like: ```json { "user": "dbadmin", "password": "$trongPassw0rdExamp1e" } ``` - To fetch the database user, specify the connector configuration value as `projects/123456789012/secrets/postgres-db-secret::user`. - To fetch the database password, specify the connector configuration value as `projects/123456789012/secrets/postgres-db-secret::password`. You can also pin a version when using a JSON key, for example `projects/123456789012/secrets/postgres-db-secret/versions/3::password`. #### NOTE Use your Google Cloud project number for the `` segment. The config key (``) must not contain the characters `$`, `{`, `}`, or `:`. ## Troubleshooting ### Connector fails to start with authentication error - Verify your Google Cloud Secret Manager secret resource name is correct and follows the format `projects//secrets/`. - Ensure your Google Cloud service account associated with the provider integration has the **Secret Manager Secret Accessor** role (`roles/secretmanager.secretAccessor`), which grants the required `secretmanager.versions.access` permission on your secret. - Check that the secret exists in your Google Cloud Secret Manager and contains a valid value. - Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see [Configuration value format](#gc-secret-manager-secret-format). - Verify that the provider integration trust is established. On the provider integration, use the **Validate** option to confirm that Confluent Cloud can access your Google Cloud account. Because Google Cloud takes time to sync policy updates, validation can take a few minutes to succeed after you grant the role.