Google Cloud Secret Manager integration

Integrate Google Cloud Secret Manager with Confluent Cloud to let fully-managed connectors retrieve authentication credentials such as database passwords and access keys from Google Cloud Secret Manager at runtime, instead of storing those values directly in the connector configuration.

Prerequisites

Before you start integrating Google Cloud Secret Manager with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources:

• Google Cloud console at https://console.cloud.google.com

• Google Cloud Secret Manager

• Confluent Cloud Console at https://confluent.cloud

Supported regions

For Dedicated, secret manager integration is supported in all Google Cloud regions.

Private networking setup for connectors using Google Cloud Secret Manager (Optional)

On Google Cloud, Confluent Cloud reaches Google Cloud Secret Manager over the *.googleapis.com endpoints. How those endpoints resolve depends on your cluster’s network type. The required setup differs from AWS and Azure, where you create one egress endpoint per service or per resource.

• VPC Peering clusters: No setup is required.

• Private Service Connect (PSC) clusters: Create an egress Private Service Connect endpoint that targets the all-google-apis bundle, and add a DNS record for googleapis.com. This single bundle fronts every *.googleapis.com endpoint, so one endpoint covers Google Cloud Secret Manager along with services such as BigQuery, Cloud Storage, and Pub/Sub. If you already created this endpoint for another Google Cloud connector, Google Cloud Secret Manager routing works without any extra setup.

  For instructions, see Use Google Cloud Egress Private Service Connect Endpoints for Dedicated Clusters on Confluent Cloud and Egress Private Service Connect Endpoints Setup Guide: First-Party Services on Google Cloud for Confluent Cloud.

If your cluster does not use private networking, skip this step.

Step 1: Set up Google Cloud Secret Manager

Create secrets in Google Cloud Secret Manager and add your authentication-related secrets such as database credentials, access keys, and similar sensitive information. For more information, see Create a secret and Access a secret version in the Google Cloud documentation.

Step 2: Configure a Google Cloud Provider Integration to access Google Cloud Secret Manager

You must configure a Google Cloud Provider Integration in Confluent Cloud to establish secure access to your Google Cloud Secret Manager before you create a connector that uses secret manager integration. On Google Cloud, the provider integration uses Workload Identity Federation and service account impersonation, so Confluent Cloud accesses your secrets without any long-lived static credentials.

For detailed instructions, see Create a Google Cloud Provider Integration in Confluent Cloud.

Tip

Note the provider integration’s id. You have to use it as secret.manager.provider.integration.id in your connector configuration.

The Confluent Cloud Console displays the id under the provider integration’s name.

Alternatively, use the Read a provider integration API to fetch the provider integration’s id.

Grant the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor) on your target secret to your Google Cloud service account associated with the provider integration. This role includes the secretmanager.versions.access permission that Confluent Cloud requires to read the secret. You can grant this role using the Google Cloud console or the gcloud CLI. For instructions, see Manage access to secrets in the Google Cloud documentation.

(Optional) Use the Validate option on the provider integration to verify that trust is established between Confluent Cloud and your Google Cloud account.

Note

On Google Cloud, policy updates take a couple of minutes to propagate across the infrastructure. If you select Validate immediately after you grant the role, validation can fail with an error similar to the one in the screenshot below. Wait a few minutes and validate again.

Step 3: Use Google Cloud Secret Manager integration with a connector

Now that you have successfully integrated Google Cloud Secret Manager with Confluent Cloud, you can use it within a connector. For example, you can add a PostgreSQL CDC Source V2 connector, and use Google Cloud Secret Manager to fetch sensitive data such as passwords for authentication.

Authenticate using secrets from Google Cloud Secret Manager

Cloud Console

At the Add Source/Sink connector screen, complete the following in the Authentication step:

1. Choose an authentication method that uses static credentials, such as Password.

2. Toggle the Use secret manager setting on to let Confluent Cloud fetch credentials from Google Cloud Secret Manager.

3. Select GCP_SECRET_MANAGER from the Secret manager dropdown.

4. From the Configurations from Secret manager multi-select dropdown, select the connector fields that Confluent Cloud should resolve from Google Cloud Secret Manager. For each selected field, enter the secret identifier for that field instead of the literal credential value. For example:

   • For a plaintext secret, if you select database.password then paste the secret resource name into the database.password field as projects/<project-number>/secrets/postgres-db-secret.

   • For a JSON secret, if you select both database.user and database.password, then paste the corresponding references into each field.

     • For database.user, paste the input as projects/<project-number>/secrets/postgres-db-secret::user.

     • For database.password, paste the input as projects/<project-number>/secrets/postgres-db-secret::password.

   For more information, see Configuration value format.

5. Select your provider integration name from the Provider integration dropdown.

6. Complete your connector’s configuration. For detailed documentation on supported connectors, see Supported connectors.

Confluent CLI

1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON:

   • The authentication type that the connector supports. For the exact authentication type, see the connector’s documentation. For example, it’s authentication.method for Postgres CDC Source V2 connector and mongodb.auth.mechanism for MongoDB Atlas Source connector.

   • secret.manager.enabled

   • secret.manager

   • secret.manager.managed.configs

   • secret.manager.provider.integration.id

   • The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the database.password and the database.user fields.

     For more information on the Google Cloud Secret Manager’s secret identifier format, see Configuration value format.

   {
      "name": "PostgresCdcSourceV2_1",
      "config": {
         "connector.class": "PostgresCdcSourceV2",
         "name": "PostgresCdcSourceV2_1",
         "kafka.auth.mode": "SERVICE_ACCOUNT",
         "kafka.service.account.id": "sa-xxxxxx",
         "authentication.method": "Password",
         "secret.manager.enabled": "true",
         "secret.manager": "GCP_SECRET_MANAGER",
         "secret.manager.managed.configs": "database.password,database.user",
         "secret.manager.provider.integration.id": "cspi-devxxxxxx",
         "database.hostname": "203.0.113.0",
         "database.port": "5432",
         "database.user": "projects/123456789012/secrets/postgres-db-secret::user",
         "database.password": "projects/123456789012/secrets/postgres-db-secret::password",
         "database.dbname": "postgres",
         "database.sslmode": "prefer",
         "output.data.format": "JSON",
         "output.key.format": "JSON",
         "table.include.list": "public.table123",
         "topic.prefix": "postgrescdc",
         "tasks.max": "1"
      }
   }
   

2. Enter the following command to load the JSON and start the connector:

   confluent connect cluster create --config-file <file-name>.json
   

   For example, if your filename is postgrescdc_with_secret_manager.json, then run:

   confluent connect cluster create --config-file postgrescdc_with_secret_manager.json
   

   Example output:

   Created connector PostgresCdcSourceV2_1 lcc-ix4dl
   

Confluent REST API

Include the following fields in the configuration payload of your Create a connector API request.

• The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s authentication.method for Postgres CDC Source V2 connector and mongodb.auth.mechanism for MongoDB Atlas Source connector.

• secret.manager.enabled

• secret.manager

• secret.manager.managed.configs

• secret.manager.provider.integration.id

• The secret identifier whose value you want Confluent Cloud to fetch. In this example, it is the database.password and database.user field values. For more information on the Google Cloud Secret Manager’s secret identifier format, see Configuration value format.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "PostgresCdcSourceV2_1",
   "config": {
      "connector.class": "PostgresCdcSourceV2",
      "name": "PostgresCdcSourceV2_1",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-xxxxxx",
      "authentication.method": "Password",
      "secret.manager.enabled": "true",
      "secret.manager": "GCP_SECRET_MANAGER",
      "secret.manager.managed.configs": "database.password,database.user",
      "secret.manager.provider.integration.id": "cspi-devxxxxxx",
      "database.hostname": "203.0.113.0",
      "database.port": "5432",
      "database.user": "projects/123456789012/secrets/postgres-db-secret::user",
      "database.password": "projects/123456789012/secrets/postgres-db-secret::password",
      "database.dbname": "postgres",
      "database.sslmode": "prefer",
      "output.data.format": "JSON",
      "output.key.format": "JSON",
      "table.include.list": "public.table123",
      "topic.prefix": "postgrescdc",
      "tasks.max": "1"
   }
}'

Verify Google Cloud Secret Manager authentication

After the connector is provisioned and running, verify the successful authorization through the credentials fetched from Google Cloud Secret Manager.

View topic messages

1. Sign in to your Confluent Cloud account.

2. In the left navigation menu, click Connectors.

3. Click the connector you just created.

4. View the messages produced to or received from a topic.

View provider integration resources

1. Go to the Environments page at https://confluent.cloud/environments and select the environment in which you created your provider integration.

2. Click the Provider integrations tab.

3. Under the Resources column, view the resources.

4. Click the resource link to view the associated connector that you created.

Configuration value format

For the connector configurations that you manage through Google Cloud Secret Manager, you must specify secret identifiers as configuration values in one of the following formats:

1. If your secret is stored as plaintext (non-JSON format), specify the configuration value as the secret’s resource name in the format projects/<project-number>/secrets/<secret-name>.

   For example, to have Confluent Cloud fetch the database password, specify projects/123456789012/secrets/postgres-db-secret.

   To pin a specific secret version, append /versions/<version>, for example projects/123456789012/secrets/postgres-db-secret/versions/3. If you omit the version, Confluent Cloud reads the latest version.

2. If your secrets are stored as a JSON object within the Google Cloud Secret Manager secret, then you must specify each configuration value as projects/<project-number>/secrets/<secret-name>::<config-key>.

   For example, if the secret projects/123456789012/secrets/postgres-db-secret stores a JSON that looks like:

   {
      "user": "dbadmin",
      "password": "$trongPassw0rdExamp1e"
   }
   

   • To fetch the database user, specify the connector configuration value as projects/123456789012/secrets/postgres-db-secret::user.

   • To fetch the database password, specify the connector configuration value as projects/123456789012/secrets/postgres-db-secret::password.

   You can also pin a version when using a JSON key, for example projects/123456789012/secrets/postgres-db-secret/versions/3::password.

Note

Use your Google Cloud project number for the <project-number> segment. The config key (<config-key>) must not contain the characters $, {, }, or :.

Troubleshooting

Connector fails to start with authentication error

• Verify your Google Cloud Secret Manager secret resource name is correct and follows the format projects/<project-number>/secrets/<secret-name>.

• Ensure your Google Cloud service account associated with the provider integration has the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor), which grants the required secretmanager.versions.access permission on your secret.

• Check that the secret exists in your Google Cloud Secret Manager and contains a valid value.

• Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see Configuration value format.

• Verify that the provider integration trust is established. On the provider integration, use the Validate option to confirm that Confluent Cloud can access your Google Cloud account. Because Google Cloud takes time to sync policy updates, validation can take a few minutes to succeed after you grant the role.