# Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes Confluent for Kubernetes (CFK) provides the ability to securely provide sensitive credentials and certificates (referred to as “secrets” in this document) to the Confluent Platform deployment. CFK supports the following two mechanisms: * [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) Your sensitive data is stored in a Kubernetes Secret, and the data is referenced via the Kubernetes Secret (`secretRef`) in the Confluent Platform component custom resource (CR). * Vaults When integrated with vaults, which provide secure key-value stores with tight access control, CFK can offer increased security of the credentials. Vault injects secrets into the CFK or Confluent Platform component pods on specific directory paths in the containers. Then, in the Confluent Platform component custom resources (CRs), the secrets are referenced via those directory paths (`directoryPathInContainer`). If secrets are changed, Vault updates them in the pod, and CFK dynamically reads the updated info. You can use the following vaults with CFK: - [HashiCorp Vault](#co-directory-path-in-container) - [Azure Key Vault](#co-azure-key-vault) ## Provide secrets in HashiCorp Vault Vault securely stores secrets as files in its container and injects the file paths in a CFK or Confluent Platform pod. CFK looks for the required secrets in the file under the specified directory path in the pod. See [CFK example repo in GitHub](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/security/configure-with-vault) for a comprehensive use case for providing credentials and certificates with Vault. #### NOTE HashiCorp Vault is a third-party software product that is not supported or distributed by Confluent. Refer to their [product documentation](https://www.vaultproject.io/docs) for configuration information. All CFK CRs and applications/workflows can take advantage of Vault. The components and applications fall into one of the following categories: * [Confluent component CRs where CFK creates pods](#co-vault-category-1) * [CFK application CRs where CFK does not create pods](#co-vault-category-2) * [CFK operations with no associated CRs](#co-vault-category-3) ### Provide secrets for Confluent Platform component CR For Confluent Platform components, their secrets are injected into the Confluent Platform component pods on the directory path in the container that you specify in the Vault annotation. The following are the component resources that fall into this category: * Kafka * ZooKeeper (Confluent Platform 7.9 or earlier only) * Schema Registry * Connect * ksqlDB * Control Center (Legacy) * Kafka REST Proxy 1. In the Confluent Platform component CR, add the [required annotations](#co-vault-annotations) to integrate with Vault. For an example, see [Example annotations in component CR](#co-example-vault-annotations-cr). 2. In the Confluent Platform component CR, set `directoryPathInContainer` property to the directory path that contains the required secrets in Vault. ### Provide secrets for Confluent Platform application CR The Confluent Platform application custom resources do not have pods associated with them. For these resources, Vault annotations should be configured in the CFK Helm values (in [Deploy CFK with Custom Values](co-deploy-custom-values.md#co-values-file)), and their secrets are injected into the CFK pod at runtime for the applications. The following are the application resources that fall into this category: * Cluster Link CRs * Schema Link CRs * Schema CRs * Kafka Topic CRs * Kafka REST Class CRs * Confluent Rolebinding CRs * Connector CRs 1. In the CFK Helm `values.yaml` file, add the [required annotations](#co-vault-annotations) to integrate with Vault. For an example, see [Example CFK annotations](#co-example-vault-annotations-cfk). 2. In the Confluent Platform application CR, set the `directoryPathInContainer` property to the directory path that contains the required secrets. ### Provide secrets for Confluent Platform operations without CRs There are operations that are not managed with application CRs, but are required to use Kafka Admin REST API. Since these operations do not have their own pods, Vault annotations are added to the CFK Helm values (in [Deploy CFK with Custom Values](co-deploy-custom-values.md#co-values-file)), their secrets are injected into the CFK pod at runtime for the operations, and CFK defaults to using the same path to search for the credentials in the CFK pod. Examples of these operations are: * Cluster shrink * Kafka roll * Internal rolebinding creation * Listing of connectors in the Connect status 1. Configure the component CR as described in [Provide secrets for Confluent Platform component CR](#co-vault-category-1). 2. You have an option to extend the default path with the following annotation: ```properties platform.confluent.io/dpic-append-path= ``` With the above annotation, if your Confluent Platform component, such as Kafka, has its secrets injected to `/vault/secrets`, CFK will look for the same credentials in `/vault/secrets/append/path` in the CFK pod. This is recommended to avoid any path or file name collisions between the component server secret and the operation secrets. 3. In the CFK Helm `values.yaml` file, add the [required annotations](#co-vault-annotations) to integrate with Vault. For an example, see [Example CFK annotations](#co-example-vault-annotations-cfk). ### Vault annotations for CFK For HashiCorp Vault, the following annotations are required. For complete information about the required annotations, see [HashiCorp Vault Annotations](https://www.vaultproject.io/docs/platform/k8s/injector/annotations). * `vault.hashicorp.com/agent-inject: "true"` Injects the Vault Agent container to the pod. * `vault.hashicorp.com/agent-inject-status: update` Updates the secret when it’s rotated. * `vault.hashicorp.com/preserve-secret-case: "true"` Preserves the secret name case when creating secret files.
For each secret being injected, add the following annotations in the component CR or the CFK `values.yaml` file. For complete information about the annotations, see [HashiCorp Vault Annotations](https://www.vaultproject.io/docs/platform/k8s/injector/annotations). * `vault.hashicorp.com/agent-inject-secret- = ` Sets to retrieve the `` secret from `` in Vault. * `vault.hashicorp.com/agent-inject-file- = ` Sets `` where `` is written. Use the same `` used in the `vault.hashicorp.com/agent-inject-secret-` annotation. * `vault.hashicorp.com/agent-inject-template- = |` `{{- with secret “" -}}` `{{ .Data.data.data }}` `{{- end }}` Formats and creates a template for the contents of the secret. This uses the temp secret name and uses the data field of the secret at the specified path. Use the same ```used in the ``vault.hashicorp.com/agent-inject-secret-` annotation. When adding annotations to CFK Helm `values.yaml`, surround the value of the `vault.hashicorp.com/agent-inject-template-` annotation as `{{``}}`. See [Configure Confluent Platform Authentication with Confluent for Kubernetes](co-authenticate-overview.md#co-authenticate) and [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption) for the required format of Confluent Platform credentials and certificates. * `vault.hashicorp.com/secret-volume-path- = ` Mounts `` to `` on the filesystem in the pod. Use the same ```used in the ``vault.hashicorp.com/agent-inject-secret-` annotation. ### Configure to provide secrets for Confluent Platform using Vault To provide secrets for Confluent Platform using Vault: 1. Create the required credential files. For example, create `keystore.jks`, `truststore.jks`, and `jksPassword.txt`. #### IMPORTANT Vault secret injection itself is format-agnostic and can inject PEM files, such as CA and public-key material. However, CFK does not convert PEM files to Java KeyStore format for the TLS key and truststore material consumed through `directoryPathInContainer`. For this material, you must provide the credential files directly in Java KeyStore format, such as `.jks` or `.p12`. See [Configure Confluent Platform Authentication with Confluent for Kubernetes](co-authenticate-overview.md#co-authenticate) and [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption) for the required Confluent Platform credentials and certificates. 2. Configure Vault. Install Vault and configure Vault policy and Vault permission according to the Vault documentation. 3. Add secrets to Vault: For example, in HashiCorp Vault: 1. Exec into the Vault pod ```bash kubectl exec vault-0 sh -it ``` 2. Put the secret value contents into the file: ```bash cat > vault kv put secret/ data=@ ``` 3. Confirm that secret contents are as expected: ```bash vault kv get secret/ ``` 4. Add annotations to have Vault inject secrets to the pod. See [Vault annotations for CFK](#co-vault-annotations) for the list of required annotations. 5. In the component or application CR, set `directoryPathInContainer` property to point to the Vault directory path. The following example is a snippet of a Kafka REST Class CR that references the Bearer authentication credentials specified in the example in [Vault annotations for CFK](#co-vault-annotations). ```yaml spec: kafkaRest: authentication: bearer: directoryPathInContainer: /vault/secrets type: bearer ``` ### Example annotations #### Example CFK annotations The following is an example snippet of the CFK Helm values (in `values.yaml`) file with the Vault annotations. Use these CFK pod annotations for the CFK applications that do not have a pod associated with them. ```yaml pod: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-file-cacerts.pem: cacerts.pem vault.hashicorp.com/agent-inject-file-fullchain.pem: fullchain.pem vault.hashicorp.com/agent-inject-file-privkey.pem: privkey.pem vault.hashicorp.com/agent-inject-secret-cacerts.pem: secret/cacerts.pem vault.hashicorp.com/agent-inject-secret-fullchain.pem: secret/fullchain.pem vault.hashicorp.com/agent-inject-secret-privkey.pem: secret/privkey.pem vault.hashicorp.com/agent-inject-status: update vault.hashicorp.com/agent-inject-template-cacerts.pem: |- {{- with secret "secret/cacerts.pem" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-fullchain.pem: |- {{- with secret "secret/fullchain.pem" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-privkey.pem: |- {{- with secret "secret/privkey.pem" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/preserve-secret-case: "true" vault.hashicorp.com/role: confluent-for-kubernetes vault.hashicorp.com/secret-volume-path-cacerts.pem: /vault/secrets vault.hashicorp.com/secret-volume-path-fullchain.pem: /vault/secrets vault.hashicorp.com/secret-volume-path-privkey.pem: /vault/secrets ``` #### Example annotations in component CR The following is an example of Kafka annotations for RBAC, TLS, and SASL/PLAIN: ```yaml kind: Kafka spec: authorization: superUsers: - User:kafka type: rbac dependencies: kafkaRest: authentication: bearer: directoryPathInContainer: /vault/secrets type: bearer podTemplate: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-file-bearer.txt: bearer.txt vault.hashicorp.com/agent-inject-file-cacerts.pem: cacerts.pem vault.hashicorp.com/agent-inject-file-jksPassword.txt: jksPassword.txt vault.hashicorp.com/agent-inject-file-keystore.jks: keystore.jks vault.hashicorp.com/agent-inject-file-ldap.txt: ldap.txt vault.hashicorp.com/agent-inject-file-mdsPublicKey.pem: mdsPublicKey.pem vault.hashicorp.com/agent-inject-file-mdsTokenKeyPair.pem: mdsTokenKeyPair.pem vault.hashicorp.com/agent-inject-file-truststore.jks: truststore.jks vault.hashicorp.com/agent-inject-secret-bearer.txt: secret/kafka/bearer.txt vault.hashicorp.com/agent-inject-secret-cacerts.pem: secret/cacerts.pem vault.hashicorp.com/agent-inject-secret-jksPassword.txt: secret/jksPassword.txt vault.hashicorp.com/agent-inject-secret-keystore.jks: secret/kafka-keystore.jks vault.hashicorp.com/agent-inject-secret-ldap.txt: secret/kafka/ldap.txt vault.hashicorp.com/agent-inject-secret-mdsPublicKey.pem: secret/mdsPublicKey.pem vault.hashicorp.com/agent-inject-secret-mdsTokenKeyPair.pem: secret/mdsTokenKeyPair.pem vault.hashicorp.com/agent-inject-secret-truststore.jks: secret/kafka-truststore.jks vault.hashicorp.com/agent-inject-status: update vault.hashicorp.com/agent-inject-template-bearer.txt: |- {{- with secret "secret/kafka/bearer.txt" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-cacerts.pem: |- {{- with secret "secret/cacerts.pem" -}} {{ .Data.data.cacerts }} {{- end }} vault.hashicorp.com/agent-inject-template-jksPassword.txt: |- {{- with secret "secret/jksPassword.txt" -}} {{ .Data.data.password }} {{- end }} vault.hashicorp.com/agent-inject-template-keystore.jks: |- {{- with secret "secret/kafka-keystore.jks" -}} {{ .Data.data.keystore | base64Decode }} {{- end }} vault.hashicorp.com/agent-inject-template-ldap.txt: |- {{- with secret "secret/kafka/ldap.txt" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-mdsPublicKey.pem: |- {{- with secret "secret/mdsPublicKey.pem" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-mdsTokenKeyPair.pem: |- {{- with secret "secret/mdsTokenKeyPair.pem" -}} {{ .Data.data.data }} {{- end }} vault.hashicorp.com/agent-inject-template-truststore.jks: |- {{- with secret "secret/kafka-truststore.jks" -}} {{ .Data.data.truststore | base64Decode }} {{- end }} vault.hashicorp.com/preserve-secret-case: "true" vault.hashicorp.com/role: confluent-operator vault.hashicorp.com/secret-volume-path-bearer.txt: /vault/secrets vault.hashicorp.com/secret-volume-path-cacerts.pem: /mnt/sslcerts vault.hashicorp.com/secret-volume-path-jksPassword.txt: /vault/secrets vault.hashicorp.com/secret-volume-path-keystore.jks: /vault/secrets vault.hashicorp.com/secret-volume-path-ldap.txt: /vault/secrets vault.hashicorp.com/secret-volume-path-mdsPublicKey.pem: /vault/secrets vault.hashicorp.com/secret-volume-path-mdsTokenKeyPair.pem: /vault/secrets vault.hashicorp.com/secret-volume-path-truststore.jks: /vault/secrets vault.hashicorp.com/tls-skip-verify: "true" services: mds: tokenKeyPair: directoryPathInContainer: /vault/secrets tls: directoryPathInContainer: /vault/secrets ``` Once Kafka starts, Vault injects the following secrets to the Kafka pod: ```text /vault/secrets/bearer.txt /vault/secrets/cacerts.pem /vault/secrets/jksPassword.txt /vault/secrets/keystore.jks /vault/secrets/ldap.txt /vault/secrets/mdsPublicKey.pem /vault/secrets/mdsTokenKeyPair.pem /vault/secrets/truststore.jks ``` ### Troubleshoot Vault integration Error: Unable to retrieve credentials from file xxx in CFK pod : You tried to use a base-64 encoded credential without first decoding it.
To resolve the issue, in the component annotation, add `base64Decode` as shown in the below example:
```yaml vault.hashicorp.com/agent-inject-template-bearer.txt: |- {{- with secret "secret/kafka/bearer.txt" -}} {{ .Data.data.data | base64Decode}} {{- end }} ``` ## Mount custom Kubernetes secrets When installing Confluent Platform components, Confluent for Kubernetes (CFK) securely stores sensitive credential configuration values you provide as [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). However, there are cases where you may wish to provide your own pre-existing secrets to be used in the configuration of Confluent Platform, rather than having CFK create the secret object for you. The most common scenario for this is when providing [custom configuration overrides](co-configure-misc.md#co-config-overrides), which CFK treats as passthrough configuration and does not consider whether that data belongs in a secret object or not. To utilize your own secret objects, specify a list of secret objects to mount in-memory into the containers of a Confluent Platform component. Combined with the [Externalizing Secrets](/platform/current/connect/security.html#connect-externalize-secrets) capability of Confluent Platform and custom configuration overrides, you can configure Confluent Platform with your secret data so that you store the information in an external file and pass the content of the a key file without exposing the sensitive information. A change in mounted secrets will roll the cluster. To provide and use mounted secrets, add the following in the component CR: ```yaml spec: mountedSecrets: - secretRef: --- [1] keyItems: --- [2] - key: --- [3] path: --- [4] configOverrides: server: - property=${file:/mnt/secrets/:} --- [5] ``` * [1] Set to the name of your secret. You must use the secret which are referenced in the other part of the component custom resource (CR). * [2] Optional. If `keyItems` are not specified, the secret is stored under `/mnt/secrets/` where `` is the value set in [1]. See [Projection of Secret keys to specific paths](https://kubernetes.io/docs/concepts/configuration/secret/#projection-of-secret-keys-to-specific-paths) for more information. * [3] Set to the secret key. * [4] Set to the relative path where the secret `key` ([3]) is stored. It may not be an absolute path and may not contain the string `..`. For example, if `secretRef` is set to `my-secret-aws`, `key` is set to `credentials`, and `path` is set to `aws/creds`, the `credentials` are stored in `/mnt/secrets/my-secret-aws/aws/creds/credentials`. * [5] Set the externalized file in the `${file:/mnt/secrets/:}` format. The following is an example workflow of configuring mounted secrets. 1. Create a file, `my_file.txt`, that contains your sensitive data: ```text username: user1 password: $ecret_passw0rd ``` #### NOTE The file must use the Linux style line ending that only uses line feed (`\n`). The Windows style line ending that uses carriage return and line feed (`\r\n`) does not work in Kubernetes secret files. 2. Create a secret, `example-secret`, which stores the content of the above file in the `my_credentials` key: ```bash kubectl create secret generic example-secret \ --from-file=my_credentials=path/to/my_file.txt ``` The contents of `my_file.txt` will be Base64-encoded and stored under the `my_credentials` key of `example-secret`. 3. Set the `configOverrides` and `mountedSecrets` in the Kafka CR: ```yaml spec: mountedSecrets: - secretRef: example-secret configOverrides: server: - kafka.user=${file:/mnt/secrets/example-secret/my_credentials:username} - kafka.password=${file:/mnt/secrets/example-secret/my_credentials:password} ``` 4. Deploy Kafka: ```bash kubectl apply -f ``` ## Provide secrets in Azure Key Vault You can integrate CFK with Azure Key Vault through the custom volume mount feature. The following is an example snippet of a Kafka CR integrated with Azure Key Vault: ```yaml kind: Kafka metadata: name: kafka namespace: operator spec: mountedVolumes: volumes: - name: secrets-store01-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-sync" volumeMounts: - name: secrets-store01-inline mountPath: "/mnt/secrets-store" readOnly: true ``` For details about the custom volume mount feature, see [Mount custom volumes](co-configure-misc.md#co-custom-volume-mounts). For more information about Azure Key Vault, see [Use the Azure Key Vault provider for Secrets Store CSI Driver in an Azure Kubernetes Service (AKS) cluster](https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#set-an-environment-variable-to-reference-kubernetes-secrets).