Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes¶

Confluent for Kubernetes (CFK) provides the ability to securely provide sensitive credentials and certificates (referred to as “secrets” in this document) to the Confluent Platform deployment. CFK supports the following two mechanisms:

Kubernetes Secrets

Your sensitive data is stored in a Kubernetes Secret, and the data is referenced via the Kubernetes Secret (secretRef) in the Confluent Platform component custom resource (CR).
Vaults

When integrated with vaults, which provide secure key-value stores with tight access control, CFK can offer increased security of the credentials.

Vault injects secrets into the CFK or Confluent Platform component pods on specific directory paths in the containers. Then, in the Confluent Platform component custom resources (CRs), the secrets are referenced via those directory paths (directoryPathInContainer).

If secrets are changed, Vault updates them in the pod, and CFK dynamically reads the updated info.

You can use the following vaults with CFK:
- HashiCorp Vault
- Azure Key Vault

Provide secrets in HashiCorp Vault¶

Vault securely stores secrets as files in its container and injects the file paths in a CFK or Confluent Platform pod. CFK looks for the required secrets in the file under the specified directory path in the pod.

See CFK example repo in GitHub for a comprehensive use case for providing credentials and certificates with Vault.

Note

HashiCorp Vault is a third-party software product that is not supported or distributed by Confluent. Refer to their product documentation for configuration information.

All CFK CRs and applications/workflows can take advantage of Vault. The components and applications fall into one of the following categories:

Confluent component CRs where CFK creates pods
CFK application CRs where CFK does not create pods
CFK operations with no associated CRs

Provide secrets for Confluent Platform component CR¶

For Confluent Platform components, their secrets are injected into the Confluent Platform component pods on the directory path in the container that you specify in the Vault annotation.

The following are the component resources that fall into this category:

Kafka
ZooKeeper
Schema Registry
Connect
ksqlDB
Control Center
Kafka REST Proxy

In the Confluent Platform component CR, add the required annotations to integrate with Vault.

For an example, see Example annotations in component CR.
In the Confluent Platform component CR, set directoryPathInContainer property to the directory path that contains the required secrets in Vault.

Provide secrets for Confluent Platform application CR¶

The Confluent Platform application custom resources do not have pods associated with them. For these resources, Vault annotations should be configured in the CFK Helm values (in Deploy customized CFK), and their secrets are injected into the CFK pod at runtime for the applications.

The following are the application resources that fall into this category:

Cluster Link CRs
Schema Link CRs
Schema CRs
Kafka Topic CRs
Kafka REST Class CRs
Confluent Rolebinding CRs
Connector CRs

In the CFK Helm values.yaml file, add the required annotations to integrate with Vault.

For an example, see Example CFK annotations.
In the Confluent Platform application CR, set the directoryPathInContainer property to the directory path that contains the required secrets.

Provide secrets for Confluent Platform operations without CRs¶

There are operations that are not managed with application CRs, but are required to use Kafka Admin REST API. Since these operations do not have their own pods, Vault annotations are added to the CFK Helm values (in Deploy customized CFK), their secrets are injected into the CFK pod at runtime for the operations, and CFK defaults to using the same path to search for the credentials in the CFK pod.

Examples of these operations are:

Cluster shrink
Kafka roll
Internal rolebinding creation
Listing of connectors in the Connect status

Configure the component CR as described in Provide secrets for Confluent Platform component CR.
You have an option to extend the default path with the following annotation:
```
platform.confluent.io/directory-path-in-container-suffix=<suffix/path>
```
With the above annotation, if your Confluent Platform component, such as Kafka, has its secrets injected to /vault/secrets, CFK will look for the same credentials in /vault/secrets/suffix/path in the CFK pod.

This is recommended to avoid any path or file name collisions between the component server secret and the operation secrets.
In the CFK Helm values.yaml file, add the required annotations to integrate with Vault.

For an example, see Example CFK annotations.

Vault annotations for CFK¶

For HashiCorp Vault, the following annotations are required. For complete information about the required annotations, see HashiCorp Vault Annotations.

vault.hashicorp.com/agent-inject: "true"

Injects the Vault Agent container to the pod.
vault.hashicorp.com/agent-inject-status: update

Updates the secret when it’s rotated.
vault.hashicorp.com/preserve-secret-case: "true"

Preserves the secret name case when creating secret files.

For each secret being injected, add the following annotations in the component CR or the CFK values.yaml file. For complete information about the annotations, see HashiCorp Vault Annotations.

vault.hashicorp.com/agent-inject-secret-<secret-name> = <secret-path>

Sets to retrieve the <secret-name> secret from <secret-path> in Vault.
vault.hashicorp.com/agent-inject-file-<secret-name> = <secret-file>

Sets <secret-file> where <secret-name> is written.

Use the same <secret-name> used in the vault.hashicorp.com/agent-inject-secret-<secret-name> annotation.
vault.hashicorp.com/agent-inject-template-<secret-name> = |

{{- with secret “<Vault-secret-path>" -}}

{{ .Data.data.data }}

{{- end }}

Formats and creates a template for the contents of the secret. This uses the temp secret name and uses the data field of the secret at the specified path.

Use the same <secret-name>``used in the ``vault.hashicorp.com/agent-inject-secret-<secret-name> annotation.

When adding annotations to CFK Helm values.yaml, surround the value of the vault.hashicorp.com/agent-inject-template-<secret-name> annotation as {{`<value>`}}.

See Configure Confluent Platform Authentication with Confluent for Kubernetes and Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes for the required format of Confluent Platform credentials and certificates.
vault.hashicorp.com/secret-volume-path-<secret-name> = <mount-path>

Mounts <secret-name> to <mount-path> on the filesystem in the pod.

Use the same <secret-name>``used in the ``vault.hashicorp.com/agent-inject-secret-<secret-name> annotation.

Configure to provide secrets for Confluent Platform using Vault¶

To provide secrets for Confluent Platform using Vault:

Create the required credential files. For example, create keystore.jks, truststore.jks, and jksPassword.txt.

Important

Only the Java KeyStore format is supported when providing secrets using Vault.

Because there is no mechanism to convert .pem files to .jks when using Vault, you have to directly use .jks files.

See Configure Confluent Platform Authentication with Confluent for Kubernetes and Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes for the required Confluent Platform credentials and certificates.
Configure Vault.

Install Vault and configure Vault policy and Vault permission according to the Vault documentation.
Add secrets to Vault:

For example, in HashiCorp Vault:
1. Exec into the Vault pod
```
kubectl exec vault-0 sh -it
```
2. Put the secret value contents into the file:
```
cat >  <filename>
vault kv put secret/<secret-name> data=@<filename>
```
3. Confirm that secret contents are as expected:
```
vault kv get secret/<secret-name>
```
Add annotations to have Vault inject secrets to the pod. See Vault annotations for CFK for the list of required annotations.
In the component or application CR, set directoryPathInContainer property to point to the Vault directory path.

The following example is a snippet of a Kafka REST Class CR that references the Bearer authentication credentials specified in the example in Vault annotations for CFK.
```
spec:
  kafkaRest:
    authentication:
      bearer:
        directoryPathInContainer: /vault/secrets
      type: bearer
```

Example annotations¶

Example CFK annotations¶

The following is an example snippet of the CFK Helm values (in values.yaml) file with the Vault annotations. Use these CFK pod annotations for the CFK applications that do not have a pod associated with them.

pod:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-file-cacerts.pem: cacerts.pem
    vault.hashicorp.com/agent-inject-file-fullchain.pem: fullchain.pem
    vault.hashicorp.com/agent-inject-file-privkey.pem: privkey.pem
    vault.hashicorp.com/agent-inject-secret-cacerts.pem: secret/cacerts.pem
    vault.hashicorp.com/agent-inject-secret-fullchain.pem: secret/fullchain.pem
    vault.hashicorp.com/agent-inject-secret-privkey.pem: secret/privkey.pem
    vault.hashicorp.com/agent-inject-status: update
    vault.hashicorp.com/agent-inject-template-cacerts.pem: |-
      {{- with secret "secret/cacerts.pem" -}}
      {{ .Data.data.data }}
      {{- end }}
    vault.hashicorp.com/agent-inject-template-fullchain.pem: |-
      {{- with secret "secret/fullchain.pem" -}}
      {{ .Data.data.data }}
      {{- end }}
    vault.hashicorp.com/agent-inject-template-privkey.pem: |-
      {{- with secret "secret/privkey.pem" -}}
      {{ .Data.data.data }}
      {{- end }}
    vault.hashicorp.com/preserve-secret-case: "true"
    vault.hashicorp.com/role: confluent-for-kubernetes
    vault.hashicorp.com/secret-volume-path-cacerts.pem: /vault/secrets
    vault.hashicorp.com/secret-volume-path-fullchain.pem: /vault/secrets
    vault.hashicorp.com/secret-volume-path-privkey.pem: /vault/secrets

Example annotations in component CR¶

The following is an example of Kafka annotations for RBAC, TLS, and SASL/PLAIN:

kind: Kafka
spec:
    authorization:
      superUsers:
      - User:kafka
      type: rbac
    dependencies:
      kafkaRest:
        authentication:
          bearer:
            directoryPathInContainer: /vault/secrets
          type: bearer
    podTemplate:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-file-bearer.txt: bearer.txt
        vault.hashicorp.com/agent-inject-file-cacerts.pem: cacerts.pem
        vault.hashicorp.com/agent-inject-file-jksPassword.txt: jksPassword.txt
        vault.hashicorp.com/agent-inject-file-keystore.jks: keystore.jks
        vault.hashicorp.com/agent-inject-file-ldap.txt: ldap.txt
        vault.hashicorp.com/agent-inject-file-mdsPublicKey.pem: mdsPublicKey.pem
        vault.hashicorp.com/agent-inject-file-mdsTokenKeyPair.pem: mdsTokenKeyPair.pem
        vault.hashicorp.com/agent-inject-file-truststore.jks: truststore.jks
        vault.hashicorp.com/agent-inject-secret-bearer.txt: secret/kafka/bearer.txt
        vault.hashicorp.com/agent-inject-secret-cacerts.pem: secret/cacerts.pem
        vault.hashicorp.com/agent-inject-secret-jksPassword.txt: secret/jksPassword.txt
        vault.hashicorp.com/agent-inject-secret-keystore.jks: secret/kafka-keystore.jks
        vault.hashicorp.com/agent-inject-secret-ldap.txt: secret/kafka/ldap.txt
        vault.hashicorp.com/agent-inject-secret-mdsPublicKey.pem: secret/mdsPublicKey.pem
        vault.hashicorp.com/agent-inject-secret-mdsTokenKeyPair.pem: secret/mdsTokenKeyPair.pem
        vault.hashicorp.com/agent-inject-secret-truststore.jks: secret/kafka-truststore.jks
        vault.hashicorp.com/agent-inject-status: update
        vault.hashicorp.com/agent-inject-template-bearer.txt: |-
          {{- with secret "secret/kafka/bearer.txt" -}}
          {{ .Data.data.data }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-cacerts.pem: |-
          {{- with secret "secret/cacerts.pem" -}}
          {{ .Data.data.cacerts }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-jksPassword.txt: |-
          {{- with secret "secret/jksPassword.txt" -}}
          {{ .Data.data.password }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-keystore.jks: |-
          {{- with secret "secret/kafka-keystore.jks" -}}
          {{ .Data.data.keystore | base64Decode }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-ldap.txt: |-
          {{- with secret "secret/kafka/ldap.txt" -}}
          {{ .Data.data.data }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-mdsPublicKey.pem: |-
          {{- with secret "secret/mdsPublicKey.pem" -}}
          {{ .Data.data.data }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-mdsTokenKeyPair.pem: |-
          {{- with secret "secret/mdsTokenKeyPair.pem" -}}
          {{ .Data.data.data }}
          {{- end }}
        vault.hashicorp.com/agent-inject-template-truststore.jks: |-
          {{- with secret "secret/kafka-truststore.jks" -}}
          {{ .Data.data.truststore | base64Decode }}
          {{- end }}
        vault.hashicorp.com/preserve-secret-case: "true"
        vault.hashicorp.com/role: confluent-operator
        vault.hashicorp.com/secret-volume-path-bearer.txt: /vault/secrets
        vault.hashicorp.com/secret-volume-path-cacerts.pem: /mnt/sslcerts
        vault.hashicorp.com/secret-volume-path-jksPassword.txt: /vault/secrets
        vault.hashicorp.com/secret-volume-path-keystore.jks: /vault/secrets
        vault.hashicorp.com/secret-volume-path-ldap.txt: /vault/secrets
        vault.hashicorp.com/secret-volume-path-mdsPublicKey.pem: /vault/secrets
        vault.hashicorp.com/secret-volume-path-mdsTokenKeyPair.pem: /vault/secrets
        vault.hashicorp.com/secret-volume-path-truststore.jks: /vault/secrets
        vault.hashicorp.com/tls-skip-verify: "true"
    services:
      mds:
        tokenKeyPair:
          directoryPathInContainer: /vault/secrets
    tls:
      directoryPathInContainer: /vault/secrets

Once Kafka starts, Vault injects the following secrets to the Kafka pod:

/vault/secrets/bearer.txt
/vault/secrets/cacerts.pem
/vault/secrets/jksPassword.txt
/vault/secrets/keystore.jks
/vault/secrets/ldap.txt
/vault/secrets/mdsPublicKey.pem
/vault/secrets/mdsTokenKeyPair.pem
/vault/secrets/truststore.jks

Troubleshoot Vault integration¶

Error: Unable to retrieve credentials from file xxx in CFK pod

You tried to use a base-64 encoded credential without first decoding it.

To resolve the issue, in the component annotation, add base64Decode as shown in the below example:

vault.hashicorp.com/agent-inject-template-bearer.txt: |-
  {{- with secret "secret/kafka/bearer.txt" -}}
  {{ .Data.data.data | base64Decode}}
  {{- end }}

Mount custom Kubernetes secrets¶

When installing Confluent Platform components, Confluent for Kubernetes (CFK) securely stores sensitive credential configuration values you provide as Kubernetes Secrets. However, there are cases where you may wish to provide your own pre-existing secrets to be used in the configuration of Confluent Platform, rather than having CFK create the secret object for you. The most common scenario for this is when providing custom configuration overrides, which CFK treats as passthrough configuration and does not consider whether that data belongs in a secret object or not.

To utilize your own secret objects, specify a list of secret objects to mount in-memory into the containers of a Confluent Platform component. Combined with the Externalizing Secrets capability of Confluent Platform and custom configuration overrides, you can configure Confluent Platform with your secret data so that you store the information in an external file and pass the content of the a key file without exposing the sensitive information.

A change in mounted secrets will roll the cluster.

To provide and use mounted secrets, add the following in the component CR:

spec:
  mountedSecrets:
  - secretRef:                                          --- [1]
    keyItems:                                           --- [2]
    - key:                                              --- [3]
      path:                                             --- [4]

  configOverrides:
    server:
      - property=${file:/mnt/secrets/<secretRef>:<key>} --- [5]

[1] Set to the name of your secret. You must use the secret which are referenced in the other part of the component custom resource (CR).
[2] Optional. If keyItems are not specified, the secret is stored under /mnt/secrets/<secretRef> where <secretRef> is the value set in [1].

See Projection of Secret keys to specific paths for more information.
[3] Set to the secret key.
[4] Set to the relative path where the secret key ([3]) is stored. It may not be an absolute path and may not contain the string ...

For example, if secretRef is set to my-secret-aws, key is set to credentials, and path is set to aws/creds, the credentials are stored in /mnt/secrets/my-secret-aws/aws/creds/credentials.
[5] Set the externalized file in the ${file:/mnt/secrets/<secretRef>:<key>} format.

The following is an example workflow of configuring mounted secrets.

Create a file, my_file.txt, that contains your sensitive data:
```
username: user1
password: $ecret_passw0rd
```
Note

The file must use the Linux style line ending that only uses line feed (\n). The Windows style line ending that uses carriage return and line feed (\r\n) does not work in Kubernetes secret files.
Create a secret, example-secret, which stores the content of the above file in the my_credentials key:
```
kubectl create secret generic example-secret \
  --from-file=my_credentials=path/to/my_file.txt
```
The contents of my_file.txt will be Base64-encoded and stored under the my_credentials key of example-secret.

Set the configOverrides and mountedSecrets in the Kafka CR:

spec:
  mountedSecrets:
    - secretRef: example-secret
  configOverrides:
    server:
      - kafka.user=${file:/mnt/secrets/example-secret/my_credentials:username}
      - kafka.password=${file:/mnt/secrets/example-secret/my_credentials:password}

Deploy Kafka:
```
kubectl apply -f <Kafka CR>
```

Provide secrets in Azure Key Vault¶

You can integrate CFK with Azure Key Vault through the custom volume mount feature.

The following is an example snippet of a Kafka CR integrated with Azure Key Vault:

kind: Kafka
metadata:
  name: kafka
  namespace: operator
spec:
  mountedVolumes:
    volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"
    volumeMounts:
    - name: secrets-store01-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true

For details about the custom volume mount feature, see Mount custom volumes.

For more information about Azure Key Vault, see Use the Azure Key Vault provider for Secrets Store CSI Driver in an Azure Kubernetes Service (AKS) cluster.