Case Study: Manage Cloud Secrets

The streaming-ops project is a simulated production environment running a streaming microservices based application targeting Apache Kafka® on Confluent Cloud. Applications and resources are managed by GitOps with declarative infrastructure, Kubernetes, and the Operator Pattern.

This case study looks at a method for managing secrets used to access Confluent Cloud for automated systems inside of a Kubernetes cluster.

Declarative Secrets

Kubernetes provides a declarative API for managing application deployments, configurations, and other related metadata. Included in the API is a Secrets type that can be used to store and manage sensitive information, such as passwords, SSH keys, or other secret data.

Note

Kubernetes objects of type Secret do not fully protect sensitive data. Data contained within a Secret is not encrypted and can be viewed by any Kubernetes user with read access permission to the object. To secure Secret objects and prevent unauthorized access, use Kubernetes Role-based access controls (RBAC) .

While there are multiple solutions available for managing sensitive data in distributed systems, the streaming-ops project uses Bitnami Sealed Secrets , an open source, Kubernetes native, solution. Sealed secrets enable the ability to manage secrets using the same GitOps and declarative approach that is used for all other Kubernetes objects. This case study will show how streaming-ops uses Sealed Secrets with a declarative-based GitOps approach to store Confluent Cloud CLI credentials for use in automated scripts within the Kubernetes cluster.

Sealed Secrets Controller

The main component of the Sealed Secrets solution is the SealedSecret controller. The controller is deployed into the Kubernetes cluster, provides a key for encrypting secrets, monitors the Kubernetes API for changes to SealedSecret types, and deploys decrypted Secret objects to the Kubernetes API for applications to use.

streaming-ops deploys the controller using kubectl and a deployment manifest provided by the Sealed Secret project:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml

For details, see the project’s Makefile.

Sealing Secrets

The first step in sealing secrets is to create a local Secret file that you want to seal. streaming-ops accomplishes this by first storing the secret values in a key-value pair environment file. The ccloud CLI looks for environment variables that can be used for automated login, so the following properties file can be used with the appropriate values:

CCLOUD_EMAIL=emailaddress@email.com
CCLOUD_PASSWORD=YYYYYYYYYY

Next, using the standard kubectl create secret command with the --dry-run=client option, create the local Secret file:

# Ensure usage of ``--dry-run=client`` to ensure creation of local file only
kubectl create secret generic secret-name --namespace=default --from-env-file=ccloud-secrets.props --dry-run=client -o yaml > ccloud-secrets-to-seal.yaml

The Secret file created in the previous step contains the same sensitive data as the input properties file, so protect both of these files in the same manner that you would protect any sensitive data.

Now we can create the SealedSecret file using the Bitnami Sealed Secrets kubeseal command, for example:

kubeseal < ccloud-secrets-to-seal.yaml > ccloud-secrets-sealed.yaml

The resulting ccloud-secrets-sealed.yaml file is encrypted and can only be decrypted by the SealedSecret controller inside your Kubernetes cluster. This makes this file safe to commit to a private or public Git repository.

Deploying Secrets with GitOps

With files sealed using the SealedSecret process, we can use existing GitOps processes to deploy these secrets to the Kubernetes cluster for consumption by automated tools. By committing the SealedSecret files to folders monitored by the GitOps process, they will deployed to the Kubernetes API. You can observe streaming-ops SealedSecret files for the dev environment on GitHub <https://github.com/confluentinc/streaming-ops/tree/master/secrets/sealed/dev>__, which is a folder monitored by the Flux GitOps controller. When Flux is deployed, it is configured to monitor the secrets/sealed folder for changes. For details, see the flux-init.sh script.

Once Flux deploys the SealedSecret objects, they are observed in the Kubernetes API by the SealedSecrets controller that will decrypt the values and post equivalent Secret objects to the Kubernetes API. These Secret objects can then be used by workloads inside Kubernetes using standard Kubernetes methods, like environment variables or volume-mounted files.

For the full details on how streaming-ops uses Sealed Secrets, see the GitHub repository Makefile and scripts.

streaming-ops in action

For more information on this, and to explore other operational use cases, see the complete DevOps for Kafka with Kubernetes and GitOps documentation and the project’s GitHub repository.