Case Study: Manage Cloud Secrets¶
The kafka-devops project is a simulated production environment running a streaming microservices based application targeting Apache Kafka® on Confluent Cloud. Applications and resources are managed by GitOps with declarative infrastructure, Kubernetes, and the Operator Pattern.
This case study looks at a method for managing secrets used to access Confluent Cloud for automated systems inside of a Kubernetes cluster.
Kubernetes provides a declarative API for managing application deployments, configurations, and other related metadata. Included in the API is a Secrets type that can be used to store and manage sensitive information, such as passwords, SSH keys, or other secret data.
Kubernetes objects of type
Secret do not fully protect sensitive data. Data contained within a
Secret is not encrypted and can be viewed by any Kubernetes user with read access permission to the object. To secure
Secret objects and prevent unauthorized access, use Kubernetes Role-based access controls (RBAC) .
While there are multiple solutions available for managing sensitive data in distributed systems, the kafka-devops project uses Bitnami Sealed Secrets , an open source, Kubernetes native, solution. Sealed secrets enable the ability to manage secrets using the same GitOps and declarative approach that is used for all other Kubernetes objects. This case study will show how kafka-devops uses Sealed Secrets with a declarative-based GitOps approach to store Confluent Cloud CLI credentials for use in automated scripts within the Kubernetes cluster.
Sealed Secrets Controller¶
The main component of the Sealed Secrets solution is the
SealedSecret controller. The controller is deployed into the Kubernetes cluster, provides a key for encrypting secrets, monitors the Kubernetes API for changes to
SealedSecret types, and deploys decrypted
Secret objects to the Kubernetes API for applications to use.
kafka-devops deploys the controller using
kubectl and a deployment manifest provided by the Sealed Secret project:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml
For details, see the project’s Makefile.
The first step in sealing secrets is to create a local
Secret file that you want to seal. kafka-devops accomplishes this by first storing the secret values in a key-value pair environment file. The
ccloud CLI looks for environment variables that can be used for automated login, so the following properties file can be used with the appropriate values:
Next, using the standard
kubectl create secret command with the
--dry-run=client option, create the local
# Ensure usage of ``--dry-run=client`` to ensure creation of local file only kubectl create secret generic secret-name --namespace=default --from-env-file=ccloud-secrets.props --dry-run=client -o yaml > ccloud-secrets-to-seal.yaml
Secret file created in the previous step contains the same sensitive data as the input properties file, so protect both of these files in the same manner that you would protect any sensitive data.
Now we can create the
SealedSecret file using the Bitnami Sealed Secrets
kubeseal command, for example:
kubeseal < ccloud-secrets-to-seal.yaml > ccloud-secrets-sealed.yaml
ccloud-secrets-sealed.yaml file is encrypted and can only be decrypted by the
SealedSecret controller inside your Kubernetes cluster. This makes this file safe to commit to a private or public Git repository.
Deploying Secrets with GitOps¶
With files sealed using the
SealedSecret process, we can use existing GitOps processes to deploy these secrets to the Kubernetes cluster for consumption by automated tools. By committing the
SealedSecret files to folders monitored by the GitOps process, they will deployed to the Kubernetes API. You can observe kafka-devops
SealedSecret files for the
dev environment on GitHub <https://github.com/confluentinc/kafka-devops/tree/master/secrets/sealed/dev>__, which is a folder monitored by the Flux GitOps controller. When Flux is deployed, it is configured to monitor the
secrets/sealed folder for changes. For details, see the flux-init.sh script.
Once Flux deploys the
SealedSecret objects, they are observed in the Kubernetes API by the
SealedSecrets controller that will decrypt the values and post equivalent
Secret objects to the Kubernetes API. These
Secret objects can then be used by workloads inside Kubernetes using standard Kubernetes methods, like environment variables or volume-mounted files.