Case Study: Graduated Environments¶
The kafka-devops project is a simulated production environment running a streaming microservices based application targeting Apache Kafka® on Confluent Cloud. Applications and resources are managed by GitOps with declarative infrastructure, Kubernetes, and the Operator Pattern.
This case study looks at how applications are managed in “graduated environments”, deploying from lower environments (
dev) to higher (
prd). For more details on this case study, see the blog post Spring Your Microservices into Production with Kubernetes and GitOps.
Customize multiple environments¶
The kafka-devops project currently manages two environments,
prd. The Apache Kafka® based microservices are deployed as Kubernetes Deployments. Controlling the version deployed in each environment is accomplished by using Kustomize to “patch” application versions prior to submitting them to the Kubernetes API. Here is an example patch overriding the deployed version of the orders-service service:
apiVersion: apps/v1 kind: Deployment metadata: name: orders-service spec: template: spec: containers: - name: orders-service image: cnfldemos/orders-service:sha-0ccce99 # 10.0.8
Notice that this is not a full Kubernetes Deployment, only a partial definition. The complete definition is defined in a base folder which contains definitions shared by all of the environments. In this partial definition, we are only overriding the orders-service container image with the version we want deployed to
kafka-devops utilizes Flux to enable GitOps, and prior to a deployment Flux will invoke Kustomize to perform the environment specific patch. This is enabled in the .flux.yaml file which instructs Flux to invoke
version: 1 patchUpdated: generators: - command: kustomize build . patchFile: flux-patch.yaml
kustomize build command patches the orders-service base Deployment definition with the patch from the appropriate environment folder prior to posting the merged manifest to the Kubernetes API.
To deploy an application to
prd, the base definition can be modified to the same version as the
dev patch, and a Pull Request (PR) is filed. Once merged, the Flux controller in the
prd environment deploys the appropriate version. Here is an example PR from the kafka-devops project itself: https://github.com/confluentinc/kafka-devops/pull/62/files
Recovering from failed deployments¶
Rolling back failed deployments is a critical operational function and GitOps can help us greatly when this is required. Using a Git repository as the source of truth for deployment manifests allows you to quickly revert a failed change. From the PR that represents the failed deployment, use your Git providers Revert function to create a new PR to revert the previous change.
Once this new PR is merged, Kubernetes will deploy the previous versions of the application. Keeping deployment changes small and reversible is important to enabling these capabilities in your distributed software operations.