Case Study: Graduated Environments

The streaming-ops project is a simulated production environment running a streaming microservices based application targeting Apache Kafka® on Confluent Cloud. Applications and resources are managed by GitOps with declarative infrastructure, Kubernetes, and the Operator Pattern.

This case study looks at how applications are managed in “graduated environments”, deploying from lower environments (dev) to higher (prd). For more details on this case study, see the blog post Spring Your Microservices into Production with Kubernetes and GitOps.

Customize multiple environments

The streaming-ops project currently manages two environments, dev and prd. The Apache Kafka® based microservices are deployed as Kubernetes Deployments. Controlling the version deployed in each environment is accomplished by using Kustomize to “patch” application versions prior to submitting them to the Kubernetes API. Here is an example patch overriding the deployed version of the orders-service service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
spec:
  template:
    spec:
      containers:
      - name: orders-service
        image: cnfldemos/orders-service:sha-0ccce99 # 10.0.8

Notice that this is not a full Kubernetes Deployment, only a partial definition. The complete definition is defined in a base folder which contains definitions shared by all of the environments. In this partial definition, we are only overriding the orders-service container image with the version we want deployed to dev.

streaming-ops utilizes Flux to enable GitOps, and prior to a deployment Flux will invoke Kustomize to perform the environment specific patch. This is enabled in the .flux.yaml file which instructs Flux to invoke kustomize build.

version: 1
patchUpdated:
  generators:
    - command: kustomize build .
  patchFile: flux-patch.yaml

The kustomize build command patches the orders-service base Deployment definition with the patch from the appropriate environment folder prior to posting the merged manifest to the Kubernetes API.

To deploy an application to prd, the base definition can be modified to the same version as the dev patch, and a Pull Request (PR) is filed. Once merged, the Flux controller in the prd environment deploys the appropriate version. Here is an example PR from the streaming-ops project itself: https://github.com/confluentinc/streaming-ops/pull/62/files

Recovering from failed deployments

Rolling back failed deployments is a critical operational function and GitOps can help us greatly when this is required. Using a Git repository as the source of truth for deployment manifests allows you to quickly revert a failed change. From the PR that represents the failed deployment, use your Git providers Revert function to create a new PR to revert the previous change.

../../../_images/revert.png

Once this new PR is merged, Kubernetes will deploy the previous versions of the application. Keeping deployment changes small and reversible is important to enabling these capabilities in your distributed software operations.

streaming-ops in action

For more information on this, and to explore other operational use cases, see the complete DevOps for Apache Kafka® with Kubernetes and GitOps documentation and the streaming-ops project’s GitHub repository.