Confluent for Kubernetes Blueprints Overview

Confluent for Kubernetes (CFK) Blueprints extends CFK with a new set of Kubernetes-native APIs (CustomResourceDefinitions) that further simplifies and automates the configuration and deployment of Confluent Platform.

Using CFK Blueprints, you define standardized Confluent deployment templates that your application teams can use to deploy Confluent Platform in one or more Kubernetes clusters.

You can use CFK Blueprints to manage Confluent Platform deployments in a single Kubernetes cluster or across multiple Kubernetes clusters. For clarity, the term Control Plane and Data Plane will be used in the CFK Blueprints user guide. In the case of single-cluster deployment, the Control Plane and Data Plane will be in the same Kubernetes cluster.

CFK Blueprints architecture

CFK Blueprints consists of Orchestrator, Agent, and CFK.

../_images/tech-overview-3.png
Orchestrator

The Orchestrator is a set of controllers that service the Kubernetes APIs to:

  • Generate and coordinate CFK APIs to deploy Confluent Platform.
  • Manage the status of the Blueprints components.
  • Support dynamic registration of Data Plane clusters.
  • Provide unified orchestration for Confluent Platform on multiple cluster regions.
  • Manage credential and certificate lifecycle that includes automatic cluster rolls.
Agent

The Agent initiates connection to the Orchestrator to perform necessary tasks, such as to:

  • Pull the needed resources from the Orchestrator to run Confluent Platform.
  • Send the health probe to the Orchestrator.
CFK
CFK is the operator to manage Confluent Platform in the Data Plane.

You can deploy CFK Blueprints in the following configurations:

  • Single Kubernetes cluster where Orchestrator, Agents, CFK, and Confluent Platform are deployed.
  • Multiple Kubernetes clusters, with one Control Plane and many Data Plane Kubernetes clusters.
    • Orchestrator is deployed in the Control Plane Kubernetes cluster to create and manage the custom resources.
    • Agents and CFK are deployed in the Data Plane Kubernetes clusters.
    • Confluent Platform components run in the Data Plane Kubernetes clusters.

CFK Blueprints custom resources

CFK Blueprints Orchestrator manages the following custom resources (CRs).

Infrastructure (KubernetesCluster)

A KubernetesCluster CR is an immutable representation of a Data Plane Kubernetes cluster. It contains a unique cluster ID, information about regions and zones of a Data Plane cluster.

The Orchestrator uses the information to propagate CFK APIs to the right Kubernetes cluster. lets also add identity as unique identifier

Health Check (CPCHealthCheck)

CPCHealthCheck Periodically receives probes from all registered Agents about their availability.

The Orchestrator uses this resource to allow or deny user requests based on the availability of services.

Confluent Platform Blueprint (ConfluentPlatformBlueprint)

A Blueprint defines the configuration and shape of how Confluent should be deployed. It specifies component configurations, such as security (authentication, authorization, encryption), networking, container registry, etc.

Use a ConfluentPlatformBlueprint CR to standardize how Confluent should be deployed across your organization.

Blueprint references Confluent component classes CRs that represent corresponding Confluent components.

../_images/tech-overview-4.png
Confluent Platform Cluster Classes

Confluent Platform cluster classes are part of a Blueprint, and they provide a way for platform administrators to describe the classes of different Confluent deployments which helps to standardize Confluent deployments.

Cluster classes CRs contain information about Confluent Platform components, such as:

  • Provisioner types: CFK or Confluent Cloud
  • Min and max replicas
  • Min and max volume capacity
  • Confluent configurations, such as config overrides, images, etc.
Confluent Platform Component Clusters

A Confluent Platform component cluster is an instantiations of a Blueprint and a component class on a target Kubernetes infrastructure.

A Deployment is an instantiation of a component cluster. You create a deployment from the Control Plane to the Data Plane.

../_images/tech-overview-2.png

Requirements

The following are the requirements for CFK Blueprints:

  • Helm 3 on your local machine.
  • kubectl command-line tool on your local machine.
  • Kubernetes versions 1.22+ in the Kubernetes clusters you want to use for the Control Plane and the Data Plane.
  • Confluent for Kubernetes, versions 2.4 and higher.
  • Confluent Platform, versions 7.4 and higher.

Workflow

A typical workflow described in this section employs two team personas:

  • The Platform Team installs and manages: Kubernetes cluster CR, Blueprint CR, Confluent Platform cluster class CR
  • The Application Dev Team installs and manages: Confluent Platform cluster CRs
Day 0
  • The Platform Team sets up the Control Plane infrastructure.
  • The Platform Team registers the target Data Plane Kubernetes infrastructure.
  • The Platform Team defines Blueprints.
Day 1
  • Either the Application Dev Team or the Platform Team selects a Blueprint for a Confluent Platform deployment.
  • Either the Application Dev Team or the Platform Team administers deployment provisioning.
Day 2
  • The Application Dev Team creates application resources for a provisioned Confluent Platform deployment.
  • The Application Dev Team creates “global” application resources across multiple provisioned deployments.