Scale Storage with Confluent for Kubernetes¶

Scale up disks¶

The ability to scale up disks is important to successfully use Confluent Platform at scale. For instance, as your organization’s usage of Kafka grows, you may experience a large growth in the number of Kafka topics, and expanding disks help you ensure you have enough disk space to accommodate the data in those topics.

Confluent for Kubernetes (CFK) enables simple, automated expansion of storage. In a supported environment, you can take a simple workflow to expand all the disks associated with a cluster, for example, ZooKeeper, Kafka. And after some time, all pods within the cluster are able to leverage expanded disk space without any interruption. In some cases, this expansion can be completed across an entire cluster in just a few seconds.

Requirements¶

Expanding a disk requires both resizing the PersistentVolumeClaim (PVC) and expanding the file system exposed to the pod where the given associated PersistentVolume (PV) is mounted.

The ability to automatically resize a PVC depends on the volume types you use and how the StorageClass associated with the PVCs is configured. For further details on PVC expansion support, see Expanding Persistent Volumes Claims in Kubernetes.

To scale up Confluent Platform storage, the following must be true of your environment:

The StorageClass associated with your Confluent Platform component cluster must have set allowVolumeExpansion: true.
The underlying volume type must support expansion.

At the time of this writing, GCE Persistent Disk, AWS EBS, Azure Disk, and others support expansion, while Host Path and other types of volumes do not.
The ExpandInUsePersistentVolumes feature must be enabled on your Kubernetes cluster.

Once a PVC is expanded, the ability to automatically expand the file system exposed to a pod without restarting the pod requires the ExpandInUsePersistentVolumes feature to be enabled on your Kubernetes cluster. For Kubernetes clusters with version 1.15 and higher, this feature is enabled by default. For further details on expanding in-use PVCs, see Resizing an in-use PersistentVolumeClaim.

Expand storage¶

To expand the disk associated with a Confluent component cluster:

Update dataVolumeCapacity in your Confluent component custom resource (CR).

For example, to increases the disk for Kafka brokers to 100 Gi:
```
kind: Kafka
metadata:
  name: kafka
spec:
  dataVolumeCapacity: 100Gi
```
Apply the CR change by running the kubectl apply command.

Note

If you need to scale up PersistentVolume disks in Azure, contact Confluent Support for guidance.

Scale down disks¶

Scaling down disks is not supported in the current version of CFK.