Configure CPU and Memory
Set proper CPU and memory requests and limits to maintain the performance of your Confluent Platform clusters and applications.
Before configuring CPU and memory resource requirements for Confluent Platform, review Plan Capacity and Sizing for Confluent for Kubernetes for resource allocation planning.
Specify CPU and memory requests and limits
Confluent for Kubernetes (CFK) allows you to define custom pod resource requirements for
Confluent Platform components it deploys. You specify these requirements using the
requests and limits properties for components in their custom
resources (CR).
Requests and limits are dependent on your workload. As a best practice, follow the minimum requirements for Confluent Platform, and then benchmark and tune if needed to suit your environment.
For more information about CPU and memory resources in Kubernetes, see Resource Management for Pods and Containers.
spec:
podTemplate:
resources:
limits:
cpu:
memory:
requests:
cpu:
memory:
Field |
Description |
|---|---|
|
Describes the compute resource requirements for this component custom resource (CR). |
|
Describes the maximum amount of compute resources allowed. If your Confluent Platform component tries to use more resources than the values set here, it throttles. |
|
Describes the minimum amount of compute resources required. If the
|
|
Limits and requests for CPU resources are measured in CPU units. 1 CPU unit is equivalent to 1 physical CPU core or 1 virtual core. Fractional CPU requests are allowed. For example,
For CPU resource units, the quantity expression |
|
Limits and requests for memory are measured in bytes. |
The following example custom resource (CR) specifies requests for 0.25 CPU and 64 MiB of memory. The limits are set to 0.5 CPU and 128 MiB of memory.
spec:
podTemplate:
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Resource definitions for CFK init container
CFK does not allow you to configure the resource requests or limits for the init container. For each Confluent Platform deployment, CFK sets the following for the init container:
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
Define Java heap size
In addition to memory sizing, you can configure the Java Virtual Machine (JVM)
heap size for Confluent components using the configuration override feature
(spec.configOverrides.jvm).
For guidance on setting JVM heap size, see the Tuning defaults for containers article.
For the full list of JVM flags that CFK sets by default and how to override or remove them, see Default JVM settings.
Recommended heap size for Kafka and KRaft
By default, CFK sets the JVM heap (-Xms/-Xmx) for Apache Kafka® brokers and
KRaft controllers to 100% of the pod memory request. This leaves no memory
for the OS page cache that Kafka relies on for efficient log reads.
As a best practice, pin a smaller heap explicitly using
spec.configOverrides.jvm, with the same ratios that later CFK
versions set automatically:
Kafka broker:
min(50% of requests.memory, 32Gi)KRaft controller:
min(80% of requests.memory, 32Gi)
For example, on a broker with a 6Gi memory request:
kind: Kafka
spec:
configOverrides:
jvm:
- "-Xms3G"
- "-Xmx3G"
If you are running CFK 3.4.0 or later, this is already the default and you don’t need to take any action.
Remove the default JVM settings
To auto scale the JVM heap size
(UseContainerSupport) or to set the JVM heap size (MaxRAMPercentage), you must remove the
existing JVM -Xms and -Xmx settings that CFK sets by default.
To remove them, add the following overrides to the component CR:
kind: <component>
spec:
configOverrides:
jvm:
- "---Xmx"
- "---Xms"
Specify the overrides exactly as shown, with no value. CFK removes the default
setting only when the override is exactly ---Xmx or ---Xms. An override
that includes a value, such as ---Xmx4G, does not remove the default.
Auto scale JVM heap size
To auto scale the JVM heap size, enable UseContainerSupport by adding the
following to the component custom resource (CR):
kind: <component>
spec:
configOverrides:
jvm:
- "-XX:+UseContainerSupport"
Set JVM heap size
To set the JVM heap size, in the component custom resource (CR), set
MaxRAMPercentage to match the desired JVM heap size in relation to the
total memory limit (spec.podTemplate.resources.limits.memory) you set
in Specify CPU and memory requests and limits.
kind: <component>
spec:
configOverrides:
jvm:
- "-XX:MaxRAMPercentage=<percentage value>"
The JVM calculates the heap as a percentage of the entire Kubernetes node
memory, not the pod memory, when you set -XX:MaxRAMPercentage but do not set
-Xms, -Xmx, pod memory requests
(spec.podTemplate.resources.requests.memory), or pod memory limits
(spec.podTemplate.resources.limits.memory). In this example, the Java heap
is 50% of the node memory.
kind: <component>
spec:
configOverrides:
jvm:
- "-XX:MaxRAMPercentage=50.0"
To make the heap a percentage of the pod memory instead of the node memory,
you must set pod memory limits (spec.podTemplate.resources.limits.memory).
In this example, the Java heap is 50% of the pod’s memory limit (4Gi), not
50% of the node memory.
kind: <component>
spec:
podTemplate:
resources:
limits:
memory: "4Gi"
configOverrides:
jvm:
- "-XX:MaxRAMPercentage=50.0"
Examples for JVM heap size configuration
The following examples show the JVM MaxRAMPercentage set to 50% of the
memory and auto-scaling enabled.
Example 1: In this example, only limits is set (4Gi) for the pod. To
remove the -Xmx and -Xms, add the ---Xmx and ---Xms overrides.
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
name: kafka
namespace: confluent
spec:
podTemplate:
resources:
limits:
memory: "4Gi"
configOverrides:
jvm:
- "---Xmx" # This removes -Xmx parameter
- "---Xms" # This removes -Xms parameter
- "-XX:+UseContainerSupport"
- "-XX:MaxRAMPercentage=50.0"
Example 2: In this example, both limits and requests are set for the
pod. To remove the -Xmx and -Xms, add the same ---Xmx and ---Xms
overrides. The limits and requests values do not change the overrides.
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
name: kafka
namespace: confluent
spec:
podTemplate:
resources:
limits:
memory: "4Gi"
requests:
memory: "2Gi"
configOverrides:
jvm:
- "---Xmx" # This removes -Xmx parameter
- "---Xms" # This removes -Xms parameter
- "-XX:+UseContainerSupport"
- "-XX:MaxRAMPercentage=50.0"
For more details about the configuration overrides feature, see Configuration overrides.