CPU and Memory Resources for Confluent Platform in Confluent for Kubernetes

Setting proper requests and limits is important for the performance of the Confluent Platform clusters and their applications.

Before configuring CPU and memory resource requirements for Confluent Platform, review Cluster sizing for resource allocation planning.

Requests and limits are dependent on your workload. The best practice should follow the minimum requirements for Confluent Platform, and then benchmark and tune if needed to suit your environment.

For more information about CPU and memory resources in Kubernetes, see Resource Management for Pods and Containers.

Confluent for Kubernetes (CFK) allows you to define custom pod resource requirements for Confluent Platform components it deploys. You specify these requirements using the requests and limits properties for components in their custom resources (CR).

spec:
  podTemplate:
    resources:  --- [1]
      limits:   --- [2]
        cpu:    --- [3]
        memory: --- [4]
      requests: --- [5]
        cpu:    --- [6]
        memory: --- [7]

  • [1] resources describe the compute resource requirements for this component CR.

  • [2] limits describe the maximum amount of compute resources allowed. Your Confluent Platform component will throttle if it tries to use more resources than the values set here.

  • [3] [6] Limits and requests for CPU resources are measured in CPU units. 1 CPU unit is equivalent to 1 physical CPU core or 1 virtual core.

    Fractional CPU requests are allowed. For example, resources.requests.cpu: 0.5 requests half as much CPU time.

    For CPU resource units, the quantity expression 0.1 is equivalent to the expression 100m, which means “one hundred milli CPU”.

  • [4] [7] Limits and requests for memory are measured in bytes.

  • [5] requests describe the minimum amount of compute resources required. If the requests section is omitted, it defaults to limits if that is explicitly specified in the same CR, and otherwise to the values defined for the Kubernetes cluster.

The following example CR specifies a request for 0.25 CPU and 64 MiB of memory. Its limits are set to 0.5 CPU and 128 MiB of memory.

spec:
  podTemplate:
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Define Java heap size

In addition to setting memory sizing, you need to define the the Java (JVM) heap size for the Confluent Platform components.

You define the Java heap size for Confluent Platform using the configuration override feature (spec.configOverrides.jvm) in the component CR.

Here’s an example where the JVM heap size is set to 6 GB:

spec:
  configOverrides:
    server:
      - auto.create.topics.enable=true
      - confluent.cluster.link.enable=true
    jvm:
      - "-Xmx6g"                # Set to 6 GB
      - "-XX:MetaspaceSize=96m"
      - "-XX:+UseG1GC"

For more details about the configuration overrides feature, see Configuration overrides.

Resource definitions for CFK Init Container

CFK does not allow you to configure the resource requests or limits for the Init Container. For each Confluent Platform deployment, CFK sets the following for the Init Container:

resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 512Mi