CPU and Memory Resources for Confluent Platform

Before configuring CPU and memory resource requirements, review Cluster sizing for resource allocation planning.

Setting proper requests and limits is important for the performance of the Confluent Platform clusters and their applications.

For more information about CPU and memory resources in Kubernetes, see Resource Management for Pods and Containers.

Confluent for Kubernetes (CFK) allows you to define custom pod resource requirements for Confluent Platform components it deploys. You specify these requirements using the requests and limits properties for components in their custom resources (CR).

spec:
  podTemplate:
    resources:  --- [1]
      limits:   --- [2]
        cpu:    --- [3]
        memory: --- [4]
      requests: --- [5]
        cpu:    --- [6]
        memory: --- [7]

  • [1] resources describe the compute resource requirements for this component CR.

  • [2] limits describe the maximum amount of compute resources allowed. Your Confluent Platform component will throttle if it tries to use more resources than the values set here.

  • [3] [6] Limits and requests for CPU resources are measured in CPU units. 1 CPU unit is equivalent to 1 physical CPU core or 1 virtual core.

    Fractional CPU requests are allowed. For example, resources.requests.cpu: 0.5 requests half as much CPU time.

    For CPU resource units, the quantity expression 0.1 is equivalent to the expression 100m, which means “one hundred milli CPU”.

  • [4] [7] Limits and requests for memory are measured in bytes.

  • [5] requests describe the minimum amount of compute resources required. If the requests section is omitted, it defaults to limits if that is explicitly specified in the same CR, and otherwise to the values defined for the Kubernetes cluster.

The following example CR specifies a request for 0.25 CPU and 64 MiB of memory. Its limits are set to 0.5 CPU and 128 MiB of memory.

spec:
  podTemplate:
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Define Java heap size

In addition to setting memory sizing, you need to define the the Java (JVM) heap size.

You define the Java heap size for Confluent Platform using the configuration override feature (spec.configOverrides.jvm) in the component CR.

Here’s an example where the JVM heap size is set to 6 GB:

spec:
  configOverrides:
    server:
      - auto.create.topics.enable=true
      - confluent.cluster.link.enable=true
    jvm:
      - "-Xmx6g"                # Set to 6 GB
      - "-XX:MetaspaceSize=96m"
      - "-XX:+UseG1GC"

For more details about the configuration overrides feature, see Configuration overrides.