<a id="co-schedule-workloads"></a>

# Configure Pod Scheduling for Confluent Platform Using Confluent for Kubernetes

To get the optimal performance out of Confluent components, you can control how
the component pods are scheduled on Kubernetes nodes. For example, you can
configure pods not to be scheduled on the same node as other resource intensive
applications, pods to be scheduled on dedicated nodes, or pods to be scheduled
on the nodes with the most suitable hardware.

To avoid components competing for the same resources, such as storage or
network, and causing performance degradation in both pods, schedule component
pods in a way that avoids sharing nodes with other critical workloads.

Confluent for Kubernetes supports the following features for scheduling Confluent component
pods on Kubernetes nodes:

* [Taints and tolerations](#co-taints-tolerations)
* [Affinity and anti-affinity](#co-affinity)
* [Pod topology spread constraints](#co-topology-spread-constrainits)
* [One replica per node](#co-one-replica-per-node)

<a id="co-taints-tolerations"></a>

Taints and tolerations
: [Taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration) work
  together to ensure that pods are not scheduled onto specific nodes in
  Kubernetes.
  <br/>
  Taints are applied to a node, and tolerations are applied to pods. Only the pods
  that do not declare a toleration for the taint cannot be scheduled to that node.

<a id="co-affinity"></a>

Affinity and anti-affinity
: A taint and toleration make sure that pods are not scheduled onto the nodes, but
  the features do not guarantee pods get scheduled onto the node where you intend
  them to run.
  <br/>
  To schedule the pods you do want on the right node, use another Kubernetes
  feature, [affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
  Confluent for Kubernetes supports the following types of Kubernetes affinity features:
  <br/>
  * Node affinity
  <br/>
    The node affinity feature allows you to schedule workloads (pods) onto specific
    nodes, for example, to optimize for various resources, such as storage, CPU,
    and networking.
  <br/>
    With Confluent for Kubernetes, you can create node affinity rules to specify which
    Kubernetes nodes your Confluent pods are eligible to be scheduled on. For
    example, if you have special hardware that you want to use to run one or more
    Confluent pods, use node affinity to pin those pods to the special hardware
    nodes.
  * Pod affinity and anti-affinity
  <br/>
    The pod affinity feature allows you to specify that a pod should be scheduled on
    the same node as another pod.
  <br/>
    The pod anti-affinity feature allows you to specify that a set of pods should be
    scheduled away from one another on different nodes, availability zones, or
    several other potential topology domains. For example, you can use pod
    anti-affinity to ensure that the Kafka brokers do not share nodes with other
    resource-intensive or critical workloads.
  <br/>
    The affinity and anti-affinity rules apply at the namespace level.

<a id="co-topology-spread-constrainits"></a>

Pod topology spread constraints
: Using pod topology spread constraints, you can control the distribution of your
  pods across nodes, zones, regions, or other user-defined topology domains,
  achieving high availability and efficient cluster resource utilization.
  <br/>
  You first label nodes to provide topology information, such as regions, zones,
  and nodes. Then in Confluent component CRs, you define spread constraints, such
  as which pods to group together, which topology domains they are spread among,
  and the acceptable skew.
  <br/>
  Pod topology constraints only apply to pods within the same namespace.

<a id="co-one-replica-per-node"></a>

One replica per node
: You can configure CFK to enforce only one replica to run on one Kubernetes
  node.
  <br/>
  This scheduling method guarantees that Confluent Platform components do not compete for system
  resources with other applications running on the same node.
  <br/>
  The rule applies at the namespace level and only to the replicas from the same
  cluster.
  <br/>
  Confluent recommends that you enable the one replica per node setting for Kafka
  and ZooKeeper (Confluent Platform 7.9 and earlier only).
  <br/>
  For other Confluent Platform components, you can enable the one replica per node setting for
  availability, but the setting is not required for reliability.

## Configure taint and toleration for Confluent components

### Configure taint

Taint the nodes using the `kubectl taint nodes` command as described in [Taint
and Toleration](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration).

### Configure toleration

When a pod is configured with a toleration, the pod “tolerates” the taint that
matches the triple `<key,value,effect>` using the matching operator
`<operator>`.

When a pod tolerates a node taint, the pod can be scheduled to the node.

Configure the tolerations property of the Confluent component CRs as follows,
and create or update the resource.

```yaml
spec:
  podTemplate:
    tolerations:
    - effect:                --- [1]
      key:                   --- [2]
      operator:              --- [3]
      value:                 --- [4]
      tolerationSeconds:     --- [5]
```

* [1] Indicates what to do with intolerant pods. Allowed values are
  `NoSchedule`, `PreferNoSchedule`, `NoExecute`.

  When set to an empty value, this toleration matches all taint effects.
* [2] The taint key that the toleration applies to.

  If the `key` is empty, `operator` must be set to `Exists`. This
  combination means to match all values and all keys.
* [3] The match operator to compare key to the value. Allowed operators are
  `Exists` and `Equal`, and `Equal` is the default.

  `Exists` is equivalent to wildcard for value, so that a pod can tolerate all
  taints of the `key`.
* [4] The taint value the toleration matches to. If the `operator` is
  `Exists`, the value must be empty.
* [5] The period of time the toleration tolerates the taint. Only applies when
  `effect` is set to `NoExecute`. Otherwise, this field is ignored.

  By default, it is not set, which means to tolerate the taint forever (do not
  evict).

  Zero and negative values will be treated as `0` (evict immediately).

For example, for the following node taint:

```yaml
kubectl taint nodes node1 myKey=myValue:NoSchedule
```

The following CR snippet matches the taint created above, and the pod can be
schedule onto the node:

```yaml
spec:
  podTemplate:
    tolerations:
    - key: "myKey"
      operator: "Equal"
      value: "myValue"
      effect: "NoSchedule"
```

For more information on the fields, see [Taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration).

## Configure node affinity for Confluent components

This section describes how to configure node affinity scheduling rules for
Confluent component pods.

Using node affinity, you specify which nodes Confluent component pods can run on
based on labels on the nodes (`matchExpressions`) or on node fields
(`matchFields`). Each match requirement is a `<key, operator, value>`
triple.

Two types of node affinity rules are supported:

* `requiredDuringSchedulingIgnoredDuringExecution`

  A pod must satisfy these rules to be scheduled onto a node.
* `preferredDuringSchedulingIgnoredDuringExecution`

  The scheduler prefers to schedule the pods to the nodes that satisfy the
  specified affinity expression, but it may choose a node that violates one or
  more of the expressions.

  The node with the greatest sum of weights is most preferred.

  An empty preferred scheduling term matches all objects with implicit weight 0.

  A null preferred scheduling term matches no objects.

### Configure node affinity with requiredDuringSchedulingIgnoredDuringExecution

The pod can be scheduled to a node if one of the `nodeSelectorTerms` matches.

A `nodeSelectorTerms` item is a match if all `matchExpressions` match.

To create the node affinity rules to match node labels, configure the
`nodeAffinity` property of the Confluent components CRs as follows and create
or update the resource. Set either `matchExpressions` or `matchFields`.

```yaml
spec:
  podTemplate:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:     --- [1]
            - matchExpressions:  --- [2]
                - key:           --- [3]
                  operator:      --- [4]
                  values:        --- [5]
              matchFields:       --- [6]
                - key:           --- [7]
                  operator:      --- [8]
                  values:        --- [9]
```

* [1] Required. A list of node selector terms.
* [2] A list of node selector requirements by node’s labels.

  A node selector requirement is a selector that contains `values`, a `key`,
  and an `operator` that relates the `key` and `values`.
* [3] Required. The label key that the selector applies to.
* [4] Required. Represents how the `key` is related to the `values`.
  Valid `operators` are `In`, `NotIn`, `Exists`, `DoesNotExist`,
  `Gt`, and `Lt`.
* [5] An array of string values.

  If the `operator` is `In` or `NotIn`, the `values` array must be
  non-empty.

  If the  `operator` is `Exists` or `DoesNotExist`, the values array must
  be empty.

  If the `operator` is `Gt` or `Lt`, the `values` array must have a
  single element, which will be interpreted as an integer.
* [6] A list of node selector requirements by node’s fields.

  A node selector requirement is a selector that contains `values`, a `key`,
  and an `operator` that relates the `key` and `values`.
* [7] Required. The field key that the selector applies to.
* [8] Required. Represents how the `values` are related to the `key`.
  Valid operators are `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`,
  and `Lt`.
* [9] An array of string values.

  If the `operator` is `In` or `NotIn`, the `values` array must be
  non-empty.

  If the `operator` is `Exists` or `DoesNotExist`, the `values` array
  must be empty.

  If the `operator` is `Gt` or `Lt`, the values array must have a single
  element, which will be interpreted as an integer.

An example CR snippet:

```yaml
spec:
  podTemplate:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: "node-role.kubernetes.io/kafka-connect"
              operator: In
              values: ["true"]
```

### Configure node affinity with preferredDuringSchedulingIgnoredDuringExecution

To create the node affinity rules to match node selector expressions or node
selector fields,  Configure the Confluent components CRs as follows and create
or update the resource. Set either `matchExpressions` or `matchFields`.

```yaml
spec:
  podTemplate:
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:            --- [1]
            matchExpressions:    --- [2]
              key:               --- [3]
              operator:          --- [4]
              values:            --- [5]
            matchFields:         --- [6]
            - key:               --- [7]
              operator:          --- [8]
              values:            --- [9]
          weight:                --- [10]
```

* [1] Required. A node selector term, associated with the corresponding weight.
* [2] A list of node selector requirements by node’s labels.
* [3] Required. The label key that the selector applies to.
* [4] Required. Represents how the `key` is related to the `values`. Valid
  operators are `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, and
  `Lt`.
* [5] An array of string values.

  If the `operator` is `In` or `NotIn`, the `values` array must be
  non-empty.

  If the `operator` is `Exists` or `DoesNotExist`, the `values` array
  must be empty.

  If the `operator` is `Gt` or `Lt`, the values array must have a single
  element, which will be interpreted as an integer.
* [6] A list of node selector requirements.
* [7] Required, The label key that the selector applies to.
* [8] Required. Represents how the `key` is related to the `values`. Valid
  operators are `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, and
  `Lt`.
* [9] An array of string values.

  If the `operator` is `In` or `NotIn`, the `values` array must be
  non-empty.

  If the `operator` is `Exists` or `DoesNotExist`, the `values` array
  must be empty.

  If the `operator` is `Gt` or `Lt`, the values array must have a single
  element, which will be interpreted as an integer.
* [10] Required. Weight associated with matching the corresponding
  `nodeSelectorTerm` in the range `1` - `100`.

## Configure pod affinity and anti-affinity for Confluent components

This section describes how to configure pod affinity and pod anti-affinity
scheduling rules, for example, whether to co-locate or to avoid putting a pod in
the same node or zone as some other pods already running on the node.

Use `topologyKey` to specify the nodes to co-locate or avoid scheduling pods.
The label on the nodes on which selected pods are running must match the
`topologyKey`.

`topologyKey` is typically used to indicate the granularity at which the
affinity rule is applied.

For example, if you do not want to run Kafka with some other applications on the
same node, set the `topologyKey` to `kubernetes.io/hostname` to specify that
the Kafka pods should not be scheduled on nodes with the same hostname.

For another example, if you set the `topologyKey` to
`topology.kubernetes.io/zone` for an anti-affinity rule, and a node in zone
`us-central1-a` is running a Kafka pod. The scheduler looks for a node in a
different zone with the node label value of `topology.kubernetes.io/zone` is
not `us-central1-a`.

Two types of pod affinity/anti-affinity rules are supported:

* `requiredDuringSchedulingIgnoredDuringExecution`

  A pod must satisfy these rules to be scheduled onto a node.
* `preferredDuringSchedulingIgnoredDuringExecution`

  The scheduler prefers to schedule the pods to nodes that satisfy the specified
  affinity expression, but it may choose a node that violates one or more of the
  expressions.

  The node with the greatest sum of weights is most preferred.

Pod affinity and anti-affinity apply within a namespace as pods are namespaced.

### Configure pod affinity with preferredDuringSchedulingIgnoredDuringExecution

Configure the `affinity` property of the Confluent components CRs as follows
and create or update the resources.

```yaml
spec:
  podTemplate:
    affinity:
      podAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:     --- [1]
              namespaces:        --- [2]
              labelSelector:     --- [3]
              topologyKey:       --- [4]
          weight:                --- [5]
```

* [1] Required. A pod affinity term, associated with the corresponding weight.
* [2] The namespaces the `labelSelector` applies to (matches
  against). Null or empty list means the namespace of this pod.
* [3] A query used to find matching pods. See [Pod label selector](#co-label-selector) for detail.
* [4] Required. The label of the cluster node.

  This pod should be co-located (affinity) or not co-located (anti-affinity)
  with the pods matching the `labelSelector` in the specified namespaces.
  Co-located is defined as running on a node whose value of the label with
  `topologyKey` matches that of any node on which any of the selected pods are
  running.

  An empty `topologyKey` is not allowed.
* [5] Required. The weight associated with matching the corresponding
  `podAffinityTerm` in the range `1` -  `100`.

### Configure pod affinity and pod anti-affinity with requiredDuringSchedulingIgnoredDuringExecution

Configure the `affinity` property of the Confluent components CRs as follows
and create or update the resources.

```yaml
spec:
  podTemplate:
    affinity:
      podAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:      --- [1]
          namespaces:           --- [2]
          topologyKey:          --- [3]
```

* [1] A query used to find matching pods. See [Pod label selector](#co-label-selector) for
  detail.
* [2] The namespaces the `labelSelector` applies to (matches
  against). Null or empty list means the namespace of this pod.
* [3] Required. The label of the cluster node.

  This pod should be co-located (affinity) or not co-located (anti-affinity)
  with the pods matching the `labelSelector` in the specified namespaces.

  An empty `topologyKey` is not allowed.

<a id="co-label-selector"></a>

### Pod label selector

Use `labelSeletor` in pod affinity, pod anti-affinity, and pod topology spread
constraints to find matching pods to apply the constraints.

```yaml
labelSelector:
  matchExpressions:    --- [1]
  - key:               --- [2]
    operator:          --- [3]
    values:            --- [4]
  matchLabels:         --- [5]
```

* [1] A list of label selector requirements. The requirements are ANDed.

  A selector contains values, a key, and an operator that relates the key and
  values.
* [2] Required. The label key that the selector applies to.
* [3] Required. Represents how the key is related to a set of values. Valid
  operators are `In`, `NotIn`, `Exists`, and ``DoesNotExist`.
* [4] An array of string values.

  If the operator is `In` or `NotIn`, the `values` array must be non-empty.

  If the operator is `Exists` or `DoesNotExist`, the `values` array must
  be empty.
* [5] A map of {key,value} pairs. The requirements are ANDed.

CFK adds the following labels you can use in `labelSelector`:

| Label key                  | Label value    |
|----------------------------|----------------|
| app                        | <cluster name> |
| clusterId                  | operator       |
| confluent-platform         | “true”         |
| platform.confluent.io/type | <CR type>      |
| type                       | <cluster name> |

An example CR snippet to not schedule two Connect apps onto the same host:

```yaml
spec:
  podTemplate:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - connect
            topologyKey: "kubernetes.io/hostname"
```

## Configure pod topology spread constraints

`TopologySpreadConstraints` describes how matching Confluent pods should be
scheduled across topology domains.

All topologySpreadConstraints are ANDed.

Specify the following in the Confluent component CRs that you want to apply pod
topology spread constraints to:

```yaml
spec:
  podTemplate:
    topologySpreadConstraints:
      - labelSelector:           --- [1]
        maxSkew:                 --- [2]
        topologyKey:             --- [3]
        whenUnsatisfiable:       --- [4]
```

* [1] Used to find matching pods. Pods that match this label selector are
  counted to determine the number of pods in their corresponding topology
  domain. See [Pod label selector](#co-label-selector) for detail.
* [2] Required. Describes the degree to which pods may be unevenly distributed.
  Default value is `1` for the most even distribution. `0` is not allowed.
* [3] Required. The key of node labels. Nodes that have a label with this
  key and identical values are considered to be in the same topology.
* [4] Required. Indicates how to deal with a pod if it doesn’t satisfy
  the spread constraint. Allowed values are `DoNotSchedule` (default) and
  `ScheduleAnyway`.

The following example specifies to evenly balance Kafka pod across all worker
nodes:

```yaml
spec:
  podTemplate:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            type: kafka-pods
```

## Configure one replica per node

The `oneReplicaPerNode` property enforces to run one pod per node through the
pod anti-affinity capability, assigning a dedicated node for the pod workload.

The `oneReplicaPerNode` is disabled by default.

When this property is enabled (`oneReplicaPerNode: true`), the pod
anti-affinities are enabled.

When you change this property in an existing cluster, the cluster will roll.

Configure the `oneReplicaPerNode` property for the Confluent CRs as follows
and create or update the resources.

```yaml
spec:
  oneReplicaPerNode:    # Set it to true or false.
```