Configure SQL Compilers and Compile Workers for Confluent Manager for Apache Flink

Confluent Manager for Apache Flink® (CMF) compiles every Flink SQL statement before it runs. Starting with version 2.5, CMF compiles statements in separate compiler processes, one for each Flink line, so that one CMF installation serves compute pools on different Flink versions. A line is a major and minor version, such as 1.19 or 2.3. There are two kinds of compilers:

Shared compilers: Containers in the CMF pod, one for each Flink line. They are on by default, with a compiler for Flink 1.19 and one for Flink 2.3. Shared compilers cannot load artifacts.
Compile workers: A separate pod, cmf-compile-worker, in the CMF namespace. Compile workers are the only compilers that load artifacts, so statements that use user-defined functions or custom connectors and formats from artifacts need them. The chart adds one compile worker by default when cmf.artifacts.enabled and cmf.sql.environmentCatalog.enabled are both true, and none otherwise.

You configure both with the cmf.statementService Helm values.

Where a statement compiles

Where CMF compiles a statement depends on your Helm values and on whether the statement’s environment has artifacts. An environment has artifacts when cmf.artifacts.enabled and cmf.sql.environmentCatalog.enabled are both true and the environment has at least one artifact of any kind, including an application JAR. All statements of such an environment compile on the compile workers, including those that use no artifact.

Helm values	Environments without artifacts	Environments with artifacts
Shared compilers and compile workers, the default when `cmf.artifacts.enabled` and `cmf.sql.environmentCatalog.enabled` are both `true`	Shared compilers	Compile workers
Shared compilers and no compile workers, the default when either feature is off, or with `workers.count: 0`	Shared compilers	Shared compilers. Statements that use an artifact fail.
`sharedCompilers: []` and compile workers	Compile workers	Compile workers
`sharedCompilers: []`, no compile workers	The `cmf` container, which compiles for Flink 1.19 only	The `cmf` container. Statements that use an artifact fail.

If an environment has artifacts but its SQL statements use none, for example because its only artifacts are application JARs, its statements still compile on the compile workers. Give the workers what these statements need, as described in Give the workers what catalogs and connectors need, or set workers.count: 0 to keep them on the shared compilers.

Shared compilers

Each entry in cmf.statementService.sharedCompilers adds a container to the CMF pod that compiles statements for one Flink line. If you do not set the value, the CMF pod has these two compilers:

cmf:
  statementService:
    sharedCompilers:
      - version: "1.19"
        port: 8095
      - version: "2.3"
        port: 8096

The containers are named after the cp-flink-sql version they compile for, with the dots replaced by dashes. In CMF 2.5 they are compiler-1-19-cmf2-5 and compiler-2-3-cmf2-5. They run from the CMF image, listen on the pod’s loopback interface only, and cannot be reached from outside the pod. They do not get a Kubernetes API token.

Setting sharedCompilers replaces the whole list. For where statements compile with sharedCompilers: [], see Where a statement compiles.

Each entry has the following fields:

Field	Default	Description
`version`	Required	The Flink line, `<major>.<minor>`, in quotes, for example `"1.19"`. Must be unique. In CMF 2.5 the lines are `"1.19"` and `"2.3"`. A compiler for another line does not start. The compiler compiles for the `cp-flink-sql` version `<version>-cmf<CMF minor version>`, for example `1.19-cmf2.5`, and the older releases of its line that CMF supports. You do not change it when you upgrade CMF.
`port`	Required	The port the compiler listens on inside the pod. Must be unique, and must not be `8080`, `cmf.internalPort` (default `8081`), `cmf.metrics.port`, or `cmf.mds.port`. All containers of the pod share its ports.
`resources`	Memory request and limit of `2Gi`. CPU request and limit of the top-level `resources` (`1` and `2` by default).	Container resources of the compiler. Setting it replaces the whole default.
`jvmArgs`	The top-level `jvmArgs` without the options for CMF only, plus a heap of `-Xmx1g`.	JVM options of the compiler. Setting it replaces the default; include a heap size, such as `-Xmx1g`.
`extraEnv`	The top-level `extraEnv`.	Environment variables of the compiler. Setting it replaces the default.
`volumeMounts`	The top-level `mountedVolumes.volumeMounts`.	Volume mounts of the compiler. Setting it replaces the default. Must not mount `/tmp` or `/var/run/secrets/kubernetes.io/serviceaccount`.
`maxConcurrentCompiles`	`16`	How many statements the compiler compiles at the same time. Further statements wait, and fail with HTTP 429 when no slot frees up.

Compilers use the same cmf.sql.classAllowlist, cmf.sql.examplesCatalog, and cmf.kafka.oauthbearerAllowedUrls settings as CMF.

Size the shared compilers

With the defaults, the two compilers add 4 GiB of memory, 2 CPU of requests, and 4 CPU of limits to the CMF pod. The nodes, the namespace’s resource quota, and any LimitRange need room for them.

To change the resources of a compiler, set resources and a heap in jvmArgs on its entry. The compiler’s maximum heap plus about 512 MiB must fit in its memory limit, so that the container is not OOMKilled; see Check the shared compilers. An entry’s jvmArgs replaces the top-level jvmArgs, so repeat the options that the compiler needs, such as a trust store. For example, to give the 1.19 compiler a larger heap and keep the trust store of the top-level jvmArgs:

cmf:
  statementService:
    sharedCompilers:
      - version: "1.19"
        port: 8095
        jvmArgs: "-Xmx3g -Djavax.net.ssl.trustStore=/mnt/truststore/truststore.jks"
        resources:
          requests:
            cpu: 1
            memory: 4Gi
          limits:
            cpu: 2
            memory: 4Gi
      - version: "2.3"
        port: 8096

If an entry sets no jvmArgs, the compiler gets a heap of half the entry’s memory limit, or -Xmx1g if the entry sets no memory limit. No heap is added when the entry’s extraEnv sets a JVM option variable such as JAVA_TOOL_OPTIONS.

What compilers take from the `cmf` container

Unless a compiler entry sets its own value, a compiler gets the top-level jvmArgs, extraEnv, and mountedVolumes.volumeMounts of the cmf container, so it can reach the same Kafka clusters and Schema Registry instances with the same trust stores. CMF leaves out the options that are meant for CMF only:

heap and memory sizing options, such as -Xmx and -XX:MaxRAMPercentage
Java agents and debug options, such as -javaagent and -agentlib
JMX options (-Dcom.sun.management.*)

These options are also left out of the JVM option variables in extraEnv, such as JAVA_TOOL_OPTIONS, except when the variable is set with valueFrom, because the chart cannot read its value. If such a variable holds options for CMF only, set extraEnv on each compiler entry to replace it. Compiler containers have a read-only root file system, so a JVM option that writes a file, such as a GC log, needs a path on a mounted volume.

To see the JVM options, environment variables, and resources each container gets, run helm template with your values. The SP_JVM_OPTS variable holds a compiler’s JVM options.

Check the shared compilers

Compiler containers have no health probes. What happens when a compiler stops depends on how it stops:

A compiler that exits, for example because its heap is full, is started again inside its container after 5 seconds. This is not a container restart, and the CMF pod stays ready.
A compiler that cannot start, for example because image.tag names an image of another CMF minor version than the chart, logs Compiler <version> is not retried. and stays stopped. The CMF pod stays ready. Fix the cause, then run helm upgrade, or restart the CMF pod with kubectl rollout restart deployment/confluent-manager-for-apache-flink.
A compiler container that Kubernetes kills because it uses more memory than its limit (OOMKilled) makes the CMF pod, and with it the CMF REST API, not ready until the container has restarted.

In the first two cases, creating a statement that needs the compiler fails until the compiler runs.

To check a compiler, read its log. A compiler that runs logs Supervisor server is ready.

kubectl logs deployment/confluent-manager-for-apache-flink --namespace <namespace> \
  -c compiler-1-19-cmf2-5

If you set nameOverride, the CMF Deployment has that name.

Compile workers

Compile workers compile statements that use your own code, such as user-defined functions and custom connectors and formats from artifacts, in a pod of their own, so that this code does not run in the CMF pod. Each worker is one container, worker-, in the cmf-compile-worker pod, and listens on port 8090 + . All workers share the one pod.

Enable compile workers

Statements can use code from artifacts only when artifact management and the environment catalog are both on, and the chart then adds one compile worker. Without compile workers, those statements fail. The worker runs code from artifacts, so take the steps in Secure the compile workers before you turn on both features in production.

Enable artifact management and the environment catalog. See Enable artifact management and Enable the environment catalog.
Optionally, set the number of workers in your Helm values file. Without count, you get one worker. A worker compiles one statement at a time, for one environment. If several environments create statements at the same time, raise count; see Size the compile workers. For example:
```
cmf:
  artifacts:
    enabled: true
    basePath: s3://my-bucket/cmf
  sql:
    environmentCatalog:
      enabled: true
  statementService:
    workers:
      count: 2
```
The workers compile for the same Flink lines as the shared compilers. With sharedCompilers: [], also set workers.compilers. Without it, the chart adds no worker by default and fails with a count above 0. For example:
```
cmf:
  statementService:
    sharedCompilers: []
    workers:
      count: 2
      compilers:
        - version: "1.19"
        - version: "2.3"
```
Give the workers the environment variables and files that your catalogs and connectors need. Workers do not get the top-level extraEnv, mountedVolumes, or jvmArgs of CMF. See Give the workers what catalogs and connectors need.
If you render the chart with helm template, or with a GitOps tool that uses it such as Argo CD, see Compile worker certificates.
Apply the values with helm install or helm upgrade, then check that the worker pod is running:
```
kubectl get deployment cmf-compile-worker --namespace <namespace>
```

When the chart adds the worker by default, helm install and helm upgrade print a note that says so. CMF logs Compile workers enabled with the number of workers and their compilers when it starts.

To run no compile workers, set count: 0. Statements that use artifacts then fail. If artifact management and the environment catalog are on but there are no workers, helm install and helm upgrade print a warning, and CMF logs a warning that statements that use artifacts fail.

Give the workers what catalogs and connectors need

A worker compiles the statements of many environments, so it needs what the catalogs and connectors of all those statements need, such as Kafka trust stores and key stores, Hadoop configuration, or cloud credentials. Set them with workers.extraEnv and workers.mountedVolumes. Set JVM options that catalogs need, such as a trust store, in workers.compilerJvmArgs, and keep a heap size there, because setting it replaces the default -Xmx1g.

cmf:
  statementService:
    workers:
      count: 2
      extraEnv:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: s3-credentials
              key: access-key
      mountedVolumes:
        volumes:
          - name: kafka-truststore
            secret:
              secretName: kafka-truststore
        volumeMounts:
          - name: kafka-truststore
            mountPath: /mnt/truststore
            readOnly: true

Code that a statement runs on a worker, such as a UDF, can read these variables and files, regardless of the statement’s environment. Give the workers only what catalogs and connectors need, and prefer to put credentials in the options of each environment’s catalogs, which a worker gets only with the statements of that environment.

The chart refuses the following, and helm install or helm upgrade fails:

a volume mount without readOnly: true
workers.extraEnv that sets LD_PRELOAD, an SP_* variable, or a JVM option variable such as JAVA_TOOL_OPTIONS. Set JVM options with workers.jvmArgs and workers.compilerJvmArgs.
a volume or secretKeyRef that uses the compile worker certificates, or a projected service account token
a mount path at, under, or above /tmp, /dev/shm, /var/cache/artifacts, /etc/cmf/compile-tls, or /etc/passwd

Compile worker settings

Set these values under cmf.statementService.workers:

Property	Default	Description
`count`	`1` when `cmf.artifacts.enabled` and `cmf.sql.environmentCatalog.enabled` are both `true` and the workers have compilers, else `0`	Number of workers. `0` creates no worker pod. A value that you set, also `0`, replaces the default.
`compilers`	The versions of `sharedCompilers`	The Flink lines each worker compiles for, as a list of `version` entries. Required with `sharedCompilers: []`.
`resources`	Memory request and limit of `3Gi`. CPU request and limit of the top-level `resources`, or else a CPU request of `500m`.	Container resources of each worker. Setting it replaces the whole default.
`jvmArgs`	`-Xmx384m`	JVM options of the worker process. Include a heap size.
`compilerJvmArgs`	`-Xmx1g`	JVM options of the process that compiles a statement. Include a heap size.
`tmpSizeLimit`	`256Mi`	Size limit of each worker’s `/tmp`, which is in memory.
`volumeSizeLimit`	`512Mi`	Size limit of each worker’s artifact cache, which is in memory. Set it to at least the size of the JAR files that one statement uses.
`compileTimeout`	`2m`	The longest a statement can compile, and the longest it waits for a free worker.
`requirePtraceProtection`	`false`	With `true`, a worker does not start on a node where `kernel.yama.ptrace_scope` is `0` or missing, and logs why. With `false`, it starts there and logs an error. See Prepare the worker nodes.
`tls.existingSecretPrefix`	`""`	Prefix of the Secrets with your own worker certificates. With `""`, the chart generates them. See Compile worker certificates.
`extraEnv`	`[]`	Environment variables of every worker container.
`mountedVolumes.volumes`, `mountedVolumes.volumeMounts`	`[]`	Volumes of the worker pod, and read-only volume mounts of every worker container.

CMF also opens an internal port for the workers, cmf.internalPort (default 8081). It accepts only the workers’ certificates.

The worker pod does not get the top-level nodeSelector, tolerations, affinity, topologySpreadConstraints, priorityClassName, podLabels, or podAnnotations. Set them for the worker pod under workerDeploymentOverrides.spec.template.spec, and its labels and annotations under workerDeploymentOverrides.spec.template.metadata. If your nodes are all tainted, or admission policies require pod labels, the worker pod stays Pending without them. See also Prepare the worker nodes.

Size the compile workers

A worker’s JVM heaps and other JVM memory, its /tmp and /dev/shm, and its artifact cache are all in memory, so all of them count towards resources.limits.memory. A worker that uses more than this limit is killed and restarts, and the statement it was compiling fails. Set resources.limits.memory to at least the sum of the following:

Part of a worker’s memory	Set by	Default
Heap of the worker process	`jvmArgs`	384 MiB
Heap of the process that compiles a statement	`compilerJvmArgs`	1 GiB
Other memory of these two processes		About 512 MiB
`/tmp`	`tmpSizeLimit`	256 MiB
`/dev/shm`		64 MiB
Artifact cache	`volumeSizeLimit`	512 MiB

With the defaults these add up to about 2.7 GiB, within the default limit of 3 GiB. The 512 MiB of other JVM memory is an estimate and grows with large artifacts. When you raise a heap, tmpSizeLimit, or volumeSizeLimit, raise the memory limit by the same amount. A worker logs a warning when it starts with a smaller limit or with no limit.

A worker compiles one statement at a time, and serves one environment and one Flink line at a time. In environments with artifacts this includes statements such as DESCRIBE and SHOW CREATE TABLE. CMF sends a statement to a free worker that already serves its environment and line. If there is none, CMF restarts a worker for it, which adds several seconds to that statement. This keeps environments apart, but when there are fewer workers than active environments, the workers keep switching and most statements wait for a restart. Choose count as follows:

Set count to at least the number of pairs of environment and Flink line that create statements at the same time, counting only environments with artifacts. For example, two such environments that each create statements on 1.19 and 2.3 compute pools need 4 workers. With sharedCompilers: [], count all environments.
All workers run in one pod, so one node must have room for count times the worker resources.
When all workers are busy, a statement waits up to compileTimeout and then fails with HTTP 429 and a message to retry later or raise cmf.statementService.workers.count.
Set the read timeout of any ingress or load balancer in front of the CMF REST API to more than (2 × compileTimeout) + 45 seconds, which is 285 seconds with the default 2m.

Compile worker certificates

CMF and the workers authenticate each other with TLS client certificates. By default, the chart generates a CA and the certificates, valid for 10 years, in the Secrets cmf-compile-ca, cmf-compile-tls-cmf, and cmf-compile-tls-worker. Upgrades keep them, and helm uninstall leaves them in place.

To renew the generated certificates, delete these Secrets and run helm upgrade. Both pods restart with new certificates. Also delete them before you install the chart again under another release name in the same namespace.
If you render the chart with helm template, or with a GitOps tool that uses it such as Argo CD, the chart generates new certificates on every render. Set workers.tls.existingSecretPrefix and use your own certificates, as described below. This also applies to the worker that the chart adds by default.

To use your own certificates, set workers.tls.existingSecretPrefix: <prefix> and create two Secrets in the CMF namespace. Each certificate needs the extended key usages server auth and client auth. Use a CA that issues no other certificates.

<prefix>-cmf: Type kubernetes.io/tls, with tls.crt, tls.key, and ca.crt in PEM format. The certificate has the common name cmf and the DNS names cmf-worker-api, cmf-worker-api.<namespace>, and cmf-worker-api.<namespace>.svc.
<prefix>-worker: Type Opaque, with ca.crt and, for each worker  from 0 to count - 1, worker-.crt and worker-.key in PEM format. Each certificate has the common name compile-worker- and the DNS names cmf-compile-worker, cmf-compile-worker.<namespace>, and cmf-compile-worker.<namespace>.svc.

kubectl create secret tls cannot add ca.crt, so create both Secrets with kubectl create secret generic, for example for one worker:

kubectl create secret generic <prefix>-cmf --namespace <namespace> --type=kubernetes.io/tls \
  --from-file=tls.crt=cmf.crt --from-file=tls.key=cmf.key --from-file=ca.crt=ca.crt
kubectl create secret generic <prefix>-worker --namespace <namespace> \
  --from-file=ca.crt=ca.crt \
  --from-file=worker-0.crt=worker-0.crt --from-file=worker-0.key=worker-0.key

Check the Secrets before you install or upgrade:

If <prefix>-cmf is missing, has no ca.crt, or holds a certificate with another common name, the CMF pod does not start, and the CMF REST API is down.
A worker whose pair is missing from <prefix>-worker does not start, and the worker pod is not ready. Add the pairs of the new workers before you raise count.

CMF and the workers read their certificates only when they start. After you replace the Secrets, restart both Deployments:

kubectl rollout restart deployment/confluent-manager-for-apache-flink --namespace <namespace>
kubectl rollout restart deployment/cmf-compile-worker --namespace <namespace>

If you set nameOverride, use it instead of confluent-manager-for-apache-flink.

Secure the compile workers

Compile workers run code from artifacts while they compile, for the statements of every environment with artifacts. Do the following before you turn on artifact management and the environment catalog in production.

Prepare the worker nodes

On the nodes that run the worker pod:

Set the kernel setting kernel.yama.ptrace_scope to 1 or higher, so that code that a statement runs cannot read the memory of the worker that compiles it. Some Linux distributions set 0 by default, so check it with sysctl kernel.yama.ptrace_scope on each node. A worker logs an error when it starts on a node with 0 or without this setting. To have workers not start on such nodes, set cmf.statementService.workers.requirePtraceProtection: true. The worker pod is then not ready on such a node, and statements that compile on the workers fail until it runs on a node with the setting.
Set the kubelet’s podPidsLimit, so that code in the worker pod cannot start an unbounded number of processes.
Keep the worker resources limits set, so that a statement cannot take more memory or CPU from the node.

You can also keep the worker pod on other nodes than the CMF pod. Set nodeSelector, tolerations, and affinity under workerDeploymentOverrides.spec.template.spec.

Limit network access

The chart creates no NetworkPolicy. If your cluster enforces NetworkPolicies, create one that lets the worker pod reach only what it needs: DNS, CMF’s internal port, and the services that your catalogs and connectors use, such as Kafka and Schema Registry. Everything else is then blocked, including the cloud metadata endpoint 169.254.169.254, the CMF REST API, and the Kubernetes API server. The following example assumes Kafka and Schema Registry run in the namespace confluent:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cmf-compile-worker
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: confluent-manager-for-apache-flink-compile-worker
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # CMF sends statements to the workers on ports 8090 and up, one port for each worker.
    # Raise endPort if you run more than 10 workers.
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: confluent-manager-for-apache-flink
      ports:
        - protocol: TCP
          port: 8090
          endPort: 8099
  egress:
    # DNS
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # CMF's internal port (cmf.internalPort), where the workers fetch artifacts
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: confluent-manager-for-apache-flink
      ports:
        - protocol: TCP
          port: 8081
    # Kafka and Schema Registry
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: confluent
      ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 8081

Adjust the example to your cluster:

If the DNS pods of your cluster have other labels or ports than k8s-app: kube-dns and 53, change the DNS rule to match them.
If you set cmf.internalPort, use its value instead of 8081 in the rule for CMF’s internal port.
For services outside the cluster, add ipBlock rules with their addresses. If you allow a wide range such as 0.0.0.0/0, add 169.254.169.254/32 to its except list.
If you set nameOverride, the pod labels use that name instead of confluent-manager-for-apache-flink.

If a NetworkPolicy selects the CMF pod, admit the internal port from the worker pod there too, as in the following rule. If it also limits the egress of the CMF pod, allow egress to the worker pod on ports 8090 and up.

ingress:
  - from:
      - podSelector:
          matchLabels:
            app.kubernetes.io/name: confluent-manager-for-apache-flink-compile-worker
    ports:
      - protocol: TCP
        port: 8081

Limit credentials and identities

Keep cloud identities off the worker pod. It runs as the namespace’s ServiceAccount default, without a Kubernetes API token, and needs no access to artifact storage, because CMF serves the artifacts. Do not annotate default for IAM roles for service accounts (IRSA) or Workload Identity.
If a catalog needs a cloud identity, give the worker pod a ServiceAccount of its own with workerDeploymentOverrides.spec.template.spec.serviceAccountName, and grant it only what all environments can share. It must not be the CMF ServiceAccount. On Google Kubernetes Engine (GKE), Workload Identity uses the metadata endpoint, so your NetworkPolicy must then allow 169.254.169.254.
Give the workers only what catalogs and connectors need. See Give the workers what catalogs and connectors need.
Restrict get on Secrets in the CMF namespace. Any certificate from the workers’ CA with the common name cmf can send statements to the workers.

Set permissions per environment

Statements of the same environment share compile workers. Grant the permissions to upload artifacts and to create statements in an environment only to users you trust with all statements and artifacts of that environment, and keep teams that must not see each other’s statements in separate environments. See Configure Access Control for Confluent Manager for Apache Flink and Access control.

All compile workers of a CMF installation run in one pod and serve all of its environments. If environments belong to parties that must be fully separated from each other, run a separate CMF installation for each of them.

Keep the CMF pod trusted

Shared compilers run in the CMF pod and get each statement together with the catalog credentials of its environment. Add containers to the CMF pod, for example with extraContainers, only if you trust them with this data.

Flink versions and compilers

CMF 2.5 compiles for Flink 2.x compute pools only on a compiler for the 2.3 line, either a shared compiler or a compile worker. Keep the "2.3" entry in sharedCompilers, and in workers.compilers if you set it, if you run statements on Flink 2.x compute pools. Compute pools of each line need a minimum cp-flink-sql image; see the image requirements of compute pools.

The following need a compute pool on Flink 2.x:

Models: CREATE MODEL, ALTER MODEL, DROP MODEL, SHOW MODELS, DESCRIBE MODEL, and queries that call ML_PREDICT or ML_EVALUATE on a model. On a Flink 1.x compute pool, CMF rejects them with a message that models require Flink 2.x.
Batch statements, with execution.runtime-mode=BATCH.
Statements that use the tables of a Kafka database with subjectNameStrategy: AssociatedNameStrategy. CMF accepts such a database only when a compiler for Flink 2.x is configured.

CMF 2.5 runs statements on compute pools with flinkVersion v1_19 or v2_3. It compiles statements of v1_19 compute pools on the 1.19 compiler and those of v2_3 compute pools on the 2.3 compiler. If no compiler serves the compute pool’s Flink version, creating the statement fails.

Statements that do not run on a compute pool, such as those that list catalogs, databases, and tables, use the compiler with the newest version.

Compute pools on Flink 1.x and 2.x in the same environment share its catalogs. A table or view that uses SQL features of Flink 2.x fails when a statement on a Flink 1.x compute pool uses it. Create and change such tables and views from a compute pool on Flink 2.x.

Troubleshoot compilers

Error when you create a statement	What to do
`Statement uses artifact '<artifact>' of environment '<environment>', which is not available to it ...`	Check that the artifact exists in the statement’s environment. If it does, CMF has no compile workers. If your values set `cmf.statementService.workers.count: 0`, remove it or set it above `0`. With `sharedCompilers: []`, also set `workers.compilers`.
`No shared compiler serves Flink version '<version>'. Configured compilers: ...`, or the same with `No compile worker`	Add a compiler for the compute pool’s Flink line to `sharedCompilers`, or to `workers.compilers` for environments with artifacts: `"1.19"` for `v1_19` compute pools, or `"2.3"` for `v2_3` compute pools.
`No shared compilers or compile workers are configured, so CMF compiled the statement with its own Flink 1.19 compiler ...`	The compute pool runs Flink 2.x. Add a `"2.3"` entry to `sharedCompilers`.
`Failed to reach the shared compiler ... See the log of container compiler-<version>.`	The compiler has not started yet or has stopped. Read the log of the container that the message names, as described in Check the shared compilers, and create the statement again when the compiler runs.
`Failed to reach the compile worker ...`	A worker is not running, or CMF and the worker do not accept each other’s certificates. CMF tries again until `compileTimeout`, so the statement fails after about 2 minutes with the default. A worker pod that is `Pending` or in `ImagePullBackOff` has no log; see its events with `kubectl describe pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink-compile-worker --namespace <namespace>`. Otherwise read the worker’s log with `kubectl logs deployment/cmf-compile-worker --namespace <namespace> -c worker-<i>`, adding `--previous` for a container that restarted. If the message says that the TLS handshake failed or that the worker closed the connection without an answer, check the Secrets in Compile worker certificates.
Connection, TLS, or credential errors from Kafka, Schema Registry, or object storage, only in environments with artifacts	These statements compile on the compile workers, which do not get the top-level `jvmArgs`, `extraEnv`, `mountedVolumes`, or the cloud identity of CMF. Give the workers what the catalogs need, as described in Give the workers what catalogs and connectors need, and check that your NetworkPolicy lets the worker pod reach these services.
`The shared compiler is busy ...` or `All <n> compile workers are busy ...` (HTTP 429)	Retry later, or raise the compiler’s `maxConcurrentCompiles` or `cmf.statementService.workers.count`.
No error, but statements in environments with artifacts take several seconds or more to create	The workers restart to switch between environments and Flink lines. Raise `cmf.statementService.workers.count` as described in Size the compile workers.
`Compilation on compile worker <i> took longer than the compile timeout ...`	The statement is created in phase `FAILED`. Raise `workers.compileTimeout` if the statement needs more time.
`Artifact '<artifact>' of environment '<environment>' ... does not fit the compile worker's artifact cache ...`	Raise `workers.volumeSizeLimit` and the workers’ memory limit by the same amount.

To follow a statement on the compile workers, find its metadata.uid in the CMF log line Dispatching compile to compile worker: requestId='<id>', uuid='<uid>', ..., worker=, then search the worker’s log for the request ID:

kubectl logs deployment/cmf-compile-worker --namespace <namespace> -c worker-<i> | grep <id>

When you operate the workers:

CMF and the workers must run the same CMF version. While helm upgrade replaces the CMF pod and the worker pod, statements that compile on the workers fail until both run the new version.
The worker pod is not ready while any worker is down, so helm install --wait and helm upgrade --wait time out when a worker cannot start, and --atomic then rolls back the whole release.
If the node of the worker pod is lost, the old pod can stay in Terminating. Delete it with kubectl delete pod <pod> --namespace <namespace> --force --grace-period=0.