# Configure SQL Compilers and Compile Workers for Confluent Manager for Apache Flink Confluent Manager for Apache Flink® (CMF) compiles every Flink SQL statement before it runs. Starting with version 2.5, CMF compiles statements in separate compiler processes, one for each Flink line, so that one CMF installation serves compute pools on different Flink versions. A line is a major and minor version, such as 1.19 or 2.3. There are two kinds of compilers: Shared compilers : Containers in the CMF pod, one for each Flink line. They are on by default, with a compiler for Flink 1.19 and one for Flink 2.3. Shared compilers cannot load [artifacts](../configure/artifacts.md#cmf-artifacts). Compile workers : A separate pod, `cmf-compile-worker`, in the CMF namespace. Compile workers are the only compilers that load artifacts, so statements that use user-defined functions or custom connectors and formats from artifacts need them. The chart adds one compile worker by default when `cmf.artifacts.enabled` and `cmf.sql.environmentCatalog.enabled` are both `true`, and none otherwise. You configure both with the `cmf.statementService` Helm values. ## Where a statement compiles Where CMF compiles a statement depends on your Helm values and on whether the statement’s environment has artifacts. An environment has artifacts when `cmf.artifacts.enabled` and `cmf.sql.environmentCatalog.enabled` are both `true` and the environment has at least one artifact of any kind, including an application JAR. All statements of such an environment compile on the compile workers, including those that use no artifact. | Helm values | Environments without artifacts | Environments with artifacts | |---------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|------------------------------------------------------------| | Shared compilers and compile workers, the default when `cmf.artifacts.enabled` and
`cmf.sql.environmentCatalog.enabled` are both `true` | Shared compilers | Compile workers | | Shared compilers and no compile workers, the default when either feature is off, or with
`workers.count: 0` | Shared compilers | Shared compilers. Statements that use an artifact fail. | | `sharedCompilers: []` and compile workers | Compile workers | Compile workers | | `sharedCompilers: []`, no compile workers | The `cmf` container, which compiles for Flink 1.19 only | The `cmf` container. Statements that use an artifact fail. | If an environment has artifacts but its SQL statements use none, for example because its only artifacts are application JARs, its statements still compile on the compile workers. Give the workers what these statements need, as described in [Give the workers what catalogs and connectors need](#cmf-compile-workers-config), or set `workers.count: 0` to keep them on the shared compilers. ## Shared compilers Each entry in `cmf.statementService.sharedCompilers` adds a container to the CMF pod that compiles statements for one Flink line. If you do not set the value, the CMF pod has these two compilers: ```yaml cmf: statementService: sharedCompilers: - version: "1.19" port: 8095 - version: "2.3" port: 8096 ``` The containers are named after the `cp-flink-sql` version they compile for, with the dots replaced by dashes. In CMF 2.5 they are `compiler-1-19-cmf2-5` and `compiler-2-3-cmf2-5`. They run from the CMF image, listen on the pod’s loopback interface only, and cannot be reached from outside the pod. They do not get a Kubernetes API token. Setting `sharedCompilers` replaces the whole list. For where statements compile with `sharedCompilers: []`, see [Where a statement compiles](#cmf-sql-compilers-routing). Each entry has the following fields: | Field | Default | Description | |-------------------------|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `version` | Required | The Flink line, `.`, in quotes, for example `"1.19"`. Must be unique.
In CMF 2.5 the lines are `"1.19"` and `"2.3"`. A compiler for another line does not
start. The compiler compiles for the `cp-flink-sql` version
`-cmf`, for example `1.19-cmf2.5`, and the older releases
of its line that CMF supports. You do not change it when you upgrade CMF. | | `port` | Required | The port the compiler listens on inside the pod. Must be unique, and must not be
`8080`, `cmf.internalPort` (default `8081`), `cmf.metrics.port`, or
`cmf.mds.port`. All containers of the pod share its ports. | | `resources` | Memory request and limit of `2Gi`. CPU request and limit of the top-level
`resources` (`1` and `2` by default). | Container resources of the compiler. Setting it replaces the whole default. | | `jvmArgs` | The top-level `jvmArgs` without the options for CMF only, plus a heap of `-Xmx1g`. | JVM options of the compiler. Setting it replaces the default; include a heap size, such
as `-Xmx1g`. | | `extraEnv` | The top-level `extraEnv`. | Environment variables of the compiler. Setting it replaces the default. | | `volumeMounts` | The top-level `mountedVolumes.volumeMounts`. | Volume mounts of the compiler. Setting it replaces the default. Must not mount `/tmp`
or `/var/run/secrets/kubernetes.io/serviceaccount`. | | `maxConcurrentCompiles` | `16` | How many statements the compiler compiles at the same time. Further statements wait, and
fail with HTTP 429 when no slot frees up. | Compilers use the same `cmf.sql.classAllowlist`, `cmf.sql.examplesCatalog`, and `cmf.kafka.oauthbearerAllowedUrls` settings as CMF. ### Size the shared compilers With the defaults, the two compilers add 4 GiB of memory, 2 CPU of requests, and 4 CPU of limits to the CMF pod. The nodes, the namespace’s resource quota, and any LimitRange need room for them. To change the resources of a compiler, set `resources` and a heap in `jvmArgs` on its entry. The compiler’s maximum heap plus about 512 MiB must fit in its memory limit, so that the container is not `OOMKilled`; see [Check the shared compilers](#cmf-shared-compilers-check). An entry’s `jvmArgs` replaces the top-level `jvmArgs`, so repeat the options that the compiler needs, such as a trust store. For example, to give the 1.19 compiler a larger heap and keep the trust store of the top-level `jvmArgs`: ```yaml cmf: statementService: sharedCompilers: - version: "1.19" port: 8095 jvmArgs: "-Xmx3g -Djavax.net.ssl.trustStore=/mnt/truststore/truststore.jks" resources: requests: cpu: 1 memory: 4Gi limits: cpu: 2 memory: 4Gi - version: "2.3" port: 8096 ``` If an entry sets no `jvmArgs`, the compiler gets a heap of half the entry’s memory limit, or `-Xmx1g` if the entry sets no memory limit. No heap is added when the entry’s `extraEnv` sets a JVM option variable such as `JAVA_TOOL_OPTIONS`. ### What compilers take from the `cmf` container Unless a compiler entry sets its own value, a compiler gets the top-level `jvmArgs`, `extraEnv`, and `mountedVolumes.volumeMounts` of the `cmf` container, so it can reach the same Kafka clusters and Schema Registry instances with the same trust stores. CMF leaves out the options that are meant for CMF only: - heap and memory sizing options, such as `-Xmx` and `-XX:MaxRAMPercentage` - Java agents and debug options, such as `-javaagent` and `-agentlib` - JMX options (`-Dcom.sun.management.*`) These options are also left out of the JVM option variables in `extraEnv`, such as `JAVA_TOOL_OPTIONS`, except when the variable is set with `valueFrom`, because the chart cannot read its value. If such a variable holds options for CMF only, set `extraEnv` on each compiler entry to replace it. Compiler containers have a read-only root file system, so a JVM option that writes a file, such as a GC log, needs a path on a mounted volume. To see the JVM options, environment variables, and resources each container gets, run `helm template` with your values. The `SP_JVM_OPTS` variable holds a compiler’s JVM options. ### Check the shared compilers Compiler containers have no health probes. What happens when a compiler stops depends on how it stops: - A compiler that exits, for example because its heap is full, is started again inside its container after 5 seconds. This is not a container restart, and the CMF pod stays ready. - A compiler that cannot start, for example because `image.tag` names an image of another CMF minor version than the chart, logs `Compiler is not retried.` and stays stopped. The CMF pod stays ready. Fix the cause, then run `helm upgrade`, or restart the CMF pod with `kubectl rollout restart deployment/confluent-manager-for-apache-flink`. - A compiler container that Kubernetes kills because it uses more memory than its limit (`OOMKilled`) makes the CMF pod, and with it the CMF REST API, not ready until the container has restarted. In the first two cases, creating a statement that needs the compiler fails until the compiler runs. To check a compiler, read its log. A compiler that runs logs `Supervisor server is ready`. ```bash kubectl logs deployment/confluent-manager-for-apache-flink --namespace \ -c compiler-1-19-cmf2-5 ``` If you set `nameOverride`, the CMF Deployment has that name. ## Compile workers Compile workers compile statements that use your own code, such as user-defined functions and custom connectors and formats from artifacts, in a pod of their own, so that this code does not run in the CMF pod. Each worker is one container, `worker-`, in the `cmf-compile-worker` pod, and listens on port `8090 + `. All workers share the one pod. ### Enable compile workers Statements can use code from artifacts only when artifact management and the environment catalog are both on, and the chart then adds one compile worker. Without compile workers, those statements fail. The worker runs code from artifacts, so take the steps in [Secure the compile workers](#cmf-compile-workers-secure) before you turn on both features in production. 1. Enable artifact management and the environment catalog. See [Enable artifact management](../configure/artifacts.md#cmf-artifacts-enable) and [Enable the environment catalog](../configure/environment-catalog.md#cmf-environment-catalog-enable). 2. Optionally, set the number of workers in your Helm values file. Without `count`, you get one worker. A worker compiles one statement at a time, for one environment. If several environments create statements at the same time, raise `count`; see [Size the compile workers](#cmf-compile-workers-sizing). For example: ```yaml cmf: artifacts: enabled: true basePath: s3://my-bucket/cmf sql: environmentCatalog: enabled: true statementService: workers: count: 2 ``` The workers compile for the same Flink lines as the shared compilers. With `sharedCompilers: []`, also set `workers.compilers`. Without it, the chart adds no worker by default and fails with a `count` above `0`. For example: ```yaml cmf: statementService: sharedCompilers: [] workers: count: 2 compilers: - version: "1.19" - version: "2.3" ``` 3. Give the workers the environment variables and files that your catalogs and connectors need. Workers do not get the top-level `extraEnv`, `mountedVolumes`, or `jvmArgs` of CMF. See [Give the workers what catalogs and connectors need](#cmf-compile-workers-config). 4. If you render the chart with `helm template`, or with a GitOps tool that uses it such as Argo CD, see [Compile worker certificates](#cmf-compile-workers-certificates). 5. Apply the values with `helm install` or `helm upgrade`, then check that the worker pod is running: ```bash kubectl get deployment cmf-compile-worker --namespace ``` When the chart adds the worker by default, `helm install` and `helm upgrade` print a note that says so. CMF logs `Compile workers enabled` with the number of workers and their compilers when it starts. To run no compile workers, set `count: 0`. Statements that use artifacts then fail. If artifact management and the environment catalog are on but there are no workers, `helm install` and `helm upgrade` print a warning, and CMF logs a warning that statements that use artifacts fail. ### Give the workers what catalogs and connectors need A worker compiles the statements of many environments, so it needs what the catalogs and connectors of all those statements need, such as Kafka trust stores and key stores, Hadoop configuration, or cloud credentials. Set them with `workers.extraEnv` and `workers.mountedVolumes`. Set JVM options that catalogs need, such as a trust store, in `workers.compilerJvmArgs`, and keep a heap size there, because setting it replaces the default `-Xmx1g`. ```yaml cmf: statementService: workers: count: 2 extraEnv: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: s3-credentials key: access-key mountedVolumes: volumes: - name: kafka-truststore secret: secretName: kafka-truststore volumeMounts: - name: kafka-truststore mountPath: /mnt/truststore readOnly: true ``` Code that a statement runs on a worker, such as a UDF, can read these variables and files, regardless of the statement’s environment. Give the workers only what catalogs and connectors need, and prefer to put credentials in the options of each environment’s catalogs, which a worker gets only with the statements of that environment. The chart refuses the following, and `helm install` or `helm upgrade` fails: - a volume mount without `readOnly: true` - `workers.extraEnv` that sets `LD_PRELOAD`, an `SP_*` variable, or a JVM option variable such as `JAVA_TOOL_OPTIONS`. Set JVM options with `workers.jvmArgs` and `workers.compilerJvmArgs`. - a volume or `secretKeyRef` that uses the compile worker certificates, or a projected service account token - a mount path at, under, or above `/tmp`, `/dev/shm`, `/var/cache/artifacts`, `/etc/cmf/compile-tls`, or `/etc/passwd` ### Compile worker settings Set these values under `cmf.statementService.workers`: | Property | Default | Description | |---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `count` | `1` when `cmf.artifacts.enabled` and `cmf.sql.environmentCatalog.enabled` are both
`true` and the workers have compilers, else `0` | Number of workers. `0` creates no worker pod. A value that you set, also `0`, replaces
the default. | | `compilers` | The versions of `sharedCompilers` | The Flink lines each worker compiles for, as a list of `version` entries. Required with
`sharedCompilers: []`. | | `resources` | Memory request and limit of `3Gi`. CPU request and limit of the top-level
`resources`, or else a CPU request of `500m`. | Container resources of each worker. Setting it replaces the whole default. | | `jvmArgs` | `-Xmx384m` | JVM options of the worker process. Include a heap size. | | `compilerJvmArgs` | `-Xmx1g` | JVM options of the process that compiles a statement. Include a heap size. | | `tmpSizeLimit` | `256Mi` | Size limit of each worker’s `/tmp`, which is in memory. | | `volumeSizeLimit` | `512Mi` | Size limit of each worker’s artifact cache, which is in memory. Set it to at least the
size of the JAR files that one statement uses. | | `compileTimeout` | `2m` | The longest a statement can compile, and the longest it waits for a free worker. | | `requirePtraceProtection` | `false` | With `true`, a worker does not start on a node where `kernel.yama.ptrace_scope` is
`0` or missing, and logs why. With `false`, it starts there and logs an error. See
[Prepare the worker nodes](#cmf-compile-workers-nodes). | | `tls.existingSecretPrefix` | `""` | Prefix of the Secrets with your own worker certificates. With `""`, the chart generates
them. See [Compile worker certificates](#cmf-compile-workers-certificates). | | `extraEnv` | `[]` | Environment variables of every worker container. | | `mountedVolumes.volumes`, `mountedVolumes.volumeMounts` | `[]` | Volumes of the worker pod, and read-only volume mounts of every worker container. | CMF also opens an internal port for the workers, `cmf.internalPort` (default `8081`). It accepts only the workers’ certificates. The worker pod does not get the top-level `nodeSelector`, `tolerations`, `affinity`, `topologySpreadConstraints`, `priorityClassName`, `podLabels`, or `podAnnotations`. Set them for the worker pod under `workerDeploymentOverrides.spec.template.spec`, and its labels and annotations under `workerDeploymentOverrides.spec.template.metadata`. If your nodes are all tainted, or admission policies require pod labels, the worker pod stays `Pending` without them. See also [Prepare the worker nodes](#cmf-compile-workers-nodes). ### Size the compile workers A worker’s JVM heaps and other JVM memory, its `/tmp` and `/dev/shm`, and its artifact cache are all in memory, so all of them count towards `resources.limits.memory`. A worker that uses more than this limit is killed and restarts, and the statement it was compiling fails. Set `resources.limits.memory` to at least the sum of the following: | Part of a worker’s memory | Set by | Default | |-----------------------------------------------|-------------------|---------------| | Heap of the worker process | `jvmArgs` | 384 MiB | | Heap of the process that compiles a statement | `compilerJvmArgs` | 1 GiB | | Other memory of these two processes | | About 512 MiB | | `/tmp` | `tmpSizeLimit` | 256 MiB | | `/dev/shm` | | 64 MiB | | Artifact cache | `volumeSizeLimit` | 512 MiB | With the defaults these add up to about 2.7 GiB, within the default limit of 3 GiB. The 512 MiB of other JVM memory is an estimate and grows with large artifacts. When you raise a heap, `tmpSizeLimit`, or `volumeSizeLimit`, raise the memory limit by the same amount. A worker logs a warning when it starts with a smaller limit or with no limit. A worker compiles one statement at a time, and serves one environment and one Flink line at a time. In environments with artifacts this includes statements such as `DESCRIBE` and `SHOW CREATE TABLE`. CMF sends a statement to a free worker that already serves its environment and line. If there is none, CMF restarts a worker for it, which adds several seconds to that statement. This keeps environments apart, but when there are fewer workers than active environments, the workers keep switching and most statements wait for a restart. Choose `count` as follows: - Set `count` to at least the number of pairs of environment and Flink line that create statements at the same time, counting only environments with artifacts. For example, two such environments that each create statements on 1.19 and 2.3 compute pools need 4 workers. With `sharedCompilers: []`, count all environments. - All workers run in one pod, so one node must have room for `count` times the worker `resources`. - When all workers are busy, a statement waits up to `compileTimeout` and then fails with HTTP 429 and a message to retry later or raise `cmf.statementService.workers.count`. - Set the read timeout of any ingress or load balancer in front of the CMF REST API to more than (2 × `compileTimeout`) + 45 seconds, which is 285 seconds with the default `2m`. ### Compile worker certificates CMF and the workers authenticate each other with TLS client certificates. By default, the chart generates a CA and the certificates, valid for 10 years, in the Secrets `cmf-compile-ca`, `cmf-compile-tls-cmf`, and `cmf-compile-tls-worker`. Upgrades keep them, and `helm uninstall` leaves them in place. - To renew the generated certificates, delete these Secrets and run `helm upgrade`. Both pods restart with new certificates. Also delete them before you install the chart again under another release name in the same namespace. - If you render the chart with `helm template`, or with a GitOps tool that uses it such as Argo CD, the chart generates new certificates on every render. Set `workers.tls.existingSecretPrefix` and use your own certificates, as described below. This also applies to the worker that the chart adds by default. To use your own certificates, set `workers.tls.existingSecretPrefix: ` and create two Secrets in the CMF namespace. Each certificate needs the extended key usages `server auth` and `client auth`. Use a CA that issues no other certificates. `-cmf` : Type `kubernetes.io/tls`, with `tls.crt`, `tls.key`, and `ca.crt` in PEM format. The certificate has the common name `cmf` and the DNS names `cmf-worker-api`, `cmf-worker-api.`, and `cmf-worker-api..svc`. `-worker` : Type `Opaque`, with `ca.crt` and, for each worker `` from `0` to `count - 1`, `worker-.crt` and `worker-.key` in PEM format. Each certificate has the common name `compile-worker-` and the DNS names `cmf-compile-worker`, `cmf-compile-worker.`, and `cmf-compile-worker..svc`. `kubectl create secret tls` cannot add `ca.crt`, so create both Secrets with `kubectl create secret generic`, for example for one worker: ```bash kubectl create secret generic -cmf --namespace --type=kubernetes.io/tls \ --from-file=tls.crt=cmf.crt --from-file=tls.key=cmf.key --from-file=ca.crt=ca.crt kubectl create secret generic -worker --namespace \ --from-file=ca.crt=ca.crt \ --from-file=worker-0.crt=worker-0.crt --from-file=worker-0.key=worker-0.key ``` Check the Secrets before you install or upgrade: - If `-cmf` is missing, has no `ca.crt`, or holds a certificate with another common name, the CMF pod does not start, and the CMF REST API is down. - A worker whose pair is missing from `-worker` does not start, and the worker pod is not ready. Add the pairs of the new workers before you raise `count`. CMF and the workers read their certificates only when they start. After you replace the Secrets, restart both Deployments: ```bash kubectl rollout restart deployment/confluent-manager-for-apache-flink --namespace kubectl rollout restart deployment/cmf-compile-worker --namespace ``` If you set `nameOverride`, use it instead of `confluent-manager-for-apache-flink`. ## Secure the compile workers Compile workers run code from artifacts while they compile, for the statements of every environment with artifacts. Do the following before you turn on artifact management and the environment catalog in production. ### Prepare the worker nodes On the nodes that run the worker pod: - Set the kernel setting `kernel.yama.ptrace_scope` to `1` or higher, so that code that a statement runs cannot read the memory of the worker that compiles it. Some Linux distributions set `0` by default, so check it with `sysctl kernel.yama.ptrace_scope` on each node. A worker logs an error when it starts on a node with `0` or without this setting. To have workers not start on such nodes, set `cmf.statementService.workers.requirePtraceProtection: true`. The worker pod is then not ready on such a node, and statements that compile on the workers fail until it runs on a node with the setting. - Set the kubelet’s `podPidsLimit`, so that code in the worker pod cannot start an unbounded number of processes. - Keep the worker `resources` limits set, so that a statement cannot take more memory or CPU from the node. You can also keep the worker pod on other nodes than the CMF pod. Set `nodeSelector`, `tolerations`, and `affinity` under `workerDeploymentOverrides.spec.template.spec`. ### Limit network access The chart creates no NetworkPolicy. If your cluster enforces NetworkPolicies, create one that lets the worker pod reach only what it needs: DNS, CMF’s internal port, and the services that your catalogs and connectors use, such as Kafka and Schema Registry. Everything else is then blocked, including the cloud metadata endpoint `169.254.169.254`, the CMF REST API, and the Kubernetes API server. The following example assumes Kafka and Schema Registry run in the namespace `confluent`: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: cmf-compile-worker namespace: spec: podSelector: matchLabels: app.kubernetes.io/name: confluent-manager-for-apache-flink-compile-worker policyTypes: - Ingress - Egress ingress: # CMF sends statements to the workers on ports 8090 and up, one port for each worker. # Raise endPort if you run more than 10 workers. - from: - podSelector: matchLabels: app.kubernetes.io/name: confluent-manager-for-apache-flink ports: - protocol: TCP port: 8090 endPort: 8099 egress: # DNS - to: - namespaceSelector: {} podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP port: 53 # CMF's internal port (cmf.internalPort), where the workers fetch artifacts - to: - podSelector: matchLabels: app.kubernetes.io/name: confluent-manager-for-apache-flink ports: - protocol: TCP port: 8081 # Kafka and Schema Registry - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: confluent ports: - protocol: TCP port: 9092 - protocol: TCP port: 8081 ``` Adjust the example to your cluster: - If the DNS pods of your cluster have other labels or ports than `k8s-app: kube-dns` and `53`, change the DNS rule to match them. - If you set `cmf.internalPort`, use its value instead of `8081` in the rule for CMF’s internal port. - For services outside the cluster, add `ipBlock` rules with their addresses. If you allow a wide range such as `0.0.0.0/0`, add `169.254.169.254/32` to its `except` list. - If you set `nameOverride`, the pod labels use that name instead of `confluent-manager-for-apache-flink`. - If a NetworkPolicy selects the CMF pod, admit the internal port from the worker pod there too, as in the following rule. If it also limits the egress of the CMF pod, allow egress to the worker pod on ports `8090` and up. ```yaml ingress: - from: - podSelector: matchLabels: app.kubernetes.io/name: confluent-manager-for-apache-flink-compile-worker ports: - protocol: TCP port: 8081 ``` ### Limit credentials and identities - Keep cloud identities off the worker pod. It runs as the namespace’s ServiceAccount `default`, without a Kubernetes API token, and needs no access to artifact storage, because CMF serves the artifacts. Do not annotate `default` for IAM roles for service accounts (IRSA) or Workload Identity. - If a catalog needs a cloud identity, give the worker pod a ServiceAccount of its own with `workerDeploymentOverrides.spec.template.spec.serviceAccountName`, and grant it only what all environments can share. It must not be the CMF ServiceAccount. On Google Kubernetes Engine (GKE), Workload Identity uses the metadata endpoint, so your NetworkPolicy must then allow `169.254.169.254`. - Give the workers only what catalogs and connectors need. See [Give the workers what catalogs and connectors need](#cmf-compile-workers-config). - Restrict `get` on Secrets in the CMF namespace. Any certificate from the workers’ CA with the common name `cmf` can send statements to the workers. ### Set permissions per environment Statements of the same environment share compile workers. Grant the permissions to upload artifacts and to create statements in an environment only to users you trust with all statements and artifacts of that environment, and keep teams that must not see each other’s statements in separate environments. See [Configure Access Control for Confluent Manager for Apache Flink](../configure/access-control.md#cmf-access-control) and [Access control](../configure/artifacts.md#cmf-artifacts-rbac). All compile workers of a CMF installation run in one pod and serve all of its environments. If environments belong to parties that must be fully separated from each other, run a separate CMF installation for each of them. ### Keep the CMF pod trusted Shared compilers run in the CMF pod and get each statement together with the catalog credentials of its environment. Add containers to the CMF pod, for example with `extraContainers`, only if you trust them with this data. ## Flink versions and compilers CMF 2.5 compiles for Flink 2.x compute pools only on a compiler for the 2.3 line, either a shared compiler or a compile worker. Keep the `"2.3"` entry in `sharedCompilers`, and in `workers.compilers` if you set it, if you run statements on Flink 2.x compute pools. Compute pools of each line need a minimum `cp-flink-sql` image; see [the image requirements of compute pools](../configure/compute-pools.md#cmf-compute-pools-image). The following need a compute pool on Flink 2.x: - Models: `CREATE MODEL`, `ALTER MODEL`, `DROP MODEL`, `SHOW MODELS`, `DESCRIBE MODEL`, and queries that call `ML_PREDICT` or `ML_EVALUATE` on a model. On a Flink 1.x compute pool, CMF rejects them with a message that models require Flink 2.x. - [Batch statements](../jobs/sql-statements/batch-statements.md#flink-sql-batch-statements), with `execution.runtime-mode=BATCH`. - Statements that use the tables of a Kafka database with `subjectNameStrategy: AssociatedNameStrategy`. CMF accepts such a database only when a compiler for Flink 2.x is configured. CMF 2.5 runs statements on compute pools with `flinkVersion` `v1_19` or `v2_3`. It compiles statements of `v1_19` compute pools on the 1.19 compiler and those of `v2_3` compute pools on the 2.3 compiler. If no compiler serves the compute pool’s Flink version, creating the statement fails. Statements that do not run on a compute pool, such as those that list catalogs, databases, and tables, use the compiler with the newest version. Compute pools on Flink 1.x and 2.x in the same environment share its catalogs. A table or view that uses SQL features of Flink 2.x fails when a statement on a Flink 1.x compute pool uses it. Create and change such tables and views from a compute pool on Flink 2.x. ## Troubleshoot compilers | Error when you create a statement | What to do | |---------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `Statement uses artifact '' of environment '', which is not
available to it ...` | Check that the artifact exists in the statement’s environment. If it does, CMF has no
compile workers. If your values set `cmf.statementService.workers.count: 0`, remove it or
set it above `0`. With `sharedCompilers: []`, also set `workers.compilers`. | | `No shared compiler serves Flink version ''. Configured compilers: ...`, or the
same with `No compile worker` | Add a compiler for the compute pool’s Flink line to `sharedCompilers`, or to
`workers.compilers` for environments with artifacts: `"1.19"` for `v1_19` compute
pools, or `"2.3"` for `v2_3` compute pools. | | `No shared compilers or compile workers are configured, so CMF compiled the statement
with its own Flink 1.19 compiler ...` | The compute pool runs Flink 2.x. Add a `"2.3"` entry to `sharedCompilers`. | | `Failed to reach the shared compiler ... See the log of container compiler-.` | The compiler has not started yet or has stopped. Read the log of the container that the
message names, as described in [Check the shared compilers](#cmf-shared-compilers-check), and create the statement
again when the compiler runs. | | `Failed to reach the compile worker ...` | A worker is not running, or CMF and the worker do not accept each other’s certificates.
CMF tries again until `compileTimeout`, so the statement fails after about 2 minutes
with the default. A worker pod that is `Pending` or in `ImagePullBackOff` has no log;
see its events with
`kubectl describe pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink-compile-worker --namespace `.
Otherwise read the worker’s log with
`kubectl logs deployment/cmf-compile-worker --namespace -c worker-`,
adding `--previous` for a container that restarted. If the message says that the TLS
handshake failed or that the worker closed the connection without an answer, check the
Secrets in [Compile worker certificates](#cmf-compile-workers-certificates). | | Connection, TLS, or credential errors from Kafka, Schema Registry, or object storage, only in
environments with artifacts | These statements compile on the compile workers, which do not get the top-level
`jvmArgs`, `extraEnv`, `mountedVolumes`, or the cloud identity of CMF. Give the
workers what the catalogs need, as described in [Give the workers what catalogs and connectors need](#cmf-compile-workers-config), and
check that your NetworkPolicy lets the worker pod reach these services. | | `The shared compiler is busy ...` or `All compile workers are busy ...` (HTTP 429) | Retry later, or raise the compiler’s `maxConcurrentCompiles` or
`cmf.statementService.workers.count`. | | No error, but statements in environments with artifacts take several seconds or more to
create | The workers restart to switch between environments and Flink lines. Raise
`cmf.statementService.workers.count` as described in [Size the compile workers](#cmf-compile-workers-sizing). | | `Compilation on compile worker took longer than the compile timeout ...` | The statement is created in phase `FAILED`. Raise `workers.compileTimeout` if the
statement needs more time. | | `Artifact '' of environment '' ... does not fit the compile
worker's artifact cache ...` | Raise `workers.volumeSizeLimit` and the workers’ memory limit by the same amount. | To follow a statement on the compile workers, find its `metadata.uid` in the CMF log line `Dispatching compile to compile worker: requestId='', uuid='', ..., worker=`, then search the worker’s log for the request ID: ```bash kubectl logs deployment/cmf-compile-worker --namespace -c worker- | grep ``` When you operate the workers: - CMF and the workers must run the same CMF version. While `helm upgrade` replaces the CMF pod and the worker pod, statements that compile on the workers fail until both run the new version. - The worker pod is not ready while any worker is down, so `helm install --wait` and `helm upgrade --wait` time out when a worker cannot start, and `--atomic` then rolls back the whole release. - If the node of the worker pod is lost, the old pod can stay in `Terminating`. Delete it with `kubectl delete pod --namespace --force --grace-period=0`. ## Related content - [Manage Artifacts in Confluent Manager for Apache Flink](../configure/artifacts.md#cmf-artifacts) - [User-Defined Functions in Confluent Manager for Apache Flink](../jobs/sql-statements/user-defined-functions.md#cmf-sql-udf) - [Custom Connectors and Formats in Confluent Manager for Apache Flink](../jobs/sql-statements/custom-connectors.md#cmf-sql-custom-connectors) - [Manage Compute Pools in Confluent Manager for Apache Flink](../configure/compute-pools.md#cmf-compute-pools) - [Upgrade to CMF 2.5](upgrade-cmf.md#upgrade-cmf-2-5) - [Install Confluent Manager for Apache Flink with Helm](helm.md#install-cmf-helm)