Flink Jobs for Confluent Manager for Apache Flink

Flink applications are deployed in Confluent Platform with Confluent Manager for Apache Flink® (CMF), which is a central management component that enables users to securely manage a fleet of Flink jobs across multiple environments.

CMF supports two ways to create and manage Flink jobs.

• Applications offer the full flexibility of Flink programs. You can use all JVM-based APIs (DataStream, Table, SQL) and submit and manage applications packaged as JAR files.

• Statements offer the ease-of-use of a native SQL interface. You can run table operations, interactive SQL statements, or submit and manage long-running SQL statements.

Applications and statements can be tuned with Flink configuration options to meet their performance, fault-tolerance, and security requirements. Applications accept all Flink configuration options except the reserved keys listed below. On a statement, only job-level options take effect. Cluster-level options, such as memory settings and metrics reporters, are ignored when set on a statement. Setting state.checkpoints.dir or state.savepoints.dir on a statement has no effect on savepoints or on the state kept across a stop and resume. For statements, set these options on the compute pool in spec.clusterSpec.flinkConfiguration or in the environment’s computePoolDefaults. For more information, see Manage Compute Pools in Confluent Manager for Apache Flink.

CMF reserves some Flink configuration keys and rejects them with HTTP 400:

• Applications: web.submit.jar-sources and keys that start with kubernetes.operator.cmf..

• Statements, in their flinkConfiguration or in the environment’s statement defaults: pipeline.jars, pipeline.classpaths, web.submit.jar-sources, and keys that start with user.artifacts. or kubernetes.operator.cmf..