Configure Storage for Confluent Manager for Apache Flink
Flink applications are deployed in Confluent Platform with Confluent Manager for Apache Flink® (CMF), a central management component that enables users to securely manage a fleet of Flink jobs across multiple environments. This topic describes how to configure storage for CMF.
Starting with CMF 2.1.1, you can configure CMF to use PostgreSQL as the metadata backend instead of the default SQLite database. PostgreSQL is recommended for production deployments and provides better support for high availability scenarios.
Starting with CMF 2.2.0, you can also configure CMF to use Microsoft SQL Server as the metadata backend.
Starting with CMF 2.4.0, you can also configure CMF to use IBM Db2 as the metadata backend.
Database options
CMF supports two database types. After you have deployed with one database type, changing to another type requires data migration. In addition, changing the database type after initial deployment can result in data loss.
Supported database types:
local (SQLite): The default option, stored in a persistent volume. Good for development, testing, and small deployments. Note that using SQLite on Network File Systems (NFS) is not recommended. For more information, see the SQLite FAQ.
jdbc (PostgreSQL, SQL Server, or Db2): External JDBC database. Recommended for production deployments and high availability.
PostgreSQL: Available starting with CMF 2.1.1
Microsoft SQL Server: Available starting with CMF 2.2.0
IBM Db2: Available starting with CMF 2.4.0
Important
SQLite is not supported in the Linux s390x architecture Docker images.
Configure PostgreSQL
To configure CMF to use PostgreSQL as the metadata backend, you need to:
Create a Kubernetes secret containing the PostgreSQL password.
Configure the database settings in your Helm values file.
Prerequisites
Before configuring PostgreSQL, ensure you have:
A PostgreSQL server accessible from your Kubernetes cluster
A database created in PostgreSQL
A PostgreSQL user with appropriate permissions
A Kubernetes secret containing the PostgreSQL password
Example configuration
The following configuration file shows how to provision Confluent Manager for Apache Flink with PostgreSQL as the metadata backend. You pass this configuration file to Helm when you install CMF.
# postgresql-values.yaml
cmf:
database:
# Set type to "jdbc" to use PostgreSQL
type: jdbc
jdbc:
# Database engine type
engine: postgresql
# Database server hostname or IP address
url: postgresql.example.com
# Database server port (default: 5432 for PostgreSQL)
port: 5432
# Database name (must be created beforehand)
database: cmf
# Database username
user: cmf
# Database options, string to append to the JDBC URL for extra parameters
# Example for PostgreSQL: options: "sslmode=require"
# Refer to your database's documentation for supported parameters and the exact format
options: ""
# Database password configuration
password:
# Name of Kubernetes secret containing the password
kubernetesSecretName: cmf-postgresql-password
# Key within the secret that contains the password
kubernetesSecretProperty: password
Create the password secret
Before installing CMF, create a Kubernetes secret containing the PostgreSQL password:
kubectl create secret generic cmf-postgresql-password \
--from-literal=password=<your-postgresql-password> \
--namespace=<cmf-namespace>
Install with Helm
When you make the helm install call, use the -f flag to pass the YAML file with the PostgreSQL configuration:
helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
-f postgresql-values.yaml
Configure SQL Server
Starting with CMF 2.2.0, you can configure CMF to use Microsoft SQL Server as the metadata backend.
To configure CMF to use SQL Server as the metadata backend, you need to:
Create a Kubernetes secret containing the SQL Server password.
Configure the database settings in your Helm values file.
Prerequisites
Before configuring SQL Server, ensure you have:
A SQL Server instance accessible from your Kubernetes cluster
A database created in SQL Server
A SQL Server user with appropriate permissions
A Kubernetes secret containing the SQL Server password
Example configuration
The following configuration file shows how to provision Confluent Manager for Apache Flink with SQL Server as the metadata backend. You pass this configuration file to Helm when you install CMF.
# sqlserver-values.yaml
cmf:
database:
# Set type to "jdbc" to use SQL Server
type: jdbc
jdbc:
# Database engine type
engine: sqlserver
# Database server hostname or IP address
url: sqlserver.example.com
# Database server port (default: 1433 for SQL Server)
port: 1433
# Database name (must be created beforehand)
database: cmf
# Database username
user: cmf
# Database options, string to append to the JDBC URL for extra parameters
# Example for SQL Server: options: "encrypt=true;trustServerCertificate=true;"
# Refer to your database's documentation for supported parameters and the exact format
options: ""
# Database password configuration
password:
# Name of Kubernetes secret containing the password
kubernetesSecretName: cmf-sqlserver-password
# Key within the secret that contains the password
kubernetesSecretProperty: password
Create the password secret
Before installing CMF, create a Kubernetes secret containing the SQL Server password:
kubectl create secret generic cmf-sqlserver-password \
--from-literal=password=<your-sqlserver-password> \
--namespace=<cmf-namespace>
Install with Helm
When you make the helm install call, use the -f flag to pass the YAML file with the SQL Server configuration:
helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
-f sqlserver-values.yaml
Configure Db2
Starting with CMF 2.4.0, you can configure CMF to use IBM Db2 as the metadata backend.
Important
The IBM Db2 JDBC driver (jcc) is licensed under IBM’s International Program License Agreement (IPLA) and is not bundled with CMF. You must supply the driver yourself under IBM’s license terms. CMF loads the driver at startup from the loader-path directory (default /opt/cmf/jdbc-drivers, override with the CMF_DRIVER_PATH environment variable) and fails fast with an actionable error if the driver class is not present.
To configure CMF to use Db2 as the metadata backend, you need to:
Create a Kubernetes secret containing the Db2 password.
Provision the IBM Db2 JDBC driver into the CMF pod at the loader-path directory.
Configure the database settings in your Helm values file.
Prerequisites
Before configuring Db2, ensure you have:
A Db2 instance accessible from your Kubernetes cluster
A database created in Db2
A Db2 user with appropriate permissions on that database
A Kubernetes secret containing the Db2 password
Access to the IBM Db2 JDBC driver (
jcc) JAR from a source you are licensed to use, such as your internal artifact repository
Provision the Db2 JDBC driver
Because the IBM Db2 JDBC driver is not bundled with CMF, you provision it into the CMF pod at startup. The recommended pattern uses an initContainer that downloads the driver JAR into a shared emptyDir volume mounted at the CMF loader path. The concrete YAML for this pattern is included in the db2-values.yaml example under Example configuration below. The example uses an example driver version and a placeholder fetch source; replace both with a driver version that matches your Db2 server and a source you are licensed to pull from, such as your internal artifact repository.
Important
mountedVolumes and initContainers are top-level Helm values. If your existing values file already sets either key, append the Db2 driver entries to your existing lists rather than replacing them.
Example configuration
The following configuration file shows how to provision Confluent Manager for Apache Flink with Db2 as the metadata backend. It combines the database settings with the driver initContainer and mounted volume from the previous step. You pass this configuration file to Helm when you install CMF.
# db2-values.yaml
cmf:
database:
# Set type to "jdbc" to use Db2
type: jdbc
jdbc:
# Database engine type
engine: db2
# Database server hostname or IP address
url: db2.example.com
# Database server port (default: 50000 for Db2)
port: 50000
# Database name (must be created beforehand)
database: cmf
# Database username
user: cmf
# Database options, string to append to the JDBC URL for extra parameters.
# For Db2, options are appended with a leading colon; multiple options are
# separated with semicolons, for example:
# options: "sslConnection=true"
# Refer to your database's documentation for supported parameters and the exact format.
options: ""
# Database password configuration
password:
# Name of Kubernetes secret containing the password
kubernetesSecretName: cmf-db2-password
# Key within the secret that contains the password
kubernetesSecretProperty: password
# Provision the IBM Db2 JDBC driver into the CMF pod at the loader path.
mountedVolumes:
volumes:
- name: db2-driver
emptyDir: {}
volumeMounts:
- name: db2-driver
mountPath: /opt/cmf/jdbc-drivers
initContainers:
- name: fetch-db2-driver
image: <image-with-curl>
command: ["sh", "-c"]
args:
- curl -fsSL -o /drivers/jcc.jar <your-repo>/com/ibm/db2/jcc/11.5.9.0/jcc-11.5.9.0.jar
volumeMounts:
- name: db2-driver
mountPath: /drivers
Create the password secret
Before installing CMF, create a Kubernetes secret containing the Db2 password:
kubectl create secret generic cmf-db2-password \
--from-literal=password=<your-db2-password> \
--namespace=<cmf-namespace>
Install with Helm
When you make the helm install call, use the -f flag to pass the YAML file with the Db2 configuration:
helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
-f db2-values.yaml
Configuration fields
The following table describes the database configuration fields:
Field | Description |
|---|---|
| Type of database to use. Set to |
| Database engine type. Supported values: |
| Database server hostname or IP address. Required when using |
| Database server port. Default is |
| Name of the database. The database must be created beforehand. |
| Database username. Required when using |
| Optional string to append to the JDBC URL for extra connection parameters. For example, for SQL Server: |
| Name of the Kubernetes secret containing the database password. Required when using |
| Key within the secret that contains the password. Required when using |