Configure Storage for Confluent Manager for Apache Flink

Flink applications are deployed in Confluent Platform with Confluent Manager for Apache Flink® (CMF), a central management component that enables users to securely manage a fleet of Flink jobs across multiple environments. This topic describes how to configure storage for CMF.

Starting with CMF 2.1.1, you can configure CMF to use PostgreSQL as the metadata backend instead of the default SQLite database. PostgreSQL is recommended for production deployments and provides better support for high availability scenarios.

Starting with CMF 2.2.0, you can also configure CMF to use Microsoft SQL Server as the metadata backend.

Database options

CMF supports two database types. After you have deployed with one database type, changing to another type requires data migration. In addition, changing the database type after initial deployment can result in data loss.

Supported database types:

  • local (SQLite): The default option, stored in a persistent volume. Good for development, testing, and small deployments. Note that using SQLite on Network File Systems (NFS) is not recommended. For more information, see the SQLite FAQ.

  • jdbc (PostgreSQL or SQL Server): External JDBC database. Recommended for production deployments and high availability. - PostgreSQL: Available starting with CMF 2.1.1 - Microsoft SQL Server: Available starting with CMF 2.2.0

Important

SQLite is not supported in the Linux s390x architecture Docker images.

Configure PostgreSQL

To configure CMF to use PostgreSQL as the metadata backend, you need to:

  1. Create a Kubernetes secret containing the PostgreSQL password.

  2. Configure the database settings in your Helm values file.

Prerequisites

Before configuring PostgreSQL, ensure you have:

  • A PostgreSQL server accessible from your Kubernetes cluster

  • A database created in PostgreSQL

  • A PostgreSQL user with appropriate permissions

  • A Kubernetes secret containing the PostgreSQL password

Example configuration

The following configuration file shows how to provision Confluent Manager for Apache Flink with PostgreSQL as the metadata backend. You pass this configuration file to Helm when you install CMF.

# postgresql-values.yaml
cmf:
  database:
    # Set type to "jdbc" to use PostgreSQL
    type: jdbc
    jdbc:
      # Database engine type
      engine: postgresql
      # Database server hostname or IP address
      url: postgresql.example.com
      # Database server port (default: 5432 for PostgreSQL)
      port: 5432
      # Database name (must be created beforehand)
      database: cmf
      # Database username
      user: cmf
      # Database options, string to append to the JDBC URL for extra parameters
      # Example for PostgreSQL: options: "sslmode=require"
      # Refer to your database's documentation for supported parameters and the exact format
      options: ""
      # Database password configuration
      password:
        # Name of Kubernetes secret containing the password
        kubernetesSecretName: cmf-postgresql-password
        # Key within the secret that contains the password
        kubernetesSecretProperty: password

Create the password secret

Before installing CMF, create a Kubernetes secret containing the PostgreSQL password:

kubectl create secret generic cmf-postgresql-password \
  --from-literal=password=<your-postgresql-password> \
  --namespace=<cmf-namespace>

Install with Helm

When you make the helm install call, use the -f flag to pass the YAML file with the PostgreSQL configuration:

helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
  -f postgresql-values.yaml

Configure SQL Server

Starting with CMF 2.2.0, you can configure CMF to use Microsoft SQL Server as the metadata backend.

To configure CMF to use SQL Server as the metadata backend, you need to:

  1. Create a Kubernetes secret containing the SQL Server password.

  2. Configure the database settings in your Helm values file.

Prerequisites

Before configuring SQL Server, ensure you have:

  • A SQL Server instance accessible from your Kubernetes cluster

  • A database created in SQL Server

  • A SQL Server user with appropriate permissions

  • A Kubernetes secret containing the SQL Server password

Example configuration

The following configuration file shows how to provision Confluent Manager for Apache Flink with SQL Server as the metadata backend. You pass this configuration file to Helm when you install CMF.

# sqlserver-values.yaml
cmf:
  database:
    # Set type to "jdbc" to use SQL Server
    type: jdbc
    jdbc:
      # Database engine type
      engine: sqlserver
      # Database server hostname or IP address
      url: sqlserver.example.com
      # Database server port (default: 1433 for SQL Server)
      port: 1433
      # Database name (must be created beforehand)
      database: cmf
      # Database username
      user: cmf
      # Database options, string to append to the JDBC URL for extra parameters
      # Example for SQL Server: options: "encrypt=true;trustServerCertificate=true;"
      # Refer to your database's documentation for supported parameters and the exact format
      options: ""
      # Database password configuration
      password:
        # Name of Kubernetes secret containing the password
        kubernetesSecretName: cmf-sqlserver-password
        # Key within the secret that contains the password
        kubernetesSecretProperty: password

Create the password secret

Before installing CMF, create a Kubernetes secret containing the SQL Server password:

kubectl create secret generic cmf-sqlserver-password \
  --from-literal=password=<your-sqlserver-password> \
  --namespace=<cmf-namespace>

Install with Helm

When you make the helm install call, use the -f flag to pass the YAML file with the SQL Server configuration:

helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
  -f sqlserver-values.yaml

Configuration fields

The following table describes the database configuration fields:

Field

Description

database.type

Type of database to use. Set to jdbc for PostgreSQL or SQL Server. Default is local (SQLite).

database.jdbc.engine

Database engine type. Supported values: postgresql (available starting with CMF 2.1.1) or sqlserver (available starting with CMF 2.2.0).

database.jdbc.url

Database server hostname or IP address. Required when using jdbc type.

database.jdbc.port

Database server port. Default is 5432 for PostgreSQL or 1433 for SQL Server.

database.jdbc.database

Name of the database. The database must be created beforehand.

database.jdbc.user

Database username. Required when using jdbc type.

database.jdbc.options

Optional string to append to the JDBC URL for extra connection parameters. For example, for SQL Server: encrypt=true;trustServerCertificate=true;. For PostgreSQL: sslmode=require. Refer to your database’s documentation for supported parameters and the exact format.

database.jdbc.password.kubernetesSecretName

Name of the Kubernetes secret containing the database password. Required when using jdbc type.

database.jdbc.password.kubernetesSecretProperty

Key within the secret that contains the password. Required when using jdbc type.

Related content