Configure Storage for Confluent Manager for Apache Flink

Starting with CMF 2.1.1, you can configure CMF to use PostgreSQL as the metadata backend instead of the default SQLite database. PostgreSQL is recommended for production deployments and provides better support for high availability scenarios.

Database options

CMF supports two database types:

local (SQLite): The default option, stored in a persistent volume. Good for development, testing, and small deployments.
jdbc (PostgreSQL): External JDBC database. Recommended for production deployments and high availability.

Important

Do not change the database type after initial deployment without proper data migration. Changing the database type can result in data loss.

Configure PostgreSQL

To configure CMF to use PostgreSQL as the metadata backend, you need to:

Create a Kubernetes secret containing the PostgreSQL password.
Configure the database settings in your Helm values file.

Prerequisites

Before configuring PostgreSQL, ensure you have:

A PostgreSQL server accessible from your Kubernetes cluster
A database created in PostgreSQL
A PostgreSQL user with appropriate permissions
A Kubernetes secret containing the PostgreSQL password

Example configuration

The following configuration file shows how to provision Confluent Manager for Apache Flink with PostgreSQL as the metadata backend. You pass this configuration file to Helm when you install CMF.

# postgresql-values.yaml
cmf:
  database:
    # Set type to "jdbc" to use PostgreSQL
    type: jdbc
    jdbc:
      # Database engine type
      engine: postgresql
      # Database server hostname or IP address
      url: postgresql.example.com
      # Database server port (default: 5432 for PostgreSQL)
      port: 5432
      # Database name (must be created beforehand)
      database: cmf
      # Database username
      user: cmf
      # Database password configuration
      password:
        # Name of Kubernetes secret containing the password
        kubernetesSecretName: cmf-postgresql-password
        # Key within the secret that contains the password
        kubernetesSecretProperty: password

Create the password secret

Before installing CMF, create a Kubernetes secret containing the PostgreSQL password:

kubectl create secret generic cmf-postgresql-password \
  --from-literal=password=<your-postgresql-password> \
  --namespace=<cmf-namespace>

Install with Helm

When you make the helm install call, use the -f flag to pass the YAML file with the PostgreSQL configuration:

helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink \
  -f postgresql-values.yaml

Configuration fields

The following table describes the database configuration fields:

Field	Description
`database.type`	Type of database to use. Set to `jdbc` for PostgreSQL. Default is `local` (SQLite).
`database.jdbc.engine`	Database engine type. Currently only `postgresql` is supported.
`database.jdbc.url`	Database server hostname or IP address. Required when using `jdbc` type.
`database.jdbc.port`	Database server port. Default is `5432` for PostgreSQL.
`database.jdbc.database`	Name of the database. The database must be created beforehand.
`database.jdbc.user`	Database username. Required when using `jdbc` type.
`database.jdbc.password.kubernetesSecretName`	Name of the Kubernetes secret containing the database password. Required when using `jdbc` type.
`database.jdbc.password.kubernetesSecretProperty`	Key within the secret that contains the password. Required when using `jdbc` type.