Configure Embedded MDS for Confluent Manager for Apache Flink
Confluent Manager for Apache Flink® (CMF) can run an embedded Metadata Service (MDS) inside its own process. The embedded MDS provides authentication and role-based access control (RBAC) for CMF resources without requiring an external Kafka cluster solely for security. Authentication and access control between Flink clusters and Kafka is still governed by the Kafka broker-hosted MDS.
Clients reach the embedded MDS through the CMF URL, which exposes the standard MDS REST surface (/security/1.0/...). Existing tools, including the Confluent CLI and CMF UI, work without modification.
Note
Confluent Control Center, which is deprecated, is not supported with the embedded MDS for authentication and authorization. Use the CMF UI if you want to use CMF with the embedded MDS.
When to use the embedded MDS
Use the embedded MDS when:
You are deploying CMF without Kafka or a compatible Kafka deployment and need authentication or RBAC for Flink resources.
You want to simplify your deployment by removing the dependency on an external MDS running alongside Kafka.
Continue using an external broker-hosted MDS (cp-mds mode) when:
You already run Confluent Platform with MDS on Kafka and want a unified RBAC surface across Kafka and Flink resources.
Your organization manages RBAC for all Confluent Platform components through a single broker-hosted MDS.
Deployment modes
CMF supports two mutually exclusive authorization modes. The cmf.authorization.authority Helm value controls where CMF authorizes Flink resources:
Mode | Authorization for Flink resources |
|---|---|
| Defers to an external broker-hosted MDS. A broker-hosted MDS is required. |
| Uses the embedded MDS in CMF. A broker-hosted MDS is not required. |
Kafka and other Confluent Platform resources are always authorized by their broker-hosted MDS, if you run one. This setting does not change how those resources are authorized.
Per CMF installation, Flink resources are managed in only one MDS. You cannot split Flink RBAC across both the embedded MDS and a broker-hosted MDS.
User stores
A user store is the authentication backend that the embedded MDS uses to validate user identities. When a client authenticates, the embedded MDS checks the provided credentials or tokens against the configured user store.
Set the user store with the cmf.mds.user-store value. The embedded MDS supports the following user stores:
OAUTHValidates tokens issued by an external identity provider (IdP) through its JWKS (JSON Web Key Set) endpoint. The embedded MDS does not issue tokens in this mode; it only validates them.
LDAPAuthenticates users against an LDAP directory through HTTP Basic authentication. The embedded MDS issues its own JWTs after validating LDAP credentials.
LDAP_WITH_OAUTHCombines LDAP and OpenID Connect (OIDC) SSO. Users can authenticate with LDAP credentials (Basic auth) or through an OIDC SSO flow. The embedded MDS issues JWTs for LDAP logins and validates IdP-issued tokens for SSO logins.
FILEReads credentials from a properties file. The embedded MDS issues its own JWTs after validating credentials from the file.
If you do not set cmf.mds.user-store, CMF auto-detects the store from other settings:
FILEwhenuser-store-file-pathis set.LDAP_WITH_OAUTHwhen bothjwks-endpoint-urlandcallback-handler-classare set.OAUTHwhenjwks-endpoint-urlis set andcallback-handler-classis not.LDAPwhen onlycallback-handler-classis set.NONEwhen none of these are set.
Configure the embedded MDS
The embedded MDS requires a PEM-encoded RSA key pair for signing and validating JWT tokens. The private key path is set with token-key-path and the public key path with public-key-path (see Configuration reference). You can use any valid PEM-encoded RSA key pair. The following example shows how to generate one with openssl:
# Generate a 2048-bit RSA private key
openssl genrsa -out token-key.pem 2048
# Extract the public key
openssl rsa -in token-key.pem -outform PEM -pubout -out public-key.pem
You can store the keys as Kubernetes Secrets and mount them into the CMF pod using mountedVolumes, as shown in the examples below.
The following sections show common configuration examples. For a full list of available Helm values, see Configuration reference.
OIDC (OAuth)
In this scenario, an external identity provider such as Keycloak, Okta, or Azure AD issues tokens. The embedded MDS validates tokens using the IdP’s JWKS endpoint and makes RBAC decisions from role bindings stored in CMF’s database.
Create a values file embedded-mds-oidc-values.yaml:
cmf:
# Enable the embedded MDS
mds:
enabled: true
port: 8090
authentication-method: BEARER
token-key-path: /mnt/secrets/mds/tokenKeypair.pem
user-store: OAUTH
jwks-endpoint-url: <idp-jwks-url> # e.g. https://idp.example.com/.well-known/jwks.json
expected-issuer: <idp-issuer-url> # e.g. https://idp.example.com/realms/my-realm
super-users: "User:<admin-principal>"
# OIDC SSO properties for the CMF UI login. If you only use
# the API directly, you can omit extra-configs and the ui block below.
# See also the "LDAP and OIDC properties for extra-configs" section below.
extra-configs:
confluent.metadata.server.sso.mode: oidc
confluent.oidc.idp.client.id: <client-id>
confluent.oidc.idp.client.secret: <client-secret>
confluent.oidc.idp.issuer: <idp-issuer-url>
confluent.oidc.idp.jwks.endpoint.uri: <idp-jwks-url>
confluent.oidc.idp.authorize.base.endpoint.uri: <idp-authorize-url>
confluent.oidc.idp.token.base.endpoint.uri: <idp-token-url>
# Enable RBAC authorization through the embedded MDS
authorization:
authority: cmf
# Configure CMF API authentication to accept IdP tokens
authentication:
type: oauth
config:
oauthbearer.jwks.endpoint.url: <idp-jwks-url>
oauthbearer.expected.issuer: <idp-issuer-url>
oauthbearer.sub.claim.name: sub
public.key.path: /mnt/secrets/mds/tokenPublicKey.pem
confluent.metadata.bootstrap.server.urls: http://localhost:8090
confluent.metadata.http.auth.credentials.provider: OAUTHBEARER
confluent.metadata.oauthbearer.token.endpoint.url: <idp-token-url>
confluent.metadata.oauthbearer.login.client.id: <client-id>
confluent.metadata.oauthbearer.login.client.secret: <client-secret>
# Configure the CMF UI login. The embedded MDS only validates
# IdP-issued tokens, so disable the Basic login form and enable SSO
# so browser users are redirected to the IdP for authentication.
ui:
auth:
basicAuthEnabled: false
ssoEnabled: true
kafka:
oauthbearerAllowedUrls: "<idp-base-url>"
# Mount the MDS key pair (contains both tokenKeypair.pem and
# tokenPublicKey.pem in a single secret).
mountedVolumes:
volumes:
- name: mds-keypair
secret:
secretName: mds-keypair
volumeMounts:
- name: mds-keypair
mountPath: /mnt/secrets/mds
readOnly: true
Install or upgrade CMF with the values file:
helm upgrade --install cmf confluent/confluent-manager-for-apache-flink \
-f embedded-mds-oidc-values.yaml
LDAP with mTLS
All connections require a client certificate, so in this scenario, mTLS secures the transport layer, while the embedded MDS authenticates users against an LDAP directory and issues its own JWTs. Users log in with LDAP credentials, the embedded MDS validates the credentials and mints a JWT, and CMF validates that token locally using the MDS public key.
Create a values file embedded-mds-ldap-values.yaml:
cmf:
# Enable the embedded MDS with LDAP user store
mds:
enabled: true
port: 8090
authentication-method: BEARER
token-key-path: /mnt/secrets/mds/tokenKeypair.pem
public-key-path: /mnt/secrets/mds/tokenPublicKey.pem
user-store: LDAP
callback-handler-class: io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler
sasl-mechanism: PLAIN
super-users: "User:<admin-principal>"
extra-configs:
ldap.java.naming.provider.url: ldap://<ldap-host>:<ldap-port>
ldap.java.naming.security.principal: <ldap-bind-dn>
ldap.java.naming.security.credentials: <ldap-bind-password>
ldap.java.naming.security.authentication: simple
ldap.user.search.base: <user-search-base-dn>
ldap.user.name.attribute: <user-name-attr> # e.g. uid or sAMAccountName
ldap.user.object.class: <user-object-class> # e.g. inetOrgPerson
# Optional: enable LDAP group resolution for group-based role bindings
ldap.search.mode: GROUPS
ldap.group.search.base: <group-search-base-dn>
ldap.group.object.class: groupOfNames
ldap.group.name.attribute: cn
ldap.group.member.attribute: member
ldap.group.member.attribute.pattern: <regex-to-extract-uid-from-member-dn>
# Enable RBAC authorization through the embedded MDS
authorization:
authority: cmf
# mTLS for transport security
ssl:
keystore: /mnt/certs/keystore.jks
keystore-password: <keystore-password>
trust-store: /mnt/certs/truststore.jks
trust-store-password: <truststore-password>
client-auth: need
# Validate MDS-issued JWTs locally using the MDS public key.
# Use "type: oauth" even with mTLS transport -- mTLS secures the connection,
# but the MDS-issued JWT carries the user identity.
authentication:
type: oauth
config:
public.key.path: /mnt/secrets/mds/tokenPublicKey.pem
confluent.metadata.bootstrap.server.urls: http://localhost:8090
kafka:
oauthbearerAllowedUrls: "http://localhost:8090"
ui:
auth:
basicAuthEnabled: true
ssoEnabled: false
# Mount the MDS key pair (contains both tokenKeypair.pem and
# tokenPublicKey.pem) and certificate stores.
mountedVolumes:
volumes:
- name: mds-keypair
secret:
secretName: mds-keypair
- name: certs
secret:
secretName: cmf-certs
volumeMounts:
- name: mds-keypair
mountPath: /mnt/secrets/mds
readOnly: true
- name: certs
mountPath: /mnt/certs
readOnly: true
Install or upgrade CMF with the values file:
helm upgrade --install cmf confluent/confluent-manager-for-apache-flink \
-f embedded-mds-ldap-values.yaml
Tip
To also enable SSO alongside LDAP (so users can log in with either LDAP credentials or OIDC), set cmf.mds.user-store to LDAP_WITH_OAUTH and add the OIDC configuration under cmf.mds.extra-configs. For the required OIDC properties, see LDAP and OIDC properties for extra-configs.
File-based user store
You can use a file-based user store instead of LDAP or OIDC. Create a users.properties file with credentials in the format username: password,GroupName:
admin: admin-secret,Admins
developer: dev-secret,Developers
Store it as a Kubernetes Secret and configure CMF:
cmf:
mds:
enabled: true
port: 8090
authentication-method: BEARER
token-key-path: /mnt/secrets/mds/tokenKeypair.pem
public-key-path: /mnt/secrets/mds/tokenPublicKey.pem
user-store: FILE
user-store-file-path: /mnt/secrets/mds-users/users.properties
super-users: "User:admin"
# Validate MDS-issued JWTs locally using the MDS public key
authentication:
type: oauth
config:
public.key.path: /mnt/secrets/mds/tokenPublicKey.pem
confluent.metadata.bootstrap.server.urls: http://localhost:8090
authorization:
authority: cmf
kafka:
oauthbearerAllowedUrls: "http://localhost:8090"
# Configure the CMF UI login affordance. The FILE user store validates
# username/password credentials, so keep the Basic login form enabled.
# There is no OIDC provider, so SSO stays off.
ui:
auth:
basicAuthEnabled: true
ssoEnabled: false
# Mount the MDS key pair and users.properties file.
mountedVolumes:
volumes:
- name: mds-keypair
secret:
secretName: mds-keypair
- name: mds-users
secret:
secretName: mds-users
volumeMounts:
- name: mds-keypair
mountPath: /mnt/secrets/mds
readOnly: true
- name: mds-users
mountPath: /mnt/secrets/mds-users
readOnly: true
Install or upgrade CMF with the values file:
helm upgrade --install cmf confluent/confluent-manager-for-apache-flink \
-f embedded-mds-file-values.yaml
Authentication-only mode (no RBAC)
You can enable the embedded MDS for authentication without enforcing RBAC. In this mode, CMF authenticates every request but authorizes all authenticated users for all operations.
To run without authorization, omit the entire cmf.authorization block. The embedded MDS still runs and authenticates users; CMF does not make RBAC decisions.
This example uses the FILE user store so that the embedded MDS can issue tokens and authenticate users. Combine it with any user store; configure the store and omit the cmf.authorization block.
cmf:
mds:
enabled: true
port: 8090
authentication-method: BEARER
token-key-path: /mnt/secrets/mds/tokenKeypair.pem
public-key-path: /mnt/secrets/mds/tokenPublicKey.pem
user-store: FILE
user-store-file-path: /mnt/secrets/mds-users/users.properties
authentication:
type: oauth
config:
public.key.path: /mnt/secrets/mds/tokenPublicKey.pem
confluent.metadata.bootstrap.server.urls: http://localhost:8090
kafka:
oauthbearerAllowedUrls: "http://localhost:8090"
ui:
auth:
basicAuthEnabled: true
ssoEnabled: false
mountedVolumes:
volumes:
- name: mds-keypair
secret:
secretName: mds-keypair
- name: mds-users
secret:
secretName: mds-users
volumeMounts:
- name: mds-keypair
mountPath: /mnt/secrets/mds
readOnly: true
- name: mds-users
mountPath: /mnt/secrets/mds-users
readOnly: true
Install or upgrade CMF with the values file:
helm upgrade --install cmf confluent/confluent-manager-for-apache-flink \
-f embedded-mds-authn-only-values.yaml
Manage role bindings
When RBAC is enabled (cmf.authorization.authority: cmf), only super-users have access by default. You must create role bindings to grant access to other users. For a full list of available Flink roles and their permissions, see Configure Access Control for Confluent Manager for Apache Flink.
Role bindings are managed through the standard MDS REST API, accessible through the CMF URL. Use the Confluent CLI or the MDS API directly to manage role bindings.
Set up a Confluent CLI context for the embedded MDS
With the embedded MDS, the MDS URL is the CMF URL itself (for example, http://cmf:8084). This is different from a broker-hosted MDS where the MDS runs on a separate endpoint (typically port 8090). CMF reverse-proxies /security/* requests to the embedded MDS automatically.
Log in to the embedded MDS through the CMF URL:
confluent login --url <cmf-url>
For LDAP or FILE user stores, the Confluent CLI prompts for username and password.
For OIDC, the Confluent CLI uses the SSO device login flow. This requires the OIDC SSO properties to be configured in cmf.mds.extra-configs, including confluent.oidc.idp.device.authorization.endpoint.uri. See LDAP and OIDC properties for extra-configs.
If CMF uses mTLS authentication (cmf.authentication.type: mtls), pass client certificate flags on every Confluent CLI command:
confluent login --url <cmf-url> \
--client-cert-path <path-to-client-cert> \
--client-key-path <path-to-client-key> \
--certificate-authority-path <path-to-ca-cert>
Create role bindings
The role binding commands are the same as with a broker-hosted MDS. For a full reference of roles and resource types, see Configure Access Control for Confluent Manager for Apache Flink.
The CMF-id in the commands below is the CMF cluster ID. With the embedded MDS, this is auto-generated from the database instance ID unless cmf.mds.cluster-id is explicitly set. Discover it with:
confluent cluster list
Grant a user the SystemAdmin role on CMF:
confluent iam rbac role-binding create \
--principal User:<user-name> \
--role SystemAdmin \
--cmf CMF-id
Grant a user access to manage Flink applications in an environment:
confluent iam rbac role-binding create \
--principal User:<user-name> \
--role DeveloperManage \
--cmf CMF-id \
--flink-environment <environment-name> \
--resource FlinkApplication:"*"
Note
When using the embedded MDS, the Confluent CLI context must point to the CMF URL (which reverse-proxies to the embedded MDS). If you also run a broker-hosted MDS for Kafka resources, use a separate Confluent CLI context for each. Switch between contexts with confluent context use <context-name>.
Migrate from broker-hosted MDS to embedded MDS
If you use a broker-hosted MDS for Flink RBAC, you can migrate to the embedded MDS. This migration applies when you want to move Flink RBAC from the broker-hosted MDS to the embedded MDS while keeping Kafka and other Confluent Platform resources on the broker-hosted MDS.
The migration is a three-step process with no authorization gap, because the embedded MDS can accept role-binding writes while CMF continues to authorize through the broker-hosted MDS.
Step 1: Pre-populate role bindings in the embedded MDS
While CMF is still in cp-mds mode, enable the embedded MDS so it can accept role-binding writes:
cmf:
mds:
enabled: true
port: 8090
authentication-method: BEARER
token-key-path: /mnt/secrets/mds/tokenKeypair.pem
public-key-path: /mnt/secrets/mds/tokenPublicKey.pem
super-users: "User:<admin-principal>"
# ... user store config (OIDC, LDAP, or FILE)
authorization:
authority: cp-mds
mdsRestConfig:
endpoint: <broker-mds-url>
authentication:
# ... existing broker-MDS auth config
Apply this configuration with helm upgrade. CMF continues to authorize through the broker-hosted MDS, but the embedded MDS is running and writable.
Copy existing Flink role bindings from the broker-hosted MDS to the embedded MDS. Set up two Confluent CLI contexts, one for each MDS:
# Log in to the broker-hosted MDS
confluent login --url <broker-mds-url>
# Name this context (for example, "broker")
confluent context update --name broker
# Log in to the embedded MDS (through the CMF URL)
confluent login --url <cmf-url>
# Name this context (for example, "cmf-mds")
confluent context update --name cmf-mds
List the existing Flink role bindings from the broker-hosted MDS and recreate them against the embedded MDS:
# List Flink role bindings from the broker context
confluent context use broker
confluent iam rbac role-binding list --cmf CMF-id
# Recreate each binding in the embedded MDS context
confluent context use cmf-mds
confluent iam rbac role-binding create \
--principal User:<principal> \
--role <role> \
--cmf CMF-id \
# ... (repeat for each binding)
CMF is still authorizing through the broker-hosted MDS. The role bindings in the embedded MDS are staged but not yet enforced.
Step 2: Switch to the embedded MDS
Update the Helm values to switch the authorization authority to cmf and remove the mdsRestConfig block:
cmf:
mds:
enabled: true
# ... (same MDS config as Step 1)
authorization:
authority: cmf
# mdsRestConfig is removed
Important
Do not use helm upgrade --reuse-values for this step. The --reuse-values flag retains all previously set values, including the nested mdsRestConfig block, which causes CMF to fail with authority=cmf is incompatible with mdsRestConfig. Instead, use helm upgrade --reset-values -f <complete-values-file> with a values file that omits mdsRestConfig.
Apply with helm upgrade. CMF restarts and begins authorizing through the embedded MDS using the role bindings pre-populated in Step 1. No authorization gap occurs because the bindings were already written to the database.
Step 3: Clean up the broker-hosted MDS (optional)
After confirming that CMF is operating correctly with the embedded MDS, optionally remove the Flink role bindings from the broker-hosted MDS to avoid stale data:
confluent context use broker
confluent iam rbac role-binding delete \
--principal User:<principal> \
--role <role> \
--cmf CMF-id
Configuration reference
The following table lists all Helm values for the embedded MDS under the cmf.mds prefix.
Property | Type | Default | Description |
|---|---|---|---|
| boolean |
| Activates the embedded MDS server. |
| integer |
| TCP port for the embedded MDS listener. |
| string |
| Network address to bind. Use |
| string |
| Advertised URL where this MDS instance is reachable. Defaults to |
| string | auto | MDS cluster identifier. Defaults to the CMF database instance ID. |
| string |
| Authentication method: |
| string | Path to the PEM-encoded RSA private key for JWT signing. Required when | |
| string | Path to the PEM-encoded RSA public key for JWT validation. Required when the MDS issues its own tokens ( | |
| string | Semicolon-separated list of super-user principals, for example | |
| string | auto | User store type: |
| string | Path to the user credentials file for the | |
| string | JWKS endpoint URL for validating IdP-issued tokens. Setting this auto-detects | |
| string | Expected JWT issuer claim for token validation. | |
| string |
| JWT claim used as the principal name. |
| string | JWT claim containing group membership for group-based RBAC. | |
| string | Fully qualified class name of the SASL | |
| string |
| SASL mechanism for the callback handler. Use |
| string | org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule | Fully qualified class name of the JAAS login module. The default Kafka OAuthBearer module works for both secured and unsecured OAuth validators. Override when using LDAP or PLAIN authentication. |
| map |
| Key-value options passed to the login module as a JAAS configuration entry. Some callback handlers, such as LDAP handlers, read parameters from here rather than from |
| map |
| Additional MDS configuration properties passed through verbatim. Use this for LDAP settings ( |
LDAP and OIDC properties for extra-configs
When using the LDAP user store, pass LDAP connection and search settings through cmf.mds.extra-configs. Common properties include:
ldap.java.naming.provider.url– LDAP server URLldap.java.naming.security.principal– Bind DNldap.java.naming.security.credentials– Bind passwordldap.user.search.base– User search base DNldap.user.name.attribute– Attribute used as the principal nameldap.user.object.class– LDAP object class for user entries
For LDAP group resolution (required for group-based role bindings):
ldap.search.mode– Set toGROUPSldap.group.search.base– Group search base DNldap.group.object.class– LDAP object class for group entriesldap.group.name.attribute– Attribute used as the group nameldap.group.member.attribute– Attribute listing group membersldap.group.member.attribute.pattern– Regex to extract the principal name from a member DN
When using LDAP_WITH_OAUTH to enable SSO alongside LDAP, additionally configure the OIDC provider through extra-configs:
confluent.metadata.server.sso.mode– Set tooidcconfluent.oidc.idp.client.id– OIDC client IDconfluent.oidc.idp.client.secret– OIDC client secretconfluent.oidc.idp.issuer– OIDC issuer URLconfluent.oidc.idp.jwks.endpoint.uri– JWKS endpointconfluent.oidc.idp.authorize.base.endpoint.uri– Authorization endpointconfluent.oidc.idp.token.base.endpoint.uri– Token endpointconfluent.oidc.idp.device.authorization.endpoint.uri– Device authorization endpoint (required for Confluent CLI SSO login through the device flow)
The following table lists the authorization-related Helm values under cmf.authorization.
Property | Type | Default | Description |
|---|---|---|---|
| string | auto | Source of RBAC authorization decisions. |
| object | Configuration for connecting to an external broker-hosted MDS. Required when | |
| boolean |
| When |
Invalid configuration combinations
CMF validates the configuration at startup and fails fast if it detects an invalid combination:
|
| Result |
|---|---|---|
| Set | Valid. CMF authorizes through the external broker-hosted MDS. |
| Not set | Valid. CMF authorizes through the embedded MDS. |
| Not set | Invalid. CMF refuses to start because |
| Set | Invalid. CMF refuses to start: |
CMF also validates the following combinations at startup:
cmf.authorization.authority: cmfwithcmf.mds.enabled: false(orcmf.mdsnot configured). The embedded MDS must be enabled whenauthorityiscmf. CMF refuses to start:authority=cmf requires mds.enabled=true.cmf.mds.authentication-method: BASICwithcmf.mds.user-store: NONE. BASIC authentication requires a user store to validate credentials against. Either configure a user store or switchauthentication-methodtoBEARER.
CMF UI behavior
The CMF UI communicates only with the CMF API and never calls the MDS directly. In cmf authority mode, CMF authorizes requests locally through the embedded MDS. In cp-mds mode, CMF proxies authorization requests to the broker-hosted MDS. The UI behavior is identical in both modes.
