Configure Role-Based Access Control for Schema Registry in Confluent Platform¶
Confluent Schema Registry supports Use Role-Based Access Control (RBAC) for Authorization in Confluent Platform (RBAC).
Users are granted access to manage, read, and write to particular topics and their associated schemas (contained in Schema Registry subjects) based on RBAC roles. User access is scoped to specified resources and Schema Registry supported operations.
With RBAC enabled, Schema Registry can authenticate incoming requests and authorize them based on role bindings. This allows schema evolution management to be restricted to administrative users, while providing users and applications with different types of access to a subset of subjects for which they are authorized (such as, write access to relevant subjects for producers, read access for consumers).
Overview¶
RBAC makes it easier and more efficient to set up and manage user access to Schema Registry subjects and topics.
Schema Management before RBAC¶
Without RBAC, an administrator must specify every subject or use * (for all) and specify each operation (SUBJECT_READ, SUBJECT_COMPATIBILITY_READ, and so forth) that a user needs. If you have 100 developers who need to read schemas, you must set up access 100 times.
Schema Management with RBAC¶
An RBAC-enabled environment addresses the following use cases:
- Jack can give limited access to Samantha by assigning her a
DeveloperRead
role and specify a set of subjects with a prefix. - Samantha can have multiple roles on different subjects. She can be a
DeveloperRead
for “transactions-value” and “orders-value”, but assume aDeveloperWrite
role for “customers-value”. - If Jack needs to serve 100 developers in his organization, he can create a group for developers and grant the group read-only access to schemas in Schema Registry.
How it Works¶
When a client communicates to the Schema Registry HTTPS endpoint, Schema Registry passes the client credentials to Metadata Service (MDS) for authentication. MDS is a REST layer on the Kafka broker within Confluent Server, and it integrates with LDAP to authenticate end users on behalf of Schema Registry and other Confluent Platform services such as Connect, Confluent Control Center, and ksqlDB. As shown in Scripted Confluent Platform Demo, clients must have predefined LDAP entries.
Once a client is authenticated, you must enforce that only authorized entities have access to the permitted resources. You can use ACLs, RBAC, or both to do so. While ACLs and RBAC can be used together or independently, RBAC is the preferred solution as it provides finer-grained authorization and a unified method for managing access across Confluent Platform.
The combined authentication and authorization workflow for a Kafka client connecting to Schema Registry is shown in the diagram below.
Setup Overview¶
To enable role-based access control (RBAC) on schemas, you must configure the schema-registry.properties
file with connection information to a metadata service (MDS) running RBAC, and use the
Confluent CLI to grant user and application access to subjects and other resources based on roles.
Typically, you can request account access and the MDS details needed for RBAC from your security administrator.
If you are in a security admin role experimenting with a fully local setup, you would first set up RBAC using MDS, then create a service principal for Schema Registry using the Confluent CLI.
Following that, you can create principal user accounts with various roles such as ResourceOwner
, DeveloperRead
, DeveloperWrite
, and DeveloperManage
bound to subjects (schemas associated with Kafka topics) and other resources.
Role Mappings to Operations for Subject Based Authorization¶
After Schema Registry is configured and running in an RBAC-enabled environment, users can read and write schemas to subjects, based on their authorization for operation on a resource (roles and rolebinding
).
RBAC supports all Schema Registry operations as listed in operations. For more details on these operations, see the Schema Registry API.
Role Name | Read | Write | Delete | ReadCompatibility | WriteCompatibility |
---|---|---|---|---|---|
SystemAdmin | Yes | Yes | Yes | Yes | Yes |
UserAdmin | No | No | No | No | No |
ClusterAdmin | Yes | Yes | Yes | Yes | Yes |
Operator | No | No | No | No | No |
SecurityAdmin | No | No | No | No | No |
ResourceOwner | Yes (based on scoping) | Yes (based on scoping) | Yes (based on scoping) | Yes (based on scoping) | Yes (based on scoping) |
DeveloperRead | Yes (based on scoping) | No | No | Yes (based on scoping) | No |
DeveloperWrite | Yes | Yes (based on scoping) | No | No | No |
DeveloperManage | No | No | No | Yes | Yes |
Tip
- Schema Registry has its own cluster.
- You may have multiple schema registries.
- A single Schema Registry cluster can be connected to multiple Kafka clusters.
- There is no notion of “global compatibility” in terms of roles. To grant permission to a user to manage global compatibility, give them
DeveloperManage
role on a subject resource named__GLOBAL
. - Starting with Confluent Platform 5.4.x, users with
developerRead
anddeveloperWrite
roles also need thedeveloperManage
role if they want to view and work with schemas on Schema Registry through the Control Center for Confluent Platform. (Previous to 5.4.x,developerRead
anddeveloperWrite
roles were sufficient to interact with Schema Registry through the Control Center.)
Example ClusterAdmin Use Case¶
Jack as a ClusterAdmin
wants to set up a Schema Registry cluster for his organization.
Step 1. Jack as a cluster administrator contacts the RBAC security administrator with the following information.
- The Kafka cluster to use (if there is more than one to choose from)
- Schema Registry cluster ID (e.g. schema-registry-a)
- Jack’s principal name
- Resources required by Schema Registry (schemas topic, schemas topic group, and so on).
Step 2. UserAdmin
creates a service principal to represent the Schema Registry cluster and does the following.
- Grants that service principal permissions to access the internal schemas topic and other required roles necessary for a Schema Registry cluster to operate
- Provides Jack with the credentials for that service principal
- Grants Jack the role
ClusterAdmin
on the Schema Registry cluster - Provides Jack with a public key that can be used to authenticate requests.
Step 3. Jack configures the Schema Registry cluster to use the provided public key, the specified group ID (for example, “schema-registry-a”), and the service principal provided by the UserAdmin
, then spins up the cluster.
Example User Experience¶
Samantha as a developer needs READ access to two subjects, “transactions-value” and “orders-value”, to understand schemas her application needs to interact with.
Step 1. Samantha as a developer contacts the user administrator with the following information:
- List of subjects (for example, “transactions-value”, “orders-value”)
- Schema Registry cluster ID (for example, “schema-registry-a”)
Step 2. UserAdmin
grants Samantha access to the subjects.
Step 3. When Samantha runs GET /subjects
, she will only see “transactions-value” and “orders-value”.
Step 4. Accidental POST
or DELETE
operations on these subjects will be prevented.
Quick Start¶
This Quick Start describes how to configure Schema Registry for Role-Based Access Control to manage user and application authorization to topics and subjects (schemas), including how to:
- Configure Schema Registry to start and connect to the RBAC-enabled Apache Kafka® cluster (edit
schema-registry.properties
use the Confluent CLI to create roles) - Use the Confluent CLI to grant a SecurityAdmin role to the Schema Registry service principal.
- Use the Confluent CLI to grant a ResourceOwner role to the Schema Registry service principal on the internal topic and group (used to coordinate across the Schema Registry cluster).
- Use the Confluent CLI to grant users access to topics (and associated subjects in Schema Registry).
The examples assume a local install of Schema Registry and shared RBAC and MDS configuration. Your production environment may differ (for example, Confluent Cloud or remote Schema Registry).
If you were to use a local Kafka, ZooKeeper, and bootstrap server as might be the case for testing, these would also need authorization through RBAC, requiring additional prerequisite setup and credentials.
See also
To get started, try the automated RBAC example that showcases the RBAC functionality in Confluent Platform.
Before You Begin¶
If you are new to Confluent Platform or to Schema Registry, consider first reading or working through these tutorials to get a baseline understanding of the platform, Schema Registry, and Role-Based Access Control across Confluent Platform.
Prerequisites¶
To run a resource like Schema Registry in an RBAC environment you need a Schema Registry service principal (user account for the resource), credentials, and location of the Metadata Service (MDS) running RBAC. This enables you to configure Schema Registry properties to talk to the RBAC-enabled Kafka cluster, and grant various types of access to Schema Registry using the Confluent CLI.
Specifically, you need the following to get started.
- An RBAC-enabled Confluent Platform environment and Schema Registry.
- Location of the Metadata Service (MDS) running RBAC. (This will be a URL for the Metadata Service (MDS), or file path if you are testing a local MDS.)
- Authorization to create and modify principals for the organization.
- A public key file to use to verify requests with token-based authorization.
- A service (user) account for Schema Registry.
In most cases, you will get this information from your Security administrator.
Install Confluent Platform and the Confluent CLI¶
- If you have not done so already, download and install Confluent Platform locally.)
- Install the Confluent CLI.
Configure Schema Registry to communicate with RBAC services¶
The next set of examples show how to connect a local Schema Registry to a remote
Metadata Service (MDS) running RBAC. The schema.registry.properties
file
configurations reflect a remote Metadata Service (MDS) URL, location, and Kafka cluster
ID. Also, the examples assume you are using credentials you got from your
Security administrator for a pre-configured schema registry principal user
(“service principal”), as mentioned in the prerequisites.
Tip
- In the examples, backslashes (
\
) are used before carriage returns to show multi-line property values. These line breaks and backslashes may cause errors when used in the actual properties file. If so, remove the backslashes and join the lines so as not to break up a property value across multiple lines with returns. - If you have multiple servers to reference, use a semicolon separated list. (For example, as values for
metadataServerUrls
orconfluent.metadata.bootstrap.server.urls
.)
Define these settings in CONFLUENT_HOME/etc/schema-registry/schema-registry.properties
:
Configure Schema Registry authorization for communicating with the RBAC Kafka cluster.
The
username
andpassword
are RBAC credentials for the Schema Registry service principal, and metadataServerUrls is the location of your RBAC Kafka cluster (for example, a URL to an ec2 server).# Authorize Schema Registry to talk to Kafka (security protocol may also be SASL_SSL if using TLS/SSL) kafkastore.security.protocol=SASL_PLAINTEXT kafkastore.sasl.mechanism=OAUTHBEARER kafkastore.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler kafkastore.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ username="<username>" \ password="<password>" \ metadataServerUrls="<https>://<metadata_server_url>:<port>";
Configure RBAC authorization, and bearer/basic authentication, for the Schema Registry resource.
These settings can be used as-is, JETTY_AUTH is the recommended authentication mechanism.
# These properties install the Schema Registry security plugin, and configure it to use RBAC for # authorization and OAuth for authentication resource.extension.class=io.confluent.kafka.schemaregistry.security.SchemaRegistrySecurityResourceExtension confluent.schema.registry.authorizer.class=io.confluent.kafka.schemaregistry.security.authorizer.rbac.RbacAuthorizer rest.servlet.initializor.classes=io.confluent.common.security.jetty.initializer.InstallBearerOrBasicSecurityHandler confluent.schema.registry.auth.mechanism=JETTY_AUTH
Tip
- The above setting for
resource.extension.class
activates the security plugin. - The above setting for
confluent.schema.registry.auth.mechanism
sets the authentication mechanism as Jetty, which is recommended for use with RBAC.
- The above setting for
Tell Schema Registry how to communicate with the Kafka cluster running the Metadata Service (MDS) and how to authenticate requests using a public key.
- The value for
confluent.metadata.bootstrap.server.urls
can be the same asmetadataServerUrls
, depending on your environment. - In this step, you need a public key file to use to verify requests with token-based authorization, as mentioned in the prerequisites.
# The location of the metadata service confluent.metadata.bootstrap.server.urls=<https>://<metadata_server_url>:<port> # Credentials to use with the MDS, these should usually match those used for talking to Kafka confluent.metadata.basic.auth.user.info=<username>:<password> confluent.metadata.http.auth.credentials.provider=BASIC # The path to public keys that should be used to verify json web tokens during authentication public.key.path=<public_key_file_path.pem>
For additional configurations available to any client communicating with MDS, see also REST client configurations in the Confluent Platform Security documentation.
- The value for
Specify the
kafkastore.bootstrap.server
you want to use.The default is a commented out line for a local server. If you do not change this or uncomment it, the default will be used.
#kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
Uncomment this line and set it to the address of your bootstrap server. This may be different from the MDS server URL. The standard port for the Kafka bootstrap server is
9092
.kafkastore.bootstrap.servers=<rbac_kafka_bootstrap_server>:9092
(Optional, Legacy) Specify the kafkastore.connection.url you want to use to connect the Schema Registry Security Plugin for Confluent Platform to ZooKeeper.
The default is shown on the commented out line for a local ZooKeeper. If you do not change this or uncomment it, the default will be used.
#kafkastore.connection.url=localhost:2181
Uncomment this line and set it to the address of your ZooKeeper server. The standard port for the ZooKeeper server is
2181
.Note
kafkastore.connection.url is deprecated. It was previously needed in older versions (5.4.x and earlier) when the Schema Registry Security Plugin was installed and configured to use ACLs. Starting with Confluent Platform 5.5.0, this is no longer the case, given Schema Registry ACL Authorizer for Confluent Platform. If you do not have the ACL Authorizer, upgrade to a Confluent Platform version that has it. To upgrade to Kafka leader election, see Migration from ZooKeeper primary election to Kafka primary election.
(Optional) Specify a custom
schema.registry.group.id
(to serve as Schema Registry cluster ID) which is different from the default, schema-registry.In the example,
schema.registry.group.id
is set to “schema-registry-cool-cluster”.# Schema Registry group id, which is the cluster id # The default for |sr| cluster ID is **schema-registry** schema.registry.group.id=schema-registry-cool-cluster
Tip
The Schema Registry cluster ID is the same as
schema-registry-group-id
, which defaults to schema-registry. This is used to specify the target resource inrolebinding
commands on the Confluent CLI. You might need to specify a custom cluster ID to differentiate your Schema Registry from others in the organization so as to avoid overwriting roles and users in multiple registries.(Optional) Specify a custom name for the Schema Registry default topic. (The default is _schemas.)
In the example,
schema.registry.group.id
is set to_jax-schemas-topic
.# The name of the topic to store schemas in # The default schemas topic is **_schemas** kafkastore.topic=_jax-schemas-topic
Tip
- The Schema Registry itself uses an internal topic to hold schemas. The default name for this topic is _schemas. You might need to specify a custom name for the schemas internal topic to differentiate it from others in the organization and avoid overwriting data.
- An underscore is not required in the name; this is a convention used to indicate an internal topic.
(Optional) Enable anonymous access to requests that occur without authentication.
Any requests that occur without authentication are automatically granted the principal
User:ANONYMOUS
# This enables anonymous access with a principal of User:ANONYMOUS confluent.schema.registry.anonymous.principal=true authentication.skip.paths=/*
If you get the following error about not having authorization when you run the
curl
command to list subjects as described in Start Schema Registry and test it, you can enable anonymous requests to bypass the authentication temporarily while you troubleshoot credentials.curl localhost:8081/subjects <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /subjects. Reason: <pre> Unauthorized</pre></p><hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.18.v20190429</a><hr/> </body> </html>
Tip
- For the above curl command to be successful, you can configure rolebindings or ACLs for
User:ANONYMOUS
. - The command bypasses the requirement to present valid credentials with a REST request, but not the authorization that is then performed on that request to ensure that the user (or, if no credentials are provided, User:ANONYMOUS) has the proper roles or ACLs to perform that action.
- For the above curl command to be successful, you can configure rolebindings or ACLs for
Get the Kafka cluster ID for the MDS server you plan to use¶
You will need this in order to specify the Kafka cluster to use in rolebinding
commands on the Confluent CLI.
- To get the Kafka cluster ID for a local host:
bin/zookeeper-shell localhost:2181 get /cluster/id
- To get the Kafka cluster ID on a remote host:
zookeeper-shell <host>:<port> get /cluster/id
For example, the output of this command currently shows the Kafka cluster ID: my-kafka-cluster-ID
:
zookeeper-shell <metadata_server_url>:2181 get /cluster/id
Your output should resemble:
Connecting to <metadata_server_url>:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"version":"1","id":"my-kafka-cluster-ID"}
...
Grant roles for the Schema Registry service principal¶
In these steps, you use the Confluent CLI to log on to MDS and create the Schema Registry service principal . After you have these roles set up, you can use the Confluent CLI to manage Schema Registry users. For this example, assume the commands use the MDS server credentials, URLs, and property values you set up on your local Schema Registry properties file. (Optionally, you can use a registered cluster name in your role bindings.)
Log on to MDS.
confluent login --url <https>://<metadata_server_url>:<port>
As a prerequisite to granting additional access, grant permission to create the topic
_schema_encoders
, which serves as themetadata.encoder.topic
as described in Schema Registry Configuration Reference for Confluent Platform.confluent iam rbac role-binding create \ --principal User:<sr-user-id> \ --role ResourceOwner \ --resource Topic:<_schema_encoders> \ --kafka-cluster <kafka-cluster-id>
For example:
confluent iam rbac role-binding create \ --principal User:jack-sr \ --role ResourceOwner \ --resource Topic:_schema_encoders \ --kafka-cluster my-kafka-cluster-ID
Grant the user the role
SecurityAdmin
on the Schema Registry cluster.confluent iam rbac role-binding create \ --role SecurityAdmin \ --principal User:<service-account-id> \ --kafka-cluster <kafka-cluster-id> \ --schema-registry-cluster-id <schema-registry-group-id>
Use the command
confluent iam rbac role-binding list <flags>
to view the role you just created.confluent iam rbac role-binding list \ --principal User:<service-account-id> \ --kafka-cluster <kafka-cluster-id> \ --schema-registry-cluster-id <schema-registry-group-id>
For example, here is a listing for a user “jack-sr” granted
SecurityAdmin
role on “schema-registry-cool-cluster”, connecting to MDS through a Kafka clustermy-kafka-cluster-ID
:confluent iam rbac role-binding list \ --principal User:jack-sr \ --kafka-cluster my-kafka-cluster-ID \ --schema-registry-cluster-id schema-registry-cool-cluster Role | ResourceType | Name | PatternType +---------------+--------------+------+-------------+ SecurityAdmin | Cluster | |
Grant the user the role
ResourceOwner
on the group that Schema Registry nodes use to coordinate across the cluster.confluent iam rbac role-binding create \ --principal User:<sr-user-id> \ --role ResourceOwner \ --resource Group:<schema-registry-group-id> \ --kafka-cluster <kafka-cluster-id>
For example:
confluent iam rbac role-binding create \ --principal User:jack-sr \ --role ResourceOwner \ --resource Group:schema-registry-cool-cluster \ --kafka-cluster my-kafka-cluster-ID
Grant the user the role
ResourceOwner
Kafka topic that Schema Registry uses to store its schemas.confluent iam rbac role-binding create \ --principal User:<sr-user-id> \ --role ResourceOwner \ --resource Topic:<schemas-topic> \ --kafka-cluster <kafka-cluster-id>
For example:
confluent iam rbac role-binding create \ --principal User:jack-sr \ --role ResourceOwner \ --resource Topic:_jax-schemas-topic \ --kafka-cluster my-kafka-cluster-ID
Use the command
confluent iam rbac role-binding list <flags>
to view the role you just created.confluent iam rbac role-binding list \ --principal User:jack-sr \ --role ResourceOwner \ --kafka-cluster my-kafka-cluster-ID
For example:
confluent iam rbac role-binding list \ --principal User:jack-sr \ --role ResourceOwner \ --kafka-cluster my-kafka-cluster-ID Role | ResourceType | Name | PatternType +-------------+--------------+----------------------------------+-------------+ ResourceOwner | Topic | _jax-schemas-topic | LITERAL ResourceOwner | Topic | __schema_encoders | LITERAL ResourceOwner | Group | schema-registry-cool-cluster | LITERAL ResourceOwner | Topic | _schemas | LITERAL ResourceOwner | Group | schema-registry | LITERAL
Client authentication and authorization¶
- Configure license client authentication
When using principal propagation and the following security types, you must configure client authentication for the license topic. For more information, see the following documentation:
- Configure license client authorization
When using principal propagation and RBAC or ACLs, you must configure client authorization for the license topic.
Note
Starting with Confluent Platform 6.2.1, the
_confluent-command
internal topic is available as the preferred alternative to the_confluent-license
topic for components such as Schema Registry, REST Proxy, and Confluent Server (which were previously using_confluent-license
). Both topics will be supported going forward. Here are some guidelines:- New deployments (Confluent Platform 6.2.1 and later) will default to using
_confluent-command
as shown below. - Existing clusters will continue using the
_confluent-license
unless manually changed. - Newly created clusters on Confluent Platform 6.2.1 and later will default to creating the
_confluent-command
topic, and only existing clusters that already have a_confluent-license
topic will continue to use it.
RBAC authorization
Run this command to add
ResourceOwner
for the component user for the Confluent license topic resource (default name is_confluent-command
).confluent iam rbac role-binding create \ --role ResourceOwner \ --principal User:<service-account-id> \ --resource Topic:_confluent-command \ --kafka-cluster <kafka-cluster-id>
ACL authorization
Run this command to configure Kafka authorization, where bootstrap server, client configuration, service account ID is specified. This grants create, read, and write on the
_confluent-command
topic.kafka-acls --bootstrap-server <broker-listener> --command-config <client conf> \ --add --allow-principal User:<service-account-id> --operation Create --operation Read --operation Write \ --topic _confluent-command
- New deployments (Confluent Platform 6.2.1 and later) will default to using
(Optional) Use a registered cluster name¶
Starting in Confluent Platform 6.0, you can register your Schema Registry Kafka cluster in the cluster registry and specify a
user-friendly cluster name, which makes it easier to create role bindings. In all of the example commands to Grant roles for the Schema Registry service principal,
you can use the registered cluster name instead of <schema-registry-group-id>
and <kafka-cluster-id>
.
For example, the role binding command for a non-registered cluster must include both the Schema Registry group ID and cluster ID:
confluent iam rbac role-binding create \
--principal User:<sr-user-id> \
--role ResourceOwner \
--resource Group:<schema-registry-group-id> \
--kafka-cluster <kafka-cluster-id>
Assuming your Schema Registry cluster has been registered in the Confluent Platform cluster registry, you can
replace the <schema-registry-group-id>
and <kafka-cluster-id>
with the user-friendly name of the registered cluster:
confluent iam rbac role-binding create \
--principal User:<sr-user-id> \
--role ResourceOwner \
--cluster-name <registered-cluster-name>
Start Schema Registry and test it¶
Make sure your local ZooKeeper and Kafka servers are shut down just to keep a clean slate. Remember, for this example you are running against a remote cluster, so now you only have to start Schema Registry.
Open a command window, change directories into your local install of Confluent Platform, and run the command to start Schema Registry.
./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
Run the following command to view subjects.
curl localhost:8081/subjects
If you get an empty
[ ]
as shown above (or a bracket with topics), that’s success! (The empty brackets indicate that no topics are created yet.)Successfully starting and running the local Schema Registry and the curl on
subjects
indicates that you have access to Schema Registry through RBAC.Tip
If the
curl
command shown above does not work, try sending Schema Registry service principal credentials along with it:curl -u <username>:<password> localhost:8081/subjects
You can create and manage topics, subjects, and schemas through an RBAC-enabled Confluent Control Center or with
curl
commands. See Tutorial: Use Schema Registry on Confluent Platform to Implement Schemas for a Client Application and Schema Registry API Usage Examples for Confluent Platform. The available resources and actions will be scoped in RBAC based on login credentials.
At this point you have successfully configured Schema Registry to run with RBAC.
Now, you can use the Confluent CLI to grant role-based access to Schema Registry users scoped to subjects.
Log on to Confluent CLI and grant access to Schema Registry users¶
Log on to MDS again.
confluent login --url <https>://<metadata_server_url>:<port>
To grant a user named “sam” the role
ResourceOwner
on a Subject named “transactions” for a Kafka cluster namedmy-kafka-cluster-ID
and a Schema Registry cluster named “schema-registry-cool-cluster”:confluent iam rbac role-binding create \ --principal User:sam \ --resource Subject:transactions \ --role ResourceOwner \ --kafka-cluster my-kafka-cluster-ID \ --schema-registry-cluster-id schema-registry-cool-cluster