Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Schema Registry Security Overview

Features

Confluent Schema Registry currently supports all Kafka security features, including:

For configuration details, check the configuration options.

Tip

For a configuration example that uses Schema Registry configured with security to a secure Kafka cluster, see the Confluent Platform demo.

Schema Registry to Kafka Cluster

Kafka Store

Kafka is used as Schema Registry storage backend. The special Kafka topic <kafkastore.topic> (default _schemas), with a single partition, is used as a highly available write ahead log. All schemas, subject/version and ID metadata, and compatibility settings are appended as messages to this log. A Schema Registry instance therefore both produces and consumes messages under the _schemas topic. It produces messages to the log when, for example, new schemas are registered under a subject, or when updates to compatibility settings are registered. Schema Registry consumes from the _schemas log in a background thread, and updates its local caches on consumption of each new _schemas message to reflect the newly added schema or compatibility setting. Updating local state from the Kafka log in this manner ensures durability, ordering, and easy recoverability.

All Kafka security features are supported by Schema Registry.

Relatively few services need access to Schema Registry, and they are likely internal, so you can restrict access to the Schema Registry itself via firewall rules and/or network segmentation.

ZooKeeper

Schema Registry supports both unauthenticated and SASL authentication to ZooKeeper.

Setting up ZooKeeper SASL authentication for Schema Registry is similar to Kafka’s setup. Namely, create a keytab for Schema Registry, create a JAAS configuration file, and set the appropriate JAAS Java properties.

In addition to the keytab and JAAS setup, be aware of the zookeeper.set.acl setting. This setting, when set to true, enables ZooKeeper ACLs, which limits access to znodes.

Important: if zookeeper.set.acl is set to true, Schema Registry’s service name must be the same as Kafka’s, which is kafka by default. Otherwise, Schema Registry will fail to create the _schemas topic, which will cause a leader not available error in the DEBUG log. Schema Registry log will show org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata when Kafka does not set ZooKeeper ACLs but Schema Registry does. Schema Registry’s service name can be set either with kafkastore.sasl.kerberos.service.name or in the JAAS file.

If Schema Registry has a different service name than Kafka, at this time zookeeper.set.acl must be set to false in both Schema Registry and Kafka.

Clients to Schema Registry

Configuring the REST API for HTTP or HTTPS

By default Schema Registry allows clients to make REST API calls over HTTP. You may configure Schema Registry to allow either HTTP or HTTPS or both at the same time.

The following configuration determines the protocol used by Schema Registry:

listeners

Comma-separated list of listeners that listen for API requests over HTTP or HTTPS or both. If a listener uses HTTPS, the appropriate SSL configuration parameters need to be set as well.

On the clients, configure schema.registry.url to match the configured Schema Registry listener.

Additional configurations for HTTPS

If you configure an HTTPS listener, there are several additional configurations for Schema Registry.

First, configure the appropriate SSL configurations for the keystore and optionally truststore. The truststore is required only when ssl.client.auth is set to true.

ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234

You may specify which protocol to use while making calls between the instances of Schema Registry. The slave to master node calls for writes and deletes will use the specified protocol.

inter.instance.protocol

The protocol used while making calls between the instances of Schema Registry. The slave to master node calls for writes and deletes will use the specified protocol. The default value would be http. When https is set, ssl.keystore. and ssl.truststore. configs are used while making the call. The schema.registry.inter.instance.protocol name is deprecated; prefer using inter.instance.protocol instead.

  • Type: string
  • Default: “http”
  • Importance: low

To configure clients to use HTTPS to Schema Registry:

  1. On the client, configure the schema.registry.url to match the configured listener for HTTPS.
  2. On the client, configure the environment variables to set the SSL keystore and truststore. You will need to set the appropriate env variable depending on the client (one of KAFKA_OPTS, SCHEMA_REGISTRY_OPTS, KSQL_OPTS). For example:
export KAFKA_OPTS="-Djavax.net.ssl.trustStore=/etc/kafka/secrets/kafka.client.truststore.jks \
          -Djavax.net.ssl.trustStorePassword=confluent \
          -Djavax.net.ssl.keyStore=/etc/kafka/secrets/kafka.client.keystore.jks \
          -Djavax.net.ssl.keyStorePassword=confluent"

Migrating from HTTP to HTTPS

To upgrade Schema Registry to allow REST API calls over HTTPS in an existing cluster:

  • Add/Modify the listeners config to include HTTPS. For example: http://0.0.0.0:8081,https://0.0.0.0:8082
  • Configure Schema Registry with appropriate SSL configurations to setup the keystore and optionally truststore
  • Do a rolling bounce of the cluster

This process enables HTTPS, but still defaults to HTTP so Schema Registry instances can still communicate before all nodes have been restarted. They will continue to use HTTP as the default until configured not to. To switch to HTTPS as the default and disable HTTP support, perform the following steps:

  • Enable HTTPS as mentioned in first section of upgrade (both HTTP & HTTPS will be enabled)
  • Configure inter.instance.protocol to https in all the nodes
  • Do a rolling bounce of the cluster
  • Remove http listener from the listeners in all the nodes
  • Do a rolling bounce of the cluster

Configuring the REST API for Basic HTTP Authentication

Schema Registry can be configured to require users to authenticate using a username and password via the Basic HTTP authentication mechanism.

Note

If you’re using Basic authentication, we recommended that you configure Schema Registry to use HTTPS for secure communication, because the Basic protocol passes credentials in plain text.

Use the following settings to configure Schema Registry to require authentication:

authentication.method=BASIC
authentication.roles=<user-role1>,<user-role2>,...
authentication.realm=<section-in-jaas_config.file>

The authentication.roles config defines a comma-separated list of user roles. To be authorized to access Schema Registry, an authenticated user must belong to at least one of these roles.

For example, if you define admin, developer, user, and sr-user roles, the following configuration assigns them for authentication:

authentication.roles=admin,developer,user,sr-user

The authentication.realm config must match a section within jaas_config.file, which defines how the server authenticates users and should be passed as a JVM option during server start:

export SCHEMA_REGISTRY_OPTS=-Djava.security.auth.login.config=/path/to/the/jaas_config.file
<path-to-confluent>/bin/schema-registry-start <path-to-confluent>/etc/schema-registry/schema-registry.properties

An example jaas_config.file is:

SchemaRegistry-Props {
  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
  file="/path/to/password-file"
  debug="false";
};

Assign the SchemaRegistry-Props section to the authentication.realm config setting:

authentication.realm=SchemaRegistry-Props

The example jaas_config.file above uses the Jetty PropertyFileLoginModule, which authenticates users by checking for their credentials in a password file.

You can also use other implementations of the standard Java LoginModule interface, such as the LdapLoginModule, or the JDBCLoginModule for reading credentials from a database.

The file parameter is the location of the password file, The format is:

<username>: <password-hash>[,<rolename> ...]

Here’s an example:

fred: OBF:1w8t1tvf1w261w8v1w1c1tvn1w8x,user,admin
harry: changeme,user,developer
tom: MD5:164c88b302622e17050af52c89945d44,user
dick: CRYPT:adpexzg3FUZAk,admin,sr-user

Get the password hash for a user by using the org.eclipse.jetty.util.security.Password utility:

bin/schema-registry-run-class org.eclipse.jetty.util.security.Password fred letmein

Your output should resemble:

letmein
OBF:1w8t1tvf1w261w8v1w1c1tvn1w8x
MD5:0d107d09f5bbe40cade3de5c71e9e9b7
CRYPT:frd5btY/mvXo6

Each line of the output is the password encrypted using different mechanisms, starting with plain text.

Once Schema Registry is configured to use Basic authentication, clients must be configured with suitable valid credentials, for example:

schema.registry.basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=fred:letmein

Governance

To provide data governance with the Confluent Schema Registry :

  1. disable auto schema registration
  2. restrict access to the _schemas topic
  3. restrict access to Schema Registry operations

Disabling Auto Schema Registration

By default, client applications automatically register new schemas. If they produce new messages to a new topic, then they will automatically try to register new schemas. This is very convenient in development environments, but in production environments we recommend that client applications do not automatically register new schemas. Register schemas outside of the client application to control when schemas are registered with Confluent Schema Registry and how they evolve.

Within the application, disable automatic schema registration by setting the configuration parameter auto.register.schemas=false, as shown in the example below.

props.put(AbstractKafkaAvroSerDeConfig.AUTO_REGISTER_SCHEMAS, false);

Once a client application disables automatic schema registration, it will no longer be able to dynamically register new schemas from within the application. However, it will still be able to retrieve existing schemas from the Schema Registry, assuming proper authorization.

Authorizing Access to the Schemas Topic

If you enable Kafka authorization, you must grant the Schema Registry’s principal read and write access to the schemas topic, along with describe configuration access to verify that the topic exists. This ensures that only authorized users can make changes to the topic.

export KAFKA_OPTS="-Djava.security.auth.login.config=<path to JAAS conf file>"

bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add \
              --allow-principal 'User:<sr-principal>' --allow-host '*' \
              --operation DescribeConfigs --topic _schemas

bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add \
               --allow-principal 'User:<sr-principal>' --allow-host '*' \
               --operation Read --topic _schemas

bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add \
               --allow-principal 'User:<sr-principal>' --allow-host '*' \
               --operation Write --topic _schemas

Note

Removing world-level permissions: In previous versions of Schema Registry, we recommended making the _schemas topic world readable and writable. Now that Schema Registry supports SASL, the world-level permissions can be dropped.

Authorizing Schema Registry Operations

The Schema Registry security plugin provides authorization for various Schema Registry operations. It authenticates the incoming requests and authorizes them via the configured authorizer. This allows schema evolution management to be restricted to administrative users, with application users provided with read-only access only.