Quick Start for Schema Management on Confluent Cloud

You can use Confluent Cloud Schema Registry to manage schemas in Confluent Cloud. You enable a single Schema Registry per Confluent Cloud environment. Schema Registry is accessed over port 443.

You can create and edit schemas in a schema editor and associate them with Kafka topics.

This quick start does not cover all the capabilities of Schema Registry, but rather is an introduction. For a complete guide, see Manage Schemas in Confluent Cloud.

Important

• Your VPC must be able to communicate with the Confluent Cloud Schema Registry public internet endpoint. For more information, see Use Confluent Cloud Schema Registry to Connect to a Public Endpoint in a Private Networking Environment.
• This quick start assumes that you have completed Quick Start for Confluent Cloud, including installing and using the Confluent CLI.
• This quick start requires that you use both the Confluent Cloud Console and the Confluent CLI.

Sign in to the Cloud Console

Log in to Confluent Cloud at https://confluent.cloud.

Add a cloud environment

1. Select Environments on the left panel, choose Add cloud environment, provide an environment name in the dialog, and click Create.

2. Choose a Stream Governance package to enable Schema Registry, Stream Catalog, and Stream Lineage, either upgrade to Advanced or accept the Essentials package:

   • Upgrade to Stream Governance Advanced starting at $1/hour
   • Now, continue with Stream Governance Essentials for free
   

   When you add a new environment, Confluent Cloud provides you with Stream Governance package options. Stream Governance Essentials and Advanced packages are available in all Confluent Cloud regions.

   • Essentials: Starting at $0 cost per hour, provides the fundamentals for getting started; including Schema Registry with 100 free schemas, stream catalog with auto-technical metadata ingestion, tags metadata, UI, REST API, and most features in stream lineage. (After the first 100 free schemas, pricing is $0.002 per schema per hour on Essentials.)
   • Advanced: Starting at $1 per hour, provides enterprise-grade features for mission-critical workloads, including Schema Registry with 20,000 included schemas and a 99.99% uptime SLA, stream catalog with everything in Essentials plus business metadata and GraphQL API, and stream lineage with everything in Essentials plus point in time lineage and searchable lineage graph. The Advanced package also includes support for Data Contracts.

   Note

   Only single-zone Dedicated Kafka clusters are available in Jio Cloud regions. Stream Governance packages purchased in Jio Cloud regions support a maximum SLA of 99.95%. For more information, see Jio Cloud.

3. Create a cluster.

   

   The Schema Registry cluster is automatically assigned to the same region as this first Kafka cluster deployed in an environment. This Schema Registry region does not change, it persists regardless of clusters created in other regions or deleted. The Schema Registry region determines where Schema Registry and Stream Catalog services will run, and corresponding metadata is stored.

Create an API Key for Confluent Cloud Schema Registry

To use Confluent Cloud Schema Registry for managing Kafka clusters, you need an API key specific to Schema Registry.

Note

• The API key for Confluent Cloud Schema Registry is distinct from the API key you created for Kafka clusters in the same environment, per the Quick Start for Confluent Cloud.
• Confluent Cloud provides one Schema Registry per environment, and each environment can contain multiple Kafka clusters. You need an API key/secret pair for each Kafka cluster and another for the Schema Registry cluster.

1. In the environment for which you want to set up Schema Registry, find Credentials on the right side panel and click <someNumber> keys to bring up the API credentials dialog. (If you are just getting started, click Add key.)

   

2. Click Add key to create a new Schema Registry API key.

   

3. When the API Key and API Secret are saved, click the checkbox next to I have saved my API key and secret and am ready to continue, and click Continue.

   

   Your new Schema Registry key is shown on the Schema Registry API access key list.

   

Schema Registry API Keys for Multi-Tenant Clusters

Confluent Cloud supports one Schema Registry per environment. In multi-tenant deployments, one physical Schema Registry per cloud and geographic region hosts many logical schema registries. In these cases, the registries for different Kafka clusters in different environments will have the same URL as the Confluent Cloud Schema Registry. Confluent Cloud uses API keys that are resource scoped for Schema Registry clusters to store schemas and route requests to the appropriate logical clusters.

Tip

To learn more about other environment level schema management options available on the Environment settings page, see Configure and Manage Schemas for an Environment.

Create a Topic in Confluent Cloud

You may already have topics available on this cluster, having worked through the prerequisite Quick Start for Confluent Cloud, but create a new topic designed to test out Schema Registry.

1. Select the environment.

   Tip

   If you have only one environment, default, it will be selected already.

2. Select the cluster.

3. Click Topics on the left menu, then click Add topic.

   

   Name the new topic employees, and click Create with defaults to add it.

Create a Schema

You can create schemas for topics from the Cloud Console, from the Confluent CLI, the Schema Registry API (see POST /subjects/(string: subject)/versions), or through the Schema Registry Maven Plugin (see schema-registry:register).

This section shows you how to create a schema from the Confluent CLI, just to provide a 360 view of cloud tools and demonstrate how you can use the web UI and the CLI to interact with the same entities, such as environments, clusters, topics, and schemas.

If you need more help with the basic CLI commands, type confluent --help, confluent schema-registry --help to drill down, or see Quick Start for Confluent Cloud and the Confluent CLI command reference.

1. Sign in to the Confluent CLI.

   confluent login
   

   Follow the prompts to enter your credentials (email and password).

   Upon login, Confluent Cloud indicates your current environment.

   Enter your |ccloud| credentials:
   Email:
   Password:
   
   Using environment env-250o2r ("default")
   

2. Select the environment where you created the employees topic. (It may already be selected upon sign-in.)

   In order to verify this, use confluent environment list to get the environment IDs, followed by confluent environment use <id> to select the environment you want (if it is not already the current environment).

   For example:

   confluent environment list
   

   Your output should resemble:

     Current |     ID     |        Name
   ----------+------------+---------------------
     *       | env-250o2r | default
             | env-abc123 | ccloud-demo
             | env-xyz123 | data-lineage-demo
             | env-def12g | my-new-environment
   

   To use “ccloud-demo”:

   confluent environment use env-abc123
   

   Tip

   • To list environments: confluent environment list
   • To select an environment: confluent environment use <id>
   • To list clusters: confluent kafka cluster list
   • To select a cluster: confluent kafka cluster use <id>
   • For the full list of confluent commands see Confluent CLI command reference.

3. Create a file containing the following JSON and name it employees.json.

   {
      "type" : "record",
      "namespace" : "Example",
      "name" : "Employee",
      "fields" : [
         { "name" : "Name" , "type" : "string" },
         { "name" : "Age" , "type" : "int" }
      ]
   }
   

4. Create a schema that uses employees.json.

   confluent schema-registry schema create --subject employees-value --schema employees.json --type avro
   

   For example:

   confluent schema-registry schema create --subject employees-value --schema employees.json --type avro
   
   Successfully registered schema with ID: 100001
   

   Tip

   • You have the option to create schemas that reference other schemas by using the --refs <ref-file> flag with confluent schema-registry schema create. To learn more about schema references, see Schema references. You can also create schema references from the web UI.

   • You must specify a schema format type as one of avro, json, or protobuf, and then supply a schema that conforms to the syntax for that type. To learn more about development using these schema formats, see Schema Formats, Serializers, and Deserializers in the Confluent Platform documentation, the JSON Schema project documentation, the Apache Avro® specification, and the Google Developer documentation on Protocol Buffers.

   • You can also create a schema from the web UI. To do so, select an environment and cluster, select a topic, then click the Schema tab for that topic. Select a schema format type (for this example, Avro), paste the schema (such as the employees.json content), and click Save.

     

View the Topic and Associated Schema

Return to the Cloud Console to view the new schema for the employees topic.

1. Navigate to the topics list for your cluster.

   

2. Click the employees topic you created in a previous step.

3. Click Schema.

   • Click the arrows next to fields to expand the elements in the tree view:
   
   • Click the button in the upper left to toggle to the code view:
   

   And there you have it! You have a new schema (employees-value) for the employees topic in Confluent Cloud Schema Registry.

Configure and Manage Schemas for an Environment

Several tasks related to schemas on Confluent Cloud are managed at the environment level. This includes creating and managing Schema Registry API keys along with options, as described below, to view and search schemas, monitor usage, and set a compatibility mode for schemas.

View and Search All Schemas in an Environment

To view a searchable list of all schemas available in a Confluent Cloud environment:

1. Select an environment.

2. Click the Schemas in the right-side panel.

   The schemas list is shown.

   
   • You can use Search to find schemas in longer lists
   • Click a schema to drill down for more options, including view, download, duplicate, or delete a schema; navigate to compatibility settings; traverse the schema version history, view each version, and compare versions.

View and Edit Global Compatibility Settings on Schemas

The default compatibility mode is Backward. From the environment level Schema Registry settings, you can change the mode, which will apply globally to all schemas in an environment. You can also change compatibility settings for a schema on a specific topic (at the subject level). Subject-level compatibility settings override the global settings.

Caution

If you change the compatibility mode of existing schemas already in production use, be aware of any possible breaking changes to your applications.

1. Select an environment.

2. Click the Schemas in the right-side panel.

3. Click Edit on the Compatibility setting card.

   

   The Compatibility settings are shown.

   

4. Select a mode option:

   • Backward (Confluent Schema Registry default)
   • Transitive backward
   • Forward and Transitive forward
   • Full and Transitive full
   • None (not recommended)

   Descriptions indicate the compatibility behavior for each option. For more information, including the changes allowed for each option, see Schema Evolution and Compatibility.

5. Click Save.

Use curl to View and Manage Schemas

You can also use curl commands to view and manage schemas on Confluent Cloud.

Schema Registry on Confluent Cloud requires that you pass the API Key and Secret with the --user (or -u) flag. For example, to view all subjects in the registry:

curl --user <schema-registry-api-key>:<schema-registry-api-secret> \
<schema-registry-url>/subjects

For more information about using curl commands with Schema Registry, see Schema Registry API Usage Examples. With the exception of this one same example, the examples are for on-premises Schema Registry but many of the commands are relevant to Confluent Cloud, once you know how to pass your API Key and Secret as shown here.

Suggested Reading

Schemas Product Documentation

• Work with schemas
• Stream Governance on Confluent Cloud
• Confluent Cloud Schema Registry Tutorial
• Quick Start for Confluent Cloud
• Confluent Cloud Console
• Confluent Cloud CLI Schema Registry commands
• Schema Management
• Migrate Schemas
• Confluent Cloud schemas limits and how to free up space
• Schema Formats, Serializers, and Deserializers (Confluent Platform tools that support application development for Confluent Cloud and self-managed systems)
• Run an automated Confluent Cloud quickstart with Avro, Protobuf, and JSON formats

Blog Posts and Podcasts

• Blog post: Getting Started with Protobuf in Confluent Cloud
• Blog post: Confluent Platform Now Supports Protobuf, JSON Schema, and Custom Formats
• Blog post: Confluent Cloud Schema Registry is Now Generally Available
• Podcast: Schema Registry Made Simple