Delete Schemas and Manage Storage Space on Confluent Cloud
Confluent Cloud Schema Registry limits the number of schema versions supported. You can free up space for new schemas by identifying and deleting unused schemas.
Maximum schema limits
In general, the limit is 20,000 schemas, including all schema versions. Confluent Cloud Schema Registry also limits the number of schema versions in the registry for Basic, Standard, and Dedicated cluster types, as described in Kafka Cluster Types in Confluent Cloud. To view per-package limits, see Stream Governance Packages, Cloud Providers, and Region Support.
It is possible to go above the limit, depending on the scenario. Confluent recommends contacting Support and your Schema Registry team to determine best practices for specifics such as cluster load and memory utilization.
When the registry reaches the limit, a request to register a new schema returns the following HTTP response:
HTTP/1.1 422 Unprocessable Entity
Content-Type: application/vnd.schemaregistry.v1+json
{
"error_code": 403,
"message": "Schema Limit Exceeded"
}
To find unused schemas, see Identify and delete unused schemas. To free up space for new schemas, see Free up storage space in the registry for new schemas.
Use the APIs to determine if a schema is deleted
To find out whether a schema is soft-deleted, read the deleted field in the
response from supported Schema Registry REST API endpoints. The field lets you identify
deleted schemas in a single API call instead of making separate lookups for
each schema’s deletion status.
The following endpoints support these additional response properties:
On Confluent Cloud, you must pass the following header to include these properties in the response:
Confluent-Accept-Unknown-Properties: true
Note
Java and non-Java Schema Registry clients add this header automatically.
The following request lists schemas with the subject prefix my-subject,
including deleted schemas:
curl --silent -u $SR_APIKEY:$SR_APISECRET --request GET \
-H "Confluent-Accept-Unknown-Properties: true" \
"$SCHEMA_REGISTRY_URL/schemas?subjectPrefix=my-subject&deleted=true" | jq .
For complete response property details, see the Schemas (V1) and Subjects (V1) API references.
Identify and delete unused schemas
You can use the schema deletion tool to find and delete schemas in the Confluent Cloud Schema Registry.
Before following the build instructions in the tool’s README, clone the
schema-deletion-tool repository. The build commands and goenv install
step in the README run from the root of this repository. Cloning only the
goenv dependency referenced in the README causes make build-plugin
to fail.
git clone https://github.com/confluentinc/schema-deletion-tool.git
cd schema-deletion-tool
The README provides full instructions on how to build and use the schema deletion tool independently, or as a Confluent CLI plugin.
The basic workflow is:
Provide the subject names to analyze and detect unused schemas. The subject names are based on
TopicNameStrategy.The tool then identifies topics using the
TopicNameStrategyand analyzes messages to identify unused schemas in those topics.
For example, if you select a subject named stocks-value, the tool analyzes
messages from the topic stocks and identifies unused schemas in the subject
stocks-value.
The tool lists any schema ID that doesn’t appear in the topic’s messages as an unused schema and a candidate for deletion.
Then enter the schemas you want to delete.
To learn more about subject naming strategies, see Subject Name Strategy.
Caution
Use the schema deletion tool with care. The tool can hard-delete schemas in Schema Registry, which removes them permanently.
Free up storage space in the registry for new schemas
To free up space in the registry, hard-delete schemas. A soft delete doesn’t free up space because Confluent Cloud keeps the schema ID and doesn’t reuse it. The schema count tracks IDs. The used number increases as you add schemas, even after you soft-delete others.
A hard delete is a two-step process: a soft delete, and then a second delete
with the query string ?permanent=true. If your Schema Registry reaches the maximum
schema limit, follow the steps in Hard delete a schema to free up space.
Delete schemas
The Confluent Cloud Schema Registry subjects APIs support deleting a specific schema version or all versions of a subject. On a soft delete, the API deletes only the version. The underlying schema ID remains available for lookups.
Soft delete a schema
A soft delete removes the specified schema versions but doesn’t free up space in the registry. Schema Registry retains the schema metadata, including schema IDs, and doesn’t reuse the IDs. You can soft-delete all schema versions registered under a subject or a specific version of a subject.
To soft-delete all schema versions registered under the subject
Kafka-value, run:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET \
$SCHEMA_REGISTRY_URL/subjects/Kafka-value
The response lists the deleted versions:
[1]
To soft-delete version 1 of the schema registered under Kafka-value, run:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET \
$SCHEMA_REGISTRY_URL/subjects/Kafka-value/versions/1
To soft-delete the most recently registered schema under Kafka-value,
run:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET \
$SCHEMA_REGISTRY_URL/subjects/Kafka-value/versions/latest
Each version-specific request returns the deleted version number, such as
1.
Tip
The Confluent Cloud Console and the Confluent CLI can report different numbers of deleted schemas. To learn more, see Hard delete a schema and Schema deletion, versioning, and compatibility.
Hard delete a schema
A hard delete removes all schema metadata, including schema IDs. Confluent Platform and
Confluent Cloud Schema Registry both support it with the query string ?permanent=true. On
Confluent Cloud, which limits the number of schema versions in the registry, a hard
delete frees up space for new schemas.
You can perform a hard delete on all schema versions registered under a subject or on a specific version of a subject.
To hard-delete a schema, or a specific version of it, first soft-delete it, and then hard-delete it:
Soft-delete all versions of the schema:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET $SCHEMA_REGISTRY_URL/subjects/<subject_name>
Hard-delete all versions of the schema by appending
?permanent=trueto the command:curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET "$SCHEMA_REGISTRY_URL/subjects/<subject_name>?permanent=true"
The following commands hard-delete all versions of the schema registered under
the subject Kafka-value:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET $SCHEMA_REGISTRY_URL/subjects/Kafka-value
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET $SCHEMA_REGISTRY_URL/subjects/Kafka-value?permanent=true
The following commands hard-delete version 1 of the schema registered under the
subject Kafka-value:
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET $SCHEMA_REGISTRY_URL/subjects/Kafka-value/versions/1
curl --silent -X DELETE -u $SR_APIKEY:$SR_APISECRET $SCHEMA_REGISTRY_URL/subjects/Kafka-value/versions/1?permanent=true
Tip
A version-specific hard delete requires a version number. A request
with latest as the version results in only a soft delete.
Recover a soft-deleted schema
To recover a soft-deleted schema, retrieve it with GET /subjects and
register it again:
Retrieve the schema with
GET /subjects?deleted=true, as described in the GET /subjects API description. For example:curl --silent -u $SR_APIKEY:$SR_APISECRET -X GET $SCHEMA_REGISTRY_URL/subjects?deleted=true | jq .
To re-register the schema, use POST /subjects/{subject}/versions.
Guidelines for deleting schemas in production
The delete and recover APIs are intended for development environments, where a schema often goes through several iterations before it’s final. They aren’t recommended for production. In production, use them with care and only in the following scenarios:
A new schema has compatibility issues with an existing schema version.
An old version of the schema needs to be registered again for the same subject.
The schemas are used only in real-time streaming systems, and the older versions are no longer required.
A topic needs to be recycled.
You want to free up space for new schemas on Confluent Cloud with a hard delete.
Note
Deleting a subject, or deleting its only schema version, also deletes the compatibility settings registered for the subject.
View available schemas on Confluent Cloud
You can view available and used schemas, and infer the number of soft-deleted schemas.
On the Confluent CLI, you can use confluent schema-registry cluster describe to view the number of available schemas for the selected environment. For example:
confluent schema-registry cluster describe
Your output should resemble:
+-------------------------+----------------------------------------------------+
| Name | Stream Governance Package |
| Cluster ID | lsrc-12345 |
| Endpoint URL | https://psrc-12345.us-central1.gcp.confluent.cloud |
| Used Schemas | 25 |
| Available Schemas | 975 |
| Global Compatibility | <Requires API Key> |
| Mode | <Requires API Key> |
| Cloud | gcp |
| Region | us-central1 |
| Package | advanced |
+-------------------------+----------------------------------------------------+
Used Schemas reports the total of active and soft-deleted schemas. If you
know how many active schemas you have, you can infer the number of
soft-deleted schemas. When you hard-delete a schema, the number of used
schemas decreases and the number of available schemas increases.
Schema deletion, versioning, and compatibility
In reality, deleting a schema removes only the versioned instance(s) of the schema. The actual schema (with the hashed ID) does not go away. The canonical MD5 hash of the schema still exists in the system. This is why you can perform a lookup on a deleted schema in the rest API.
When you register a schema that matches a soft-deleted schema, Schema Registry creates a new version and reuses the original schema ID, because the hash still matches. For example, if a subject has three versions and you soft delete version 3, registering the same schema again creates version 4 with the same schema ID. Soft-deleted versions still count toward the version numbering of the subject.
If you hard delete version 3, re-registering the same schema reuses version 3. The schema keeps its previous ID only if the same schema is still registered under another subject. Otherwise, Schema Registry assigns a new ID.
When the Avro serializer serializes Avro data, it stores the ID (not the version, which is specific to the subject). If the version (in a subject) is deleted, the schema and its ID remain in Schema Registry so that the data and can still be deserialized using Schema Registry
This has implications for versioning and compatibility. Suppose, for a given subject, you have two schemas:
Schema ID: 23, version 1 with fields (a,b)
Schema ID: 45, version 2 with fields (a,b,c)
If you delete version 2, version 1 becomes the latest version.
If the compatibility level is BACKWARD, the next schema that is registered to the subject need only be backward compatible with version 1, not version 2.