Delete Schemas in Confluent Platform

The Schema Registry API supports deleting a specific schema version or all versions of a subject. A soft delete removes only the version, and the underlying schema ID remains available for lookup.

Soft delete a schema

You can perform a soft delete on all schema versions registered under a subject or on a specific version of a subject.

A soft delete removes the specified schema versions but doesn’t free up space in the registry. The registry retains the schema metadata, including schema IDs, and schema IDs aren’t reusable.

# Deletes all schema versions registered under the subject "Kafka-value"
  curl -X DELETE http://localhost:8081/subjects/Kafka-value
  [1]

# Deletes version 1 of the schema registered under subject "Kafka-value"
  curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/1
  1

# Deletes the most recently registered schema under subject "Kafka-value"
  curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/latest
  1

Hard delete a schema

Both Confluent Platform and Confluent Cloud Schema Registry support hard delete of a schema with the ?permanent=true query string. A hard delete removes all metadata, including schema IDs.

You can perform a hard delete on all schema versions registered under a subject or on a specific version of a subject.

To hard delete a schema, either all versions or a specific version, use the following two-step process. First soft delete the schema, and then hard delete it.

  1. Perform a soft delete of all versions of the schema.

    curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/my-existing-subject
    
  2. Perform a hard delete of all versions of the schema by appending ?permanent=true to the command.

    curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/<my-existing-subject>?permanent=true
    

The following commands hard-delete all versions of the schema registered under the subject “Kafka-value”:

curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/Kafka-value
curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/Kafka-value?permanent=true

To hard-delete version 1 of the schema registered under the subject Kafka-value:

curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/Kafka-value/versions/1
curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/Kafka-value/versions/1?permanent=true

Tip

A hard delete of a specific version requires a version number as input. Using version:latest results in a soft delete only.

Recover a soft-deleted schema

To recover a soft-deleted schema, use the Schema Registry GET /subjects and POST /subjects APIs:

  1. Retrieve the schema with GET /subjects?deleted=true, as described in GET /subjects API description. For a usage example, see List all subjects.

    For example:

    curl -s -X GET -H "Content-Type: application/vnd.schemaregistry.v1+json" \
    -u <username>:<password> \
    "<Schema Registry hostname>:<Schema Registry port>/subjects/<subject-name>/versions/<version-number>?deleted=true"
    

    The output should resemble:

    {"subject":"deletion-test-value","version":1,"id":14,"schema":"{\"type\":\"record\",\"name\":\"test\",\"fields\":[{\"name\":\"field1\",\"type\":\"string\"},{\"name\":\"field2\",\"type\":\"int\"}]}"}
    
  2. Use POST/subjects/(string: subject)/versions to re-register the schema.

    • You can POST from a file using this syntax:

      curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -u <username>:<password>
      --data '@schema.json'
      "<Schema Registry hostname>:<Schema Registry port>/subjects/<subject-name>/versions"
      

      The following example shows schema.json from the preceding command:

      {
        "schema": "{\"type\":\"record\",\"name\":\"test\",\"fields\":[{\"name\":\"field1\",\"type\":\"string\"},{\"name\":\"field2\",\"type\":\"int\"}]}"
      }
      
    • Or you can input the schema directly:

      curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -u <username>:<password>
      --data '{"schema":"{\"type\":\"record\",\"name\":\"test\",\"fields\":[{\"name\":\"field1\",\"type\":\"string\"},{\"name\":\"field2\",\"type\":\"int\"}]}"}'
      "<Schema Registry hostname>:<Schema Registry port>/subjects/<subject-name>/versions"
      

To learn more, see the Schema Registry API Reference for Confluent Platform and Schema Registry API Usage Examples for Confluent Platform.

Workaround to virtually recover a hard-deleted schema

Sometimes you hard delete a schema and later find that it’s still required to read messages. If you have the schema definition, you can re-register it at its previous ID so that consumers keep functioning.

To restore the schema, recreate it with the same definition and put the schema into IMPORT mode. Then manually register it at a specified ID on a Schema Registry. This approach assumes that the schema ID isn’t already taken. It’s useful in recovery operations when a schema was deleted but is backed up on another cluster.

To learn how to “restore” a hard-deleted schema using this method, see Migrate an Individual Schema to an Already Populated Schema Registry (subject-level migration).

Guidelines for using these APIs

The preceding APIs are primarily intended for development environments, where it’s common to iterate before finalizing a schema. Avoid using them in a production environment. A few scenarios exist where you can use these APIs in production, but with utmost care:

  • A new schema to be registered has compatibility issues with one of the existing schema versions.

  • An old version of the schema needs to be registered again for the same subject.

  • The schemas are used only in real-time streaming systems and the older version(s) are absolutely no longer required.

  • A topic needs to be recycled.

Deleting a subject, or deleting its only available schema version, also deletes any compatibility settings registered for the subject.

Schema deletion, versioning, and compatibility

Deleting a schema removes only the versioned instances of the schema. The actual schema, with the hashed ID, remains. The canonical MD5 hash of the schema still exists in the system, which is why you can look up a deleted schema in the REST API.

When you register a schema that matches a soft-deleted schema, Schema Registry creates a new version and reuses the original schema ID, because the hash still matches. For example, if a subject has three versions and you soft delete version 3, registering the same schema again creates version 4 with the same schema ID. Soft-deleted versions still count toward the version numbering of the subject.

If you hard delete version 3, re-registering the same schema reuses version 3. The schema keeps its previous ID only if the same schema is still registered under another subject. Otherwise, Schema Registry assigns a new ID.

When the Avro serializer serializes Avro data, it stores the schema ID, not the version, which is specific to the subject. If you delete the version in a subject, the schema and its ID remain in Schema Registry so that Schema Registry can still deserialize the data.

This has implications for versioning and compatibility. Suppose, for a given subject, you have two schemas:

  • Schema ID: 23, version 1 with fields (a,b)

  • Schema ID: 45, version 2 with fields (a,b,c)

If you delete version 2, version 1 becomes the latest version.

If the compatibility level is BACKWARD, the next schema you register to the subject needs to be backward compatible only with version 1, not version 2.

Use the APIs to determine if a schema is deleted

Some Schema Registry REST API endpoints return more schema metadata, including a deleted field that indicates whether a schema is soft-deleted. With this field, you can identify deleted schemas in a single API call without separate lookups for each schema’s deletion status.

The following endpoints support these additional response properties:

For Confluent Platform, the response always includes these properties.

Identify and delete unused schemas

The schema deletion tool finds and deletes unused schemas in Confluent Platform Schema Registry.

The README provides full instructions for building and using the schema deletion tool independently, or as a Confluent CLI plugin.

On Confluent Platform, run the tool with the --platform cp flag and a configuration file that specifies the Schema Registry connection details and the Kafka clusters to scan:

./schema-deletion-tool --platform cp --cp-config-file /path/to/cp-config.json --all

The tool follows this basic workflow:

  1. The tool takes subject names, based on TopicNameStrategy, to analyze and detect unused schemas.

  2. It identifies topics using the TopicNameStrategy and analyzes messages to identify unused schemas in those topics.

The tool identifies any schema IDs not present in the topic’s messages as unused schemas and lists them as candidates for deletion, with a confirmation prompt before deletion.

Caution

The tool can hard delete schemas in Schema Registry. Use this option with caution.