Stream Catalog

The key to unlocking the value within data in motion and increasing productivity across an organization is a self-service tool for data discovery.

Available through both the Cloud Console and API, stream catalog allows users across teams to collaborate within a centralized, organized library designed for sharing, finding, and understanding the data needed to drive their projects forward quickly and efficiently. It’s like a digital library for data in motion allowing any user, experienced with Kafka or not, to search for what they need, find what’s already been built, and put it to use right away.

With classifications, teams can constantly increase the value of the business’s stream catalog by adding contextual details to data; for example, by labeling schema fields as “PII” (personal identifiable information) or “Sensitive”.

Searching Data and Schemas

Stream catalog in Confluent Cloud centralizes all schemas-related metadata and makes it available for discovery via the global search bar. In particular, with regard to schemas you can search for:

  • Schema subject name
  • Schema record name
  • Schema field name

Search with Filters

Click See all.. or hit return on your initial search results to view a more detailed list, with an option apply more filters.

This view shows “All Results” for a search on stocks.

../_images/dg-global-search-list-all.png

You can filter this further by selecting an entity on the left menu; for example, “Schemas”.

../_images/dg-global-search-schemas-filtered-list.png

Also, you can use the filters across the top to filter by “Environment”, “Tag”, or “Entities”.

This example narrows the search result based on entity type to show only schema subject names that include stocks.

../_images/dg-catalog-filter-by-schema-subject.png

This example filters on schema subjects, records, and fields in the demos environment that are tagged with my_stocks.

../_images/dg-catalog-filter-multiple.png

You can change the search on this detailed list view to search for other names of a selected entity type. For example, if you worked through the steps in the Quick Start to create a schema for the employees topic, search for employees in schemas. (Make sure you clear any other filters like tags that may not match the employees schema.)

../_images/dg-global-search-schemas-filtered-list-new.png

Tagging Data and Schemas

Stream catalog features enables entity tagging. Tags are searchable like any other entity.

A fundamental aspect of governance is the ability to organize data based on a shared vocabulary, including multiple concepts and categories. Confluent Cloud now provides the option to create and apply tags to schemas and fine-grained entities like data records and fields. On this version of Confluent Cloud, you can:

  • Create instances of provided tags (Public, Private, Sensitive, PII) and custom (“free form”) tags
  • Associate tags with schema versions, records, and fields
  • Apply multiple tags to a single field, record, or schema version

Create tags

To create a tag:

  1. Navigate to the Schema Registry tab on the settings for a selected Environment.

    ../_images/dg-tags-view-manage.png
  2. Click View & manage tags.

  3. Click Create tag, select either free-form or commonly used tags.

    • If you chose commonly used tags, select one or more tags that you want to create. (PII, which is personally identifiable information, Sensitive, Private, Public).
    • If you chose free-form, provide a tag name and optional description. Tag names must start with an alphabetic character, and can then include alpha characters, numbers and underscores.
    ../_images/dg-tags-name-rules.png

    When the name and description is properly filled in, the Create button is active. Click Create to create the tag with the current name and description.

    ../_images/dg-tags-create.png
  4. View available tags.

    After a tag is created, it shows in the list under Tag management.

    ../_images/dg-tags-management-list.png

Apply tags

To apply a tag:

  1. Navigate to a schema, data record, or data field for which you want to apply a tag.

    There are a few of different ways to do this:

    • From the Search bar, start typing the name of a schema, record, field, or topic where you want to apply a tag.
    • From the same environment level view on the Schema Registry tab, click View and manage schemas and select a schema from the list.
    • From within a cluster, navigate to a topic, then click the Schema tab for that topic.
  2. On the Schemas tab for a topic, you can add tags as follows.

    • To add tags to selected schema version as a whole, select Add tags to this version on the right panel, and select a tag from the drop-down list of available tags.
    ../_images/dg-tags-schema-version.png
    • To add tags to a selected record or field, expand the tree view of the schema (the default view), click the plus icon image_reference next to the entity, and select a tag from the drop-down list of available tags.
    ../_images/dg-tags-data-field.png
  3. View applied tags.

    Applied tags show next to the schema version, records, and fields with which they are associated.

Remove a tag from a schema version or data field

To remove a previously applied tag from an entity:

  1. Navigate to the schema that includes the tag.
  2. Click the x on the applied tag.

Edit a tag

To edit a tag description:

  1. Navigate to the Schema Registry tab on the settings for a selected Environment.
  2. Click View & manage tags.
  3. Select the tag you want to edit from the Tag management list.
  4. Click the edit icon next to the description, edit, and click Save.

Tip

You cannot edit the name of an existing tag, only its description. To rename a tag, remove it from any entities to which it is applied, delete the tag, and create a new one.

Delete a tag

If you want to delete a tag, first make sure that the tag is not currently applied to any entities. If the tag is in use, the delete operation will fail.

To delete a tag from an environment:

  1. Navigate to the Schema Registry tab on the settings for a selected Environment.
  2. Click View & manage tags.
  3. Select the tag you want to delete from the Tag management list.
  4. Click the trashcan icon in the upper right.
    • If the tag is in use (applied to one or more entities), you will get a warning and the tag will not be deleted.
    • If the tag is not in use, it is deleted.

Catalog API Documentation

The stream catalog API allows you to create, retrieve, update, and delete catalog entities. To learn more, see the Confluent Cloud CATALOG API (V1).

Note

The catalog API provides search capabilities for schema entity types (subjects, records, and fields) only. The API currently does not include search capabilities for other entity types like topics, connectors, and so forth; whereas, the global search in the Cloud Console searches all entity types.

Catalog API Usage Examples

This section provides examples of calls to the Confluent Cloud CATALOG API (V1) using curl commands.

For your API testing, you may want to use jq along with --silent flag for curl to get nicely formatted output for the given commands. The examples below show the calls with these options included, and the expected output.

To get properly formatted output for search calls that include question mark (?) or ampersand (&), you must enclose the URL and search parameters in either single or double quotes so that the shell does not interpret them This syntax is also shown in the examples.

If you don’t use --silent and jq, the content of the output will be the same, just unformatted.

If you walked through the stocks examples for the Stream lineage tutorial, you will have most of the example entities shown below in your environment already. You can further enhance discoverability of those resources by tagging schemas, records, and fields, and then feed those tags into some of the searches shown below.

Setup

You’ll need the following to make API calls to the Catalog API:

  • API endpoint URL for the Schema Registry cluster in the environment you want to use. In the examples below, this is referred to as the <SCHEMA-REGISTRY-URL>. You can find this on Cloud Console by navigating to the Schema Registry tab for your environment, or log on the Confluent Cloud command line, by navigating to your environment and entering ccloud ccloud schema-registry cluster describe. (A handy list of CLI commands is here.)
  • API key and secret for Schema Registry in the environment you want to use. (If needed, see Create an API Key for Confluent Cloud Schema Registry.) Schema subjects live at the level of an environment, in per-environment Schema Registry clusters, so just choose the environment that contains the Kafka clusters you want to use. In the examples below, these are referred to as <API-KEY>:<API-SECRET>; you will provide these as colon-separated pair in the curl commands.

Create tag definitions

curl -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "<TAG_NAME>","description" : "Test API call"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs

For example, to create one of the commonly-used, pre-fab tags (PII, Sensitive, Private, Public), provide that tag name in the API call. The following command creates a PII tag:

curl -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "PII","description" : "Personally Identifiable Information"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs

To create a custom tag called PCI with which to tag stock symbols on your buy list, using curl --silent and piping it through jq:

curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "PCI","description" : "Payment Card Industry Data Security Standard"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .

Your output should resemble:

"category": "CLASSIFICATION",
  "createdBy": "root",
  "updatedBy": "root",
  "createTime": 1631914851910,
  "updateTime": 1631914851910,
  "version": 1,
  "name": "PCI",
  "description": "Payment Card Industry Data Security Standard",
  "typeVersion": "1.0",
  "attributeDefs": [],
  "superTypes": [],
  "entityTypes": [
    "sr_field",
    "sr_schema",
    "sr_record"
    "sr_subject_version"
  ]

To create a custom tag called stocks_buy with which to tag stock symbols on your buy list, using curl --silent and piping it through jq:

curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "stocks_buy","description" : "stock symbols on buy list"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .

Your output should resemble:

"category": "CLASSIFICATION",
  "createdBy": "root",
  "updatedBy": "root",
  "createTime": 1631914483922,
  "updateTime": 1631914483922,
  "version": 1,
  "name": "stocks_buy",
  "description": "stock symbols on buy list",
  "typeVersion": "1.0",
  "attributeDefs": [],
  "superTypes": [],
  "entityTypes": [
    "sr_field",
    "sr_schema",
    "sr_record"
    "sr_subject_version"
  ]

Create another custom tag called my_stocks:

curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "my_stocks","description" : "stocks I am tracking"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .

Log on to https://confluent.cloud and check the Cloud Console to verify that the new tags show up there. Navigate to the environment in which you are working, click the Schema Registry tab, and scroll down to find the Tags.

../_images/dg-catalog-api-view-created-tags.png

Click View and manage tags to see all tags.

../_images/dg-catalog-api-view-created-tags-detail.png

Update a tag definition by adding attributes (key, value pairs)

The following call updates the PCI tag with several new attributes.

curl --silent -u <API-KEY>:<API-SECRET> --request PUT \
  --url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs \
  --header 'Content-Type: application/json' \
  --data '[{ "entityTypes" : [ "sr_record", "sr_field" ],
           "name" : "PCI", "description" : "Payment Card Industry Data Security Standard" ,
           "attributeDefs" : [ { "name" : "key", "cardinality" : "SINGLE", "typeName" : "string" },
           { "name" : "region", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string" },
           { "name" : "keytype", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string" } ] }]' | jq .

Your output should resemble:

"category": "CLASSIFICATION",
 "name": "PCI",
 "description": "Payment Card Industry Data Security Standard",
 "attributeDefs": [
   {
     "name": "key",
     "typeName": "string",
     "isOptional": false,
     "cardinality": "SINGLE",
     "valuesMinCount": -1,
     "valuesMaxCount": -1,
     "isUnique": false,
     "isIndexable": false,
     "includeInNotification": false,
     "searchWeight": -1
   },
   {
     "name": "region",
     "typeName": "string",
     "isOptional": true,
     "cardinality": "SINGLE",
     "valuesMinCount": -1,
     "valuesMaxCount": -1,
     "isUnique": false,
     "isIndexable": false,
     "includeInNotification": false,
     "searchWeight": -1
   },
   {
     "name": "keytype",
     "typeName": "string",
     "isOptional": true,
     "cardinality": "SINGLE",
     "valuesMinCount": -1,
     "valuesMaxCount": -1,
     "isUnique": false,
     "isIndexable": false,
     "includeInNotification": false,
     "searchWeight": -1
   }
 ],
 "superTypes": [],
 "entityTypes": [
   "sr_field",
   "sr_record"
 ]

List tag definitions

curl --silent -u <API-KEY>:<API-SECRET> \
  --request GET \
  --url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .

Your output will resemble:

"category": "CLASSIFICATION",
    "guid": "f94803c3-6aba-4b41-a003-2f8fc1957c29",
    "createdBy": "root",
    "updatedBy": "root",
    "createTime": 1631593629096,
    "updateTime": 1631593629096,
    "version": 1,
    "name": "PII",
    "description": "Personally identifiable information",
    "typeVersion": "1.0",
    "attributeDefs": [],
    "superTypes": [],
    "entityTypes": [
      "cf_entity"
    ]
  },
  {
    "category": "CLASSIFICATION",
    "createdBy": "root",
    "updatedBy": "root",
    "createTime": 1631914851910,
    "updateTime": 1631914851910,
    "version": 1,
    "name": "PCI",
    "description": "Payment Card Industry Data Security Standard",
    "typeVersion": "1.0",
    "attributeDefs": [],
    "superTypes": [],
    "entityTypes": [
      "sr_field",
      "sr_schema",
      "sr_record",
      "sr_subject_version"
    ]
  },

  {
    "category": "CLASSIFICATION",
    "createdBy": "root",
    "updatedBy": "root",
    "createTime": 1631914483920,
    "updateTime": 1631914483920,
    "version": 1,
    "name": "stocks_buy",
    "description": "stock symbols on buy list",
    "typeVersion": "1.0",
    "attributeDefs": [],
    "superTypes": [],
    "entityTypes": [
      "sr_field",
      "sr_schema",
      "sr_record",
      "sr_subject_version"
    ]
  },

  ...

Get tag definition

Get the definition for the PII (personally identifiable information) tag.

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs/PII | jq .

Your output should resemble:

 "category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631593629105,
"updateTime": 1631593629105,
"version": 1,
"name": "PII",
"description": "Personally identifiable information",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
  "cf_entity"
]

Search fields by name

Search for fields named credit_card:

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&query=credit_card' | jq .

Your output should resemble:

"searchParameters": {
  "includeDeleted": false,
  "limit": 0,
  "offset": 0
},
"types": [
  "sr_field"
],
"entities": [
  {
    "typeName": "sr_field",
    "attributes": {
      "createTime": 1632516635687,
      "qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.credit_card",
      "name": "credit_card",
      "context": ".",
      "id": 100007,
      "nameLower": "credit_card",
      "tenant": "lsrc-g2p81"
    },
    "guid": "541fae87-7fe5-4cab-b509-0f76cbf47515",
    "status": "ACTIVE",
    "displayText": "credit_card",
    "classificationNames": [
      "credit_card",
      "PII"
    ],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": [

Search fields by tag

Search for fields tagged as PII.

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&tag=PII' | jq .

Your output should resemble:

"searchParameters": {
   "includeDeleted": false,
   "limit": 0,
   "offset": 0
 },
 "types": [
   "sr_field"
 ],
 "entities": [
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1614710010055,
       "qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.userid",
       "name": "userid",
       "context": ".",
       "id": 100001,
       "nameLower": "userid",
       "tenant": "lsrc-g2p81"
     },
     "guid": "99a87d47-f55c-4cff-ad36-cf870ebab382",
     "status": "ACTIVE",
     "displayText": "userid",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1614710010055,
       "qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.account",
       "name": "account",
       "context": ".",
       "id": 100001,
       "nameLower": "account",
       "tenant": "lsrc-g2p81"
     },
     "guid": "e8b96055-30ef-4203-adbc-30e28fa419c9",
     "status": "ACTIVE",
     "displayText": "account",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1614902469162,
       "qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.ACCOUNT",
       "name": "ACCOUNT",
       "context": ".",
       "id": 100003,
       "nameLower": "account",
       "tenant": "lsrc-g2p81"
     },
     "guid": "152e502a-98ac-484b-86ef-b789d9eb8bf7",
     "status": "ACTIVE",
     "displayText": "ACCOUNT",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1627756849389,
       "qualifiedName": "lsrc-g2p81:.:100004:Example.Employee.Name",
       "name": "Name",
       "context": ".",
       "id": 100004,
       "nameLower": "name",
       "tenant": "lsrc-g2p81"
     },
     "guid": "93673886-cea1-4f8d-ad2b-3e4574e70ee1",
     "status": "ACTIVE",
     "displayText": "Name",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1627756849389,
       "qualifiedName": "lsrc-g2p81:.:100004:Example.Employee.Age",
       "name": "Age",
       "context": ".",
       "id": 100004,
       "nameLower": "age",
       "tenant": "lsrc-g2p81"
     },
     "guid": "5a94a4dd-e930-46d2-b548-82f91f55f38a",
     "status": "ACTIVE",
     "displayText": "Age",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1632516635687,
       "qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.balance",
       "name": "balance",
       "context": ".",
       "id": 100007,
       "nameLower": "balance",
       "tenant": "lsrc-g2p81"
     },
     "guid": "b5652a8c-4509-4477-a45f-8dc7a504707c",
     "status": "ACTIVE",
     "displayText": "balance",
     "classificationNames": [
       "credit_card",
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1632516635687,
       "qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.last_name",
       "name": "last_name",
       "context": ".",
       "id": 100007,
       "nameLower": "last_name",
       "tenant": "lsrc-g2p81"
     },
     "guid": "175a6542-5af9-43f1-a9c6-a83491db94d9",
     "status": "ACTIVE",
     "displayText": "last_name",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1632516635687,
       "qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.credit_card",
       "name": "credit_card",
       "context": ".",
       "id": 100007,
       "nameLower": "credit_card",
       "tenant": "lsrc-g2p81"
     },
     "guid": "541fae87-7fe5-4cab-b509-0f76cbf47515",
     "status": "ACTIVE",
     "displayText": "credit_card",
     "classificationNames": [
       "credit_card",
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []
   },
   {
     "typeName": "sr_field",
     "attributes": {
       "createTime": 1632516635687,
       "qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.first_name",
       "name": "first_name",
       "context": ".",
       "id": 100007,
       "nameLower": "first_name",
       "tenant": "lsrc-g2p81"
     },
     "guid": "ddc35b11-056e-4a41-8a5b-69d2366e6455",
     "status": "ACTIVE",
     "displayText": "first_name",
     "classificationNames": [
       "PII"
     ],
     "meaningNames": [],
     "meanings": [],
     "isIncomplete": false,
     "labels": []

Search schema record by name

Search for a schema record in namespace ksql with record name StockTrade.

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_record&query=ksql.StockTrade' | jq .

Your output should resemble:

 "searchParameters": {
  "includeDeleted": false,
  "limit": 0,
  "offset": 0
},
"types": [
  "sr_record"
],
"entities": [
  {
    "typeName": "sr_record",
    "attributes": {
      "createTime": 1614710010055,
      "qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade",
      "name": "StockTrade",
      "context": ".",
      "id": 100001,
      "nameLower": "stocktrade",
      "tenant": "lsrc-g2p81"
    },
    "guid": "d757148e-28f6-465b-b7de-088732848ab5",
    "status": "ACTIVE",
    "displayText": "StockTrade",
    "classificationNames": [
      "my_stocks",
      "Sensitive"
    ],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []

Search schema by tag

Search for schema records tagged with my_stocks.

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_record&tag=my_stocks' | jq .

Your output should resemble:

"searchParameters": {
  "includeDeleted": false,
  "limit": 0,
  "offset": 0
},
"types": [
  "sr_record"
],
"entities": [
  {
    "typeName": "sr_record",
    "attributes": {
      "createTime": 1614902469162,
      "qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema",
      "name": "KsqlDataSourceSchema",
      "context": ".",
      "id": 100003,
      "nameLower": "ksqldatasourceschema",
      "tenant": "lsrc-g2p81"
    },
    "guid": "dee2acf2-08e5-43af-a2de-927a06f17c33",
    "status": "ACTIVE",
    "displayText": "KsqlDataSourceSchema",
    "classificationNames": [
      "my_stocks"
    ],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_record",
    "attributes": {
      "createTime": 1614710010055,
      "qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade",
      "name": "StockTrade",
      "context": ".",
      "id": 100001,
      "nameLower": "stocktrade",
      "tenant": "lsrc-g2p81"
    },
    "guid": "d757148e-28f6-465b-b7de-088732848ab5",
    "status": "ACTIVE",
    "displayText": "StockTrade",
    "classificationNames": [
      "my_stocks",
      "Sensitive"
    ],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []

Tag a field in Avro

Suppose you want to update the stocks_buy-value schema by tagging a field in one of its records. Specifically, the stocks_buy-value schema contains the record KsqlDataSourceSchema, and you want to tag the SYMBOL field with the stocks_buy tag.

First, you must get the fully qualified name of the field you want to tag because you will need this to specify the "entityName". You cannot get this from the Cloud Console. To get the fully qualified name of a field in a schema, send an API call that returns the details of that field.

For example, search for fields named SYMBOL:

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&query=SYMBOL' | jq .

Your output will resemble the following. The fully qualified name for the SYMBOL field in the stocks_buy-value schema KsqlDataSourceSchema record is highlighted:

"searchParameters": {
    "includeDeleted": false,
    "limit": 0,
    "offset": 0
  },
  "types": [
    "sr_field"
  ],
  "entities": [
    {
      "typeName": "sr_field",
      "attributes": {
        "createTime": 1614710010055,
        "qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.symbol",
        "name": "symbol",
        "context": ".",
        "id": 100001,
        "nameLower": "symbol",
        "tenant": "lsrc-g2p81"
      },
      "guid": "446a0d3f-b77e-4ce5-a0d2-628076cd4a13",
      "status": "ACTIVE",
      "displayText": "symbol",
      "classificationNames": [],
      "meaningNames": [],
      "meanings": [],
      "isIncomplete": false,
      "labels": []
    },
    {
      "typeName": "sr_field",
      "attributes": {
        "createTime": 1614902466440,
        "qualifiedName": "lsrc-g2p81:.:100002:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
        "name": "SYMBOL",
        "context": ".",
        "id": 100002,
        "nameLower": "symbol",
        "tenant": "lsrc-g2p81"
      },
      "guid": "8cfacbc2-8ae8-44c0-b860-e3161bdb012a",
      "status": "ACTIVE",
      "displayText": "SYMBOL",
      "classificationNames": [],
      "meaningNames": [],
      "meanings": [],
      "isIncomplete": false,
      "labels": []
    },
    {
      "typeName": "sr_field",
      "attributes": {
        "createTime": 1614902469162,
        "qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
        "name": "SYMBOL",
        "context": ".",
        "id": 100003,
        "nameLower": "symbol",
        "tenant": "lsrc-g2p81"
      },
      "guid": "16854b0c-580b-408a-9cb6-49e5505900cc",
      "status": "ACTIVE",
      "displayText": "SYMBOL",
      "classificationNames": [
        "stocks_buy"
      ],
      "meaningNames": [],
      "meanings": [],
      "isIncomplete": false,
      "labels": []

Now tag the SYMBOL field with the stocks_buy tag.

curl --silent -u <API-KEY>:<API-SECRET> \
--request POST \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/entity/tags \
--header 'Content-Type: application/json' \
--data '[ {  "entityType" : "sr_field",
  "entityName" : "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
  "typeName" : "stocks_buy"} ]' | jq .

Your output should resemble:

"typeName": "stocks_buy",
"entityStatus": "ACTIVE",
"entityType": "sr_field",
"entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"

To verify on the Cloud Console, log on to https://confluent.cloud, navigate to the schema and field you just tagged, and you should see the stocks_buy tag associated with the SYMBOL field.

../_images/dg-catalog-api-add-tag-in-avro-ui.png

Add another tag, my_stocks, to the same field either through the Cloud Console or through the API. If you want to use the API call, submit the curl command the same as above and just substitute my_stocks for stocks_buy.

Get the tag attributes from a field

Get the tag attributes for the same field (SYMBOL in the stocks_buy-value schema in the record KsqlDataSourceSchema).

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/entity/type/sr_field/name/100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL/tags' | jq .

Your output should resemble:

   "typeName": "my_stocks",
  "entityGuid": "cf62db9b-caec-40dd-928d-6dd98f803b4e",
  "entityStatus": "ACTIVE",
  "propagate": true,
  "removePropagationsOnEntityDelete": false,
  "entityType": "sr_field",
  "entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"
},
{
  "typeName": "stocks_buy",
  "entityGuid": "cf62db9b-caec-40dd-928d-6dd98f803b4e",
  "entityStatus": "ACTIVE",
  "propagate": true,
  "removePropagationsOnEntityDelete": false,
  "entityType": "sr_field",
  "entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"

Tag a schema version

First, create a new custom tag called favorites:

curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_subject_version" ],"name" : "favorites","description" : "my favorite stocks"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs' | jq .

Your output should resemble:

"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1632974329626,
"updateTime": 1632974329626,
"version": 1,
"name": "favorites",
"description": "my favorite stocks",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
  "sr_field",
  "sr_schema",
  "sr_record",
  "sr_subject_version"

Tag the current schema version of the stocks_buy-value schema you’ve been working on as favorites. The following examples assumes the current version is v2. If your stocks_buy-value version is v1, replace v2 with v1 in the API call.

curl --silent -u <API-KEY>:<API-SECRET> \
--request POST \
--url <SCHEMA-REGISTRY-URL>/catalog/v2/entity/tags \
--header 'Content-Type: application/json' \
--data '[ {  "entityType" : "sr_schema",  "entityName" : "lsrc-g2p81:.:100003",  "typeName" : "favorites"  } ]' | jq .

Your output will resemble:

"typeName": "favorites",
"entityStatus": "ACTIVE",
"entityType": "sr_schema",
"entityName": "lsrc-g2p81:.:100003"

Navigate to the stocks_buy-value schema on the Cloud Console to verify that favorites is associated with the version you tagged.

../_images/dg-catalog-api-view-tagged-schema-version.png

Get a schema given a subject version

List all version 1 value schemas with the subject name prefix of stocks.

curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url 'https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=sr_subject_version&attrName=name&attrValuePrefix=stocks' | jq .

Your output should resemble:

  "searchParameters": {
  "includeDeleted": false,
  "limit": 0,
  "offset": 0
},
"types": [
  "sr_subject_version"
],
"entities": [
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614710010055,
      "qualifiedName": "lsrc-g2p81:.:stocks-value:1",
      "name": "stocks-value",
      "context": ".",
      "id": 100001,
      "nameLower": "stocks-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "ba600e7a-0093-41b3-8d68-38ba730323bc",
    "status": "ACTIVE",
    "displayText": "stocks-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902357913,
      "qualifiedName": "lsrc-g2p81:.:stocks_under_100-value:1",
      "name": "stocks_under_100-value",
      "context": ".",
      "id": 100002,
      "nameLower": "stocks_under_100-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "7e2e652b-3b53-4461-b634-c8a58186b28d",
    "status": "ACTIVE",
    "displayText": "stocks_under_100-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902369755,
      "qualifiedName": "lsrc-g2p81:.:stocks_under_100-value:2",
      "name": "stocks_under_100-value",
      "context": ".",
      "id": 100003,
      "nameLower": "stocks_under_100-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "799fde4c-a43b-4820-896c-e1f5fa2ba399",
    "status": "ACTIVE",
    "displayText": "stocks_under_100-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902422050,
      "qualifiedName": "lsrc-g2p81:.:stocks_buy-value:2",
      "name": "stocks_buy-value",
      "context": ".",
      "id": 100003,
      "nameLower": "stocks_buy-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "59f3ced1-a303-4712-ae7a-f720b9261084",
    "status": "ACTIVE",
    "displayText": "stocks_buy-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902418968,
      "qualifiedName": "lsrc-g2p81:.:stocks_buy-value:1",
      "name": "stocks_buy-value",
      "context": ".",
      "id": 100002,
      "nameLower": "stocks_buy-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "785f7f6f-5b7b-4695-bea7-d9b0a6df1947",
    "status": "ACTIVE",
    "displayText": "stocks_buy-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902469162,
      "qualifiedName": "lsrc-g2p81:.:stocks_sell-value:2",
      "name": "stocks_sell-value",
      "context": ".",
      "id": 100003,
      "nameLower": "stocks_sell-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "4abf45bd-46e9-4ec5-a959-e95a6010b166",
    "status": "ACTIVE",
    "displayText": "stocks_sell-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  },
  {
    "typeName": "sr_subject_version",
    "attributes": {
      "createTime": 1614902466440,
      "qualifiedName": "lsrc-g2p81:.:stocks_sell-value:1",
      "name": "stocks_sell-value",
      "context": ".",
      "id": 100002,
      "nameLower": "stocks_sell-value",
      "tenant": "lsrc-g2p81"
    },
    "guid": "bfea548e-a99f-48f1-b310-e99f25e908c4",
    "status": "ACTIVE",
    "displayText": "stocks_sell-value",
    "classificationNames": [],
    "meaningNames": [],
    "meanings": [],
    "isIncomplete": false,
    "labels": []
  }