Stream Catalog REST API for Confluent Cloud¶
The stream catalog API allows you to create, retrieve, update, and delete catalog entities through a REST API, and add business metadata to entities. To learn more, see the Confluent Cloud CATALOG API (V1).
Note
The catalog API provides search capabilities for schema entity types (subjects, records, and fields) only. The API currently does not include search capabilities for other entity types like topics, connectors, and so forth; whereas, the global search in the Cloud Console searches all entity types.
Catalog API Usage Examples¶
This section provides examples of calls to the Confluent Cloud CATALOG API (V1) using curl commands.
Prerequisites and API examples are provided in these sections:
Setup and Suggestions¶
You’ll need to know the following to make API calls to the CATALOG API per the usage examples provided here:
API endpoint URL for the Confluent Cloud Schema Registry cluster in the environment you want to use. In the examples below, this is referred to as the
<SCHEMA-REGISTRY-URL>
. You can find this on Cloud Console by navigating to the Schema Registry tab for your environment, or log on to Confluent Cloud with the unified Confluent CLI, make sure you have the appropriate environment selected (confluent env list
,confluent env use <environment-ID>
), and enterconfluent schema-registry cluster describe
. (A handy list of CLI commands is here.)If you are using API keys, you need an API key and secret for Schema Registry in the environment you want to use. (If needed, see Create an API Key for Confluent Cloud Schema Registry.) Schema subjects live at the level of an environment, in per-environment Schema Registry clusters, so just choose the environment that contains the Kafka clusters you want to use.
If you are using OAuth, you must provide your access token for authorization in the proper format, as described in Access token format. An OAuth example is provided below.
To get properly formatted output for search calls that include question mark (
?
) or ampersand (&
), you must enclose the URL and search parameters in either single or double quotes so that the shell does not interpret them This syntax is also shown in the examples.(Optional) You may want to store your API keys and Schema Registry URL in local shell environment variables to make testing easier. For example, to store your API key and secret:
$MYCOOLKEYS=xyz:abc
You can check the contents of the variable with
echo $MYCOOLKEYS
then, use it for authorization in subsequent commands:curl --silent -u $MYCOOLKEYS -X POST -H "Content-Type: application/json" ...
.(Optional) For your API testing, you may want to use the
--silent
flag with the curl commands and pipe the entire command through jq . to get nicely formatted output. Most of the examples below show the calls with these options included, and the expected output. If you don’t use--silent
andjq
, the content of the output will be the same, it just won’t be formatted as human-friendly readable.(Optional) A Confluent Cloud login is handy for comparing API results with the Confluent Cloud Console (https://confluent.cloud/).
OAuth for Confluent Cloud Stream Catalog REST API¶
Stream Catalog REST API now supports OAuth, an open-standard authorization protocol for secure access. To learn more about OAuth, see OAuth for Confluent Cloud and Configure Schema Registry clients for OAuth.
Here is an example of using OAuth to call to get a list of Tags from Stream Catalog.
curl --request GET \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs \
--header 'Confluent-Identity-Pool-Id: <POOL-ID>' \
--header 'target-sr-cluster: <CLUSTER-ID' \
--header 'Authorization: Bearer $TOKEN'
The rest of the examples on this page use an API key and secret. If you want to use OAuth instead,
remove the -u <API-KEY>:<API-SECRET>
and follow the format shown above (specify identity pool ID, Schema Registry cluster, and OAuth bearer token).
Tags API Examples¶
Stream catalog features enables entity tagging. Tags are searchable like any other entity. Tags concepts are fully covered in the Confluent Cloud Console user guide under Tagging Data and Schemas, so if you have not read that overview you might want to have a look at it first.
This section provides usage examples for working with tags through the Confluent Cloud CATALOG API (V1).
If you walked through the stocks examples for the Stream lineage tutorial, you will have most of the example entities shown below in your environment already. You can further enhance discoverability of those resources by tagging schemas, records, and fields, and then feed those tags into some of the searches shown below.
Tip
Be sure to review the Setup and Suggestions notes for these API usage examples before you begin.
Create tag definitions¶
curl -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "<TAG_NAME>","description" : "Test API call"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs
For example, to create one of the commonly-used, pre-fab tags (PII, Sensitive, Private, Public), provide that tag name in the API call. The following command creates a PII tag:
curl -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "PII","description" : "Personally Identifiable Information"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs
To create a custom tag called PCI
with which to tag stock symbols on your buy list,
using curl --silent
and piping it through jq
:
curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "PCI","description" : "Payment Card Industry Data Security Standard"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .
Your output should resemble:
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631914851910,
"updateTime": 1631914851910,
"version": 1,
"name": "PCI",
"description": "Payment Card Industry Data Security Standard",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_schema",
"sr_record"
"sr_schema"
]
To create a custom tag called stocks_buy
with which to tag stock symbols on your buy list,
using curl --silent
and piping it through jq
:
curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "stocks_buy","description" : "stock symbols on buy list"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .
Your output should resemble:
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631914483922,
"updateTime": 1631914483922,
"version": 1,
"name": "stocks_buy",
"description": "stock symbols on buy list",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_schema",
"sr_record"
"sr_schema"
]
Create another custom tag called my_stocks
:
curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "my_stocks","description" : "stocks I am tracking"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .
Log on to https://confluent.cloud and check the Cloud Console to verify that the new tags show up there. Navigate to the environment in which you are working, and click Tags on the right side panel.
All tags are shown on this view.

Update a tag definition by adding attributes (key, value pairs)¶
The following call updates the PCI
tag with several new attributes.
curl --silent -u <API-KEY>:<API-SECRET> --request PUT \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs \
--header 'Content-Type: application/json' \
--data '[{ "entityTypes" : [ "sr_record", "sr_field" ],
"name" : "PCI", "description" : "Payment Card Industry Data Security Standard" ,
"attributeDefs" : [ { "name" : "key", "cardinality" : "SINGLE", "typeName" : "string" },
{ "name" : "region", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string" },
{ "name" : "keytype", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string" } ] }]' | jq .
Your output should resemble:
"category": "CLASSIFICATION",
"name": "PCI",
"description": "Payment Card Industry Data Security Standard",
"attributeDefs": [
{
"name": "key",
"typeName": "string",
"isOptional": false,
"cardinality": "SINGLE",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "region",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "keytype",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
}
],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_record"
]
List tag definitions¶
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs | jq .
Your output will resemble:
"category": "CLASSIFICATION",
"guid": "f94803c3-6aba-4b41-a003-2f8fc1957c29",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631593629096,
"updateTime": 1631593629096,
"version": 1,
"name": "PII",
"description": "Personally identifiable information",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"cf_entity"
]
},
{
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631914851910,
"updateTime": 1631914851910,
"version": 1,
"name": "PCI",
"description": "Payment Card Industry Data Security Standard",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_schema",
"sr_record",
"sr_schema"
]
},
{
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631914483920,
"updateTime": 1631914483920,
"version": 1,
"name": "stocks_buy",
"description": "stock symbols on buy list",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_schema",
"sr_record",
"sr_schema"
]
},
...
Get tag definition¶
Get the definition for the PII (personally identifiable information) tag.
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs/PII | jq .
Your output should resemble:
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1631593629105,
"updateTime": 1631593629105,
"version": 1,
"name": "PII",
"description": "Personally identifiable information",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"cf_entity"
]
Search fields by name¶
Search for fields named credit_card
:
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&query=credit_card' | jq .
Your output should resemble:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_field"
],
"entities": [
{
"typeName": "sr_field",
"attributes": {
"createTime": 1632516635687,
"qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.credit_card",
"name": "credit_card",
"context": ".",
"id": 100007,
"nameLower": "credit_card",
"tenant": "lsrc-g2p81"
},
"guid": "541fae87-7fe5-4cab-b509-0f76cbf47515",
"status": "ACTIVE",
"displayText": "credit_card",
"classificationNames": [
"credit_card",
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": [
Search fields by tag¶
Search for fields tagged as PII
.
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&tag=PII' | jq .
Your output should resemble:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_field"
],
"entities": [
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.userid",
"name": "userid",
"context": ".",
"id": 100001,
"nameLower": "userid",
"tenant": "lsrc-g2p81"
},
"guid": "99a87d47-f55c-4cff-ad36-cf870ebab382",
"status": "ACTIVE",
"displayText": "userid",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.account",
"name": "account",
"context": ".",
"id": 100001,
"nameLower": "account",
"tenant": "lsrc-g2p81"
},
"guid": "e8b96055-30ef-4203-adbc-30e28fa419c9",
"status": "ACTIVE",
"displayText": "account",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614902469162,
"qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.ACCOUNT",
"name": "ACCOUNT",
"context": ".",
"id": 100003,
"nameLower": "account",
"tenant": "lsrc-g2p81"
},
"guid": "152e502a-98ac-484b-86ef-b789d9eb8bf7",
"status": "ACTIVE",
"displayText": "ACCOUNT",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1627756849389,
"qualifiedName": "lsrc-g2p81:.:100004:Example.Employee.Name",
"name": "Name",
"context": ".",
"id": 100004,
"nameLower": "name",
"tenant": "lsrc-g2p81"
},
"guid": "93673886-cea1-4f8d-ad2b-3e4574e70ee1",
"status": "ACTIVE",
"displayText": "Name",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1627756849389,
"qualifiedName": "lsrc-g2p81:.:100004:Example.Employee.Age",
"name": "Age",
"context": ".",
"id": 100004,
"nameLower": "age",
"tenant": "lsrc-g2p81"
},
"guid": "5a94a4dd-e930-46d2-b548-82f91f55f38a",
"status": "ACTIVE",
"displayText": "Age",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1632516635687,
"qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.balance",
"name": "balance",
"context": ".",
"id": 100007,
"nameLower": "balance",
"tenant": "lsrc-g2p81"
},
"guid": "b5652a8c-4509-4477-a45f-8dc7a504707c",
"status": "ACTIVE",
"displayText": "balance",
"classificationNames": [
"credit_card",
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1632516635687,
"qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.last_name",
"name": "last_name",
"context": ".",
"id": 100007,
"nameLower": "last_name",
"tenant": "lsrc-g2p81"
},
"guid": "175a6542-5af9-43f1-a9c6-a83491db94d9",
"status": "ACTIVE",
"displayText": "last_name",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1632516635687,
"qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.credit_card",
"name": "credit_card",
"context": ".",
"id": 100007,
"nameLower": "credit_card",
"tenant": "lsrc-g2p81"
},
"guid": "541fae87-7fe5-4cab-b509-0f76cbf47515",
"status": "ACTIVE",
"displayText": "credit_card",
"classificationNames": [
"credit_card",
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1632516635687,
"qualifiedName": "lsrc-g2p81:.:100007:com.mycorp.mynamespace.sampleRecord.first_name",
"name": "first_name",
"context": ".",
"id": 100007,
"nameLower": "first_name",
"tenant": "lsrc-g2p81"
},
"guid": "ddc35b11-056e-4a41-8a5b-69d2366e6455",
"status": "ACTIVE",
"displayText": "first_name",
"classificationNames": [
"PII"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
Search schema record by name¶
Search for a schema record in namespace ksql
with record name StockTrade
.
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_record&query=ksql.StockTrade' | jq .
Your output should resemble:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_record"
],
"entities": [
{
"typeName": "sr_record",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade",
"name": "StockTrade",
"context": ".",
"id": 100001,
"nameLower": "stocktrade",
"tenant": "lsrc-g2p81"
},
"guid": "d757148e-28f6-465b-b7de-088732848ab5",
"status": "ACTIVE",
"displayText": "StockTrade",
"classificationNames": [
"my_stocks",
"Sensitive"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
Search schema by tag¶
Search for schema records tagged with my_stocks
.
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_record&tag=my_stocks' | jq .
Your output should resemble:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_record"
],
"entities": [
{
"typeName": "sr_record",
"attributes": {
"createTime": 1614902469162,
"qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema",
"name": "KsqlDataSourceSchema",
"context": ".",
"id": 100003,
"nameLower": "ksqldatasourceschema",
"tenant": "lsrc-g2p81"
},
"guid": "dee2acf2-08e5-43af-a2de-927a06f17c33",
"status": "ACTIVE",
"displayText": "KsqlDataSourceSchema",
"classificationNames": [
"my_stocks"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_record",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade",
"name": "StockTrade",
"context": ".",
"id": 100001,
"nameLower": "stocktrade",
"tenant": "lsrc-g2p81"
},
"guid": "d757148e-28f6-465b-b7de-088732848ab5",
"status": "ACTIVE",
"displayText": "StockTrade",
"classificationNames": [
"my_stocks",
"Sensitive"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
Tag a field in Avro¶
Suppose you want to update the stocks_buy-value
schema by tagging a field in one of its records. Specifically,
the stocks_buy-value
schema contains the record KsqlDataSourceSchema
, and you want to tag the SYMBOL field
with the stocks_buy
tag.
First, you must get the fully qualified name of the field you want to tag because you
will need this to specify the "entityName"
. You cannot get this from the Cloud Console.
To get the fully qualified name of a field in a schema, send an API call that returns the details of that field.
For example, search for fields named SYMBOL
:
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/search/basic?type=sr_field&query=SYMBOL' | jq .
Your output will resemble the following. The fully qualified name for the SYMBOL field in the
stocks_buy-value
schema KsqlDataSourceSchema
record is highlighted:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_field"
],
"entities": [
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:100001:ksql.StockTrade.symbol",
"name": "symbol",
"context": ".",
"id": 100001,
"nameLower": "symbol",
"tenant": "lsrc-g2p81"
},
"guid": "446a0d3f-b77e-4ce5-a0d2-628076cd4a13",
"status": "ACTIVE",
"displayText": "symbol",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614902466440,
"qualifiedName": "lsrc-g2p81:.:100002:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
"name": "SYMBOL",
"context": ".",
"id": 100002,
"nameLower": "symbol",
"tenant": "lsrc-g2p81"
},
"guid": "8cfacbc2-8ae8-44c0-b860-e3161bdb012a",
"status": "ACTIVE",
"displayText": "SYMBOL",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_field",
"attributes": {
"createTime": 1614902469162,
"qualifiedName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
"name": "SYMBOL",
"context": ".",
"id": 100003,
"nameLower": "symbol",
"tenant": "lsrc-g2p81"
},
"guid": "16854b0c-580b-408a-9cb6-49e5505900cc",
"status": "ACTIVE",
"displayText": "SYMBOL",
"classificationNames": [
"stocks_buy"
],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
Now tag the SYMBOL field with the stocks_buy
tag.
curl --silent -u <API-KEY>:<API-SECRET> \
--request POST \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/entity/tags \
--header 'Content-Type: application/json' \
--data '[ { "entityType" : "sr_field",
"entityName" : "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL",
"typeName" : "stocks_buy"} ]' | jq .
Your output should resemble:
"typeName": "stocks_buy",
"entityStatus": "ACTIVE",
"entityType": "sr_field",
"entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"
To verify on the Cloud Console, log on to https://confluent.cloud,
navigate to the schema and field you just tagged, and you should see the stocks_buy
tag associated with the SYMBOL field.

Add another tag, my_stocks
, to the same field either through the Cloud Console or through the API.
If you want to use the API call, submit the curl command the same as above and just substitute my_stocks
for stocks_buy
.
Get the tag attributes from a field¶
Get the tag attributes for the same field (SYMBOL in the stocks_buy-value
schema
in the record KsqlDataSourceSchema
).
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url '<SCHEMA-REGISTRY-URL>/catalog/v1/entity/type/sr_field/name/100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL/tags' | jq .
Your output should resemble:
"typeName": "my_stocks",
"entityGuid": "cf62db9b-caec-40dd-928d-6dd98f803b4e",
"entityStatus": "ACTIVE",
"propagate": true,
"removePropagationsOnEntityDelete": false,
"entityType": "sr_field",
"entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"
},
{
"typeName": "stocks_buy",
"entityGuid": "cf62db9b-caec-40dd-928d-6dd98f803b4e",
"entityStatus": "ACTIVE",
"propagate": true,
"removePropagationsOnEntityDelete": false,
"entityType": "sr_field",
"entityName": "lsrc-g2p81:.:100003:io.confluent.ksql.avro_schemas.KsqlDataSourceSchema.SYMBOL"
Tag a schema version¶
First, create a new custom tag called favorites
:
curl --silent -u <API-KEY>:<API-SECRET> \
--header 'Content-Type: application/json' \
--data '[ { "entityTypes" : [ "sr_schema", "sr_record", "sr_field", "sr_schema" ],"name" : "favorites","description" : "my favorite stocks"} ]' \
--url <SCHEMA-REGISTRY-URL>/catalog/v1/types/tagdefs' | jq .
Your output should resemble:
"category": "CLASSIFICATION",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1632974329626,
"updateTime": 1632974329626,
"version": 1,
"name": "favorites",
"description": "my favorite stocks",
"typeVersion": "1.0",
"attributeDefs": [],
"superTypes": [],
"entityTypes": [
"sr_field",
"sr_schema",
"sr_record",
"sr_schema"
Tag the current schema version of the stocks_buy-value
schema you’ve been working on as favorites
.
The following examples assumes the current version is v2. If your stocks_buy-value
version is v1, replace v2
with v1
in the API call.
curl --silent -u <API-KEY>:<API-SECRET> \
--request POST \
--url <SCHEMA-REGISTRY-URL>/catalog/v2/entity/tags \
--header 'Content-Type: application/json' \
--data '[ { "entityType" : "sr_schema", "entityName" : "lsrc-g2p81:.:100003", "typeName" : "favorites" } ]' | jq .
Your output will resemble:
"typeName": "favorites",
"entityStatus": "ACTIVE",
"entityType": "sr_schema",
"entityName": "lsrc-g2p81:.:100003"
Navigate to the stocks_buy-value
schema on the Cloud Console to verify that favorites
is associated with the version you tagged.

Get a schema given a subject version¶
List all version 1 value schemas with the subject name prefix of stocks
.
curl --silent -u <API-KEY>:<API-SECRET> \
--request GET \
--url 'https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=sr_schema&attrName=name&attrValuePrefix=stocks' | jq .
Your output should resemble:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_schema"
],
"entities": [
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614710010055,
"qualifiedName": "lsrc-g2p81:.:stocks-value:1",
"name": "stocks-value",
"context": ".",
"id": 100001,
"nameLower": "stocks-value",
"tenant": "lsrc-g2p81"
},
"guid": "ba600e7a-0093-41b3-8d68-38ba730323bc",
"status": "ACTIVE",
"displayText": "stocks-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902357913,
"qualifiedName": "lsrc-g2p81:.:stocks_under_100-value:1",
"name": "stocks_under_100-value",
"context": ".",
"id": 100002,
"nameLower": "stocks_under_100-value",
"tenant": "lsrc-g2p81"
},
"guid": "7e2e652b-3b53-4461-b634-c8a58186b28d",
"status": "ACTIVE",
"displayText": "stocks_under_100-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902369755,
"qualifiedName": "lsrc-g2p81:.:stocks_under_100-value:2",
"name": "stocks_under_100-value",
"context": ".",
"id": 100003,
"nameLower": "stocks_under_100-value",
"tenant": "lsrc-g2p81"
},
"guid": "799fde4c-a43b-4820-896c-e1f5fa2ba399",
"status": "ACTIVE",
"displayText": "stocks_under_100-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902422050,
"qualifiedName": "lsrc-g2p81:.:stocks_buy-value:2",
"name": "stocks_buy-value",
"context": ".",
"id": 100003,
"nameLower": "stocks_buy-value",
"tenant": "lsrc-g2p81"
},
"guid": "59f3ced1-a303-4712-ae7a-f720b9261084",
"status": "ACTIVE",
"displayText": "stocks_buy-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902418968,
"qualifiedName": "lsrc-g2p81:.:stocks_buy-value:1",
"name": "stocks_buy-value",
"context": ".",
"id": 100002,
"nameLower": "stocks_buy-value",
"tenant": "lsrc-g2p81"
},
"guid": "785f7f6f-5b7b-4695-bea7-d9b0a6df1947",
"status": "ACTIVE",
"displayText": "stocks_buy-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902469162,
"qualifiedName": "lsrc-g2p81:.:stocks_sell-value:2",
"name": "stocks_sell-value",
"context": ".",
"id": 100003,
"nameLower": "stocks_sell-value",
"tenant": "lsrc-g2p81"
},
"guid": "4abf45bd-46e9-4ec5-a959-e95a6010b166",
"status": "ACTIVE",
"displayText": "stocks_sell-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
},
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1614902466440,
"qualifiedName": "lsrc-g2p81:.:stocks_sell-value:1",
"name": "stocks_sell-value",
"context": ".",
"id": 100002,
"nameLower": "stocks_sell-value",
"tenant": "lsrc-g2p81"
},
"guid": "bfea548e-a99f-48f1-b310-e99f25e908c4",
"status": "ACTIVE",
"displayText": "stocks_sell-value",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
}
Business Metadata API Examples¶
Stream catalog allows you to define a number of tag definitions and apply the tags to entities. Support for business metadata enriches the management and search of entities. Business metadata gives you a way to attach key-value pairs to specific entities. Start by creating a business metadata definition, which is a “group” of attributes. Business metadata definitions can be queried, updated, or deleted. Then you can apply the attributes (or a subset of attributes, if some are marked optional) within the business metadata definition to entities. You can further query all the key-value pairs attached to specific entities, update them, or delete them.
Business metadata concepts are fully covered in the Confluent Cloud Console user guide under Business Metadata for Schemas.
Following are usage examples for working with business metadata through the Confluent Cloud CATALOG API (V1).
The examples include creating a hamburgers
schema and a business metadata
definition called team
with several attributes, attaching specific team
metadata to hamburgers
, and then updating the values for the attributes. Try
tracking the changes you make on the Confluent Cloud Console to validate the API
output, as suggested in the examples.
Tip
Be sure to review the Setup and Suggestions notes for these API usage examples before you begin.
Entity types¶
Business metadata can be applied to the same set of entities as can tags. This currently includes customized definitions of schema registry elements:
kafka_topic
sr_schema
sr_schema
sr_record
sr_type
sr_field
sr_primitive
sr_fixed
sr_enum
sr_array
sr_map
sr_combined
All these types except kafka-topic
are defined under the super class of sr_entity
, which can be
used to indicate “any type of SR entity”. In future versions, the set of
entities may be extended to include topics, clusters, connectors, and so on.
Tip
Entities are identified in different ways. For example:
- A schema subject (
sr_subject_version
) is identified by its name and version (for version one of ahamburgers
schema, this would behamburgers:1
). Business metadata attached to a subject version does not show up on the Confluent Cloud Console rendering of the schema. - A schema (
sr_schema
, not any specific version) is identified by its schema ID. This type of entity is used for the examples below.
Create a schema¶
To start with, create a new schema. In later steps, you will create business metadata and apply it to this new schema.
Here is an example of a schema file called hamburgers.json, properly formatted for the API call to the catalog.
{
"schema":
"{
\"type\": \"record\",
\"name\": \"Hamburger\",
\"fields\":
[
{
\"type\": \"string\",
\"name\": \"Sandwich\"
},
{
\"type\": \"string\",
\"name\": \"Size\"
}
]
}"
}
The API call to create the schema is as follows:
curl -u <API-KEY>:<API-SECRET> -X POST -H "Content-Type: application/json" \
--data @/path/to/your/schema <SCHEMA-REGISTRY-URL>/subjects/<subject-name>/versions
For the example, the API call to create the “hamburgers” schema would look similar to this:
curl --silent -u <API-KEY>:<API-SECRET> -X POST -H "Content-Type: application/json" \
--data @/Users/vicky/hamburgers.json <SCHEMA-REGISTRY-URL>/subjects/Hamburgers/versions | jq .
After you’ve created the schema, verify that it is available on the Confluent Cloud Console
(https://confluent.cloud/). Navigate to your environment,
and click Schemas on the right side panel to see the hamburgers
schema in the list.
Create your first business metadata definition¶
Here is an example business metadata definition called team
, which with two fields, teamName
and slackChannel
.
You can provide this definition in a file called team.txt, and pass the file in the API call.
[ {
"name" : "team",
"description" : "Team information as business metadata",
"attributeDefs" : [
{ "name" : "teamName", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string", "options": {"maxStrLength": 25, "applicableEntityTypes": "[\"sr_schema\"]"} },
{ "name" : "slackChannel", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string", "options": {"maxStrLength": 30, "applicableEntityTypes": "[\"sr_schema\"]"} }
]
} ]
Here is an example of a POST request to add the above definition to the catalog for a Schema Registry. Sending this POST request is analogous to adding a new definition through the Confluent Cloud Console.
curl --silent -u <API-KEY>:<API-SECRET> -X POST -H "Content-Type: application/json" \
--data @/Users/vicky/team.txt https://<SCHEMA-REGISTRY-URL>/catalog/v1/types/businessmetadatadefs | jq .
- The
--data
flag provides the path to the file. In this example the data path isUsers/vicky/team.txt
. You must include the@
sign before the path with no spaces in between. - The only required field (from the request perspective) is
name
. Ifdescription
not specified, it defaults to the same asname
. - The
attributeDefs
field is optional. It can be updated later, but you can then only add optional attribute definitions (isOptional
field set totrue
). - For each attribute definition in
attributeDefs
, the required fields (from the request perspective) includename
andtypeName
.name
is the unique identifier of an attribute within one business metadata definition (each attribute is accessed in the form of<business metadata def>.<attribute name>
).typeName
is the type of the attribute, including builtin types such aslong
,float
,boolean
,string
, and more.
- Business metadata requires that each attribute have
applicableEntityTypes
as a key in theoptions
field.applicableEntityTypes
is a stringified JSON that is a list of all applicable entities of this attribute. - You can specify
cf_entity
as the applicable entity type, to specify that the business metadata definition could apply to any type of entities you define. - If
typeName
of an attribute isstring
, options must havemaxStrLength
specified. For other builtin types, there is no such constraint.
If you used the curl --silent
flag and piped the API call through jq
(like the example),
you will get human-readable output as shown below.
{
"category": "BUSINESS_METADATA",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1656010917403,
"updateTime": 1656010917403,
"version": 1,
"name": "team",
"description": "Team information as business metadata",
"typeVersion": "1.0",
"attributeDefs": [
{
"name": "teamName",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1,
"options": {
"applicableEntityTypes": "[\"sr_schema\"]",
"maxStrLength": "25"
}
},
{
"name": "slackChannel",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1,
"options": {
"applicableEntityTypes": "[\"sr_schema\"]",
"maxStrLength": "30"
}
After creating the new business metadata definition, verify that it is available
on the Confluent Cloud Console. Navigate to View environments <your-environment> > Schema Registry tab > View & manage business metadata > team
to see the new team
definition.
Get all the business metadata definitions created so far¶
The following call retrieves all business metadata definitions on the target
registry catalog. Adding the curl --silent
flag and piping through jq
provides human-readable output.
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
https://<SCHEMA-REGISTRY-URL>/catalog/v1/types/businessmetadatadefs | jq .
Get a specific business metadata definition by its name¶
This call gets the specified definition by name; in this case, team
.
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
https://<SCHEMA-REGISTRY-URL>/catalog/v1/types/businessmetadatadefs/team | jq .
Update business metadata definitions¶
To update a definition, use the PUT
, and pass in the updated file.
curl -u <API-KEY>:<API-SECRET> -X PUT -H "Content-Type: application/json" \
--data --data @/Users/vicky/team.txt https://<SCHEMA-REGISTRY-URL>/catalog/v1/types/businessmetadatadefs
- The contents of team.txt follows the same format as that used during creation. You pass a list of business metadata definitions to PUT.
- As mentioned above, you can only add optional attribute definitions with PUT.
- Attributes cannot be deleted once they are added to business metadata definitions.
- Some fields for attributes (such as
description
) can be updated, while others (such astypeName
) cannot be updated.
Here is an example of the team.txt file, updated to add an email
field.
[ {
"name" : "team",
"description" : "Team information as business metadata",
"attributeDefs" : [
{ "name" : "teamName", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string", "options": {"maxStrLength": 25, "applicableEntityTypes": "[\"sr_schema\"]"} },
{ "name" : "slackChannel", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string", "options": {"maxStrLength": 30, "applicableEntityTypes": "[\"sr_schema\"]"} },
{ "name" : "email", "isOptional" : "true", "cardinality" : "SINGLE", "typeName" : "string", "options": {"maxStrLength": 30, "applicableEntityTypes": "[\"sr_schema\"]"} }
]
} ]
Add business metadata to an entity¶
To add business metadata to an entity, specify:
- the target entity type (such as a schema,
sr_schema
) - name of the specific entity (hamburgers schema, specified by schema ID)
- metadata definition name
- values for each of the business metadata attributes.
This example shows how to add team
metadata to the hamburgers
schema,
all of which you created in previous steps. You will provide values for the metadata fields
(teamName
, slackChannel
, and email
) as shown in the example below.
You will need to identify the schema by its ID. To get the schema ID, you can either look at the
schema on the Confluent Cloud Console or run this API call against the hamburgers
schema to retrieve it:
curl --silent -u <API-KEY>:<API-SECRET> GET https://<SCHEMA-REGISTRY-URL>/subjects/hamburgers/versions/1 | jq .
Your output should resemble:
"subject": "hamburgers",
"version": 1,
"id": 100012,
"schema": "{\"type\":\"record\",\"name\":\"Hamburger\",\"fields\":[{\"name\":\"Sandwich\",\"type\":\"string\"},{\"name\":\"Size\",\"type\":\"string\"}]}"
Following is an example file called team-data.txt you can pass into the POST request.
[ {
"entityType": "sr_schema",
"entityName": "100012",
"typeName": "team",
"attributes": {"teamName": "Design", "slackChannel": "Design-all", "email": "design@coolideas.org"}
} ]
The file contains a list of business metadata instances, specifies that the
target entity is a schema (sr_schema
), provides the schema ID for hamburgers
,
and will attach the team
metadata to that schema, filling in the fields as shown
to specify the Design
team details.
- All fields (
entityType
,entityName
,typeName
, andattributes
are mandatory.) typeName
is the business metadata definition name.- You must pass all attributes that are not marked as optional in the
attributes
field.
Use the following API call to apply the metadata to the schema.
Tip
Remember that the particular instance of the entity (in this case the
schema ID of the hamburgers
schema) is specified in the file you are
passing to the API call (team-data.txt
). This is easy to overlook.
curl --silent -u <API-KEY>:<API-SECRET> -X POST -H "Content-Type: application/json" \
--data @/Users/vicky/team-data.txt
https://<SCHEMA-REGISTRY-URL>/catalog/v1/entity/businessmetadata | jq .
Your output should resemble:
"typeName": "team",
"attributes": {
"teamName": "Design",
"slackChannel": "Design-all",
"email": "design@coolideas.org"
},
"entityType": "sr_schema",
"entityName": "lsrc-g2p81:.:100012"
After applying the data to the schema, verify it on the Confluent Cloud Console.
Navigate to the hamburgers
schema (select the environment, click Schemas on the right side panel,
click into the hamburgers schema) to see the metadata attached to the schema with your specified values:
- teamName: Design
- slackChannel: Design-all
- email: design@coolideas.org
Get business metadata associated with an instance of an entity¶
To retrieve the business metadata associated with a specific instance of an entity:
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
https://<SCHEMA-REGISTRY-URL>/catalog/v1/entity/type/<entityType>/<value-identifying-the-instance-of-the-entity>/businessmetadata | jq .
For example, the following API call which specifies the schema entity type and schema ID for the hamburgers
schema
will return the same attribute details as the previous call, where you applied that metadata.
curl --silent -u <API-KEY>:<API-SECRET> -X GET https://<SCHEMA-REGISTRY-URL>/catalog/v1/entity/type/sr_schema/name/100012/businessmetadata | jq .
"typeName": "team",
"attributes": {
"teamName": "Design",
"slackChannel": "Design-all",
"email": "design@coolideas.org"
},
"entityType": "sr_schema",
"entityName": "lsrc-g2p81:.:100012"
Update business metadata associated with entity¶
Use a PUT request with new data to update business metadata associated with an entity.
For example, use this modified version of the team-data.txt to update values for teamName
and email
:
[ {
"entityType": "sr_schema",
"entityName": "100012",
"typeName": "team",
"attributes": {"teamName": "DesignSuperStars", "slackChannel": "Design-all", "email": "design@coolestideas.org"}
} ]
Then pass the data file as a PUT request.
curl -- silent -u <API-KEY>:<API-SECRET> -X PUT -H "Content-Type: application/json" \
--data @/Users/vicky/team-data.txt https://<SCHEMA-REGISTRY-URL>/catalog/v1/entity/businessmetadata | jq .
Your output should resemble:
"typeName": "team",
"attributes": {
"teamName": "DesignSuperStars",
"slackChannel": "Design-all",
"email": "design@coolestideas.org"
},
"entityType": "sr_schema",
"entityName": "lsrc-g2p81:.:100012",
"error": {
"error_code": 4000151,
"message": "Business-metadata attribute already exists in entity: lsrc-g2p81:.:100012"
Tip
Updating only a subset of fields will not delete other fields.
Note that this is analogous to editing these values on the Confluent Cloud Console.
The path on the UI would be: View environments > <your-environment> > Schemas > hamburgers, then click the context-sensitive edit button for the team metadata on the right panel.
The dialog to edit those same values from the UI looks like this.
Search for business metadata associated with entity¶
To search for business metadata on a particular entity, use the following API call to specify an entity type and value in a field.
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
"https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=<entityType>&attrName=<business-metadata-definition>.<field-name>&attrValuePrefix=<value-for-a-field>" | jq .
- Currently, only prefix search on a string attributes is supported. (This may be extended to multiple data types with more search semantics in the future).
- Attributes in a business metadata definition are specified as
<business-metadata-definition>.<attribute-name>
; the attribute name is the field name.
For example, you might want to search for the team name, Slack channel, or email address associated with a team.
Here are examples of all of those searches on the updated team
metadata associated with the hamburgers
schema:
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
"https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=sr_schema&attrName=team.teamName&attrValuePrefix=DesignSuperStars" | jq .
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
"https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=sr_schema&attrName=team.slackChannel&attrValuePrefix=Design-all" | jq .
curl --silent -u <API-KEY>:<API-SECRET> -X GET \
"https://<SCHEMA-REGISTRY-URL>/catalog/v1/search/attribute?type=sr_schema&attrName=team.email&attrValuePrefix=design@coolestideas.org" | jq .
Here is an example of the output for the last search, on the team.email
associated with hamburgers
:
"searchParameters": {
"includeDeleted": false,
"limit": 0,
"offset": 0
},
"types": [
"sr_schema"
],
"entities": [
{
"typeName": "sr_schema",
"attributes": {
"createTime": 1655158742047,
"qualifiedName": "lsrc-g2p81:.:100012",
"name": "Hamburger",
"context": ".",
"nameLower": "hamburger",
"id": 100012,
"tenant": "lsrc-g2p81",
"lsrc_g2p81__team.email": "design@coolestideas.org"
},
"guid": "a36c35c7-86b1-4f7d-a6f7-e3b86efe7928",
"status": "ACTIVE",
"displayText": "Hamburger",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": []
}
Remove business metadata associated with an entity¶
For now, you can only delete all attributes at once from the business metadata definition associated with an entity.
For example, the following API call removes the team
metadata definition and associated attributes from the hamburgers
schema.
curl --silent -u <API-KEY>:<API-SECRET> -X DELETE https://<SCHEMA-REGISTRY-URL>/catalog/v1/entity/type/sr_schema/name/100012/businessmetadata/team | jq .
Tip
This does not delete the business metadata definition from the catalog; it only removes the particular definition from the entity (schema, in this case).
Delete business metadata definitions¶
To entirely delete a business metadata definition (for example, team
) from the Schema Registry catalog:
curl --silent -u <API-KEY>:<API-SECRET> -X DELETE https://<SCHEMA-REGISTRY-URL>/catalog/v1/types/businessmetadatadefs/team | jq .
Suggested Reading¶
- CATALOG API (V1)
- Stream Catalog GraphQL API
- Tagging Data and Schemas (concepts and steps for working tags on the Confluent Cloud Console)
- Business Metadata for Schemas (concepts and steps for working with business metadata on the Confluent Cloud Console)