.. _topicschema: ====================== Manage Schemas in |cp| ====================== Use the |sr| feature in |c3-short| to manage |cp| topic schemas. .. include:: ../../includes/topics-schemas/schemas-intro.rst The |sr| feature in |c3-short| is :ref:`enabled` by default. Disabling the feature disables both viewing and editing of schemas. .. _c3_schema_create: Create a topic schema in |c3-short| ----------------------------------- .. include:: ../../includes/topics-schemas/create-schema-intro.rst ----------------------------- Create a topic value schema ----------------------------- #. Select a cluster. #. Click **Topics** on the menu. The :ref:`c3_all_topics` appears. #. Select a topic. .. figure:: ../../images/c3-schemas-topics-list.png :scale: 80% :alt: Topics list The topic overview page is displayed. .. figure:: ../../images/c3-schemas-topic-overview.png :scale: 55% :alt: Topics overview #. Click the **Schema** tab. You are prompted to set a message value schema. .. figure:: ../../images/c3-set-msg-value-schema.png :scale: 65% :alt: Prompt to click and define a schema for the selected topic #. Click **Set a schema**. The Schema editor appears pre-populated with the basic structure of an `Avro schema `__ to use as a starting point, if desired. .. figure:: ../../images/c3-schema-value-editor.png :scale: 65% :alt: Schema editor where you can define a schema #. Select a schema format type: - Avro - JSON - Protobuf .. figure:: ../../images/c3-schema-select-format.png :scale: 70% Choose **Avro** if you want to try out the code examples provided in the next steps. .. tip:: To learn more about each of the schema types, see :ref:`sr-serializer`. #. Enter the schema in the editor: .. include:: ../../includes/topics-schemas/schema-basic-structure.rst Copy and paste the following example schema. .. code-block:: JSON { "type": "record", "namespace": "my.examples", "name": "Payment", "fields": [ { "name": "id", "type": "string" }, { "name": "amount", "type": "double" } ] } .. figure:: ../../images/c3-entered-schema.png :scale: 65% :alt: Schema for a topic entered into Control Center editor #. In edit mode, you have options to: - Validate the schema for syntax and structure before you create it. - :ref:`Add schema references ` with a guided wizard. #. Click **Create**. - If the entered schema is valid, the **Schema updated** message is briefly displayed in the banner area. - If the entered schema is not valid, an **Input schema is an invalid Avro schema** error is displayed in the banner area. If applicable, repeat the procedure as appropriate for the topic key schema. .. _cp-sr-schema-references: ------------------------------ Working with schema references ------------------------------ You can add a reference to another schema, using the wizard to help locate available schemas and versions. .. figure:: ../../images/c3-schema-reference-add.png :scale: 65% :alt: Add a schema reference The **Reference name** you provide must match the target schema, based on guidelines for the schema format you are using: - In JSON Schema, the name is the value on the ``$ref`` field of the referenced schema - In Avro, the name is the value on the ``type`` field of the referenced schema - In Protobuf, the name is the value on the ``Import`` statement referenced schema First, locate the schema you want to reference, and get the reference name for it. **Add a schema reference to the current schema in the editor** #. Click **Add reference**. #. Provide a Reference name per the rules described above. #. Select the schema fro the Subject list. #. Select the Version of the schema you want to use. #. Click **Validate** to check if the reference will pass. #. Click **Save** to save the reference. For example, to reference the schema for the ``my-transactions`` topic (``my-transactions-value``) from the ``widget`` schema, you can configure a reference to type, ``record`` as shown. .. figure:: ../../images/c3-schema-reference-in-widget.png :scale: 65% :alt: Add a schema reference To learn more, see :ref:`referenced-schemas` in the schema formats developer documentation. **View, edit, or delete schema references for a topic** Existing schema references show up on editable versions of the schema where they are configured. #. Navigate to a topic; for example, the ``widget-value`` schema associated with the ``widget`` topic in the previous example. #. Click into the editor as if to edit the schema. If there are references to other Schemas configured in this schema, they will display in the **Schema references** list below the editor. You can also add more references to this schema, modify existing, or delete references from this view. ------------------------- Create a topic key schema ------------------------- #. Click the **Key** option. You are prompted to set a message key schema. .. figure:: ../../images/c3-set-msg-key-schema.png :scale: 65% :alt: Prompt to define a schema for a message key #. Click **Set a schema**. #. Choose **Avro** format and/or delete the sample formatting and simply paste in a string UUID. #. Enter the schema into the editor and click **Save**. Copy and paste the following example schema, and save it. .. code-block:: JSON { "namespace": "io.confluent.examples.clients.basicavro", "name": "key_my_transactions", "type": "string" } .. figure:: ../../images/c3-key-schema.png :scale: 65% :alt: Example of a schema for a message key entered in the schema editor #. In edit mode, you have options to: - Validate the schema for syntax and structure before you create it. - :ref:`Add schema references ` with a guided wizard. .. _c3-schemas-best-practices-key-value-pairs: Best Practices and Pitfalls for Key Values ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |ak| messages are key-value pairs. Message keys and message values can be serialized independently. For example, the value may be using an `Avro `_ ``record``, while the key may be a primitive (``string``, ``integer``, and so forth). Typically message keys, if used, are primitives. How you set the key is up to you and the requirements of your implementation. As a best practice, keep key value schema complexity to a minimum. Use either a simple, non-serialized data type such as a string UUID or long ID, or an Avro record that does not use maps or arrays as fields, as shown in the example below. Do not use Protobuf messages and JSON objects for key values. Avro does not guarantee deterministic serialization for maps or arrays, and Protobuf and JSON schema formats do not guarantee deterministic serialization for any object. Using these formats for key values will break topic partitioning. If you do decide to use a complex format for a key value schema, set ``auto.register.schemas=false`` to prevent registration of new valid and compatible schemas that, because of the complex key value format, will break your partitioning. To learn more, see :ref:`auto-schema-registration` in the On-Premises |sr| Tutorial, and `Partitioning gotchas `__ in the Confluent Community Forum. For detailed examples of key and value schemas, see the discussion under `Formats, Serializers, and Deserializers `__ in the |sr| documentation. .. _c3_schema_view: Viewing a schema in |c3-short| ------------------------------ .. include:: ../../includes/topics-schemas/view-schema.rst #. Select a cluster from the navigation bar. #. Click the **Topics** menu. The :ref:`c3_all_topics` appears. #. Select a topic. .. figure:: ../../images/c3-schemas-topics-list.png :scale: 80% :alt: Topics list The topic overview page appears. #. Click the **Schema** tab. The Value schema is displayed by default. .. figure:: ../../images/c3-view-value-schema.png :scale: 65% :alt: Currently defined value schema for the selected topic .. tip:: The |c3-short| schema display may re-order the placement of meta information about the schema. Comparing this view with the example provided in the previous section, note that schema name, namespace, and record type are shown below the field definitions. This is an artifact of the display in |c3-short|; the schema definition is the same. #. Click the **Key** tab to view the key schema if present. .. _c3_schema_edit: Editing a schema in |c3-short| ------------------------------ .. include:: ../../includes/topics-schemas/edit-schema.rst #. Select a cluster from the navigation bar. #. Click the **Topics** menu. The :ref:`c3_all_topics` appears. #. Select a topic. #. Click the **Schema** tab. #. Select the **Value** or **Key** tab for the schema. #. Click anywhere in the schema to enable edit mode and make changes in the schema editor. For example, if you are following along with the example: - Select the topic ``my-transactions``, click **Schema**, and select the **Value** tab. - Edit the schema by copy-pasting the following definition for a new ``region`` field, after the ``id`` and ``amount`` fields. Precede your new definition with a comma, per the syntax. .. code-block:: JSON { "name": "region", "type": "string", "default": "" } Note that the new ``region`` field includes a default value, which makes it backward compatible. By plugging in the default value, consumers can use the new schema to read data submitted by producers that use the older schema (without the ``region`` field). #. Click **Save**. - If the schema update is valid and compatible with its prior versions (assuming a backward-compatible mode), the schema is updated and the version count is incremented. You can :ref:`compare the different versions ` of a schema. .. figure:: ../../images/c3-schema-version-updated.png :scale: 65% - If the schema update is invalid or incompatible with an earlier schema version, an error is displayed. The example below shows the addition of another new field, ``country``, with no default provided for backward compatibility. .. figure:: ../../images/c3-schema-incompatible.png :scale: 65% .. tip:: You can also add schema references as a part of editing a schema, as described in :ref:`cp-sr-schema-references`. .. _c3_schema_compare_versions: Comparing schema versions in |c3-short| --------------------------------------- .. include:: ../../includes/topics-schemas/compare-versions.rst #. Select a cluster from the navigation bar. #. Click the **Topics** menu. The :ref:`c3_all_topics` appears. #. Select a topic. #. Click the **Schema** tab. #. Select the **Key** or **Value** tab for the schema. #. Select **Version history** from the inline menu. .. figure:: ../../images/c3-schema-version-history-01.png :scale: 65% The current version number of the schema is indicated on the version menu. .. figure:: ../../images/c3-schema-version-history-02.png :scale: 65% #. Select the **Turn on version diff** check box. #. Select the versions to compare from each version menu. The differences are highlighted for comparison. .. figure:: ../../images/c3-schema-compare.png :scale: 65% .. _c3_schema_compat_mode: Changing the compatibility mode of a schema in |c3-short| --------------------------------------------------------- .. include:: ../../includes/topics-schemas/compat-mode.rst :start-after: compat-intro-start :end-before: compat-intro-end #. Select a cluster from the navigation bar. #. Click the **Topics** menu. The :ref:`c3_all_topics` appears. #. Select a topic. #. Click the **Schema** tab. #. Select the **Key** or **Value** tab for the schema. #. Select **Compatibility setting** from the inline menu. |compat-menu| The Compatibility settings are displayed. .. |compat-menu| image:: ../../images/c3-schema-compat-mode-menu.png .. figure:: ../../images/c3-schema-compat-mode.png :scale: 60% #. Select a mode option: .. include:: ../../includes/topics-schemas/compat-mode.rst :start-after: mode-details-start :end-before: mode-details-end #. Click **Save**. .. _c3_schema_download: Downloading a schema from |c3-short| ------------------------------------ #. Select a cluster from the navigation bar. #. Click the **Topics** menu. The :ref:`c3_all_topics` appears. #. Select a topic. #. Click the **Schema** tab. #. Select the **Key** or **Value** tab for the schema. #. Click **Download**. A schema JSON file for the topic is downloaded into your Downloads directory. .. include:: ../../includes/topics-schemas/download-schema-example.rst .. _c3_sr_troubleshoot: Troubleshoot error "|sr| is not set up" ---------------------------------------- If you get an error message on |c3-short| when you try to access a topic schema ("|sr| is not set up"), first make sure that |sr| is running. Then verify that the |sr| ``listeners`` configuration matches the |c3-short| ``confluent.controlcenter.schema.registry.url`` configuration. Also check the HTTPS configuration parameters. .. figure:: ../../images/c3-SR-not-set-up.png :scale: 65% For more information, see :ref:`c3_schema_registry_not_set_up`, and start-up procedures for :ref:`quickstart`, or :ref:`installation`, depending on which one of these you are using to run |cp|. .. _ff_c3_edit_schema: Enabling and disabling |sr| in |c3-short| ----------------------------------------- The feature that allows working with schemas in |c3-short| is enabled by default. The feature can be disabled if an organization does not want any users to access the feature. After disabling the feature, the Topics Schema menu and the Schema tab are no longer visible in the |c3-short| UI. The ability to view and edit schemas is disabled. To disable the edit schema feature in |c3-short|: #. Set the ``confluent.controlcenter.schema.registry.enable`` option in your ``control-center.properties`` file to ``false``. .. sourcecode:: bash confluent.controlcenter.schema.registry.enable=false .. note:: Make the change in the appropriate |c3-short| properties file or files configured for your environments, including ``control-center-dev.properties`` or ``control-center-production.properties``. The properties files are located in ``/path-to-confluent/etc/confluent-control-center/``. #. Restart |c3-short| and pass in the properties file for the configuration to take effect: .. sourcecode:: bash ./bin/control-center-stop ./bin/control-center-start ../etc/confluent-control-center/control-center.properties .. tip:: If you are using a |cp| development environment with a :confluent-cli:`confluent local|command-reference/local/index.html`, stop and start as follows: .. codewithvars:: bash |confluent_local_stop_control_center| |confluent_local_stop_control_center| ../etc/confluent-control-center/control-center-dev.properties To enable the feature again, set the option back to ``true`` and restart |c3-short| with the updated properties file. .. _multi-cluster-sr: Enabling Multi-Cluster |sr| --------------------------- Starting with version 5.4.1, |cp| supports the ability to run multiple schema registries and associate a unique |sr| to each |ak| cluster in multi- cluster environments. The ability to scale up schema registries in conjunction with |ak| clusters is useful for evolving businesses; and particularly supports data governance, organization, and management across departments in large enterprises. When multi-cluster |sr| is configured and running, you can create and manage schemas per topics in |c3-short| as usual. ---------------------------------- Configuration Properties and Files ---------------------------------- Multiple |sr| clusters may be specified with ``confluent.controlcenter.schema.registry.{name}.url`` in the appropriate |c3-short| properties file. To use a |sr| cluster identified in this way, add or verify the following broker and |c3-short| configurations. A new endpoint ``/v1/metadata/schemaRegistryUrls`` has been exposed by |ak| to return the ``confluent.schema.registry.url`` field from the |ak| broker configurations. |c3-short| uses this field to look up the registries from |ak| broker configurations. To use this, you must configure unique listener endpoints for each cluster: - In the broker ``server.properties`` files (unique for each |ak| cluster), specify the REST endpoint with the ``confluent.http.server.listeners`` field, which defaults to ``http://0.0.0.0:8090``. - In the appropriate |c3-short| properties file, use ``confluent.controlcenter.streams.cprest.url`` to define the REST endpoint for ``controlcenter.cluster``. - For additional clusters, define REST endpoints using ``confluent.controlcenter.kafka.{name}.cprest.url``. This should be consistent with the |ak| cluster name used for other |ak| |c3-short| configurations; for example, ``confluent.controlcenter.kafka.{name}.bootstrap.servers``. A minimal viable configuration touches the following files, and includes settings for these properties (example names and ports are given): |c3-short| properties file ^^^^^^^^^^^^^^^^^^^^^^^^^^ The :ref:`c3_properties_files` file includes: - ``confluent.controlcenter.schema.registry.url=http://localhost:8081`` - ``confluent.controlcenter.schema.registry.sr-1.url=http://localhost:8082`` - ``confluent.controlcenter.streams.cprest.url=http://localhost:8090`` - ``confluent.controlcenter.kafka.AK1.cprest.url=http://localhost:8091`` See :ref:`Control Center configuration reference` for a full description of ``confluent.controlcenter.schema.registry.url``. Broker configuration file for the |c3-short| cluster ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The |ak| broker configuration file for ``controlcenter.cluster``, such as ``server0.properties``, includes: - ``confluent.http.server.listeners=http://localhost:8090`` - ``confluent.schema.registry.url=http://localhost:8081`` Broker configuration file for the |ak| cluster ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The |ak| broker configuration file for ``AK1``, ``server1.properties``) includes: - ``confluent.http.server.listeners=http://localhost:8091`` - ``confluent.schema.registry.url=http://localhost:8082`` With these configurations, editing the schema through the |c3-short| UI will connect to ``http://localhost:8081`` for ``controlcenter.cluster`` and ``http://localhost:8082`` for ``AK1``. Defaults and Fallback ^^^^^^^^^^^^^^^^^^^^^ If ``confluent.schema.registry.url`` fields are not specified for any brokers, the ``confluent.controlcenter.schema.registry.url`` |sr| URL is applied. For example, if the |sr| URL was not provided for ``AK1``, ``AK1``'s associated |sr| cluster would also be specified at ``http://localhost:8081``. If ``confluent.controlcenter.schema.registry.url`` is not explicitly specified in the |c3-short| properties file, it defaults to ``http://localhost:8081``. ------- Example ------- Following is a detailed example of a functional multi-cluster |sr| setup with two |ak| clusters connected to |c3-short|, one the ``controlcenter.cluster``, and the other named ``AK1``, each with one broker. Example instructions assume refer to the location of your |cp| installation as ``$CONFLUENT_HOME``. The table below shows files and configurations in each for the two cluster example. These are not complete properties files, but rather excerpts from each file that define multi-cluster |sr| specifics, port numbers, and configurations that are important to the example setup or generally not found in default properties files. To run the example, copy default configuration files to new files per the example filenames below, add/modify properties as shown, and run the components as described in :ref:`run-example-multi-cluster-sr`. .. important:: As of |cp| 7.5, |zk| is deprecated for new deployments. Confluent recommends |kraft| mode for new deployments. To learn more about running |ak| in |kraft| mode, see :ref:`kraft-overview`, the |kraft| steps in the :ref:`Platform Quick Start `, and :ref:`config-cp-components-kraft`. The following example provides both |kraft| (*combined mode*) and |zk| configurations. Another example of running multi-cluster |sr| on either |kraft| or |zk| mode is shown in the :ref:`Schema Linking Quick Start for Confluent Platform `. Key Configurations ^^^^^^^^^^^^^^^^^^ .. tabs:: .. group-tab:: |kraft| mode The example properties files are based on the defaults. You can copy the default properties files to use as a basis for the specialized versions of them shown here. The example assumes the new files are in the same directories as the originals. - |kraft| server file is in ``$CONFLUENT_HOME/etc/kafka/kraft/server.properties`` (|kraft| combined mode). Copy this to create ``server0.properties`` and ``server1.properties``. - |sr| properties file is ``$CONFLUENT_HOME/etc/schema-registry/schema-registry.properties``. Copy this to create ``schema-registry0.properties`` and ``schema-registry1.properties``. - |c3-short| properties file is ``$CONFLUENT_HOME/etc/confluent-control-center/control-center-dev.properties``. Copy this to create ``$CONFLUENT_HOME/etc/confluent-control-center/control-center-multi-sr.properties``. In addition to the configs, shown below for this file, it is recommended to comment out the ``zookeeper.connect`` line, as it doesn't apply in this mode. .. raw:: html
File Properties
server0.properties
  • node.id=1
  • listeners=PLAINTEXT://:9092, CONTROLLER://:9094
  • log.dirs=/tmp/kraft-combined-logs (fresh logging directory)
  • confluent.metrics.reporter.bootstrap.servers=localhost:9092
The following configurations are specific to multi-cluster Schema Registry setup for this broker:
  • confluent.http.server.listeners=http://0.0.0.0:8090
  • confluent.schema.registry.url=http://localhost:8081
server1.properties
  • node.id=1
  • listeners=PLAINTEXT://:9093, CONTROLLER://:9095
  • og.dirs=/tmp/kraft-combined-logs-1(fresh logging directory)
  • confluent.metrics.reporter.bootstrap.servers=localhost:9093
The following configurations are specific to multi-cluster Schema Registry setup for this broker:
  • confluent.http.server.listeners=http://0.0.0.0:8091
  • confluent.schema.registry.url=http://localhost:8082
schema-registry0.properties
  • listeners=http://0.0.0.0:8081
  • kafkastore.bootstrap.servers=localhost:9092
schema-registry1.properties
  • listeners=http://0.0.0.0:8082
  • kafkastore.bootstrap.servers=localhost:9093
  • kafkastore.topic=_schemas1 (with both Schema Registry clusters on localhost, this value must be different from the default _schemas in schema-registry0.properties so that the registries do not overwrite each other)
  • schema.registry.group.id=schema-registry-dest (with both Schema Registry clusters on localhost, this value must be different from the group ID used by schema-registry0.properties, which takes the default ID schema-registry)
control-center-multi-sr.properties
  • bootstrap.servers=localhost:9092
  • confluent.controlcenter.kafka.AK1.bootstrap.servers=localhost:9093
  • confluent.controlcenter.streams.cprest.url=http://0.0.0.0:8090
  • confluent.controlcenter.schema.registry.url=http://localhost:8081
  • confluent.controlcenter.kafka.AK1.cprest.url=http://0.0.0.0:8091
  • confluent.controlcenter.schema.registry.SR-AK1.url=http://localhost:8082
The configurations for cpcrest.url, confluent.controlcenter.kafka.AK1.cprest.url, and confluent.controlcenter.schema.registry.SR-AK1.url are new properties, specific to multi-cluster Schema Registry.
.. tip:: The values for ``kafkastore.topic`` and ``schema.registry.group.id`` must be unique for each |sr| properties file because in this example the two registries are colocated on ``localhost``. If the |sr| clusters were on different hosts, you would not need to make these changes. .. group-tab:: |zk| mode The example properties files are based on the defaults. You can copy the default properties files to use as a basis for the specialized versions of them shown here. The example assumes the new files are in the same directories as the originals. - |zk| ``zookeeper.properties`` file is in ``$CONFLUENT_HOME/etc/kafka/zookeeper.properties``. Copy this to create ``zookeeper0.properties`` and ``zookeeper1.properties``. - |kraft| ``server.properties`` file is in ``$CONFLUENT_HOME/etc/kafka/``. Copy this to create ``server0.properties`` and ``server1.properties``. - |sr| properties file is ``$CONFLUENT_HOME/etc/schema-registry/schema-registry.properties``. Copy this to create ``schema-registry0.properties`` and ``schema-registry1.properties``. - |c3-short| properties file is ``$CONFLUENT_HOME/etc/confluent-control-center/control-center-dev.properties``. Copy this to create ``$CONFLUENT_HOME/etc/confluent-control-center/control-center-multi-sr.properties``. .. raw:: html
File Properties
zookeeper0.properties
  • dataDir=/tmp/zookeeper/zk-0 (fresh logging directory)
  • clientPort=2181 (same as zookeeper.properties)
zookeeper1.properties
  • dataDir=/tmp/zookeeper/zk-1(fresh logging directory)
  • clientPort=2182 (since zookeeper0 is already using 2181)
server0.properties
  • broker.id=0
  • listeners=PLAINTEXT://:9092
  • log.dirs=/tmp/kafka-logs-bk-0 (fresh logging directory)
  • zookeeper.connect=localhost:2181
  • confluent.metrics.reporter.bootstrap.servers=localhost:9092
The following configurations are specific to multi-cluster Schema Registry setup for this broker:
  • confluent.http.server.listeners=http://0.0.0.0:8090
  • confluent.schema.registry.url=http://localhost:8081
server1.properties
  • broker.id=1
  • listeners=PLAINTEXT://:9093
  • log.dirs=/tmp/kafka-logs-bk-1(fresh logging directory)
  • zookeeper.connect=localhost:2182
  • confluent.metrics.reporter.bootstrap.servers=localhost:9093
The following configurations are specific to multi-cluster Schema Registry setup for this broker:
  • confluent.http.server.listeners=http://0.0.0.0:8091
  • confluent.schema.registry.url=http://localhost:8082
schema-registry0.properties
  • listeners=http://0.0.0.0:8081
  • kafkastore.bootstrap.servers=localhost:9092
schema-registry1.properties
  • listeners=http://0.0.0.0:8082
  • kafkastore.bootstrap.servers=localhost:9093
  • kafkastore.topic=_schemas1 (with both Schema Registry clusters on localhost, this value must be different from the default _schemas in schema-registry0.properties so that the registries do not overwrite each other)
  • schema.registry.group.id=schema-registry-dest (with both Schema Registry clusters on localhost, this value must be different from the group ID used by schema-registry0.properties, which takes the default ID schema-registry)
control-center-multi-sr.properties
  • bootstrap.servers=localhost:9092
  • zookeeper.connect=localhost:2181
  • confluent.controlcenter.kafka.AK1.bootstrap.servers=localhost:9093
  • confluent.controlcenter.kafka.AK1.zookeeper.connect=localhost:2182
  • confluent.controlcenter.streams.cprest.url=http://0.0.0.0:8090
  • confluent.controlcenter.schema.registry.url=http://localhost:8081
  • confluent.controlcenter.kafka.AK1.cprest.url=http://0.0.0.0:8091
  • confluent.controlcenter.schema.registry.SR-AK1.url=http://localhost:8082
The configurations for cpcrest.url, confluent.controlcenter.kafka.AK1.cprest.url, and confluent.controlcenter.schema.registry.SR-AK1.url are new properties, specific to multi-cluster Schema Registry.
.. tip:: The values for ``kafkastore.topic`` and ``schema.registry.group.id`` must be unique for each |sr| properties file because in this example the two registries are colocated on ``localhost``. If the |sr| clusters were on different hosts, you would not need to make these changes. .. _run-example-multi-cluster-sr: Run the Example ^^^^^^^^^^^^^^^ .. tabs:: .. group-tab:: |kraft| mode To run the example in |kraft| mode: #. Configure cluster IDs and format log directories for the |ak| clusters. #. Start the |ak| brokers in dedicated command windows, one per broker. #. Start the |sr| clusters in dedicated command windows, one per |sr| cluster. #. Start |c3| in its own dedicated command window. **Configure KRaft specific settings for server0 and server1** The following configuration commands must be run from ``$CONFLUENT_HOME`` (the top level directory where you installed |cp|). Assuming you configured your example |kraft| servers in the same directory as the default ``server.properties`` file, this would be ``$CONFLUENT_HOME/etc/kafka/kraft``. #. In a new command window where you plan to run server0, generate a ``random-uuid`` for server0 using the kafka-storage tool. .. code:: bash KAFKA_CLUSTER_ID="$(bin/kafka-storage random-uuid)" #. Format the log directories for server0: .. code:: bash ./bin/kafka-storage format -t $KAFKA_CLUSTER_ID -c $CONFLUENT_HOME/etc/kafka/kraft/server0.properties --ignore-formatted This is the dedicated window in which you will run server0. #. In a new command window where you plan to run server1, a ``random-uuid`` for server1 using the kafka-storage tool. .. code:: bash KAFKA_CLUSTER_ID="$(bin/kafka-storage random-uuid)" #. Format the log directories for server0: .. code:: bash ./bin/kafka-storage format -t $KAFKA_CLUSTER_ID -c $CONFLUENT_HOME/etc/kafka/kraft/server1.properties --ignore-formatted This is the dedicated window in which you will run server0. **Start the Kafka brokers** :: kafka-server-start etc/kafka/server0.properties :: kafka-server-start etc/kafka/server1.properties **Start Schema Registry clusters** :: schema-registry-start etc/schema-registry/schema-registry0.properties :: schema-registry-start etc/schema-registry/schema-registry1.properties **Start Control Center** :: control-center-start etc/confluent-control-center/control-center-multi-sr.properties .. group-tab:: |zk| mode To run the example in |zk| mode: #. Start the ZooKeepers in dedicated command windows, one per |zk|. #. Start the |ak| brokers in dedicated command windows, one per broker. #. Start the |sr| clusters in dedicated command windows, one per |sr| cluster. #. Start |c3| in its own dedicated command window. **Start ZooKeepers** :: zookeeper-server-start etc/kafka/zookeeper0.properties :: zookeeper-server-start etc/kafka/zookeeper1.properties **Start the Kafka brokers** :: kafka-server-start etc/kafka/server0.properties :: kafka-server-start etc/kafka/server1.properties **Start Schema Registry clusters** :: schema-registry-start etc/schema-registry/schema-registry0.properties :: schema-registry-start etc/schema-registry/schema-registry1.properties **Start Control Center** :: control-center-start etc/confluent-control-center/control-center-multi-sr.properties Manage Schemas for Both Clusters on |c3-short| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #. When the example clusters are running and |c3-short| finishes initialization, open |c3-short| in your web browser. (|c3-short| runs at ``http://localhost:9021/`` by default, as described in :ref:`control-center-access`.) #. Select a cluster from the navigation bar, click the **Topics** menu, and explore the schema management options for one or both clusters, starting with :ref:`c3_schema_create`. -------- Security -------- Any other configurations used to set up a |sr| client with |c3-short| can be configured for an additional |sr| cluster by simply appending the |sr| cluster's name to the ``confluent.controlcenter.schema.registry`` prefix. For example, for HTTP Basic authentication with multi-cluster |sr|, specify the following in the |c3| configuration file: - Use ``confluent.controlcenter.schema.registry.basic.auth.credentials.source`` and ``confluent.controlcenter.schema.registry.basic.auth.user.info`` to define authentication for the ``confluent.controlcenter.schema.registry.url`` cluster. - Use ``confluent.controlcenter.schema.registry.{name}.basic.auth.credentials.source`` and ``confluent.controlcenter.schema.registry.{name}.basic.auth.user.info`` for additional |sr| clusters (associated with the URL fields by ``{name}``). Some |sr| client configurations also include a ``schema.registry`` prefix. For TLS/SSL security settings, specify the following in the |c3| configuration file: - Use ``confluent.controlcenter.schema.registry.schema.registry.ssl.truststore.location`` and ``confluent.controlcenter.schema.registry.schema.registry.ssl.truststore.password`` for the ``confluent.controlcenter.schema.registry.url`` cluster. - Use ``confluent.controlcenter.schema.registry.{name}.schema.registry.ssl.truststore.location`` and ``confluent.controlcenter.schema.registry.{name}.schema.registry.ssl.truststore.password`` for additional |sr| clusters (associated with the URL fields by ``{name}``). To learn more, see |sr| authentication properties in the |c3-short| Configuration Reference under :ref:`controlcenter_configuration_encryption` and the section on :ref:`basic-auth-sr` in HTTP Basic authentication. To learn more, see :ref:`How to configure clients to Schema Registry ` in the |sr| Security Overview. -------------------------- Errors and Troubleshooting -------------------------- If the brokers for the cluster have matching |sr| URLs, but these URLs were not defined in the |c3-short| properties file, an error message is displayed on the cluster overview page. .. figure:: ../../images/c3-schema-multi-sr-error.png :align: center :alt: Multi-cluster schema registry error message on Control Center --------------------- Version Compatibility --------------------- The ``confluent.controlcenter.schema.registry.url`` configuration in the |c3-short| properties file acts as a default if a cluster's broker configurations do not contain ``confluent.schema.registry.url fields``. Multiple |sr| clusters may be specified with ``confluent.controlcenter.schema.registry.{name}.url`` fields. Multi-cluster |sr| cannot be used with |ak| versions prior to |ak| 2.4.x, the version current with |cp| 5.4.0. However, using a single cluster |sr| setup will work with earlier |ak| versions. To learn more, see :ref:`cp-ak-compatibility`. ----------------- Suggested Reading ----------------- - See :ref:`schema_registry_onprem_tutorial` for hands-on examples of developing schemas, mapping to topics, and sending messages. - See :ref:`schemaregistry_intro` for an overview of |sr| and schema management. - :ref:`controlcenter_configuration` - :ref:`c3_properties_files` - :ref:`c3_schema_registry_not_set_up` in :ref:`controlcenter_troubleshooting` - :ref:`schema_validation` - :ref:`sr-per-topic-subject-name-strategy`