Unified Stream Manager overview¶

Unified Stream Manager (USM) is a hybrid solution designed to integrate your self-managed Confluent Platform cluster deployment with Confluent Cloud. It provides a unified interface within the Confluent Cloud UI, allowing you to monitor, manage, and govern both self-managed (on-premises) and fully-managed cloud clusters from a single location.

USM establishes Confluent Cloud as the authoritative control plane for governance tasks, while your on-premises Confluent Platform continues to serve as the data plane for data processing. This operational model empowers organizations to centralize control without moving actual message data to the cloud.

The primary benefits of USM include the following:

Reduce operational overhead. You can gain a single, unified view of your entire hybrid data streaming landscape. By monitoring the health, performance, and configuration of all your clusters from one place, you can simplify troubleshooting and decreases the management burden.
Strengthen data governance. You can centrally manage and enforce consistent rules across all your on-premises and cloud clusters. This approach establishes a single source of truth for schemas, data quality, and security policies.
Accelerate your cloud migration. You can manage on-premises and cloud clusters as a single system. This unique operational model lets you migrate data streams and applications incrementally, which reduces the complexity and risk of the transition.

Key capabilities¶

USM extends Confluent Cloud’s advanced features to your self-managed environment, enabling you to:

Unify schema management. Combines your on-premises and cloud Schema Registries into a single, global registry managed from Confluent Cloud. This establishes a single source of truth for all schemas across your hybrid landscape.
Monitor your self-managed clusters: Observe and monitor your on-premises clusters from a unified view within the Confluent Cloud UI.
Discover topics and schemas globally: Enable developers to search for and discover topics and schemas across both self-managed and cloud infrastructure through a global, searchable data catalog.
Visualize your end-to-end data lineage. Get a complete view of your data’s journey with stream lineage graphs that visualize data flows for topics, connectors, and applications within your connected on-premises clusters.
Define comprehensive data contracts. Enforce data quality by bundling a schema, validation rules, and metadata into a single, enforceable definition called a data contract.
Protect sensitive data with field-level encryption. Centrally manage encryption keys and policies in the Confluent Cloud UI. These policies are automatically enforced on clients connected to your on-premises clusters.
Organize data with metadata tagging. Apply tags and other metadata to resources from the Confluent Cloud UI to improve organization and discoverability across your entire hybrid environment.

USM components¶

USM has two main components: the USM agent and the USM cloud console.

USM agent¶

The USM agent is a lightweight component deployed in your on-premises Confluent Platform environment. It authenticates using a Confluent Cloud service account and establishes a secure, outbound connection to Confluent Cloud, which lets you monitor and manage your self-managed resources from a central UI.

The USM agent runs in your self-managed Confluent Platform 8.1 or later environment. You deploy and manage it by using a Confluent toolset: Confluent for Kubernetes for containerized deployments, or Ansible Playbooks for Confluent Platform for environments on virtual machines or bare-metal servers.

The USM agent performs the following tasks:

Collects telemetry: The agent securely streams operational and governance data from Confluent Platform to Confluent Cloud, including cluster health metrics, topic states, connector configurations, and schema metadata.
Communicates securely: It maintains an encrypted communication channel that is authenticated with service accounts and supports private networking to help keep data secure. Your message data remains in your self-managed environment.
Acts as a secure data proxy: The agent serves as a single egress point for funneling metadata and metrics to Confluent Cloud. This architecture supports read operations from the cloud for monitoring purposes but does not allow Confluent Cloud to send management commands to your self-managed cluster.

For detailed deployment guidance, including resource sizing, high availability configurations, and monitoring setup, see the USM agent: sizing, high availability, and monitoring.

USM cloud console¶

The USM cloud console is the central hub in the Confluent Cloud for managing your entire hybrid environment. It aggregates and visualizes data from the USM agent, letting you do the following:

Get a unified view: See and manage all your clusters—both in Confluent Cloud and on-premises—from a single dashboard. This provides a complete view of your data streaming environment.
Monitor from a central location: Track the health, performance, and key metrics of your Confluent Platform clusters without leaving the Confluent Cloud. This simplifies troubleshooting and reduces the need to switch between different monitoring tools.
Extend governance: Apply Confluent Cloud’s governance features to your on-premises data. You can discover, manage, and trace data lineage for topics and schemas across your hybrid landscape from one location.

USM architecture¶

The USM architecture uses two core principles: a hybrid model that separates management from data processing, and a global mechanism for synchronizing schemas.

Unified Stream Manager for Confluent Platform and Confluent Cloud.¶

Hybrid governance model¶

This model separates the system into a cloud-based control plane for governance and a self-managed data plane for data processing.

Confluent Cloud (control plane): Serves as the central hub for all governance and management. All schema write operations and monitoring data are consolidated here to establish a single source of truth.
Confluent Platform (data plane): Your self-managed cluster that handles the core data streaming workload. Your brokers, connectors, and message data remain within your own security perimeter.
USM agent: The component in your Confluent Platform environment that securely forwards operational telemetry and metadata to the Confluent Cloud control plane. This connection is established over a secure, private network to ensure your data remains isolated.

Global Schema Registry model¶

The cornerstone of USM is its hybrid Schema Registry model, which establishes a primary/replica pattern.

Confluent Cloud Schema Registry as primary: The Confluent Cloud Schema Registry is the primary, authoritative read/write source of truth for all schemas.
Confluent Platform Schema Registry as Read-Cache: Your on-premises Confluent Platform Schema Registry is configured in FORWARD mode to act as a low-latency, read-only cache for local clients. All write requests it receives are automatically and transparently forwarded to Confluent Cloud.
Synchronization: To keep the on-premise cache up-to-date, a pull-based schema importer is configured on the Confluent Platform Schema Registry. This importer continuously fetches the latest schema updates from the Confluent Cloud Schema Registry, ensuring governance rules defined in the cloud are propagated efficiently.
Schema synchronization and contexts: To prevent schema ID collisions during the migration of an existing, populated Confluent Platform Schema Registry, USM uses schema contexts. This feature namespaces the on-premise schemas when they are copied to the cloud, ensuring a clean merge without conflicts.

Deployment and networking¶

All communication between your Confluent Platform environment and Confluent Cloud is handled exclusively over AWS PrivateLink. This ensures that all metadata and telemetry traffic is secure and does not traverse the public internet. For datacenter deployments, you may need to establish a connection like AWS Direct Connect to your AWS environment first.