Stream Governance on Confluent Cloud

Stream Governance is built upon three key strategic pillars:

  • Stream lineage - Understand complex data relationships and uncover more insights with interactive, end-to-end maps of event streams.
  • Stream catalog - Increase collaboration and productivity with self-service data discovery that allows teams to classify, organize, and find the event streams they need.
  • Stream quality - Deliver trusted, high-quality event streams to the business and maintain data integrity as services evolve.

To learn more about data governance, stream governance, and the strategies behind these new tools and products, see the brief introduction below.

Data governance

Data governance initiatives aim to manage the availability, integrity, and security of data used across an organization. With the explosion in volume, variety, and velocity of data powering the modern enterprise, it’s no surprise that the management of that data has become paramount to the success of companies across the globe. Data governance is now a mainstream, mandatory, and critical force behind the “data everywhere” movement.

At its core, a data governance program consists of three key areas:

icon-people The team that manages all aspects of your organization’s data, with clearly defined roles and responsibilities.

icon-processes The processes and procedures for how data should be controlled, audited, secured, and monitored.

icon-technology The technology that enables people, and streamlines processes, with tools like standardization, monitoring, collaboration, and reporting, just to name a few.

Governance for Data in Motion

With the sharp rise of real-time data, the need for organizational governance over data in motion is growing quickly. As investments into microservices increase and data sources scale, it becomes more and more challenging for any individual or team to understand or govern the full scope of streams flowing across the business.

To enable a generation of event-centric enterprises, Confluent developed Stream Governance: governance built for data in motion, allowing businesses to operate around a central nervous system of real-time data.

With Confluent’s Stream Governance, companies can confidently share the power of data in motion across the organization with a governance solution designed for the intricacies of streaming data.

Stream quality

Confluent’s stream quality tools enable teams to deliver a scalable supply of trusted event streams throughout the business, enabling reliable delivery of mission-critical applications, confident decision making, and a simplified design for data standards. These tools set and control the data rules and definitions by which the entire system operates. This determines what data gets in and what data does not, all in the spirit of maintaining high data integrity.

  • Schema Registry allows teams to define and enforce universal data standards that enable scalable data compatibility while reducing operational complexity. Avro, JSON, and Protobuf serialization formats are supported.
  • Data Contracts and Data Quality Rules are supported in Schema Registry to expand on schemas and provide formal agreement between producers and consumers on the structure, evolution, and integrity constraints on data exchange. Data quality rules can specify the semantics of data fields, detail constraints on the data (such as requiring positive values for certain fields), describe the evolution and compatibility of schemas, and indicate data ownership.
  • Schema Validation, enabled at the topic-level, ensures broker/registry coordination by verifying that schemas tied to incoming messages are both valid and assigned to the specific destination topic in order to publish.
  • Schema Linking keeps schemas in sync across Schema Registry clusters. Optionally, use it in combination with Geo-replication with Cluster Linking on Confluent Cloud to keep both schemas and topic data in sync across Schema Registry and Kafka clusters.

Demo

This video provides a real-world use case for Stream Governance, highlighting various aspects of stream catalog, stream lineage, data discovery, and data quality.

®

Get started

To learn about Stream Governance packages and feature offerings, see Stream Governance Packages, Features, and Limits.

To get started with Stream Governance, sign up for Confluent Cloud and then try out the tutorials and workflows in the sections below.

Related content