.. title:: Confluent Platform Overview .. meta:: :description: Confluent Platform is an enterprise grade and full-scale data streaming platform that builds on the features of open-source Apache Kafka and enables you to easily access, store, and manage data as continuous, real-time streams. .. _cp_platform: |cp| Overview ############## |cp| is a full-scale :term:`streaming platform ` that enables you to easily access, store, and manage data as continuous, real-time streams. Built by the original creators of |ak-tm|, |cp| is an enterprise-ready platform that completes |ak| with advanced capabilities designed to help accelerate application development and connectivity. |cp| enables transformations through stream processing, simplifies enterprise operations at scale, and meets stringent architectural requirements. |cp| is software you download and manage yourself. Any |ak| use cases are also |cp| use cases. |cp| is a specialized distribution of |ak| that includes additional features and APIs. Many of the commercial |cp| features are built into |ak| brokers as a function of |cs|. By integrating historical and real-time data into a single, central source of truth, Confluent makes it easy to build an entirely new category of modern, event-driven applications, gain a universal data pipeline, and unlock powerful new use cases with full scalability, performance, and reliability. .. include:: includes/cp-cta.rst |ak| Use Cases *************** |ak| is used for a wide array of use cases across numerous industries, such as: - Financial services - Omnichannel retail - Autonomous cars - Fraud detection services - Microservices - IoT You can use |ak| to collect user activity data, system logs, application metrics, stock ticker data, and device instrumentation signals. Regardless of the use case, |cp| lets you focus on how to derive business value from your data rather than worrying about the underlying mechanics, such as how data is being transported or integrated between disparate systems. Specifically, |cp| simplifies connecting data sources to |ak|, building streaming applications, as well as securing, monitoring, and managing your |ak| infrastructure. .. _cp-features: |cp| Features ************** At the core of |cp| is |ak|, the most popular open source distributed streaming platform. |ak| enables you to: - Publish and subscribe to streams of records - Store streams of records in a fault tolerant way - Process streams of records Each |cp| release includes the latest release of |ak| and additional tools and services that make it easier to build and manage an event streaming platform. |cp| provides community and commercially licensed features such as :ref:`Schema Registry`, :ref:`Cluster Linking`, a :ref:`REST Proxy`, :ref:`100+ pre-built Kafka connectors`, and :ref:`ksqlDB `. For more information about Confluent components and the license that applies to them, see :ref:`Confluent Licenses`. .. figure:: images/confluentPlatform.png :align: center |cp| Components |ak| capabilities +++++++++++++++++ |cp| provides all of |ak|'s open-source features plus additional proprietary components. Following is a summary of |ak| features. For an overview of |ak| use cases, features and terminology, see :kafka-common:`Kafka Introduction|introduction.html`. - At the core of |ak| is the :term:`Kafka broker`. A broker stores data in a durable way from clients in one or more topics that can be consumed by one or more clients. |ak| also provides several :ref:`command-line tools` that enable you to start and stop |ak|, create topics and more. - |ak| provides security features such as :ref:`data encryption` between producers and consumers and brokers using SSL / TLS. :ref:`Authentication ` using SSL or SASL and authorization using ACLs. These security features are disabled by default. - Additionally, |ak| provides the following :kafka-common:`Java APIs|kafka-apis.html`. - The Producer API that enables an application to send messages to |ak|. To learn more, see :ref:`Producer`. - The Consumer API that enables an application to subscribe to one or more topics and process the stream of records produced to them. To learn more, see :ref:`Consumer`. - :ref:`kafka_connect`, a component that you can use to stream data between |ak| and other data systems in a scalable and reliable way. It makes it simple to configure connectors to move data into and out of |ak|. |kconnect-long| can ingest entire databases or collect metrics from all your application servers into |ak| topics, making the data available for stream processing. Connectors can also deliver data from |ak| topics into secondary indexes like Elasticsearch or into batch systems such as Hadoop for offline analysis. - The :ref:`Streams API` that enables applications to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. It has a very low barrier to entry, easy operationalization, and a high-level DSL for writing stream processing applications. As such it is the most convenient yet scalable option to process and analyze data that is backed by |ak|. - The :kafka-common:`Admin API|kafka-apis.html#admin-client-api` that provides the capability to create, inspect, delete, and manage topics, brokers, ACLs, and other |ak| objects. To learn more, see :ref:`kafkarest_intro`, which leverages the Admin API. Development and connectivity features ++++++++++++++++++++++++++++++++++++++ To supplement |ak|'s Java APIs, and to help you connect all of your systems to |ak|, |cp| provides the following features: - Confluent Connectors, which leverage the |ak| Connect API to connect |ak| to other systems such as databases, key-value stores, search indexes, and file systems. Confluent Hub has downloadable connectors for the most popular data sources and sinks. These include fully tested and supported versions of these connectors with |cp|. See the following documentation for more information: - :ref:`connect_userguide` - :ref:`connect_bundled_connectors` - :ref:`connect_preview_connectors` Confluent provides both commercial and community licensed connectors. For details, and to download connectors , see `Confluent Hub `__. - :ref:`Non-java clients` such as a :clients:`C/C++|librdkafka/current/overview.html`, :clients:`Python|python/current/overview.html`, :clients:`Go|go/current/overview.html`, and :clients:`.NET client|dotnet/current/overview.html` libraries in addition to the Java client. These clients are full-featured and performant. For more information, see the :ref:`kafka_clients`. In addition, Confluent provides a JMS-compatible client for |ak|. For more information, see :ref:`client_jms`. - A :ref:`REST Proxy `, which leverages the Admin API and makes it easy to work with |ak| from any language by providing a RESTful HTTP service for interacting with |ak| clusters. The REST Proxy supports all the admin core functionality: sending messages to |ak|, reading messages, both individually and as part of a consumer group, and inspecting cluster metadata, such as the list of topics and their settings. You get the full benefits of the high quality, officially maintained Java clients from any language. The REST Proxy also integrates with Schema Registry. Because it automatically translates JSON data to and from Avro, you can get all the benefits of centralized schema management from any language using only HTTP and JSON. - All of the |ak| command-line tools and additional tools, including the `Confluent CLI `__. You can find a list of all of these tools in :ref:`CLI Tools Bundled With Confluent Platform `. - :ref:`Schema Registry `, which provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of data over a network. With a messaging service like |ak|, services that interact with each other must agree on a common format, called a schema, for messages. Schema Registry helps enable safe, zero-downtime evolution of schemas by centralizing schema management. It provides a RESTful interface for storing and retrieving Avro®, JSON Schema, and Protobuf schemas. Schema Registry tracks all versions of schemas and enables the evolution of schemas according to user-defined compatibility settings. Schema Registry also includes plugins for |ak| clients that handle schema storage and retrieval for |ak| messages that are sent in the Avro format. For more information, see the :ref:`Schema Registry Documentation `. For a hands-on introduction to working with schemas, see the :ref:`On-Premises Schema Registry Tutorial `. For a deep dive into supported serialization and deserialization formats, see Formats, Serializers, and Deserializers. - :ref:`ksqlDB `, a streaming SQL engine for |ak|. It provides an interactive SQL interface for stream processing on |ak|, without the need to write code in a programming language such as Java or Python. ksqlDB is scalable, elastic, fault-tolerant, and real-time. It supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing, and sessionization. For more information, see the `ksqlDB Documentation `_, or the `ksqlDB getting started guide `_. - A :ref:`MQTT Proxy`, which provides a way to publish data directly to |ak| from MQTT devices and gateways without the need for a MQTT Broker in the middle. For more information, see :ref:`MQTT Proxy`. Management and monitoring features +++++++++++++++++++++++++++++++++++++ |cp| provides several features to supplement |ak|'s Admin API, and built-in JMX monitoring. - :ref:`Confluent Control Center `, which is a web-based system for managing and monitoring |ak|. It allows you to easily manage |ak| Connect, to create, edit, and manage connections to other systems. |c3-short| also enables you to monitor data streams from producer to consumer, assuring that every message is delivered, and measuring how long it takes to deliver messages. Using |c3-short|, you can build a production data pipeline based on |ak| without writing a line of code. |c3-short| also has the capability to :ref:`define alerts ` on the latency and completeness statistics of data streams, which can be delivered by email or queried from a centralized alerting system. - :ref:`Health+`, also a web-based tool to help ensure the health of your clusters and minimize business disruption with intelligent alerts, monitoring, and proactive support. - :ref:`Metrics reporter ` for collecting various metrics from a |ak| cluster. The metrics are produced to a topic in a |ak| cluster. Performance and scalability features +++++++++++++++++++++++++++++++++++++ Confluent offers a number of features to scale effectively and get the maximum performance for your investment. - To help save money, you can use the :ref:`tiered-storage` feature, which automatically tiers data to cost-effective object storage, and scale brokers only when you need more compute resources. - :ref:`sbc` provides automated load balancing, failure detection and self-healing for your clusters. It provides support for adding or decommissioning brokers as needed, with no manual tuning. Self-Balancing Clusters are the next iteration of Auto Data Balancer in that Self-Balancing auto-monitors clusters for imbalances, and automatically triggers rebalances based on your configurations. Partition reassignment plans and execution are taken care of for you. - To deploy |cp| at scale, you can deploy with `Ansible Playbooks `_ or `Confluent for Kubernetes `_. - `Confluent for Kubernetes `_ is a Kubernetes operator. Kubernetes operators extend the orchestration capabilities of Kubernetes by providing the unique features and requirements for a specific platform application. For |cp|, this includes greatly simplifying the deployment process of |ak| on Kubernetes and automating typical infrastructure lifecycle tasks. - `Ansible Playbooks `_ enable you to automatically configure, provision, and deploy |cp| clusters using YAML files. Security and resilience features +++++++++++++++++++++++++++++++++ |cp| also offers a number of features that build on |ak|’s security features to help ensure your deployment stays secure and resilient. - You can set authorization by role with Confluent's :ref:`Role-based Access Control (RBAC)` feature. - If you use |c3-short|, you can set up :ref:`Single Sign On (SSO)` that integrates with a supported OIDC identity provider, and enable additional security measures such as multi-factor authentication. - The :ref:`kafka-rest-security-plugins-install` and :ref:`confluentsecurityplugins_schema_registry_security_plugin` add security capabilities to the |cp| REST Proxy and Schema Registry. The Confluent REST Proxy Security Plugin helps in authenticating the incoming requests and propagating the authenticated principal to requests to |ak|. This enables Confluent REST Proxy clients to utilize the multi-tenant security features of the |ak| broker. The Schema Registry Security Plugin supports authorization for both role-based access control (RBAC) and ACLs. - :ref:`Audit logs ` provide the ability to capture, protect, and preserve authorization activity into topics in |ak| clusters on |cp| using :ref:`Confluent Server Authorizer `. - The :ref:`Cluster Linking ` feature enables you to directly connect clusters and mirror topics from one cluster to another. This makes it easier to build multi-datacenter, multi-region and hybrid cloud deployments. - :ref:`Confluent Replicator ` makes it easier to maintain multiple |ak| clusters in multiple data centers. Managing replication of data and topic configuration between data centers enables use-cases such as active geo-localized deployments, centralized analytics and cloud migration. You can use Replicator to configure and manage replication for all these scenarios from either |c3-short| or command-line tools. To get started, see the :ref:`Replicator documentation `, including the :ref:`Replicator Quick Start`. Install |cp| ************* You can install |cp| and its components manually using :ref:`ZIP or TAR installation ` packages, using package managers on :ref:`CentOS, RHEL, Rocky Linux `, :ref:`Ubuntu, Debian, ` or :ref:`Docker images `. The :ref:`quickstart` uses Docker Compose to get you up and running quickly. Docker images of |cp| are available on `Docker Hub `_. To learn more about the packages, see :ref:`image_reference`. Some of these images contain proprietary components that require a Confluent enterprise license. You can also install using an orchestrator like `Ansible Playbooks `_ or `Confluent for Kubernetes `_.. For more information about all of the installation options, see :ref:`installation`. If you would rather take advantage of all of |cp|'s features in a managed cloud environment, you can use :ccloud-cta:`Confluent Cloud|` and get started for free using the :cloud:`Cloud quick start|get-started/index.html`.