.. _cp_platform:
What is |cp|?
=============
.. tip:: Sign up for :ccloud-cta:`Confluent Cloud|` or download
`Confluent Platform `_ to get started.
Confluent provides the industry’s only enterprise-ready |streaming|, driving a new paradigm for application and data
infrastructure. With |cp| you can leverage data as a continually updating stream of events rather than as discrete snapshots.
Over 60% of the Fortune 100 leverage event streaming – and the majority of those leverage |cp|. Confluent provides a
single platform for real-time and historical events, enabling you to build an entirely new category of event-driven
applications and gain a universal event pipeline. Confluent is founded by the original creators of |ak-tm|.
|cp| makes it easy to build real-time data pipelines and streaming applications by integrating data
from multiple sources and locations into a single, central |streaming| for your company.
|cp| lets you focus on how to derive business value from your data rather than worrying about the
underlying mechanics such as how data is being transported or massaged between various systems. Specifically,
|cp| simplifies connecting data sources to |ak|, building applications with |ak|, as well as securing,
monitoring, and managing your |ak| infrastructure.
.. figure:: images/confluentPlatform.png
:align: center
Confluent Platform Components
--------
Overview
--------
At the core of |cp| is `Apache Kafka `_, the most popular open source distributed streaming
platform. The key capabilities of |ak| are:
* Publish and subscribe to streams of records
* Store streams of records in a fault tolerant way
* Process streams of records
|ak| is used by `40% of Fortune 500 companies `_ for a variety of
use cases, including collecting user activity data, system logs, application metrics, stock ticker data, and device
instrumentation signals.
The key components of the |ak| open source project are |ak| Brokers and |ak| Java Client APIs.
|ak| Brokers
|ak| brokers that form the messaging, data persistency and storage tier of |ak|.
|ak| Java Client APIs
* :ref:`Producer API` is a Java Client that allows an application to publish a stream records to one or
more |ak| topics.
* :ref:`Consumer API` is a Java Client that allows an application to subscribe to one or more topics and
process the stream of records produced to them.
* :ref:`Streams API` allows applications to act as a stream processor, consuming an input stream from
one or more topics and producing an output stream to one or more output topics, effectively transforming the input
streams to output streams. It has a very low barrier to entry, easy operationalization, and a high-level DSL for
writing stream processing applications. As such it is the most convenient yet scalable option to process and analyze
data that is backed by |ak|.
* :ref:`Connect API` is a component that you can use to stream data between |ak| and other data
systems in a scalable and reliable way. It makes it simple to configure connectors to move data into and out of |ak|.
|kconnect-long| can ingest entire databases or collect metrics from all your application servers into |ak| topics,
making the data available for stream processing. Connectors can also deliver data from |ak| topics into secondary
indexes like Elasticsearch or into batch systems such as Hadoop for offline analysis.
Each release of |cp| includes the latest release of |ak| and additional tools and services that make it easier
to build and manage an |streaming|. |cp| delivers both community and commercially licensed features that
complement and enhance your |ak| deployment.
.. include:: includes/cp-demo-tip.rst
-------------------
Commercial Features
-------------------
|c3|
""""
:ref:`control_center` is a GUI-based system for managing and monitoring |ak|. It allows you to easily manage |kconnect-long|, to create, edit, and manage connections to other systems. It also allows you to monitor data streams
from producer to consumer, assuring that every message is delivered, and measuring how long it takes to deliver messages. Using |c3-short|, you can build a production data pipeline based on |ak| without writing a line of code.
|c3-short| also has the capability to define alerts on the latency and completeness statistics of data streams, which can be delivered by email or queried from a centralized alerting system.
|co-long|
"""""""""
|co-long| is a *Kubernetes operator*. Kubernetes operators extend the
orchestration capabilities of Kubernetes by providing the unique features and
requirements for a specific platform application. For |cp|, this includes greatly simplifying the deployment process of |ak| on Kubernetes and automating typical infrastructure lifecycle tasks.
See :ref:`About Confluent Operator ` for more information.
|crep-full|
"""""""""""
|cp| makes it easier than ever to maintain multiple |ak| clusters in multiple data centers. Managing replication of data and topic configuration between data centers enables use-cases such as:
* Active-active geo-localized deployments: allows users to access a near-by data center to optimize their architecture for low latency and high performance
* Centralized analytics: Aggregate data from multiple |ak| clusters into one location for organization-wide analytics
* Cloud migration: Use |ak| to synchronize data between on-prem applications and cloud deployments
You can use :ref:`Confluent Replicator ` to configure and manage replication for all these scenarios
from either |c3| or command-line tools.
|cadb-full|
"""""""""""
As clusters grow, topics and partitions grow at different rates, brokers are added and removed and over time this leads to
unbalanced workload across datacenter resources. Some brokers are not doing much at all, while others are heavily taxed with
large or many partitions, slowing down message delivery. When executed, :ref:`Confluent Auto Data Balancer ` monitors
your cluster for number of brokers, size of partitions, number of partitions and number of leaders within the cluster. It allows
you to shift data to create an even workload across your cluster, while throttling rebalance traffic to minimize impact on
production workloads while rebalancing.
For more information, see the :ref:`automatic data balancing documentation `.
|cjms-full|
"""""""""""
|cp| includes a JMS-compatible client for |ak|. This |ak| client implements the JMS 1.1 standard API, using
|ak| brokers as the backend. This is useful if you have legacy applications using JMS, and you would like to
replace the existing JMS message broker with |ak|. By replacing the legacy JMS message broker with |ak|,
existing applications can integrate with your modern streaming platform without a major rewrite of the application.
For more information, see :ref:`client_jms`.
|cmqtt-full|
""""""""""""
Provides bidirectional access to |ak| from MQTT devices and gateways without the need for a MQTT Broker in the middle.
For more information, see :ref:`mqtt_proxy`.
Confluent Security Plugins
""""""""""""""""""""""""""
Confluent Security Plugins are used to add security capabilities to various |cp| tools and products.
Currently, there is a plugin available for |crest-long| which helps in authenticating the incoming requests and propagating
the authenticated principal to requests to |ak|. This enables |crest-long| clients to utilize the multi-tenant security
features of the |ak| broker. For more information, see :ref:`confluentsecurityplugins_Introduction`.
------------------
Community Features
------------------
.. _cp_platform_ksql:
|cksql-full|
""""""""""""
|cksql-full| is the streaming SQL engine for |ak|. It provides an easy-to-use yet powerful interactive SQL
interface for stream processing on |ak|, without the need to write code in a programming language such as Java or Python.
KSQL is scalable, elastic, fault-tolerant, and real-time. It supports a wide range of streaming operations, including data
filtering, transformations, aggregations, joins, windowing, and sessionization.
KSQL supports these use cases:
Streaming ETL
|ak| is a popular choice for powering data pipelines. KSQL makes it simple to transform data within the pipeline,
readying messages to cleanly land in another system.
Real-time Monitoring and Analytics
Track, understand, and manage infrastructure, applications, and data feeds by quickly building real-time dashboards,
generating metrics, and creating custom alerts and messages.
Data exploration and discovery
Navigate and browse through your data in |ak|.
Anomaly detection
Identify patterns and spot anomalies in real-time data with millisecond latency, allowing you to properly surface out
of the ordinary events and to handle fraudulent activities separately.
Personalization
Create data driven real-time experiences and insight for users.
Sensor data and IoT
Understand and deliver sensor data how and where it needs to be.
Customer 360-view
Achieve a comprehensive understanding of your customers across every interaction through a variety of channels, where
new information is continuously incorporated in real-time.
For more information, see the :ref:`KSQL documentation `.
Confluent Connectors
""""""""""""""""""""
Connectors leverage the |kconnect-long| API to connect |ak| to other data systems such as Apache Hadoop. |cp|
provides connectors for the most popular data sources and sinks, and include fully tested and supported versions of these
connectors with |cp|. The available connectors include JDBC, HDFS, Elasticsearch, and S3.
For more information and the full list of connectors, see :ref:`Kafka Connectors documentation `.
Confluent Clients
"""""""""""""""""
""""""""""""""""""""
C/C++ Client Library
""""""""""""""""""""
The library *librdkafka* is the C/C++ implementation of the |ak| protocol, containing both Producer and Consumer
support. It was designed with message delivery, reliability and high performance in mind. Current benchmarking figures
exceed 800,000 messages per second for the producer and 3 million messages per second for the consumer. This library includes
support for many new features |ak| 0.10, including message security. It also integrates easily with libserdes, our C/C++
library for Avro data serialization (supporting |sr|).
For more information, see the :ref:`kafka_clients` documentation.
"""""""""""""""""""""
Python Client Library
"""""""""""""""""""""
A high-performance client for Python.
For more information, see the :ref:`kafka_clients` documentation.
"""""""""""""""""
Go Client Library
"""""""""""""""""
|cp| includes of a full-featured, high-performance client for Go.
For more information, see the :ref:`kafka_clients` documentation.
"""""""""""""""""""
.Net Client Library
"""""""""""""""""""
|cp| bundles a full featured, high performance client for .Net.
For more information, see the :ref:`kafka_clients` documentation.
.. _cp_platform_schemaregistry:
|sr-long|
"""""""""
One of the most difficult challenges with loosely coupled systems is ensuring
compatibility of data and code as the system grows and evolves. With a messaging
service like |ak|, services that interact with each other must agree on a common format, called a
*schema*, for messages. In many systems, these formats are ad
hoc, only implicitly defined by the code, and often are duplicated
across each system that uses that message type.
As requirements change, it becomes necessary to evolve these formats. With only an ad-hoc definition, it
is very difficult for developers to determine what the impact of their change might be.
:ref:`Confluent Schema Registry ` enables safe, zero downtime evolution of schemas by
centralizing the management of
schemas written for the `Avro `_ serialization system. It
tracks all versions of schemas used for every topic in |ak| and only allows
evolution of schemas according to user-defined compatibility settings. This
gives developers confidence that they can safely modify schemas as necessary
without worrying that doing so will break a different service they may not even
be aware of.
|sr| also includes plugins for |ak| clients that handle schema
storage and retrieval for |ak| messages that are sent in the Avro format. This
integration is seamless -- if you are already using |ak| with Avro data, using
|sr| only requires including the serializers with your
application and changing one setting.
For more information, see the :ref:`Schema Registry documentation `.
|crest-long|
""""""""""""
|ak| and Confluent provide native clients for Java, C, C++, and Python that make it fast and easy to
produce and consume messages through |ak|. These clients are usually the easiest, fastest, and most secure way to
communicate directly with |ak|.
But sometimes, it isn't practical to write and maintain an application that uses the native clients. For example,
an organization might want to connect a legacy application written in PHP to |ak|. Or suppose that a company is
running point-of-sale software that runs on cash registers, written in C# and running on Windows NT 4.0, maintained
by contractors, and needs to post data across the public internet. To help with these cases, |cp|
includes a REST Proxy. The REST Proxy addresses these problems.
The |crest-long| makes it easy to work with |ak| from any language by providing a
RESTful HTTP service for interacting with |ak| clusters. The REST Proxy
supports all the core functionality: sending messages to |ak|, reading
messages, both individually and as part of a consumer group, and inspecting
cluster metadata, such as the list of topics and their settings. You can get the
full benefits of the high quality, officially maintained Java clients from any
language.
The REST Proxy also integrates with |sr|. It can read and write
Avro data, registering and looking up schemas in |sr|. Because it
automatically translates JSON data to and from Avro, you can get all the
benefits of centralized schema management from any language using only HTTP and
JSON.
For more information, see the :ref:`Confluent REST Proxy documentation `.