# Stream Processing with Confluent Cloud for Apache Flink Confluent Cloud for Apache Flink provides fully managed, serverless stream processing on Confluent Cloud. Filter, join, enrich, and transform your Apache Kafka® data streams in real time without managing infrastructure. Built on [Apache Flink](https://flink.apache.org/), the industry standard for stateful stream processing, Confluent Cloud for Apache Flink handles provisioning, autoscaling, and fault tolerance so you can focus on your stream processing logic. Your Kafka topics appear automatically as queryable Flink tables, with schemas and metadata attached by Confluent Cloud. You can start processing data immediately by using Flink SQL, the [Flink Table API](reference/table-api.md#flink-table-api) (Java and Python), or custom [user-defined functions](concepts/user-defined-functions.md#flink-sql-udfs). Confluent Cloud for Apache Flink integrates with [Schema Registry](../sr/schemas-manage.md#sr-prv), [Connectors](../connectors/overview.md#kafka-connect-cloud), and [Tableflow](../topics/tableflow/overview.md#cloud-tableflow) to provide a complete data streaming platform. To run Flink on-premises with Confluent Platform, see [Confluent Platform for Apache Flink](/platform/current/flink/overview.html). - [What is Confluent Cloud for Apache Flink?](#ccloud-flink-overview-what-is-flink) - [Cloud native](#ccloud-flink-overview-cloud-native) - [Enterprise-ready](#ccloud-flink-overview-complete) - [Everywhere](#ccloud-flink-overview-everywhere) - [Program Flink with SQL, Java, and Python](#ccloud-flink-overview-program-flink) - [Confluent for VS Code](#ccloud-flink-overview-vs-code) ## What is Confluent Cloud for Apache Flink?  Confluent Cloud for Apache Flink is Flink re-imagined as a truly cloud-native service. Confluent’s fully managed Flink service enables you to: - Easily filter, join, and enrich your data streams with Flink - Enable high-performance and efficient stream processing at any scale, without the complexities of managing infrastructure - Experience Kafka and Flink as a unified platform, with fully integrated monitoring, security, and governance When bringing Flink to Confluent Cloud, the goal was to provide a uniquely serverless experience superior to just “cloud-hosted” Flink. Kafka on Confluent Cloud goes beyond Kafka by using the [Kora engine](https://www.confluent.io/resources/report/kora-a-cloud-native-event-streaming-platform-for-kafka/), which showcases Confluent’s engineering expertise in building cloud-native data systems. Confluent’s goal is to deliver the same simplicity, security, and scalability for Flink that you expect for Kafka. Confluent Cloud for Apache Flink is engineered to be: - [Cloud-native](#ccloud-flink-overview-cloud-native): Flink is fully managed on Confluent Cloud and autoscales up and down with your workloads. - [Enterprise-ready](#ccloud-flink-overview-complete): Flink is integrated deeply with Confluent Cloud to provide an enterprise-ready experience. - [Everywhere](#ccloud-flink-overview-everywhere): Flink is available in AWS, Azure, and Google Cloud. Get started with Confluent Cloud for Apache Flink: * [Flink SQL Quick Start with Confluent Cloud Console](get-started/quick-start-cloud-console.md#flink-sql-quick-start-cloud-console) * [Flink SQL Shell Quick Start](get-started/quick-start-shell.md#flink-sql-quick-start-shell) ## Confluent Cloud for Apache Flink is cloud-native  Confluent Cloud for Apache Flink provides a cloud-native experience for Flink. This means you can focus fully on your business logic, encapsulated in Flink SQL [statements](concepts/statements.md#flink-sql-statements), and Confluent Cloud takes care of what’s needed to run them in a secure, resource-efficient and fault-tolerant manner. You don’t need to know about or interact with Flink clusters, state backends, checkpointing, or any of the other aspects that are usually involved when operating a production-ready Flink deployment. Fully Managed : On Confluent Cloud, you don’t need to choose a runtime version of Flink. You’re always using the latest version and benefit from continuous improvements and innovations. All of your running statements automatically and transparently receive security patches and minor upgrades of the Flink runtime. Autoscaling : All of your Flink SQL statements on Confluent Cloud are monitored continuously and [auto-scaled](concepts/autopilot.md#flink-sql-autopilot) to keep up with the rate of their input topics. The resources required by a statement depend on its complexity and the throughput of topics it reads from. Usage-based billing : You pay only for what you use, not what you provision. Flink compute in Confluent Cloud is elastic: once you stop using the compute resources, they are deallocated, and you no longer pay for them. Coupled with the elasticity provided by scale-to-zero, you can benefit from unbounded scalability while maintaining cost efficiency. For more information, see [Billing](concepts/flink-billing.md#flink-sql-billing). ## Confluent Cloud for Apache Flink is enterprise-ready  Confluent has integrated Flink deeply with Confluent Cloud to provide an enterprise-ready experience that enables data discovery and processing using familiar SQL semantics. Confluent Cloud for Apache Flink includes integrated monitoring, [role-based access control](operate-and-deploy/flink-rbac.md#flink-rbac), and [private networking](concepts/flink-private-networking.md#flink-sql-private-networking) for production workloads. ### Confluent Cloud for Apache Flink is a regional service Confluent Cloud for Apache Flink is a regional service. When you run a Flink SQL statement, Confluent Cloud for Apache Flink automatically uses a default [compute pool](concepts/compute-pools.md#flink-sql-compute-pools) in your environment and region. Compute pools represent a set of resources that scale automatically between zero and their maximum size to provide all of the power required by your statements. A compute pool is bound to a region, and the resources provided by a compute pool are shared among all statements that use them. For advanced use cases like workload isolation or cost control, you can create compute pools manually. See [Compute Pools in Confluent Cloud for Apache Flink](concepts/compute-pools.md#flink-sql-compute-pools) for more information. You can query data in any topic in your Confluent Cloud organization, even if the data is in a different environment, as long as it’s in the same region. This enables Flink to do cross-cluster, cross-environment queries while providing low latency. Of course, access control with [RBAC](operate-and-deploy/flink-rbac.md#flink-rbac) still determines the data that can be read or written. Flink can read from and write to any Kafka cluster in the same region, but by design, Confluent Cloud doesn’t allow you to query across regions. This helps you to avoid expensive data transfer charges, and also protects data locality and sovereignty by keeping reads and writes in-region. For a list of available regions, see [Supported Cloud Regions](reference/cloud-regions.md#flink-cloud-regions). ### Metadata mapping between Kafka cluster, topics, schemas, and Flink Kafka topics and schemas are always in sync with Flink, simplifying how you can process your data. Any topic created in Kafka is visible directly as a table in Flink, and any table created in Flink is visible as a topic in Kafka. Effectively, Flink provides a SQL interface on top of Confluent Cloud. Because Flink follows the SQL standard, the terminology is slightly different from Kafka. The following table shows the mapping between Kafka and Flink terminology. | Kafka | Flink | Notes | |----------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Environment | Catalog | Flink can query and join data that are in any environments/catalogs | | Cluster | Database | Flink can query and join data that are in different clusters/databases | | Topic + Schema | Table | Kafka topics and Flink tables are always in sync. You never need to declare tables manually for existing topics. Creating a table in Flink creates a topic and the associated schema. | As a result, when you start using Flink, you can directly access all of the environments, clusters, and topics that you already have in Confluent Cloud, without any additional metadata creation.  Compared with Apache Flink, the main difference is that the [Data Definition Language (DDL) statements](reference/statements/overview.md#flink-sql-statements-overview) related to catalogs, databases, and tables act on physical objects and not only on metadata. For example, when you create a table in Flink, the corresponding topic and schema are created immediately in Confluent Cloud. Confluent Cloud provides a unified approach to metadata management. There is one object definition, and Flink integrates directly with this definition, avoiding unnecessary duplication of metadata and making all topics immediately queryable with Flink SQL. Also, any existing schemas in [Schema Registry](../sr/schemas-manage.md#sr-prv) are used to surface fully-defined entities in Confluent Cloud. If you’re already on Confluent Cloud, you see tables automatically that are ready to query using Flink, simplifying data discovery and exploration. ### Observability Confluent Cloud provides you with a curated set of metrics, exposing them through Confluent’s existing [Metrics API](../monitoring/metrics-api.md#metrics-api). If you have established observability platforms in place, Confluent Cloud provides first-class integrations with New Relic, Datadog, Grafana Cloud, and Dynatrace. You can also monitor workloads directly within the Confluent Cloud Console. Clicking into a [compute pool](concepts/compute-pools.md#flink-sql-compute-pools) gives you insight into the health and performance of your applications, in addition to the resource consumption of your compute pool. ### Security Confluent Cloud for Apache Flink has a deep integration with [Role-Based Access Control (RBAC)](operate-and-deploy/flink-rbac.md#flink-rbac), ensuring that you can easily access and process the data that you have access to, and no other data. #### Access from Flink to the data - For ad-hoc queries, you can use your user account, because the permissions of the current user are applied automatically without any additional setting needed. - For long-running statements that need to run 24/7, like [INSERT INTO](reference/queries/insert-into-from-select.md#flink-sql-insert-into-from-select-statement), you should use a [service account](../security/authenticate/workload-identities/service-accounts/overview.md#service-accounts), so the statements are not affected by a user leaving the company or changing teams. #### Access to Flink To manage Flink access, Confluent has introduced two roles. In both cases, RBAC of the user on the underlying data is still applied. - [FlinkDeveloper](../security/access-control/rbac/predefined-rbac-roles.md#flinkdeveloper-role): Granted by default to all users. Enables querying data and managing their own statements. - [FlinkAdmin](../security/access-control/rbac/predefined-rbac-roles.md#flinkadmin-role): Enables creating and managing explicit Flink compute pools for workload isolation or cost control. #### Service accounts Service accounts are available for running statements permanently. If you want to run a statement with service account permissions, an OrganizationAdmin must create an **Assigner** role binding for the user on the service account. For more information, see [Production workloads (service accounts)](operate-and-deploy/flink-rbac.md#flink-rbac-grant-sa-and-user-permission-for-sql-statements). #### Private networking Confluent Cloud for Apache Flink supports private networking on AWS, Azure, and Google Cloud, providing a simple, secure, and flexible solution that enables new scenarios while keeping your data securely in private networking. All Kafka cluster types are supported, with any type of connectivity (public, Private Links, VPC Peering, and Transit Gateway). For more information, see [Private Networking with Flink](concepts/flink-private-networking.md#flink-sql-private-networking). ## Cross-environment queries Flink can perform cross-environment queries when using both public and private networking. This can be useful if you want to enable a single networking route from your VPC or VNET.  In this case, you can use a single environment and a single PLATT where you run all your Flink workloads and use three-part name queries, to query data in other environments, for example: ```sql SELECT * FROM `myEnvironment`.`myDatabase`.`myTable`; ``` As a result, a single routing rule is necessary on the VPC or VNet side, per region, to redirect all traffic to the Flink regional endpoint(s) using this PrivateLink Attachment Connection. To isolate different workloads, you can create compute pools manually, which enables you to control budget and scale of these workloads independently. See [Compute Pools in Confluent Cloud for Apache Flink](concepts/compute-pools.md#flink-sql-compute-pools) for more information. Data access is protected by RBAC at the Kafka cluster (Flink database) or Kafka topic (Flink table) level. If your user account or service account that runs the query doesn’t have access, Flink can’t access sources and destinations. To access Flink statements and workspaces, you must access them from a public IP address, if authorized, or from a PLATT or Confluent Cloud Network from the same environment and region. Flink statements themselves can then access all the environments in the same organization and region. ## Program Flink with SQL, Java, and Python Confluent Cloud for Apache Flink supports programming your streaming applications in these languages: - [SQL](reference/sql-syntax.md#flink-sql-syntax) - [Java Table API](get-started/quick-start-java-table-api.md#flink-java-table-api-quick-start) - [Python Table API](get-started/quick-start-python-table-api.md#flink-python-table-api-quick-start) Also, you can create custom user-defined functions and call them in your SQL statements. For more information, see [User-defined Functions](concepts/user-defined-functions.md#flink-sql-udfs). #### NOTE The Flink Table API is available for preview. A Preview feature is a Confluent Cloud component that is being introduced to gain early feedback from developers. Preview features can be used for evaluation and non-production testing purposes or to provide feedback to Confluent. The warranty, SLA, and Support Services provisions of your agreement with Confluent do not apply to Preview features. Confluent may discontinue providing preview releases of the Preview features at any time in Confluent’s’ sole discretion. Comments, questions, and suggestions related to the Table API are encouraged and can be submitted through the [established channels](get-help.md#ccloud-flink-help). ## Confluent for VS Code Install [Confluent for VS Code](../client-apps/vs-code-extension.md#cc-vscode-extension) to access Smart Project Templates that accelerate project setup by providing ready-to-use templates tailored for common development patterns. These templates enable you to launch new projects quickly with minimal configuration, significantly reducing setup time. ## Next steps - [Get Started](get-started/overview.md#flink-sql-get-started) - [Quick Start with Cloud Console](get-started/quick-start-cloud-console.md#flink-sql-quick-start-cloud-console) - [Quick Start with Flink SQL Shell](get-started/quick-start-shell.md#flink-sql-quick-start-shell) - [Java Table API Quick Start](get-started/quick-start-java-table-api.md#flink-java-table-api-quick-start) - [Python Table API Quick Start](get-started/quick-start-python-table-api.md#flink-python-table-api-quick-start) ## Related content - [Stream Processing Concepts](concepts/overview.md#flink-sql-stream-processing-concepts) - [How-to Guides](how-to-guides/overview.md#flink-sql-how-to-guides) - [Operate and Deploy](operate-and-deploy/overview.md#flink-sql-operate-and-deploy) - [Flink SQL Reference](reference/overview.md#flink-sql-reference) - Blog post: [Introducing Confluent Cloud for Apache Flink](https://www.confluent.io/blog/introducing-flink-on-confluent-cloud/) - Blog post: [Your Guide to Flink SQL: An In-Depth Exploration](https://www.confluent.io/blog/getting-started-with-apache-flink-sql/) - Blog post: [How to Use Flink SQL, Streamlit, and Kafka: Part 1](https://www.confluent.io/blog/how-to-use-flinksql-streamlit-kafka-part-1/) - Blog post: [How to Use Flink SQL, Streamlit, and Kafka: Part 2](https://www.confluent.io/blog/how-use-flinksql-streamlit-kafka-part-2/) - Blog post: [Data Products, Data Contracts, and Change Data Capture](https://www.confluent.io/blog/implementing-streaming-data-products/) - Course: [Apache Flink SQL](https://developer.confluent.io/courses/flink-sql/overview/) - Course: [Apache Flink 101](https://developer.confluent.io/courses/apache-flink/intro/) - Course: [Building Flink Applications in Java](https://developer.confluent.io/courses/flink-java/overview/) - Course: [Apache Flink® Table API: Processing Data Streams in Java](https://developer.confluent.io/courses/flink-table-api-java/exercise-connecting-to-confluent-cloud/) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).