Frequently Asked Questions for Confluent Cloud for Apache Flink

Q: How do I get started with Confluent Cloud for Apache Flink?

Get started by clicking SQL Workspaces in the Confluent Cloud Console. For more information, see Flink SQL Quick Start with Confluent Cloud Console. Also, you can run the confluent flink shell command to start the Flink SQL shell. For more information, see Flink SQL Shell Quick Start.

Q: Does dropping a table also delete the Kafka topic?

Yes, in most cases. In Apache Flink, DROP TABLE only removes catalog metadata. In Confluent Cloud for Apache Flink, dropping a table backed by an Kafka topic permanently deletes the underlying topic and every record in it. Depending on your topic naming strategy, the associated schemas in Schema Registry are also deleted. Stop any dependent statements and confirm that you no longer need the data before running DROP TABLE. For the full behavior, including how external tables differ, see the DROP TABLE statement reference.

This topic provides answers to frequently asked questions about Confluent Cloud for Apache Flink®.

What is Confluent Cloud for Apache Flink?

Confluent Cloud for Apache Flink is a fully managed, cloud-native service for stream processing using Flink SQL. It enables you to process, analyze, and transform data in real time directly on your Confluent Cloud-managed Kafka clusters.

How do I get started with Confluent Cloud for Apache Flink?

Get started by clicking SQL Workspaces in the Confluent Cloud Console. For more information, see Flink SQL Quick Start with Confluent Cloud Console.

Also, you can run the confluent flink shell command to start the Flink SQL shell. For more information, see Flink SQL Shell Quick Start.

What is a compute pool?

A compute pool is a dedicated set of resources, measured in CFUs, that runs your Flink SQL statements. You must create a compute pool before running statements. Multiple statements can share a compute pool, and you can scale pools up or down as needed. For more information, see Compute Pools.

What is the maximum size of a compute pool?

Each compute pool has a configurable maximum capacity, up to 50 CFUs per pool.

Note

Compute pools with up to 1,000 CFU capacity are available as a Limited Availability feature for customers with large Flink job fleets. To participate in the Limited Availability Program, sign up at Scale Apache Flink® compute pools to 1,000 CFUs.

The maximum size of a single job remains 50 CFUs. The total CFU usage of all concurrently running jobs in a pool cannot exceed the pool’s CFU capacity. For example, a 1,000 CFU pool can run up to twenty 50-CFU jobs concurrently, or a larger number of smaller jobs, as long as the total CFUs in use do not exceed 1,000.

With higher CFU limits available during Limited Availability, you can consolidate multiple smaller compute pools into a single larger pool to simplify architecture and management. For more information about moving statements between pools, see Update metadata for a statement.

How is Confluent Cloud for Apache Flink billed?

Billing is based on the number of CFUs provisioned in your compute pools and the duration for which they are running. You are charged for the resources allocated, not per statement. For more information, see Billing.

What are the prerequisites for using Confluent Cloud for Apache Flink?

You need a Confluent Cloud account and an environment with Stream Governance enabled.
You must have the appropriate roles and permissions, for example, the FlinkDeveloper role to run statements.
You need access to at least one compute pool.

What sources and sinks are supported?

Confluent Cloud for Apache Flink supports reading from and writing to Kafka topics in your Confluent Cloud environment. In addition, you use Confluent’s AI/ML features to perform searches on external tables. And you can use Confluent Tableflow to materialize streams to external tables.

Does dropping a table also delete the Kafka topic?

Yes, in most cases. In Apache Flink, DROP TABLE only removes catalog metadata. In Confluent Cloud for Apache Flink, dropping a table backed by an Kafka topic permanently deletes the underlying topic and every record in it. Depending on your topic naming strategy, the associated schemas in Schema Registry are also deleted. Stop any dependent statements and confirm that you no longer need the data before running DROP TABLE. For the full behavior, including how external tables differ, see the DROP TABLE statement reference.

Can I run custom Flink jobs using the DataStream API?

No. Confluent Cloud for Apache Flink supports Flink SQL, the Java and Python Table APIs, and custom user-defined functions, but it doesn’t support the DataStream API or submitting custom JAR-based Flink jobs. For the full list of supported programming interfaces, see Program Flink with SQL, Java, and Python. If your use case requires the DataStream API, consider Confluent Platform for Apache Flink, which runs on infrastructure that you manage.

How do I monitor my Flink SQL statements?

You can monitor statements using the Cloud Console, which provides status, metrics, and logs. For advanced monitoring, use the Metrics API and Notifications for Confluent Cloud to set up alerts for failures, lag, and resource utilization. For more information, see Best practices for alerting.

What happens if my statement fails?

If a statement fails, you will see an error message in the Cloud Console. You can view logs and metrics to diagnose the issue. Statements can be restarted after resolving the underlying problem.

Why does my statement stay in PENDING for a long time?

A statement can stay in PENDING briefly while Flink provisions resources for it. Confluent Cloud for Apache Flink sends a notification if a statement remains PENDING longer than expected: 10 minutes on AWS and Google Cloud, or 30 minutes on Azure. For more information about this and other statement notifications, see Notifications. A prolonged PENDING state is often caused by a compute pool that has reached its maximum CFU limit. Check your compute pool’s CFU usage, and consider increasing its maximum capacity or moving the statement to a pool with available capacity.

Can I use Flink SQL to join multiple topics?

Yes, you can use Flink SQL to join multiple Kafka topics, perform aggregations, windowing, filtering, and more. For more information, see the Flink SQL statements.

How do I manage schema evolution?

Flink SQL integrates with Confluent’s Schema Registry. When reading from or writing to topics with Avro, Protobuf, or JSON Schema, Flink SQL uses the registered schemas and handles compatible schema evolution.

How do I control access to Flink resources?

Access to Flink resources is managed using Role-Based Access Control (RBAC) in Confluent Cloud. Assign users and service accounts the appropriate roles, such as FlinkAdmin or FlinkDeveloper, to control what actions they can perform. For more information, see Grant Role-Based Access.

How do I secure my Flink SQL jobs and data?

Confluent Cloud for Apache Flink uses the same security model as the rest of Confluent Cloud, including RBAC, API keys, and network controls. Make sure to assign the minimum required permissions to users and service accounts. For more information, see Grant Role-Based Access in Confluent Cloud for Apache Flink.

How do I move my SQL statements to production?

To move your Flink SQL statements to production, follow best practices such as using service accounts, applying least-privilege permissions, and thoroughly testing your statements in a development environment before deploying them to production compute pools. For detailed guidance, see Best Practices for Moving SQL Statements to Production.

You can use GitHub Actions and Terraform to deploy your Flink SQL statements to production. For more information, see Deploy a Flink SQL Statement Using CI/CD. If your team uses dbt, you can manage Flink SQL transformations as dbt models. For more information, see Deploy Flink SQL Statements with dbt and Confluent Cloud for Apache Flink.

Where can I get help or support?

If you have questions or need support, you can use the in-product help in the Confluent Cloud Console, visit the Flink documentation, or reach out through the established channels. You can also ask questions in the Confluent Community forums or contact Confluent Support if you have a support plan.