# Flink SQL Autopilot for Confluent Cloud Autopilot is the elastic autoscaling service for Confluent Cloud for Apache Flink®. It continuously adjusts every SQL statement so that it keeps up with its input data. The resource demand of a long-running statement changes over time. Inbound traffic varies with business activity, time-based operations introduce bursty load, and a statement might alternate between reprocessing historical data and processing baseline throughput. Autopilot adds capacity when demand increases and releases it when demand decreases, so a statement is provisioned for its current load rather than its peak load. Autopilot reevaluates and reapplies its decisions regularly. This keeps the statement caught up without overprovisioning for the worst case. Autopilot manages the whole scaling process and covers individual data path operations, user-defined functions, state management, and auxiliary services. #### NOTE Statements that use Java [user-defined functions](user-defined-functions.md#flink-sql-udfs) (UDFs) respond more slowly to large increases in workload than statements that don’t. The statement keeps processing data while Autopilot scales it up. If scaling response time affects your workload, contact Confluent Support. For more information, see [Get Help with Confluent Cloud for Apache Flink](../get-help.md#ccloud-flink-help). Each scaling change is reflected in the number of Confluent Flink Units (CFUs) that the statement consumes. ## Scaling status The scaling status shows you how the statement resources are scaling. These are the possible scaling statuses. | Scaling Status | Description | |------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Fine | The SQL statement has enough resources to run at the required scale. | | Pending Scale Down | The SQL statement has more resources than required and will be scaled down. | | Pending Scale Up | The SQL statement doesn’t have enough resources and will be scaled up. | | Compute Pool Exhausted | There aren’t enough resources in the compute pool for the statement to run at the required scale, or the compute pool has reached the maximum number of statements that it can run concurrently. | Scaling status is displayed on the Statement detail page and in the SQL workspace. ### Compute Pool Exhausted The Compute Pool Exhausted status indicates that the compute pool can’t provide the capacity that a statement requires for optimal performance. The affected statement can run slower or might not be able to run at all. A compute pool can become exhausted for either of the following reasons: - The compute pool has reached its CFU limit, so there aren’t enough resources for the statement to run at the required scale. - The compute pool has reached the maximum number of statements that it can run, even if the CFU limit hasn’t been reached. When this happens, a new statement can’t start until capacity becomes available. There are a number of ways to resolve this situation: - You can add more resources by increasing the [CFU limit](flink-billing.md#flink-sql-cfus) on the compute pool. - You can stop some running statements to free up existing resources or to free up capacity for additional statements. - You can create compute pools to separate workloads. See [Compute Pools in Confluent Cloud for Apache Flink](compute-pools.md#flink-sql-compute-pools) for more information. ## Messages Behind Messages Behind is a strong indicator of how the statement is performing. The overall goal of Autopilot is to ensure that the SQL statement keeps up with the throughput of the source tables and topics, and to keep Messages Behind as close to zero as possible. In Apache Kafka® terms, Messages Behind is the [Consumer Lag](../../monitoring/monitor-lag.md#cloud-monitoring-lag). A low or decreasing Messages Behind value indicates that Autopilot is doing its job successfully. The following table describes scenarios in which Autopilot is scaling resources correctly or where it might be struggling. | Messages Behind and
Scaling Status | Description | |--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Messages
Behind is
increasing

Scaling status
= “Pending
Scale Up” | Autopilot has identified a need for scaling up and will increase the
statement resources. Once resources have been scaled up, the Messages
Behind should start decreasing. | | Messages
Behind is
increasing

Scaling status
= “Fine” | There is likely a problem. Reach out to Confluent Support. For more
information, see [Get Help with Confluent Cloud for Apache Flink](../get-help.md#ccloud-flink-help). | | Messages
Behind is not
increasing

Compute Pool
is Exhausted | The statement resources can keep up with throughput, but Autopilot
needs to assign more resources to improve performance capacity.
For user-created compute pools, you can add more resources by
increasing the CFU limit. Alternatively, you can stop some running
statements to free up existing resources. | ## State size The state size is the amount of state that a statement has accumulated. Confluent Cloud for Apache Flink enforces a soft limit of 500 GB and a hard limit of 1,000 GB on the state size for each statement. For more information, see [Limits on state size](statements.md#flink-sql-statements-state-size-limits). Starting at 80% of the soft limit (400 GB), you get warnings in Confluent Cloud Console and notifications from the Metrics API. The state size also includes data in flight between operators. As a result, a limit notification can show a higher state size than the value you get by adding up individual operator state size metrics. Use the notification as a signal that the state size is approaching the limit. Autopilot factors in state size, along with other metrics, when it adjusts resources, and it also accounts for the overhead of managing that state. The soft and hard limits add guardrails on top of this behavior. When a statement’s state grows large, Autopilot forces scale-ups, as long as CFUs are available in the compute pool. If a statement’s state requires more CFUs than the pool can provide, Autopilot sets the scaling status to POOL_EXHAUSTED, and autoscaling stops adding CFUs before reaching the limits. When a statement’s total state exceeds the soft limit, Autopilot sets `scaling_status` to POOL_EXHAUSTED and stops the statement, marking it as STOPPED due to state limit. You can resume the statement, and after you do, it can continue above 500 GB, but uptime SLAs/guarantees don’t apply after state is greater than 500 GB. Autopilot still considers state size, but your statement is beyond the supported envelope. When the statement’s total state exceeds the soft limit but is less than the hard limit (1000 GB), Autopilot continues to use state size to prevent downscaling and forces scale-ups while CFUs are still available. In this “best effort” zone, the platform doesn’t guarantee performance or uptime. When a statement’s total state exceeds the hard limit, the statement fails and you can’t resume it. Autopilot sets `scaling_status` to POOL_EXHAUSTED and removes the statement from further autoscaling. Cloud Console and notifications show a hard-limit breach with a red warning and a non-resumable state. ## Baseline CFU By default, Autopilot scales a statement down when demand is low, to as little as 1 CFU. A *Baseline CFU* sets a minimum scaling target for a single statement. After you set it, Autopilot keeps the statement at or above the Baseline CFU as a best-effort lower bound. Autopilot still scales up freely above it. #### NOTE Baseline CFU is a Limited Availability feature in Confluent Cloud. It applies only to Flink statements, not to [materialized tables](materialized-tables.md#flink-sql-materialized-tables), and you manage it in Confluent Cloud Console or with the Flink SQL REST API. Terraform and the Confluent CLI aren’t supported. Set a Baseline CFU when a statement needs predictable performance during quiet periods, or headroom to absorb a sudden increase in load without waiting for a scale-up. Latency-sensitive statements and statements with bursty, time-based traffic are typical candidates. Autopilot picks the smallest uniform CFU value at or above the Baseline CFU, so a statement often settles above the number you set. The Baseline CFU is a best-effort lower bound, not a guarantee: if the compute pool can’t supply the capacity, the statement runs below it. A Baseline CFU affects your bill. A statement is billed for at least its Baseline CFU even when it’s idle, and usage above the floor is billed on top. For more information, see [Billing on Confluent Cloud for Apache Flink](flink-billing.md#flink-sql-billing). You manage the Baseline CFU on a statement in Confluent Cloud Console or with the [Flink SQL REST API](../operate-and-deploy/flink-rest-api.md#flink-rest-api). For step-by-step instructions, see [Set a Baseline CFU for a Flink SQL Statement](../how-to-guides/manage-baseline-cfu.md#flink-sql-manage-baseline-cfu). #### NOTE Changing the Baseline CFU triggers a rescale, so the statement pauses briefly while capacity changes, and the change can take up to 10 minutes to take effect. You don’t need to stop or resume the statement. ## Related content - [Compute Pools](compute-pools.md#flink-sql-compute-pools) - [DDL Statements](statements.md#flink-sql-statements) - [Determinism in Continuous Queries](determinism.md#flink-sql-determinism) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).