Flink SQL Autopilot for Confluent Cloud
Autopilot is the elastic autoscaling service for Confluent Cloud for Apache Flink®. It continuously adjusts every SQL statement so that it keeps up with its input data.
The resource demand of a long-running statement changes over time. Inbound traffic varies with business activity, time-based operations introduce bursty load, and a statement might alternate between reprocessing historical data and processing baseline throughput.
Autopilot adds capacity when demand increases and releases it when demand decreases, so a statement is provisioned for its current load rather than its peak load. Autopilot reevaluates and reapplies its decisions regularly. This keeps the statement caught up without overprovisioning for the worst case.
Autopilot manages the whole scaling process and covers individual data path operations, user-defined functions, state management, and auxiliary services.
Each scaling change is reflected in the number of Confluent Flink Units (CFUs) that the statement consumes.
Scaling status
The scaling status shows you how the statement resources are scaling. These are the possible scaling statuses.
Scaling Status | Description |
|---|---|
Fine | The SQL statement has enough resources to run at the required scale. |
Pending Scale Down | The SQL statement has more resources than required and will be scaled down. |
Pending Scale Up | The SQL statement doesn’t have enough resources and will be scaled up. |
Compute Pool Exhausted | There aren’t enough resources in the compute pool for the statement to run at the required scale, or the compute pool has reached the maximum number of statements that it can run concurrently. |
Scaling status is displayed on the Statement detail page and in the SQL workspace.
Compute Pool Exhausted
The Compute Pool Exhausted status indicates that the compute pool can’t provide the capacity that a statement requires for optimal performance. The affected statement can run slower or might not be able to run at all.
A compute pool can become exhausted for either of the following reasons:
The compute pool has reached its CFU limit, so there aren’t enough resources for the statement to run at the required scale.
The compute pool has reached the maximum number of statements that it can run, even if the CFU limit hasn’t been reached. When this happens, a new statement can’t start until capacity becomes available.
There are a number of ways to resolve this situation:
You can add more resources by increasing the CFU limit on the compute pool.
You can stop some running statements to free up existing resources or to free up capacity for additional statements.
You can create compute pools to separate workloads. See Compute Pools in Confluent Cloud for Apache Flink for more information.
Messages Behind
Messages Behind is a strong indicator of how the statement is performing. The overall goal of Autopilot is to ensure that the SQL statement keeps up with the throughput of the source tables and topics, and to keep Messages Behind as close to zero as possible. In Apache Kafka® terms, Messages Behind is the Consumer Lag.
A low or decreasing Messages Behind value indicates that Autopilot is doing its job successfully. The following table describes scenarios in which Autopilot is scaling resources correctly or where it might be struggling.
Messages Behind and Scaling Status | Description |
|---|---|
Messages Behind is increasing Scaling status = “Pending Scale Up” | Autopilot has identified a need for scaling up and will increase the statement resources. Once resources have been scaled up, the Messages Behind should start decreasing. |
Messages Behind is increasing Scaling status = “Fine” | There is likely a problem. Reach out to Confluent Support. For more information, see Get Help with Confluent Cloud for Apache Flink. |
Messages Behind is not increasing Compute Pool is Exhausted | The statement resources can keep up with throughput, but Autopilot needs to assign more resources to improve performance capacity. For user-created compute pools, you can add more resources by increasing the CFU limit. Alternatively, you can stop some running statements to free up existing resources. |
State size
The state size is the amount of state that a statement has accumulated. Confluent Cloud for Apache Flink enforces a soft limit of 500 GB and a hard limit of 1,000 GB on the state size for each statement. For more information, see Limits on state size.
Starting at 80% of the soft limit (400 GB), you get warnings in Confluent Cloud Console and notifications from the Metrics API. The state size also includes data in flight between operators. As a result, a limit notification can show a higher state size than the value you get by adding up individual operator state size metrics. Use the notification as a signal that the state size is approaching the limit.
Autopilot factors in state size, along with other metrics, when it adjusts resources, and it also accounts for the overhead of managing that state. The soft and hard limits add guardrails on top of this behavior.
When a statement’s state grows large, Autopilot forces scale-ups, as long as CFUs are available in the compute pool. If a statement’s state requires more CFUs than the pool can provide, Autopilot sets the scaling status to POOL_EXHAUSTED, and autoscaling stops adding CFUs before reaching the limits.
When a statement’s total state exceeds the soft limit, Autopilot sets scaling_status to POOL_EXHAUSTED and stops the statement, marking it as STOPPED due to state limit. You can resume the statement, and after you do, it can continue above 500 GB, but uptime SLAs/guarantees don’t apply after state is greater than 500 GB. Autopilot still considers state size, but your statement is beyond the supported envelope.
When the statement’s total state exceeds the soft limit but is less than the hard limit (1000 GB), Autopilot continues to use state size to prevent downscaling and forces scale-ups while CFUs are still available. In this “best effort” zone, the platform doesn’t guarantee performance or uptime.
When a statement’s total state exceeds the hard limit, the statement fails and you can’t resume it. Autopilot sets scaling_status to POOL_EXHAUSTED and removes the statement from further autoscaling. Cloud Console and notifications show a hard-limit breach with a red warning and a non-resumable state.