Monitor Flink SQL Statements in Confluent Cloud for Apache Flink¶
You start a stream-processing app on Confluent Cloud for Apache Flink®️ by running a SQL statement. Once a statement is running, you can monitor its progress by using the Confluent Cloud Console. Also, you can set up integrations with monitoring services like Prometheus and Datadog.
View statement details in Cloud Console¶
Cloud Console shows details about your statements on the Flink page.
If you don’t have running statements currently, run a SQL query like INSERT INTO FROM SELECT in the Flink SQL shell or in a workspace.
Log in to the Confluent Cloud Console.
Navigate to the Environments page.
Click the tile that has the environment where your Flink compute pools are provisioned.
Click Flink, and in the Flink page, click Flink statements.
The Statements list opens.
You can filter for running statements by clicking the Status filter above the statements list.
For each statement, you can see the rate at which records are read (Messages in per min) and written (Messages out per min). Also, the consumer lag for each statement is displayed (Messages behind). For all three values, the sum over all inputs for output tables/topics and partitions is shown.
A low, close-to-zero consumer lag is the best indicator that your statement is running smoothly and keeping up with all of its inputs. A growing consumer lag indicates a problem: check the exceptions (see next steps).
When you click on a particular statement, the details view opens. In the Overview section, the statement’s SQL source code is displayed.
Click Exceptions.
The error log opens. If your statement is experiencing issues, errors and runtime exceptions are listed with the corresponding timestamp for when it occurred.
Click X to dismiss the details view.
Click Compute pools.
The details view opens and shows details for the current Flink compute pool.
Flink metrics integrations¶
Confluent Cloud for Apache Flink supports metrics integrations with services like Prometheus and Datadog.
If you don’t have running statements currently, run a SQL query like INSERT INTO FROM SELECT in the Flink SQL shell or in a workspace.
Log in to the Confluent Cloud Console.
Open the Administration menu (
) and select Metrics to
open the Metrics integration page.In the Explore available metrics section, click the Metric dropdown.
Scroll until you find the Flink compute pool and Flink statement metrics, for example, Messages behind.
Click the Resource dropdown and select the corresponding compute pool or statement that you want to monitor.
A graph showing the most recent data for your selected Flink metric displays.
Click New integration to export your metrics to a monitoring service. For more information, see Integrate with third-party monitoring.