# Flink SQL Development Lifecycle in Confluent Cloud for Apache Flink A Flink SQL *development lifecycle* on Confluent Cloud for Apache Flink® moves statements through four stages: develop, test, deploy, and monitor. Structuring your workflow around these stages, with isolated Confluent Cloud environments for each, helps your team deliver reliable streaming applications and manage them over time. ## Overview A typical Flink SQL development lifecycle involves the following stages: 1. **Develop**: Write and iterate on Flink SQL statements in a development environment using the [Confluent for VS Code](../../client-apps/vs-code-extension.md#cc-vscode-extension) extension, the Cloud Console SQL workspace, or the Confluent CLI Flink shell. You can also author your pipelines in Java with the [Table API](../reference/table-api.md#flink-table-api) from your IDE. 2. **Test**: Validate your statements against realistic data in a staging environment before promoting to production. 3. **Deploy**: Use an infrastructure-as-code tool such as Terraform or dbt to deploy statements programmatically through a CI/CD pipeline, or deploy [Table API programs](table-api-deploy.md#flink-table-api-deploy) by running their JAR from your pipeline. 4. **Monitor**: Observe statement health, resource usage, and data quality in production. ## Organize environments Use separate Confluent Cloud environments to isolate development, staging, and production workloads. Each environment has its own Kafka clusters, [compute pools](../concepts/compute-pools.md#flink-sql-compute-pools), and access controls, which prevents development activity from affecting production systems. A common pattern is to create three environments: - **Development**: For interactive exploration and rapid iteration. The [Confluent for VS Code](../../client-apps/vs-code-extension.md#cc-vscode-extension) extension provides IntelliSense for Flink SQL authoring, the ability to submit statements and view results directly from your editor, and built-in templates for starting new Flink projects and UDFs. You can also use the Cloud Console SQL workspace or the Confluent CLI Flink shell for interactive development. - **Staging**: For integration testing with realistic data volumes and schemas. Deploy statements using the same tooling and process you use in production. - **Production**: For running your live streaming applications. Use [service account API keys](best-practices.md#flink-sql-best-practices-for-statements-sa-keys) and [RBAC](flink-rbac.md#flink-rbac) to control access. You can provision these environments using the Confluent Terraform Provider or the Confluent CLI. For an example of creating environments with Terraform, see [Deploy a Flink SQL Statement Using CI/CD and Confluent Cloud for Apache Flink](../how-to-guides/deploy-flink-sql-statement.md#flink-deploy-sql-statement). ## Develop with Confluent for VS Code The [Confluent for VS Code](../../client-apps/vs-code-extension.md#cc-vscode-extension) extension integrates your Flink SQL development workflow directly into your editor. With the extension, you can: - Start new Flink projects and UDF projects from built-in Smart Project Templates. - Author Flink SQL statements with IntelliSense support for syntax completion and validation. - Submit Flink SQL statements to Confluent Cloud and view results without leaving your editor. - Develop, build, and test UDF artifacts locally before uploading them to Confluent Cloud. - Browse and manage your Confluent Cloud resources, including topics, schemas, and compute pools. The extension is available on the [VS Code Marketplace](https://marketplace.visualstudio.com/) for VS Code and VS Code Insiders, and on the [OpenVSX Registry](https://open-vsx.org/) for Cursor and other VS Code-derived editors. ## Structure your project Store your Flink SQL statements in version control alongside your deployment configuration. A well-organized project makes it easier to review changes, track history, and reproduce deployments. The following example shows a project structure for Terraform-managed Flink SQL statements: ```text my-flink-project/ ├── main.tf # Provider config and shared resources ├── variables.tf # Input variables (API keys, regions) ├── environments/ │ ├── dev.tfvars # Dev environment settings │ ├── staging.tfvars # Staging environment settings │ └── prod.tfvars # Production environment settings └── statements/ ├── enrichment.tf # Enrichment pipeline statements └── aggregation.tf # Aggregation pipeline statements ``` If you use dbt to manage your Flink SQL statements, see [Deploy Flink SQL Statements with dbt and Confluent Cloud for Apache Flink](deploy-flink-dbt.md#flink-deploy-dbt) for the project structure. ## Version control your statements Treat your Flink SQL statements as code. Store them in a Git repository and use pull requests to review changes before deploying them. This practice provides: - **Change history**: Track who changed what and when. - **Code review**: Catch errors and ensure consistency before deployment. - **Rollback capability**: Revert to a previous version if a deployment causes issues. - **Collaboration**: Enable multiple team members to work on different parts of the pipeline simultaneously. ## Test your statements Before deploying to production, validate your Flink SQL statements in a staging environment: - **Schema validation**: Ensure that your statements are compatible with the current schema versions in Schema Registry. Consider using `FULL_TRANSITIVE` compatibility to prevent breaking changes. For more information, see [Choose the correct Schema Registry compatibility type](best-practices.md#flink-sql-best-practices-for-statements-compatibility-type). - **Dry-run validation**: Use the `--dry-run` flag with the Confluent CLI or the REST API to validate statement syntax without executing it. - **Integration testing**: Run statements against a staging environment with representative data to verify correctness and performance before promoting to production. - **Watermark validation**: Confirm that your watermark strategy produces correct results. For more information, see [Validate your watermark strategy](best-practices.md#flink-sql-best-practices-for-statements-validate-watermarks). - **Unit testing with dbt**: If you use dbt, write unit tests that verify model logic with mock data before deploying to any environment. Run `dbt test --select "test_type:unit"` to execute unit tests. For more information, see [Step 5: Test your models](deploy-flink-dbt.md#flink-deploy-dbt-test). - **Unit testing with the Table API**: If you author pipelines with the Java Table API, unit-test your business logic and pipeline functions locally with JUnit, and run integration tests against a staging environment. For more information, see [Deploy and Manage Table API Programs on Confluent Cloud for Apache Flink](table-api-deploy.md#flink-table-api-deploy). ## Choose a deployment tool Confluent supports several tools for deploying Flink SQL statements programmatically: | Tool | Best for | Learn more | |-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Terraform** | Infrastructure-as-code deployments where you manage Confluent Cloud resources
(environments, clusters, compute pools) alongside your Flink SQL
statements. | [Deploy a Flink SQL Statement Using CI/CD and Confluent Cloud for Apache Flink](../how-to-guides/deploy-flink-sql-statement.md#flink-deploy-sql-statement) | | **dbt** | Teams already using dbt for analytics workflows who want to manage
Flink SQL transformations as dbt models. | [Deploy Flink SQL Statements with dbt and Confluent Cloud for Apache Flink](deploy-flink-dbt.md#flink-deploy-dbt) | | **Table API** | Teams that author pipelines in Java and deploy them as regular
applications: the same JAR submits statements and manages their
lifecycle from a CI/CD pipeline. | [Deploy and Manage Table API Programs on Confluent Cloud for Apache Flink](table-api-deploy.md#flink-table-api-deploy) | | **REST API** | Custom integrations and programmatic access from any language or
platform. | [Flink SQL REST API for Confluent Cloud for Apache Flink](flink-rest-api.md#flink-rest-api) | | **Confluent CLI** | Scripted deployments and automation in shell-based CI/CD pipelines. | [Confluent CLI commands with Confluent Cloud for Apache Flink](../reference/flink-sql-cli.md#flink-sql-confluent-cli) | ## Automate with CI/CD Integrate your deployment tool with a CI/CD system like GitHub Actions, GitLab CI, or Jenkins to automate the promotion of Flink SQL statements across environments. A typical CI/CD pipeline includes: 1. **Trigger**: A push to a specific branch or a pull request merge. 2. **Validate**: Run syntax checks and dry-run validation. 3. **Plan**: Preview the changes to apply (for example, `terraform plan`). 4. **Apply**: Deploy the changes to the target environment. 5. **Verify**: Confirm that statements are running and producing expected output. For a complete CI/CD example using Terraform and GitHub Actions, see [Deploy a Flink SQL Statement Using CI/CD and Confluent Cloud for Apache Flink](../how-to-guides/deploy-flink-sql-statement.md#flink-deploy-sql-statement). For dbt-based CI/CD, see [Deploy Flink SQL Statements with dbt and Confluent Cloud for Apache Flink](deploy-flink-dbt.md#flink-deploy-dbt). For CI/CD with Java Table API programs, see [Deploy and Manage Table API Programs on Confluent Cloud for Apache Flink](table-api-deploy.md#flink-table-api-deploy). ## Monitor in production After deployment, monitor your Flink SQL statements to ensure they are running correctly: - Use the Cloud Console to [monitor statement status](monitor-statements.md#flink-sql-monitor-statements-with-cloud-console). - [Assign custom names](best-practices.md#flink-sql-best-practices-for-statements-custom-names) to your statements so they are easy to identify. - Review [error handling and monitoring best practices](best-practices.md#flink-sql-best-practices-monitoring). - Use the [Query Profiler](query-profiler.md#flink-sql-query-profiler) to analyze statement performance. ## Related content - [Deploy a Flink SQL Statement Using CI/CD and Confluent Cloud for Apache Flink](../how-to-guides/deploy-flink-sql-statement.md#flink-deploy-sql-statement) - [Deploy Flink SQL Statements with dbt and Confluent Cloud for Apache Flink](deploy-flink-dbt.md#flink-deploy-dbt) - [Deploy and Manage Table API Programs on Confluent Cloud for Apache Flink](table-api-deploy.md#flink-table-api-deploy) - [Move SQL Statements to Production in Confluent Cloud for Apache Flink](best-practices.md#flink-sql-best-practices-for-statements) - [Grant Role-Based Access in Confluent Cloud for Apache Flink](flink-rbac.md#flink-rbac) - [Compute Pools in Confluent Cloud for Apache Flink](../concepts/compute-pools.md#flink-sql-compute-pools) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).