Get Started with Confluent Platform for Apache Flink¶

Confluent Platform for Apache Flink® is compatible with open-source Apache Flink®. To get started, you must use the Helm chart to install the Confluent Manager for Apache Flink (CMF) application and the Flink Kubernetes operator. After you have installed both, you can deploy a Flink job. The next few sections will describe necessary prerequisites and walk you through the steps to install CMF and deploy a Flink job.

For requirement and compatibility details, see Confluent Platform for Apache Flink compatibility.

Prerequisites¶

To use Confluent Platform for Apache Flink, you must meet the following prerequisites:

Kubernetes¶

Confluent Platform for Apache Flink® is designed for Kubernetes. Therefore, you must have access to a Kubernetes cluster to deploy the Flink Kubernetes operator and individual Flink jobs:

A Kubernetes cluster conforming to one of the supported versions.
kubectl installed, initialized, with the context set. You also must have the kubeconfig file configured for your cluster.
Helm 3 installed.

To learn more about deploying Flink on Kubernetes, see Kubernetes deployment.

Confluent CLI¶

If you have not installed Confluent Platform 7.8, you must install the latest version of the Confluent CLI and set the CONFLUENT_HOME environment variable. For more information, see Install the Confluent CLI.

Durable storage¶

In production, Flink requires durable storage that is readable and writeable from each Flink cluster. Durable storage is used to store consistent snapshots of the state of your Flink jobs (checkpoints and savepoints).

Examples of durable storage include:

S3-compatible storage (S3, MinIO)
Microsoft Azure Blob Storage
Google Cloud Storage

For more information on configuring durable storage, see Checkpoint Storage in the Flink documentation.

Supported components¶

Check to make sure that you are using components supported by Confluent Platform for Apache Flink. See, Confluent Platform for Apache Flink Features and Support.

Step 1: Install Confluent Manager for Apache Flink¶

To install Confluent Manager for Apache Flink, you have to use Helm to add the Confluent repository, install the certificate manager and then install the Flink Kubernetes operator, and finally install CMF.

Warning

This quickstart guide assumes that everything will be installed into the same Kubernetes namespace. In this example it is default. You can use a different namespace, as long as it is consistent across all the commands. Please refer to the Helm installation guide for more information on how to setup Confluent Platform for Apache Flink in a multi-namespace environment.

Add the Confluent Platform repository.

helm repo add confluentinc https://packages.confluent.io/helm
helm repo update
Copy

Install the certificate manager.

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
Copy

Install the Flink Kubernetes Operator

helm upgrade --install cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator
Copy

Install Confluent Manager for Apache Flink.

helm upgrade --install cmf \
confluentinc/confluent-manager-for-apache-flink \
--namespace default
Copy

Check that everything deployed correctly.
```
kubectl get pods
```
Copy

Step 2: Deploy Flink jobs¶

To deploy Flink jobs, you should open port forwarding to CMF, and create an environment and application. You can then use the Web UI to confirm that your application has been created.

Open port forwarding to CMF.

kubectl port-forward svc/cmf-service 8080:80
Copy

Create an environment using the Confluent CLI. For a list of Confluent CLI commands for Confluent Platform for Apache Flink, see confluent flink (On-Premises tab).
```
confluent flink environment create env1 --url http://localhost:8080 --kubernetes-namespace default
```
Copy

Create the JSON application file. Following is an example.

{
"apiVersion": "cmf.confluent.io/v1alpha1",
"kind": "FlinkApplication",
"metadata": {
   "name": "basic-example"
},
"spec": {
   "flinkConfiguration": {
      "metrics.reporter.prom.factory.class": "org.apache.flink.metrics.prometheus.PrometheusReporterFactory",
      "metrics.reporter.prom.port": "9249-9250",
      "taskmanager.numberOfTaskSlots": "1"
   },
   "flinkVersion": "v1_19",
   "image": "confluentinc/cp-flink:1.19.1-cp2",
   "job": {
      "jarURI": "local:///opt/flink/examples/streaming/StateMachineExample.jar",
      "parallelism": 3,
      "state": "running",
      "upgradeMode": "stateless"
   },
   "jobManager": {
      "resource": {
      "cpu": 1,
      "memory": "1024m"
      }
   },
   "serviceAccount": "flink",
   "taskManager": {
      "resource": {
      "cpu": 1,
      "memory": "1024m"
      }
   }
}
}
Copy

Use the application command, passing the application.json file to create the application.

confluent flink application create application.json --environment env1 --url http://localhost:8080
Copy

Access the Flink Web UI to confirm you have successfully created the application:

confluent flink application web-ui-forward basic-example --environment env1 --port 8090 --url http://localhost:8080
Copy

Step 3: Cleanup¶

Delete the Flink Application:

confluent flink application delete basic-example --environment env1 --url http://localhost:8080
Copy

Uninstall the Flink Kubernetes Operator and CMF:

helm uninstall cp-flink-kubernetes-operator
helm uninstall cmf
Copy