Get Started with Confluent Platform for Apache Flink¶

Confluent Platform for Apache Flink® is compatible with open-source Apache Flink®. To get started, you must use the Helm chart to install the Confluent Manager for Apache Flink (CMF) application and the Flink Kubernetes operator. After you have installed both, you can deploy a Flink job. The next few sections will describe necessary prerequisites and walk you through the steps to install CMF and deploy a Flink job.

For requirement and compatibility details, see Confluent Platform for Apache Flink compatibility.

Prerequisites¶

To use Confluent Platform for Apache Flink, you must meet the following prerequisites:

Kubernetes¶

Confluent Platform for Apache Flink® is designed for Kubernetes. Therefore, you must have access to a Kubernetes cluster to deploy the Flink Kubernetes operator and individual Flink jobs:

A Kubernetes cluster conforming to one of the supported versions.
kubectl installed, initialized, with the context set. You also must have the kubeconfig file configured for your cluster.
Helm 3 installed.

To learn more about deploying Flink on Kubernetes, see Kubernetes deployment.

Confluent CLI¶

If you have not installed Confluent Platform 7.8, you must install the latest version of the Confluent CLI and set the CONFLUENT_HOME environment variable. For more information, see Install the Confluent CLI.

Durable storage¶

In production, Flink requires durable storage that is readable and writeable from each Flink cluster. Durable storage is used to store consistent snapshots of the state of your Flink jobs (checkpoints and savepoints).

Examples of durable storage include:

S3-compatible storage (S3, MinIO)
Microsoft Azure Blob Storage
Google Cloud Storage

For more information on configuring durable storage, see Checkpoint Storage in the Flink documentation.

Supported components¶

Check to make sure that you are using components supported by Confluent Platform for Apache Flink. See, Confluent Platform for Apache Flink Features and Support.

Step 1: Install Confluent Manager for Apache Flink¶

To install Confluent Manager for Apache Flink, you have to use Helm to add the Confluent repository, install the certificate manager and then install the Flink Kubernetes operator, and finally install CMF.

Add the Confluent Platform repository.

helm repo add confluentinc https://packages.confluent.io/helm
helm repo update

Install the certificate manager.

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Install the Flink Kubernetes Operator

helm upgrade --install cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator

Install Confluent Manager for Apache Flink.

helm upgrade --install cmf \
confluentinc/confluent-manager-for-apache-flink \
--namespace <namespace>

Check that everything deployed correctly.
```
kubectl get pods
```

Step 2: Deploy Flink jobs¶

To deploy Flink jobs, you should open port forwarding to CMF, and create an environment and application. You can then use the Web UI to confirm that your application has been created.

Open port forwarding to CMF.

kubectl port-forward svc/cmf-service 8080:80

Create an environment using the Confluent CLI. For a list of Confluent CLI commands for Confluent Platform for Apache Flink, see confluent flink (On-Premises tab).
```
confluent flink environment create env1 --url http://localhost:8080 --kubernetes-namespace default
```

Create the JSON application file. Following is an example.

{
"apiVersion": "cmf.confluent.io/v1alpha1",
"kind": "FlinkApplication",
"metadata": {
   "name": "basic-example"
},
"spec": {
   "flinkConfiguration": {
      "metrics.reporter.prom.factory.class": "org.apache.flink.metrics.prometheus.PrometheusReporterFactory",
      "metrics.reporter.prom.port": "9249-9250",
      "taskmanager.numberOfTaskSlots": "1"
   },
   "flinkVersion": "v1_19",
   "image": "confluentinc/cp-flink:1.19.1-cp1",
   "job": {
      "jarURI": "local:///opt/flink/examples/streaming/StateMachineExample.jar",
      "parallelism": 3,
      "state": "running",
      "upgradeMode": "stateless"
   },
   "jobManager": {
      "resource": {
      "cpu": 1,
      "memory": "1048m"
      }
   },
   "serviceAccount": "flink",
   "taskManager": {
      "resource": {
      "cpu": 1,
      "memory": "1048m"
      }
   }
}
}

Use the application command, passing the application.json file to create the application.

confluent flink application create application.json --environment env1 --url http://localhost:8080

Access the Flink Web UI to confirm you have successfully created the application:

confluent flink application web-ui-forward basic-example --environment env1 --port 8090 --url http://localhost:8080