Get Started with Confluent Platform for Apache Flink¶

Confluent Platform for Apache Flink® is compatible with open-source Apache Flink®. To get started, you must use the Helm chart to install the Flink Kubernetes operator. After you have installed the Flink Kubernetes operator, you can deploy a Flink job and easily migrate from open-source Flink to Confluent Platform for Apache Flink.

For requirement and compatibility details, see Confluent Platform for Apache Flink compatibility.

Important

Confluent Platform for Apache Flink is in Limited Availability and is fully supported for select customers.

Prerequisites¶

To use Confluent Platform for Apache Flink, you must meet the following prerequisites:

Kubernetes¶

Confluent Platform for Apache Flink® is designed for Kubernetes. Therefore, you must have access to a Kubernetes cluster to deploy the Flink Kubernetes operator and individual Flink jobs:

A Kubernetes cluster conforming to one of the supported versions.
kubectl installed, initialized, with the context set. You also must have the kubeconfig file configured for your cluster.
Helm 3 installed.

To learn more about deploying Flink on Kubernetes, see Kubernetes deployment.

Note

Confluent for Kubernetes (CFK) is not the same as the Kubernetes operator provided by Confluent Platform for Apache Flink. You should use standard Kubernetes and not CFK to deploy Confluent Platform for Apache Flink.

Durable storage¶

Flink requires durable storage that is readable and writeable from each Flink cluster. Durable storage is used to store consistent snapshots of the state of your Flink jobs (checkpoints and savepoints).

Examples of durable storage include:

S3-compatible storage (S3, MinIO)
Microsoft Azure Blob Storage
Google Cloud Storage

Supported components¶

Check to make sure that you are using components supported by Confluent Platform for Apache Flink. See, Confluent Platform for Apache Flink Features and Support.

Get Confluent Platform for Apache Flink¶

Step 1: Install Confluent Platform for Apache Flink¶

Add the Confluent Platform for Apache Flink Helm repository.

helm repo add confluentinc https://packages.confluent.io/helm
helm repo update

Install the certificate manager.

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Install Confluent Platform for Apache Flink.

helm upgrade --install cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator

Check that everything is deployed:
```
kubectl get pods
```

Step 2: Deploy Flink jobs¶

Create a basic FlinkDeployment in flink-basic.yaml:

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: basic-example
spec:
  image: confluentinc/cp-flink:1.19.1-cp1
  flinkVersion: v1_19
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
  serviceAccount: flink
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "2048m"
      cpu: 1
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 2
    upgradeMode: stateless

Deploy the Flink job:
```
kubectl create -f flink-basic.yaml
```

Follow guidelines in the Kubernetes operator documentation to deploy a Flink job.

Package your Flink application¶

The Flink configuration overview provides instructions on how to configure your project. Follow the linked instructions to create a Flink project for Maven. For more on configuring your project with Maven, see Maven configuration.

To use the Flink dependencies that are provided as part of Confluent Platform for Apache Flink®, you need to add the Confluent Maven repository and change the Maven group ID and version for supported components in the POM file for your project.

Add the Confluent Platform for Apache Flink Maven repository to the pom.xml file like shown in the following code:

<repositories>
  <repository>
    <id>cp-flink-releases</id>
    <url>https://packages.confluent.io/maven</url>
    <releases>
      <enabled>true</enabled>
    </releases>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
</repositories>

Replace the Maven group IDs and versions.

In the dependency section of the pom.xml file, change the group ID to io.confluent.flink and update the version for each supported component like the following code:
```
 <dependency>
  <groupId>io.confluent.flink</groupId>
  <artifactId>flink-streaming-java</artifactId>
  <version>1.19.1-cp1</version>
</dependency>
```

After you have changed these settings, your project will use Confluent Platform for Apache Flink.

Set up port forwarding¶

Retrieve and forward the REST endpoint, which enables you to interact with the REST endpoint and Flink WebUI.

Get the endpoints with the following command:

kubectl get endpoints

You will see output like the following:

NAME                             ENDPOINTS                                 AGE
basic-example                    192.168.228.91:6124,192.168.228.91:6123   8m22s
basic-example-rest               192.168.228.91:8081                       8m22s
flink-operator-webhook-service   192.168.179.202:9443                      12m
kubernetes                       192.168.238.57:443,192.168.29.233:443     16d

Find the REST endpoint, and set up port forwarding with a command like the following:
```
kubectl port-forward svc/basic-example-rest 8081
```
After setting up port forwarding, access the Flink WebUI by navigating to http://localhost:8081 in your web browser.

Uninstall the Flink operator¶

Once you have finished with the Flink operator, you can uninstall it.

Use the following command to uninstall the Flink operator.

helm uninstall cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator