Get Started with Confluent Platform for Apache Flink

Confluent Platform for Apache Flink®️ is a compatible with open-source Apache Flink®. To get started, you must use the Helm chart to install the Flink Kubernetes operator. After you have have installed the Flink Kubernetes operator, you can deploy a Flink job and easily migrate from open-source Flink to Confluent Platform for Apache Flink.

For requirement and compatibility details, see Confluent Platform for Apache Flink compatibility.

Important

Confluent Platform for Apache Flink is in Limited Availability and is fully supported for select customers.

Prerequisites

To use Confluent Platform for Apache Flink, you must meet the following prerequisites:

Kubernetes

Confluent Platform for Apache Flink®️ is designed for Kubernetes. Therefore, you must have access to a Kubernetes cluster to deploy the Flink Kubernetes operator and individual Flink jobs:

  • A Kubernetes cluster conforming to one of the supported versions.
  • kubectl installed, initialized, with the context set. You also must have the kubeconfig file configured for your cluster.
  • Helm 3 installed.

To learn more about deploying Flink on Kubernetes, see Kubernetes deployment.

Note

Confluent for Kubernetes (CFK) is not the same as the Kubernetes operator provided by Confluent Platform for Apache Flink. You should use standard Kubernetes and not CFK to deploy Confluent Platform for Apache Flink.

Durable storage

Flink requires durable storage that is readable and writeable from each Flink cluster. Durable storage is used to store consistent snapshots of the state of your Flink jobs (checkpoints and savepoints).

Examples of durable storage include:

  • S3-compatible storage (S3, MinIO)
  • Microsoft Azure Blob Storage
  • Google Cloud Storage

Supported components

Check to make sure that you are using components supported by Confluent Platform for Apache Flink. See, Confluent Platform for Apache Flink Features and Support.

Get Confluent Platform for Apache Flink

Step 1: Install Confluent Platform for Apache Flink

  1. Add the Confluent Platform for Apache Flink Helm repository.

    helm repo add confluentinc https://packages.confluent.io/helm
    helm repo update
    
  2. Install the certificate manager.

    kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
    
  3. Install Confluent Platform for Apache Flink.

    helm upgrade --install cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator
    
  4. Check that everything is deployed:

    kubectl get pods
    

Step 2: Deploy Flink jobs

  1. Create a basic FlinkDeployment in flink-basic.yaml:

    apiVersion: flink.apache.org/v1beta1
    kind: FlinkDeployment
    metadata:
      name: basic-example
    spec:
      image: confluentinc/cp-flink:1.19.1-cp1
      flinkVersion: v1_19
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: "2"
      serviceAccount: flink
      jobManager:
        resource:
          memory: "2048m"
          cpu: 1
      taskManager:
        resource:
          memory: "2048m"
          cpu: 1
      job:
        jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
        parallelism: 2
        upgradeMode: stateless
    
  2. Deploy the Flink job:

    kubectl create -f flink-basic.yaml
    

Follow guidelines in the Kubernetes operator documentation to deploy a Flink job.

Package your Flink application

The Flink configuration overview provides instructions on how to configure your project. Follow the linked instructions to create a Flink project for Maven. For more on configuring your project with Maven, see Maven configuration.

To use the Flink dependencies that are provided as part of Confluent Platform for Apache Flink®️, you need to add the Confluent Maven repository and change the Maven group ID and version for supported components in the POM file for your project.

  1. Add the Confluent Platform for Apache Flink Maven repository to the pom.xml file like shown in the following code:

    <repositories>
      <repository>
        <id>cp-flink-releases</id>
        <url>https://packages.confluent.io/maven</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
    
  2. Replace the Maven group IDs and versions.

    In the dependency section of the pom.xml file, change the group ID to io.confluent.flink and update the version for each supported component like the following code:

     <dependency>
      <groupId>io.confluent.flink</groupId>
      <artifactId>flink-streaming-java</artifactId>
      <version>1.19.1-cp1</version>
    </dependency>
    

After you have changed these settings, your project will use Confluent Platform for Apache Flink.

Set up port forwarding

Retrieve and forward the REST endpoint, which enables you to interact with the REST endpoint and Flink WebUI.

  1. Get the endpoints with the following command:

    kubectl get endpoints
    

    You will see output like the following:

    NAME                             ENDPOINTS                                 AGE
    basic-example                    192.168.228.91:6124,192.168.228.91:6123   8m22s
    basic-example-rest               192.168.228.91:8081                       8m22s
    flink-operator-webhook-service   192.168.179.202:9443                      12m
    kubernetes                       192.168.238.57:443,192.168.29.233:443     16d
    
  2. Find the REST endpoint, and set up port forwarding with a command like the following:

    kubectl port-forward svc/basic-example-rest 8081
    
  3. After setting up port forwarding, access the Flink WebUI by navigating to http://localhost:8081 in your web browser.

Uninstall the Flink operator

Once you have finished with the Flink operator, you can uninstall it.

Use the following command to uninstall the Flink operator.

helm uninstall cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator