Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Upgrading Confluent Operator, Confluent Platform, and Helm

Upgrading Confluent Platform version 5.3.x to 5.4

Complete the following procedures to upgrade your operating environment.

Important

  • Migrate Helm 2 to Helm 3. Helm 3 is recommended for Confluent Operator and Confluent Platform Kubernetes deployments. Review this information before migrating: Helm 2 to Helm 3 changes.
  • Make sure that $PATH points to the Helm 3 binary.
  • Operator and Confluent Platform components restart during the upgrade procedure.
  • You must complete the procedure for each namespace in your environment.
  • Extract the new Confluent Operator 5.4 bundle. Consider renaming any old extracted directories before starting the procedure.

Upgrade Confluent Operator

You start the migration and upgrade process by upgrading Confluent Operator.

  1. Download and extract the Confluent Operator 5.4 bundle.

    Bundle download link:

    https://platform-ops-bin.s3-us-west-1.amazonaws.com/operator/confluent-operator-20200115-v0.142.1.tar.gz

  2. Go to the helm directory where the files were extracted.

  3. Go to the scripts > upgrade directory.

  4. Run the disable_reconcile.sh script.

    Important

    If you do not disable the reconcile operations before you run the upgrade, components will begin rolling restarts and some pods may go into CrashLoopBackOff status.

    The script disables reconcile operations so that Confluent Platform components do not restart. Only Operator restarts during this upgrade.

    ./disable_reconcile.sh <namespace>

    Your output should resemble the following:

    ./disable_reconcile.sh operator

physicalstatefulcluster.operator.confluent.cloud/connectors annotated
physicalstatefulcluster.operator.confluent.cloud/controlcenter annotated
physicalstatefulcluster.operator.confluent.cloud/kafka annotated
physicalstatefulcluster.operator.confluent.cloud/ksql annotated
physicalstatefulcluster.operator.confluent.cloud/replicator annotated
physicalstatefulcluster.operator.confluent.cloud/schemaregistry annotated
physicalstatefulcluster.operator.confluent.cloud/zookeeper annotated

  5. Go to the base helm directory and enter the following command.

    helm upgrade --install -f <provider.yaml> \
--set operator.enabled=true \
--wait <helm-deployment-name> <path-to-operator>/helm/confluent-operator \
--namespace <namespace>

    <helm-deployment-name> is the NAME displayed when running the command helm ls.

    NAME        REVISION    UPDATED              STATUS   CHART                APP VERSION    NAMESPACE
operator    2           Mon Jan 6            DEPLOYED confluent-operator   1.0            operator

Upgrade ZooKeeper

After Operator is upgraded, you can continue the upgrade for ZooKeeper.

Important

The cluster will roll during this procedure.

  1. Go to the helm directory where the files were extracted.

  2. Update the ZooKeeper component image. Complete the steps in Updating a single component image.

  3. Go to the scripts > upgrade directory.

  4. Run the pre_upgrade_cp54_zookeeper.sh script:

    ./pre_upgrade_cp54_zookeeper.sh <namespace>

    For example:

    ./pre_upgrade_cp54_zookeeper.sh operator

CONFLUENT_VERSION=5.3.1
echo ==> running zookeeper cluster on namespace operator is part of CP 5.3 release..check if snapshot file exists..
==> snapshot files already exist for a pod zookeeper-0 in namespace operator
==> copying snapshot.0 successful for a pod zookeeper-1 in namespace operator
==> copying snapshot.0 successful for a pod zookeeper-2 in namespace operator
==> zookeeper cluster is ready to upgrade to CP 5.4 version

  5. Enter the following command:

    ./enable_reconcile.sh <namespace> <cluster-name>

    For example:

    ./enable_reconcile.sh operator zookeeper

Upgrade Kafka and other Confluent Platform components

Complete the following steps for the remaining components in the Confluent Platform cluster.

Note

The cluster will roll during this procedure.

  1. For non-Kafka components, specifically ksqlDB, Connect, Replicator, Schema Registry, and Control Center, update the their clients running outside of Kubernetes with the changed load balancer ports.

    When upgrading to release 5.4, the port where the component is reachable via the load balancer is changed to the standard HTTPS (443) or HTTP (80) port, depending on whether TLS is enabled or not. This is a breaking change and requires the clients that were accessing the component via the load balancer to be updated to use the new port.

    If it is not feasible to update the clients to use the new port, set the <component>.tls.port field to 8443 (which was the previous default value) or some other value of choice in your Helm values file. For more details, refer to Load balancer ports in Confluent Operator.

  2. For non-Kafka components, specifically ksqlDB, Connect, Replicator, Schema Registry, and Control Center, update their clients running within Kubernetes with the changed component ports.

    When upgrading to release 5.4, the external and internal ports for the component containers and ClusterIP services change. Unlike the load balancer ports discussed in the previous step, these ports cannot be changed. When TLS is enabled for a component, the HTTPS listener is available on the container and ClusterIP service external port. When TLS is disabled for a component, the HTTP listener is available on the container and ClusterIP service internal port.

    Clients accessing the components from within the Kubernetes cluster must be reconfigured to use the newer ports, as listed in Ports in Confluent Operator.

  3. Update the remaining Confluent Platform component releases by following the instructions in Updating a single component image. Complete these steps for each component release in the cluster.

  4. Go to the scripts > upgrade directory.

  5. Run the enable_reconcile.sh script:

    ./enable_reconcile.sh <namespace> <cluster-name>

    For example:

    ./enable_reconcile.sh operator kafka

  6. Complete the previous step for all remaining namespaces (if applicable).

Updating a single component image

You can update a single Confluent Platform component image version without affecting cluster performance. Special care is taken when upgrading Kafka brokers. The upgrade process verifies topic leader re-election and zero under-replicated partitions before beginning the upgrade process for the next Kafka broker. This ensures that the Kafka cluster continues to perform well while being upgraded.

A component rolling upgrade begins after you run the helm upgrade command shown below, unless you have disabled reconciliation. (See the workflow above on upgrading to 5.4 for an example where you might disable reconciliation):

helm upgrade \
-f ./providers/<provider>.yaml \
--set <component>.enabled=true \
--set <component>.image.tag=<new-version-#> <component> \
./confluent-operator
  • Replace <provider> with the provider platform for your cluster.
  • Replace <new-version-#> with the new image tag provided by Confluent.
  • Replace <component> with the component you are updating (for example, kafka, connect, etc.).

Migrating from Helm 2 to Helm 3

Important

Keep Confluent Operator and Confluent Platform running during Helm version migration.