Deploying Kafka Connectors with Helm
The following highlights a simple Helm chart that can be used to deploy Kafka Connect Connector configurations using the standard Kafka Connect REST Interface. This is how this demonstration deploys the Confluent Replicator configuration, however, the same method could be used to deploy any Kafka Connect configuration. In future versions of Confluent for Kubernetes, Kafka Connect connectors will be managed by the Operator Controller.
The Helm chart is located in the
kubernetes/common/helm/replicator-cc folder of this demonstration. The
templates/replicator-configmap.yaml file contains a
data section with a templated JSON value that conforms to the Kafka Connect connectors API. The Destination and Source cluster configuration values are filled in at runtime by the
helm templating system, and are proivded by your
my-values.yaml file created in the demo instructions above.
In the same directory as the ConfigMap manifest file is a Kubernetes Job manifest (
replicator-connector-deploy-job.yaml). This defines a Kubernetes Job that will successfully execute a job to completion and terminate. Using a Docker image that contains the
curl program, the ConfigMap defined above is mounted to the batch job Pod, and then
curl executes a
POST to the Kafka Connect REST API to deploy Replicator.
- name: replicator-connector
- name: replicator-connector-deploy
"-H", "Content-Type: application/json",
- name: replicator-connector
Of note is the way in which the ConfigMap is associated to the Job Pod by name. The value,
replicator-connector in the
volumes stanza of the Job manifest matches the ConfigMap name in the metadata section of the ConfigMap manifest.
Deploying Connectors with Operator
You can deploy any Kafka connector (or single message transformation (SMT) or converter) in your Kubernetes environment.
Search in Confluent Hub, an online library of pre-packaged and ready-to-install connectors, transformations, and converters, to find the one that suits your needs.
The Confluent for Kubernetes image for Kafka Connect, confluentinc/cp-server-connect-operator, includes a small number of those connectors but may not have the specific connector you want to deploy.
Therefore, to deploy a new connector type in your Kubernetes environment, you will need to get the jars onto the Connect image.
The recommended way is to create a custom Docker image that extends the base Connect Docker image with the desired jars.
The custom Docker image builds the dependencies into a single artifact, which is more self-sufficient and portable such that it can be run on any pod despite totally ephemeral disk.
See the documentation to learn how to use the Confluent Hub client to create a custom Docker image that extends one of Confluent’s Kafka Connect images with a specific set of ready-to-install connectors.
As an example, see how the Kafka Connect Datagen connector, which generates mock events, can be pulled from Confluent Hub and bundled into a Docker image using this Dockerfile.
Once you build the custom Docker image, Kubernetes will need to pull this image from a Docker Registry to create the Pods.
It is not recommended to use volumes to place the desired jars onto the Connect image because it is less self-sufficient, less portable, and harder to match up versions between the base image and jars.
For more advanced use cases where you want to use a custom connector instead of a pre-packaged one available at Confluent Hub, you may create a Docker image with a custom connector from a local archive.
The demonstration uses this more advanced workflow.
We use the Kafka Connect Datagen connector to generate mock events, and this Dockerfile builds the Docker image with a local archive of the Kafka Connect Datagen connector compiled from source code (versus pulling directly from Confluent Hub).
We publish this image to Docker Hub, but in your environment, publish to your own Docker Hub repo.
Your Operator Helm values will need to be updated to pull the custom Connect Docker image for your Pods. You can accomplish this by overriding the
connect image to instead use the one published to Docker Hub in the demo’s value.yaml configuration file.