Configure Kafka Connect¶
Add connectors¶
Confluent for Kubernetes provides the following image for Connect: cp-server-connect-operator
.
This contains Connect and all of its dependencies. It does not contain any Connector
jars.
To add new connectors to this image, you need to build a new Docker image that has the new connectors installed.
Create a
Dockerfile
in<dockerfile-dir>
to add one or more connectors to thecp-server-connect-operator
image.You can either:
- Pull connectors from Confluent Hub.
- Use a connector JAR downloaded to the machine you are running the Docker build from.
To pull connectors from Confluent Hub, create a
Dockerfile
as follows:FROM confluentinc/cp-server-connect-operator:6.1.0.0 USER root RUN confluent-hub install --no-prompt <connector1>:<connector1-version> \ && confluent-hub install --no-prompt <connector2>:<connector2-version> \ && ... USER 1001
An example
Dockerfile
to create a Docker image with the data-gen connector from Confluent Hub:FROM confluentinc/cp-server-connect-operator:6.1.0.0 USER root RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.3.3 USER 1001
To use connector JAR files that are on your machine that you are running the Docker build from, create a
Dockerfile
as follows:FROM confluentinc/cp-server-connect-operator:6.1.0.0 ADD <local-connector1-path> /usr/share/java/<connector1> \ && <local-connector2-path> /usr/share/java/<connector2> \ && ... USER 1001
An example
Dockerfile
to use the data-gen connector existing on your local machine in the<connector-dir>
directory:FROM confluentinc/cp-server-connect-operator:6.1.0.0 ADD my-connector-dir/confluentinc-kafka-connect-datagen /usr/share/java/confluentinc-kafka-connect-datagen USER 1001
Build and push the image with the following commands;
docker build <dockerfile-dir> -t <someregistry>/<somerepository>:<sometag> docker push <someregistry>/<somerepository>:<sometag>
Get the Docker image details from the output of the above process and specify the repository and tag in the Connect CR.
connect: spec: image: application: <someregistry>/<somerepository>:<sometag>