Configure Kafka Connect¶

Add connectors¶

Confluent for Kubernetes provides the following image for Connect: cp-server-connect-operator. This contains Connect and all of its dependencies. It does not contain any Connector jars.

To add new connectors to this image, you need to build a new Docker image that has the new connectors installed.

Create a Dockerfile in <dockerfile-dir> to add one or more connectors to the cp-server-connect-operator image.

You can either:

Pull connectors from Confluent Hub.
Use a connector JAR downloaded to the machine you are running the Docker build from.

To pull connectors from Confluent Hub, create a Dockerfile as follows:

FROM confluentinc/cp-server-connect-operator:6.1.0.0
USER root
RUN confluent-hub install --no-prompt <connector1>:<connector1-version> \
  && confluent-hub install --no-prompt <connector2>:<connector2-version> \
  && ...
USER 1001

An example Dockerfile to create a Docker image with the data-gen connector from Confluent Hub:

FROM confluentinc/cp-server-connect-operator:6.1.0.0
USER root
RUN  confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.3.3
USER 1001

To use connector JAR files that are on your machine that you are running the Docker build from, create a Dockerfile as follows:

FROM confluentinc/cp-server-connect-operator:6.1.0.0
ADD <local-connector1-path> /usr/share/java/<connector1> \
  && <local-connector2-path> /usr/share/java/<connector2> \
  && ...
USER 1001

An example Dockerfile to use the data-gen connector existing on your local machine in the <connector-dir> directory:

FROM confluentinc/cp-server-connect-operator:6.1.0.0
ADD​ my-connector-dir/confluentinc-kafka-connect-datagen /usr/share/java/confluentinc-kafka-connect-datagen
USER 1001

Build and push the image with the following commands;

docker build <dockerfile-dir> -t <someregistry>/<somerepository>:<sometag>

docker push <someregistry>/<somerepository>:<sometag>

Get the Docker image details from the output of the above process and specify the repository and tag in the Connect CR.
```
connect:
  spec:
    image:
      application: <someregistry>/<somerepository>:<sometag>
```