Configure Kafka Connect

Add connectors

Confluent for Kubernetes provides the following image for Connect: cp-server-connect-operator. This contains Connect and all of its dependencies. It does not contain any Connector jars.

To add new connectors to this image, you need to build a new Docker image that has the new connectors installed.

  1. Create a Dockerfile in <dockerfile-dir> to add one or more connectors to the cp-server-connect-operator image.

    You can either:

    1. Pull connectors from Confluent Hub.
    2. Use a connector JAR downloaded to the machine you are running the Docker build from.
    • To pull connectors from Confluent Hub, create a Dockerfile as follows:

      FROM confluentinc/cp-server-connect-operator:6.1.0.0
      USER root
      RUN confluent-hub install --no-prompt <connector1>:<connector1-version> \
        && confluent-hub install --no-prompt <connector2>:<connector2-version> \
        && ...
      USER 1001
      

      An example Dockerfile to create a Docker image with the data-gen connector from Confluent Hub:

      FROM confluentinc/cp-server-connect-operator:6.1.0.0
      USER root
      RUN  confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.3.3
      USER 1001
      
    • To use connector JAR files that are on your machine that you are running the Docker build from, create a Dockerfile as follows:

      FROM confluentinc/cp-server-connect-operator:6.1.0.0
      ADD <local-connector1-path> /usr/share/java/<connector1> \
        && <local-connector2-path> /usr/share/java/<connector2> \
        && ...
      USER 1001
      

      An example Dockerfile to use the data-gen connector existing on your local machine in the <connector-dir> directory:

      FROM confluentinc/cp-server-connect-operator:6.1.0.0
      ADD​ my-connector-dir/confluentinc-kafka-connect-datagen /usr/share/java/confluentinc-kafka-connect-datagen
      USER 1001
      
  2. Build and push the image with the following commands;

    docker build <dockerfile-dir> -t <someregistry>/<somerepository>:<sometag>
    
    docker push <someregistry>/<somerepository>:<sometag>
    
  3. Get the Docker image details from the output of the above process and specify the repository and tag in the Connect CR.

    connect:
      spec:
        image:
          application: <someregistry>/<somerepository>:<sometag>