Configure Kafka Connect

Add connectors

Add a new connector to one of the following Connect images:

  • For Confluent Platform 6.2.x and above, use the cp-server-connect image.

    The remainder of this document uses this image.

  • For Confluent Platform 6.1.x and below, use the cp-server-connect-operator image.

The image contains Connect and all of its dependencies. It does not contain any Connector jars.

To add new connectors to the Connect image, you need to build a new Docker image that has the new connectors installed.

  1. Create a Dockerfile in <dockerfile-dir> to add one or more connectors to the cp-server-connect image.

    You can either:

    • Pull connectors from Confluent Hub.
    • Use the connector JAR downloaded to the machine you are running the Docker build from.

    To pull connectors from Confluent Hub:

    Create a Dockerfile as follows:

    FROM confluentinc/cp-server-connect:<Confluent Platform release>
    USER root
    RUN confluent-hub install --no-prompt <connector1>:<connector1-version> \
      && confluent-hub install --no-prompt <connector2>:<connector2-version> \
      && ...
    USER 1001
    

    An example Dockerfile to create a Docker image with the data-gen connector from Confluent Hub:

    FROM confluentinc/cp-server-connect:6.2.0
    USER root
    RUN  confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.3.3
    USER 1001
    

    To use the connector JAR downloaded to the machine you are running the Docker build from:

    Create a Dockerfile as follows:

    FROM confluentinc/cp-server-connect:<Confluent Platform release>
    ADD <local-connector1-path> /usr/share/java/<connector1> \
      && <local-connector2-path> /usr/share/java/<connector2> \
      && ...
    USER 1001
    

    An example Dockerfile to use the data-gen connector existing on your local machine in the <connector-dir> directory:

    FROM confluentinc/cp-server-connect:6.2.0
    ADD​ my-connector-dir/confluentinc-kafka-connect-datagen /usr/share/java/confluentinc-kafka-connect-datagen
    USER 1001
    
  2. Build and push the image with the following commands;

    docker build <dockerfile-dir> -t <someregistry>/<somerepository>:<sometag>
    
    docker push <someregistry>/<somerepository>:<sometag>
    
  3. Get the Docker image details from the output of the above process and specify the repository and tag in the Connect CR.

    connect:
      spec:
        image:
          application: <someregistry>/<somerepository>:<sometag>