.. _development: Docker Developer Guide for |cp| =============================== .. _image_design_overview : This document assumes knowledge of Docker and ``Dockerfiles``. To review best practices for writing ``Dockerfiles``, see `Docker's best practices guide `__. If you want to contribute back to the project, review `the contributing guidelines `__. |cp| image bootup process ------------------------- Upon startup of Docker, the entry point ``/etc/confluent/docker/run`` runs three executable scripts found in the ``/etc/confluent/docker``. They are run in the following sequence: #. Configure script The ``/etc/confluent/docker/configure`` script does all the necessary configuration for each image. This includes the following: - Create all configuration files and copying them to their proper location. - Ensure that mandatory configuration properties are present. - If required, handle service discovery. #. Ensure script The ``/etc/confluent/docker/ensure`` scripts makes sure that all the prerequisites for launching the service are in place. This includes: - Ensure the configuration files are present and readable. - Ensure that you can write/read to the data directory. The directories must be world writable. - Ensure that supporting services are in the READY state. For example, ensure that the |ak| controller is ready before launching a |ak| broker. - Ensure supporting systems are configured properly. For example, make sure all topics required for |c3| are created with proper replication, security and partition settings. #. Launch The ``/etc/confluent/docker/launch`` script runs the actual process. The script should ensure that: - The process is run with process id 1. Your script should use ``exec`` so the program takes over the shell process rather than running as a child process. This is so that your program will receive signals like SIGTERM directly rather than its parent shell process receiving them. - Log to stdout. .. _setup : Prerequisites ------------- #. Install Docker: - Docker version 1.11 or later is `installed and running `_. - Docker Compose is `installed `_. Docker Compose is installed by default with Docker for Mac. - Docker memory is allocated minimally at 8 GB. When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. You can change the default allocation to 8 GB in Docker. Navigate to **Preferences** > **Resources** > **Advanced**. #. Install Maven: .. codewithvars:: bash brew install maven .. _building_the_images : Build the |cp| images --------------------- Refer to :ref:`image_reference` for a list of GitHub repos for the |cp| components. For each |cp| image you want to build: #. Clone the repo. #. Checkout the release branch. #. Get the values for the required and optional arguments for the build command. For the list of supported arguments, see `this README file `__. The following are required arguments: - ``CONFLUENT_PACKAGES_REPO``: Specify the location of the |cp| packages repository. Depending on the type of OS for the image you are building, you might need to provide a Debian or RPM repository. - ``CONFLUENT_VERSION``: Specify the full |cp| release version, e.g., |release|. - ``docker.upstream-registry``: Registry to pull base images from. Trailing ``/`` is required. Used as ``DOCKER_UPSTREAM_REGISTRY`` during ``docker build``. - ``docker.upstream-tag``: Use the given tag when pulling base images. Used as ``DOCKER_UPSTREAM_TAG`` during ``docker build``. #. Optionally, you can choose an operating system you want your docker image based on, specifically Debian or RHEL UBI. To build a RHEL UBI image pass the following argument to the Maven command: .. codewithvars:: bash -Ddocker.os_type=ubi8 #. From the root folder of the repo, build the |cp| images using Maven. For example: .. codewithvars:: bash mvn clean package \ -DskipTests -Pdocker \ -DCONFLUENT_PACKAGES_REPO='https://packages.confluent.io/rpm/|version|' \ -DCONFLUENT_VERSION=|release| \ -Ddocker.upstream-registry=docker.io/ \ -Ddocker.upstream-tag=|release| .. _running_tests : .. _extending_images: Extend |cp| images ------------------ You can extend the images to add or customize connectors, add new software, change the configuration management, and set up external service discovery. The following sections provide examples of extending |cp| images. .. note:: You can apply the similar steps to extend custom Docker images that you built. However, if you run into an issue with a custom Docker image, you need to work with Confluent Support to narrow down the issue. Confluent Support can only assist with the issues involving |cp| software. The issues with Docker or any 3rd party software cannot be further debugged by Confluent Support. .. _prerequisites: .. _adding_connectors_to_images: .. include:: includes/docker-extend-connector-image.rst .. _change_configuration_management : Change configuration management ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example describes how to change the configuration management. To accomplish this, you override the ``configure`` script to download the scripts from a URL. For example, with the |zk| image, you need the following ``Dockerfile`` and configure script. This example assumes that each property file has a URL. **Dockerfile:** .. sourcecode:: bash FROM confluentinc/cp-zookeeper COPY include/etc/confluent/docker/configure /etc/confluent/docker/configure **Example Configure Script:** Location: ``include/etc/confluent/docker/configure`` .. sourcecode:: bash . /etc/confluent/docker/bash-config # Ensure that URL locations are available. dub ensure ZOOKEEPER_SERVER_CONFIG_URL dub ensure ZOOKEEPER_SERVER_ID_URL dub ensure ZOOKEEPER_LOG_CONFIG_URL # Ensure that the config location is writable. dub path /etc/kafka/ writable curl -XGET ZOOKEEPER_SERVER_CONFIG_URL > /etc/kafka/zookeeper.properties curl -XGET ZOOKEEPER_SERVER_ID_URL > /var/lib/zookeeper/data/myid curl -XGET ZOOKEEPER_LOG_CONFIG_URL > /etc/kafka/log4j.properties Build the image: docker build -t foo/zookeeper:latest . Enter the command. .. sourcecode:: bash docker run \ -e ZOOKEEPER_SERVER_CONFIG_URL=http://foo.com/zk1/server.properties \ -e ZOOKEEPER_SERVER_ID_URL =http://foo.com/zk1/myid \ -e ZOOKEEPER_LOG_CONFIG_URL =http://foo.com/zk1/log4j.properties \ foo/zookeeper:latest .. _log-to-external-volumes: Log to external volumes ~~~~~~~~~~~~~~~~~~~~~~~ The images only expose volumes for data and security configuration. But you might want to write to external storage for some use cases. The following example shows how to write the |ak| authorizer logs to a volume for auditing. **Dockerfile:** .. sourcecode:: bash FROM confluentinc/cp-server # Make sure the log directory is world-writable RUN echo "===> Creating authorizer logs dir ..." \\ && mkdir -p /var/log/kafka-auth-logs \\ && chmod -R ag+w /var/log/kafka-auth-logs VOLUME \["/var/lib/$\{COMPONENT}/data", "/etc/$\{COMPONENT}/secrets", "/var/log/kafka-auth-logs"] COPY include/etc/confluent/log4j.properties.template /etc/confluent/docker/log4j.properties.template **log4j.properties.template:** Location: ``include/etc/confluent/log4j.properties.template`` .. sourcecode:: bash log4j.rootLogger={{ env["KAFKA_LOG4J_ROOT_LOGLEVEL"] | default('INFO') }}, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH log4j.appender.authorizerAppender.File=/var/log/kafka-auth-logs/kafka-authorizer.log log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.additivity.kafka.authorizer.logger=false {% set loggers = { 'kafka': 'INFO', 'kafka.network.RequestChannel$': 'WARN', 'kafka.producer.async.DefaultEventHandler': 'DEBUG', 'kafka.request.logger': 'WARN', 'kafka.controller': 'TRACE', 'kafka.log.LogCleaner': 'INFO', 'state.change.logger': 'TRACE', 'kafka.authorizer.logger': 'WARN, authorizerAppender' } -%} {% if env['KAFKA_LOG4J_LOGGERS'] %} {% set loggers = parse_log4j_loggers(env['KAFKA_LOG4J_LOGGERS'], loggers) %} {% endif %} {% for logger,loglevel in loggers.iteritems() %} log4j.logger.\{\{logger}}=\{\{loglevel}} {% endfor %} Build the image. .. sourcecode:: bash docker build -t foo/kafka-auditable:latest . Write garbage collection logs to an external volume ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following example shows how to log heap dumps and GC logs to an external volume. This is useful for debugging the |ak| image. **Dockerfile:** .. sourcecode:: bash FROM confluentinc/cp-server # Make sure the jvm log directory is world-writable RUN echo "===> Creating jvm logs dir ..." \ && mkdir -p /var/log/jvm-logs && chmod -R ag+w /var/log/jvm-logs VOLUME ["/var/lib/${COMPONENT}/data", "/etc/${COMPONENT}/secrets", "/var/log/jvm-logs"] #. Build the image. .. sourcecode:: bash docker build -t foo/kafka-verbose-jvm:latest . #. Enter the command. .. sourcecode:: bash docker run \ -e KAFKA_HEAP_OPTS="-Xmx256M -Xloggc:/var/log/jvm-logs/verbose-gc.log -verbose:gc -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/jvm-logs" \ foo/kafka-verbose-jvm:latest Use external service discovery ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can extend the images to support for any service discovery mechanism either by overriding relevant properties or by overriding the ``configure`` script as shown in :ref:`Change Configuration Management `. The Docker images provide Mesos support by overriding relevant properties for Mesos service discovery. See ``debian/kafka-connect/includes/etc/confluent/docker/mesos-overrides`` for examples. .. _oracle_jdk : Use the Oracle JDK ~~~~~~~~~~~~~~~~~~ The |cp| images ship with Azul Zulu OpenJDK. If you are required to use Oracle's version, follow the steps below to modify the cp-base-new image to include Oracle JDK instead of Zulu OpenJDK. #. Obtain a download URL from `Oracle's website `__. #. In the ``base`` directory of the `Confluent Base image `__ repo, edit the ``Dockerfile.ubi8`` file in the correct version branch, :litwithvars:`|release|-post`. #. Replace the following lines for Zulu OpenJDK: .. codewithvars:: dockerfile ENV ZULU_OPENJDK="zulu-11-11.990-1" RUN microdnf install yum \ && rpm --import https://www.azul.com/files/0xB1998361219BD9C9.txt \ && yum -y install https://cdn.azul.com/zulu/bin/zulu-repo-1.0.0-1.noarch.rpm \ && yum -y install ${ZULU_OPENJDK} \ With the following lines for Oracle JDK: .. codewithvars:: dockerfile RUN microdnf install yum \ && curl -L --output /tmp/jre1.8-latest.rpm \ && yum -y install /tmp/jre1.8-latest.rpm \ && rm -f /tmp/jre1.8-latest.rpm \ #. Rebuild the cp-base-new image. Refer to :ref:`building_the_images` for the steps. .. _utility_scripts : Utility scripts --------------- Because there are dependencies between the various |cp| components, you might need to check the status of different services. The following utilities are used during the bootup sequence of the images and in the testing framework. Docker Utility Belt (dub) ~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Template .. codewithvars:: bash usage: dub template [-h] input output Generate template from env vars. positional arguments: input Path to template file. output Path of output file. 2. ensure .. codewithvars:: bash usage: dub ensure [-h] name Check if env var exists. positional arguments: name Name of env var. 3. wait .. codewithvars:: bash usage: dub wait [-h] host port timeout wait for network service to appear. positional arguments: host Host. port Host. timeout timeout in secs. 4. path .. codewithvars:: bash usage: dub path [-h] path {writable,readable,executable,exists} Check for path permissions and existence. positional arguments: path Full path. {writable,readable,executable,exists} One of [writable, readable, executable, exists]. 5. path-wait .. codewithvars:: bash usage: dub path-wait [-h] path timeout Wait for a path to exist. positional arguments: path Full path. timeout Time in secs to wait for the path to exist. optional arguments: -h, --help show this help message and exit Confluent Platform Utility Belt (cub) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. kafka-ready Used to check if |ak| is ready. .. codewithvars:: bash usage: cub kafka-ready [-h] (-b BOOTSTRAP_BROKER_LIST | -z ZOOKEEPER_CONNECT) [-c CONFIG] [-s SECURITY_PROTOCOL] expected_brokers timeout Check if |ak| is ready. positional arguments: expected_brokers Minimum number of brokers to wait for timeout Time in secs to wait for service to be ready. optional arguments: -h, --help show this help message and exit -b BOOTSTRAP_BROKER_LIST, --bootstrap_broker_list BOOTSTRAP_BROKER_LIST List of bootstrap brokers. -z ZOOKEEPER_CONNECT, This option is deprecated in Confluent Platform 5.5.0 and later. --zookeeper_connect ZOOKEEPER_CONNECT ZooKeeper connect string. -c CONFIG, --config CONFIG Path to config properties file (required when security is enabled). -s SECURITY_PROTOCOL, --security-protocol SECURITY_PROTOCOL Security protocol to use when multiple listeners are enabled. #. sr-ready Used to check if |sr| is ready. If you have multiple |sr| nodes, you may need to check their availability individually. .. codewithvars:: bash usage: cub sr-ready [-h] host port timeout positional arguments: host Hostname for Schema Registry. port Port for Schema Registry. timeout Time in secs to wait for service to be ready. #. kr-ready Used to check if the REST Proxy is ready. If you have multiple REST Proxy nodes, you may need to check their availability individually. .. codewithvars:: bash usage: cub kr-ready [-h] host port timeout positional arguments: host Hostname for REST Proxy. port Port for REST Proxy. timeout Time in secs to wait for service to be ready. #. connect-ready Used to check if |kconnect-long| is ready. .. codewithvars:: bash usage: cub connect-ready [-h] host port timeout positional arguments: host Hostname for Connect worker. port Port for Connect worker. timeout Time in secs to wait for service to be ready. #. ksql-server-ready Used to check if |ksqldb| is ready. .. codewithvars:: bash usage: cub ksql-server-ready [-h] host port timeout positional arguments: host Hostname for KSQL server. port Port for KSQL server. timeout Time in secs to wait for service to be ready. #. control-center-ready Used to check if |c3| is ready. .. codewithvars:: bash usage: cub control-center-ready [-h] host port timeout positional arguments: host Hostname for Control Center. port Port for Control Center. timeout Time in secs to wait for service to be ready. #. zk-ready .. note:: .. include:: ../../includes/zk-deprecation.rst Used to check if |zk| is ready. .. codewithvars:: bash usage: cub zk-ready [-h] connect_string timeout retries wait Check if ZK is ready. positional arguments: connect_string ZooKeeper connect string. timeout Time in secs to wait for service to be ready. retries No of retries to check if leader election is complete. wait Time in secs between retries Client properties ~~~~~~~~~~~~~~~~~ The following properties may be configured when using the ``kafka-ready`` utility described above. ``bootstrap.servers`` A list of host/port pairs to use for establishing the initial connection to the |ak| cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form ``host1:port1,host2:port2,...``. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). * Type: list * Default: * Importance: high ``ssl.key.password`` The password of the private key in the key store file. This is optional for client. * Type: password * Importance: high ``ssl.keystore.location`` The location of the key store file. This is optional for client and can be used for bidirectiona client authentication. * Type: string * Importance: high ``ssl.keystore.password`` The store password for the key store file.This is optional for client and only needed if ssl.keystore.location is configured. * Type: password * Importance: high ``ssl.truststore.location`` The location of the trust store file. * Type: string * Importance: high ``ssl.truststore.password`` The password for the trust store file. * Type: password * Importance: high ``sasl.kerberos.service.name`` The Kerberos principal name that |ak| runs as. This can be defined either in |ak|'s JAAS config or in |ak|'s config. * Type: string * Importance: medium ``sasl.mechanism`` SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism. * Type: string * Default: "GSSAPI" * Importance: medium ``security.protocol`` Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. * Type: string * Default: "PLAINTEXT" * Importance: medium ``ssl.enabled.protocols`` The comma-separated list of protocols enabled for TLS connections. The default value is ``TLSv1.2,TLSv1.3`` when running with Java 11 or later, ``TLSv1.2`` otherwise. With the default value for Java 11 (``TLSv1.2,TLSv1.3``), |ak| clients and brokers prefer TLSv1.3 if both support it, and falls back to TLSv1.2 otherwise (assuming both support at least TLSv1.2). * Type: list * Default: ``TLSv1.3,TLSv1.3`` * Importance: medium ``ssl.keystore.type`` The file format of the key store file. This is optional for client. * Type: string * Default: "JKS" * Importance: medium ``ssl.protocol`` The TLS protocol used to generate the SSLContext. The default is ``TLSv1.3`` when running with Java 11 or newer, ``TLSv1.2`` otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are ``TLSv1.2`` and ``TLSv1.3``. ``TLS``, ``TLSv1.1``, ``SSL``, ``SSLv2`` and ``SSLv3`` might be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this configuration and ``ssl.enabled.protocols``, clients downgrade to ``TLSv1.2`` if the server does not support ``TLSv1.3``. If this configuration is set to ``TLSv1.2``, clients do not use ``TLSv1.3``, even if it is one of the values in ``ssl.enabled.protocols`` and the server only supports ``TLSv1.3``. * Type: string * Default: ``TLSv1.3`` * Importance: medium ``ssl.provider`` The name of the security provider used for TLS connections. Default value is the default security provider of the JVM. * Type: string * Importance: medium ``ssl.truststore.type`` The file format of the trust store file. * Type: string * Default: "JKS" * Importance: medium ``sasl.kerberos.kinit.cmd`` Kerberos kinit command path. * Type: string * Default: "/usr/bin/kinit" * Importance: low ``sasl.kerberos.min.time.before.relogin`` Login thread sleep time between refresh attempts. * Type: long * Default: 60000 * Importance: low ``sasl.kerberos.ticket.renew.jitter`` Percentage of random jitter added to the renewal time. * Type: double * Default: 0.05 * Importance: low ``sasl.kerberos.ticket.renew.window.factor`` Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. * Type: double * Default: 0.8 * Importance: low ``ssl.cipher.suites`` A list of cipher suites. This is a named combination of authentication, encryption, MAC, and key exchange algorithms used to negotiate the security settings for a network connection using TLS. By default, all the available cipher suites are supported. * Type: list * Importance: low ``ssl.endpoint.identification.algorithm`` The endpoint identification algorithm to validate server hostname using server certificate. * Type: string * Importance: low ``ssl.keymanager.algorithm`` The algorithm used by key manager factory for TLS connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. * Type: string * Default: "SunX509" * Importance: low ``ssl.trustmanager.algorithm`` The algorithm used by trust manager factory for TLS connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. * Type: string * Default: "PKIX" * Importance: low .. _references : Related content --------------- - :ref:`cpdocker_intro` - :ref:`image_reference` - :ref:`docker_operations_logging` - :ref:`use-jmx-monitor-docker-deployments` - :ref:`external_volumes`