Deploy Confluent Platform in Air-Gapped Environment Using Ansible Playbooks

This topic guides you through the steps to use Ansible Playbooks to deploy Confluent Platform without internet connectivity to outside networks.

This scenario requires:

  • A distribution server with internet connectivity

    You download the Ansible Playbooks for Confluent Platform (Confluent Ansible) collection and the Confluent Platform packages from the Confluent site to your distribution server.

    You need to install ansible-galaxy on the distribution server.

  • A control node to run Ansible

    This internal repository should be accessible from your Confluent Platform hosts and the distribution server.

  • Internal hosts to run Confluent Platform.

    The Confluent Platform hosts must be able to reach the control node at port 8080.

At the high level, the required steps are:

  1. Download the Confluent Platform packages and other required files, such as monitoring JARs, Confluent CLI, and the Cryptography pip package, along with Ansible collections. (This step requires internet connectivity.)
  2. Put the packages and other required files that you downloaded in the previous step on your repository server.
  3. Configure the variables for Ansible to use an internal custom repository server for Confluent Platform installation.
  4. Install Confluent Platform.

Deploy using tarball archives

Step 1: Distribute Confluent Ansible collection

Build the Confluent Ansible collection and install the collection on the control node.

  1. Create a directory with the following structure on the distribution server:

    mkdir -p <path_to_cp-ansible>/ansible_collections/confluent/
    Copy

    You can put <path_to_cp-ansible> anywhere in your directory structure, but the directory structure under <path_to_cp-ansible> should be set up exactly as specified above.

  2. Clone the Ansible Playbooks for Confluent Platform repo into the platform directory inside the directory you created in the previous step:

    git clone https://github.com/confluentinc/cp-ansible \
  <path_to_cp-ansible>/ansible_collections/confluent/platform
    Copy

  3. Check out the 7.9.0-post branch:

    cd <path_to_cp-ansible>/ansible_collections/confluent/platform
git fetch
git checkout 7.9.0-post
    Copy

  4. Create the collection:

    ansible-galaxy collection build
    Copy

    The command builds a tarball of the collection in the current directory.

  5. Upload the collection tarball to your control node:

    scp <collection tarball> <control-node>:/path/to/confluent-collection
    Copy

  6. Install the collection on the control node:

    ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
    Copy

    For more information about Ansible collection distribution, see Building your collection tarball.

  7. Download and install the cryptography pip package.

    1. Download the cryptography pip package to the distribution server:

      pip download cryptography -d <local directory on distribution server>
      Copy

    2. Transfer the cryptography package to all the Confluent Platform hosts (managed nodes):

      scp <local directory on distribution server> <managed-node>:/<some-path-on-managed-node-to-keep-pip-packages>
      Copy

    3. Install the cryptography package:

      cd <some-path-on-managed-node-to-keep-pip-packages>
pip install <crpyptography-wheel-file-name> -f ./ --no-index
      Copy
      • -f argument takes in the local path of directory where .whl files are present.
      • --no-index is for ignoring the remote repo and only looking inside the current directory.

Step 2: Download Confluent Platform archives

  1. Download the Confluent Platform archives to the distribution server:

    curl -O http://packages.confluent.io/archive/7.9/confluent-7.9.0.tar.gz
    Copy

    If enabling monitoring, download the required JARs:

    curl -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \
  http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar
    Copy
    curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
    Copy

  2. Move the packages and files to the control node:

    scp confluent-packages.tar.gz <control-node>:/path/to/confluent-packages.tar.gz
    Copy

  3. On the control node, extract the contents of the Confluent Platform archive:

    tar xzf confluent-7.9.0.tar.gz
    Copy

Step 3: Install Confluent Ansible

  1. Configure Confluent Ansible as described in Configure Ansible Playbooks for Confluent Platform.

  2. Add the following in the Ansible inventory file.

    • To install Confluent Platform using a tarball hosted on a web server:

      all:
  vars:
    repository_configuration: custom
    installation_method: archive
    confluent_archive_file_source: <web server url>/path/to/confluent-7.9.0.tar.gz
      Copy

    • To install Confluent Platform using a tarball placed on the control node:

      all:
  vars:
    installation_method: archive
    confluent_archive_file_source: /path/to/confluent-7.9.0.tar.gz
    confluent_archive_file_remote: false
      Copy

  3. Configure monitoring in the Ansible inventory file:

    • To enable Jolokia:

      jolokia_enabled: true
jolokia_url_remote: false
jolokia_jar_url: /path/to/jolokia-jvm-1.6.2-agent.jar
      Copy

    • To enable JMX exporter:

      jmxexporter_enabled: true
jmxexporter_url_remote: false
jmxexporter_jar_url: /path/to/jmx_prometheus_javaagent-0.12.0.jar
      Copy

  4. Install Confluent Ansible as described in the Installation Guide.

Deploy using RHEL or CentOS packages

Step 1: Distribute Confluent Ansible collection

Build the Confluent Ansible collection and install the collection on the control node.

  1. Create a directory with the following structure on the distribution server:

    mkdir -p <path_to_cp-ansible>/ansible_collections/confluent/
    Copy

    You can put <path_to_cp-ansible> anywhere in your directory structure, but the directory structure under <path_to_cp-ansible> should be set up exactly as specified above.

  2. Clone the Ansible Playbooks for Confluent Platform repo into the platform directory inside the directory you created in the previous step:

    git clone https://github.com/confluentinc/cp-ansible \
  <path_to_cp-ansible>/ansible_collections/confluent/platform
    Copy

  3. Check out the 7.9.0-post branch:

    cd <path_to_cp-ansible>/ansible_collections/confluent/platform
git fetch
git checkout 7.9.0-post
    Copy

  4. Create the collection:

    ansible-galaxy collection build
    Copy

    The command builds a tarball of the collection in the current directory.

  5. Upload the collection tarball to your control node:

    scp <collection tarball> <control-node>:/path/to/confluent-collection
    Copy

  6. Install the collection on your control node:

    ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
    Copy

    For more information about Ansible collection distribution, see Building your collection tarball.

  7. Download and install the cryptography pip package.

    1. Download the cryptography pip package to the distribution server:

      pip download cryptography -d <local directory on distribution server>
      Copy

    2. Transfer the cryptography package to all the Confluent Platform hosts (managed nodes):

      scp <local directory on distribution server> <managed-node>:/<some-path-on-managed-node-to-keep-pip-packages>
      Copy

    3. Install the cryptography package:

      cd <some-path-on-managed-node-to-keep-pip-packages>
pip install <crpyptography-wheel-file-name> -f ./ --no-index
      Copy
      • -f argument takes in the local path of directory where .whl files are present.
      • --no-index is for ignoring the remote repo and only looking inside the current directory.

Step 2: Download Confluent Platform packages

On the distribution server:

  1. Install the epel-release package:

    yum -y install epel-release
    Copy

  2. Create the confluent.repo file for pulling the Confluent Platform packages:

    echo "[Confluent.dist]
baseurl = https://packages.confluent.io/rpm/7.9
enabled = 1
gpgcheck = 1
gpgkey = https://packages.confluent.io/rpm/7.9/archive.key
name = Confluent repository (dist)

[Confluent]
baseurl = https://packages.confluent.io/rpm/7.9
enabled = 1
gpgcheck = 1
gpgkey = https://packages.confluent.io/rpm/7.9/archive.key
name = Confluent repository" \
> /etc/yum.repos.d/confluent.repo
    Copy

  3. Create a directory and download the Confluent Platform packages into that directory:

    mkdir /tmp/confluent-packages/
    Copy
    yum clean all

yum install --downloadonly --downloaddir=/tmp/confluent-packages/ \
  confluent-platform \
  confluent-security \
  nc \
  ansible \
  createrepo \
  java-11-openjdk
    Copy

  4. Download the required JARs if you want to enable monitoring:

    curl -L -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \
  http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar

curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
    Copy

Step 3: Host Confluent Platform packages on control node

Copy the Confluent Platform packages to the control node and run a simple repository server over a Python web server.

  1. On the distribution server, package and move the the files to the control node:

    cd /tmp

tar -zcvf confluent-packages.tar.gz confluent-packages

scp /tmp/confluent-packages.tar.gz <control-node>:/tmp/confluent-packages.tar.gz
    Copy

  2. On the control node, decompress and install the packages:

    cd /tmp
tar xvzf confluent-packages.tar.gz -C /tmp
    Copy
    cd /tmp/confluent-packages/
yum -y localinstall createrepo-*.rpm
    Copy

  3. Create a repo, and host the control node with python3 http.server:

    createrepo .
    Copy
    python3 -m http.server 8080
    Copy

Step 4: Install Confluent Ansible

Install Ansible on the control node, create a repo file for the rest of the Confluent Platform hosts to use, and install Confluent Platform.

  1. On the control node, install Confluent Ansible:

    cd /tmp/confluent-packages/
yum -y localinstall ansible-*.rpm
    Copy

  2. On the control node, create a .repo file that each host will use. If your organization has a repository server, that file should be configured for the existing repository server.

    echo "[confluent-repo]
name=Confluent Repo
baseurl=http://<ansible-control-node>:8080
enabled=1
gpgcheck=0" \
> /tmp/confluent.repo
    Copy

  3. Configure Confluent Ansible as described in Configure Ansible Playbooks for Confluent Platform, and add the following in the Ansible inventory file:

    all:
  vars:
    repository_configuration: custom
    custom_yum_repofile_filepath: /tmp/confluent.repo
    Copy

  4. Configure monitoring in the Ansible inventory file:

    • To enable Jolokia:

      jolokia_enabled: true
jolokia_url_remote: false
jolokia_jar_url: http://<control-node>:8080/jolokia-jvm-1.6.2-agent.jar
      Copy

    • To enable JMX exporter:

      jmxexporter_enabled: true
jmxexporter_url_remote: false
jmxexporter_jar_url: http://<control-node>:8080/jmx_prometheus_javaagent-0.12.0.jar
      Copy

  5. Install Confluent Ansible as described in the Installation Guide.

Deploy using Ubuntu or Debian packages

Step 1: Distribute Confluent Ansible collection

Build the Confluent Ansible collection and install the collection on the control node.

  1. Create a directory with the following structure on the distribution server:

    mkdir -p <path_to_cp-ansible>/ansible_collections/confluent/
    Copy

    You can put <path_to_cp-ansible> anywhere in your directory structure, but the directory structure under <path_to_cp-ansible> should be set up exactly as specified above.

  2. Clone the Ansible Playbooks for Confluent Platform repo into the platform directory inside the directory you created in the previous step:

    git clone https://github.com/confluentinc/cp-ansible \
  <path_to_cp-ansible>/ansible_collections/confluent/platform
    Copy

  3. Check out the 7.9.0-post branch:

    cd <path_to_cp-ansible>/ansible_collections/confluent/platform
git fetch
git checkout 7.9.0-post
    Copy

  4. Create the collection:

    ansible-galaxy collection build
    Copy

    The command builds a tarball of the collection in the current directory.

  5. Upload the collection tarball to your control node:

    scp <collection tarball> <control-node>:/path/to/confluent-collection
    Copy

  6. Install the collection on your control node:

    ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
    Copy

    For more information about Ansible collection distribution, see Building your collection tarball.

  7. Download and install the cryptography pip package.

    1. Download the cryptography pip package to the distribution server:

      pip download cryptography -d <local directory on distribution server>
      Copy

    2. Transfer the cryptography package to all the Confluent Platform hosts (managed nodes):

      scp <local directory on distribution server> <managed-node>:/<some-path-on-managed-node-to-keep-pip-packages>
      Copy

    3. Install the cryptography package:

      cd <some-path-on-managed-node-to-keep-pip-packages>
pip install <crpyptography-wheel-file-name> -f ./ --no-index
      Copy
      • -f argument takes in the local path of directory where .whl files are present.
      • --no-index is for ignoring the remote repo and only looking inside the current directory.

Step 2: Download Confluent Platform packages

  1. Download the Confluent Platform packages and copy them into a directory for distribution:

    # Downloads debs and their dependencies into /var/cache/apt/archives
apt-get install --download-only --yes \
  confluent-platform \
  confluent-security \
  ansible \
  openjdk-11-jdk \
  rsync \
  ca-certificates \
  apt-transport-https \
  python-apt \
  dpkg-dev
    Copy
    mkdir /tmp/confluent-packages

cp /var/cache/apt/archives/*.deb /tmp/confluent-packages
    Copy

  2. Add the Confluent repository key and the repository on the distribution server:

    wget -qO - https://packages.confluent.io/deb/7.9/archive.key | sudo apt-key add -
    Copy
    add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/7.9 stable main"
    Copy
    apt install software-properties-common --yes

apt-add-repository --yes --update ppa:ansible/ansible

apt-get update
    Copy

  3. Download the required JARs if you want to enable monitoring:

    curl -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \
  http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar

curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
    Copy

Step 3: Host Confluent Platform packages on control node

Copy the Confluent Platform packages to the control node and run a simple repository server over a Python web server.

  1. On the distribution server, package and move the the files to the control node:

    cd /tmp

tar -zcvf confluent-packages.tar.gz confluent-packages

scp /tmp/confluent-packages.tar.gz <control-node>:/tmp/confluent-packages.tar.gz
    Copy

  2. On the control node, decompress and install the packages:

    cd /tmp
tar xvzf confluent-packages.tar.gz -C /tmp

cd /tmp/confluent-packages/
apt install ./dpkg-dev*.deb --yes

dpkg-scanpackages ./ /dev/null | gzip > Packages.gz
    Copy

  3. On the control node, host a repository server with http.server:

    python3 -m http.server 8080
    Copy

Step 4: Install Confluent Ansible

Install Ansible on the control node, create a repo file for the rest of the Confluent Platform hosts to use, and install Confluent Ansible.

  1. On the control node, install Ansible:

    cd /tmp/confluent-packages/
apt install ./ansible-*.deb --yes
    Copy

  2. Create a .list file that each Confluent Platform host will use:

    echo "deb [trusted=yes] http://<control-node>:8080 ./" > /tmp/custom_confluent.list
    Copy

  3. Configure Confluent Ansible as described in Configure Ansible Playbooks for Confluent Platform.

  4. Add the following in the Ansible inventory file:

    all:
  vars:
    repository_configuration: custom
    custom_apt_repo_filepath: /tmp/custom_confluent.list
    Copy

  5. Configure monitoring in the Ansible inventory file:

    • To enable Jolokia:

      jolokia_enabled: true
jolokia_url_remote: false
jolokia_jar_url: http://<control-node>:8080/jolokia-jvm-1.6.2-agent.jar
      Copy

    • To enable JMX exporter:

      jmxexporter_enabled: true
jmxexporter_url_remote: false
jmxexporter_jar_url: http://<control-node>:8080/jmx_prometheus_javaagent-0.12.0.jar
      Copy

  6. Install Confluent Ansible as described in the Installation Guide.