Air-gapped Deployment of Ansible Playbooks for Confluent Platform¶
This topic guides you through the steps to use Ansible Playbooks to deploy Confluent Platform without internet connectivity to outside networks.
This scenario requires:
A distribution server with internet connectivity
You download the Ansible Playbooks for Confluent Platform collection and the Confluent Platform packages from the Confluent site to your distribution server.
You need to install
ansible-galaxy
on the distribution server.A control node to run Ansible
This internal repository should be accessible from your Confluent Platform hosts and the distribution server.
Confluent Platform hosts
Internal hosts to run Confluent Platform.
The Confluent Platform hosts must be able to reach the control node at port 8080.
At the high level, the required steps are:
- Download the Confluent Platform packages and other required files, such as monitoring JARs and Confluent CLI, along with Ansible collections. (This step requires internet connectivity.)
- Put the packages and other required files, such as monitoring JARs and Confluent CLI, along with Ansible collections on your repository server.
- Configure the variables for Ansible to use an internal custom repository server for Confluent Platform installation.
- Install Confluent Platform.
Deploy using tarball archives¶
Step 1: Distribute Ansible Playbooks for Confluent Platform collection¶
Build the Ansible Playbooks for Confluent Platform (cp-ansible) collection and install the collection on the control node.
Download the Ansible Playbooks for Confluent Platform code to the distribution server:
git clone https://github.com/confluentinc/cp-ansible.git cd cp-ansible git checkout 7.4.0-post
Create the collection:
ansible-galaxy collection build
The command builds a tarball of the collection in the current directory.
Upload the collection tarball to your control node:
scp <collection tarball> <control-node>:/path/to/confluent-collection
Install the collection on the control node:
ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
For more information about Ansible collection distribution, see Building your collection tarball.
Step 2: Download Confluent Platform archives¶
Download the Confluent Platform archives to the distribution server:
curl -O http://packages.confluent.io/archive/7.4/confluent-7.4.0.tar.gz
If enabling monitoring, download the required JARs:
curl -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \ http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar
curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
Move the packages and files to the control node:
scp confluent-packages.tar.gz <control-node>:/path/to/confluent-packages.tar.gz
On the control node, extract the contents of the Confluent Platform archive:
tar xzf confluent-7.4.0.tar.gz
Step 3: Install Ansible Playbooks for Confluent Platform¶
Configure Ansible Playbooks for Confluent Platform as described in Configure Ansible Playbooks for Confluent Platform.
Add the following in the Ansible inventory file.
To install Confluent Platform using a tarball hosted on a web server:
all: vars: repository_configuration: custom installation_method: archive confluent_archive_file_source: <web server url>/path/to/confluent-7.4.0.tar.gz
To install Confluent Platform using a tarball placed on the control node:
all: vars: installation_method: archive confluent_archive_file_source: /path/to/confluent-7.4.0.tar.gz confluent_archive_file_remote: false
Configure monitoring in the Ansible inventory file:
To enable Jolokia:
jolokia_enabled: true jolokia_url_remote: false jolokia_jar_url: /path/to/jolokia-jvm-1.6.2-agent.jar
To enable JMX exporter:
jmxexporter_enabled: true jmxexporter_url_remote: false jmxexporter_jar_url: /path/to/jmx_prometheus_javaagent-0.12.0.jar
Install Ansible Playbooks for Confluent Platform as described in the Installation Guide.
Deploy using RHEL or CentOS packages¶
Step 1: Distribute Ansible Playbooks for Confluent Platform collection¶
Build the Ansible Playbooks for Confluent Platform (cp-ansible) collection and install the collection on the control node.
Download the Ansible Playbooks for Confluent Platform code to the distribution server:
git clone https://github.com/confluentinc/cp-ansible.git cd cp-ansible git checkout 7.4.0-post
Create the collection:
ansible-galaxy collection build
The command builds a tarball of the collection in the current directory.
Upload the collection tarball to your control node:
scp <collection tarball> <control-node>:/path/to/confluent-collection
Install the collection on your control node:
ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
For more information about Ansible collection distribution, see Building your collection tarball.
Step 2: Download Confluent Platform packages¶
On the distribution server:
Install the
epel-release
package:yum -y install epel-release
Create the
confluent.repo
file for pulling the Confluent Platform packages:echo "[Confluent.dist] baseurl = https://packages.confluent.io/rpm/7.4 enabled = 1 gpgcheck = 1 gpgkey = https://packages.confluent.io/rpm/7.4/archive.key name = Confluent repository (dist) [Confluent] baseurl = https://packages.confluent.io/rpm/7.4 enabled = 1 gpgcheck = 1 gpgkey = https://packages.confluent.io/rpm/7.4/archive.key name = Confluent repository" \ > /etc/yum.repos.d/confluent.repo
Create a directory and download the Confluent Platform packages into that directory:
mkdir /tmp/confluent-packages/
yum clean all yum install --downloadonly --downloaddir=/tmp/confluent-packages/ \ confluent-platform \ confluent-security \ nc \ ansible \ ntp \ createrepo \ java-1.8.0-openjdk
Download the required JARs if you want to enable monitoring:
curl -L -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \ http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
Step 3: Host Confluent Platform packages on control node¶
Copy the Confluent Platform packages to the control node and run a simple repository server over a Python web server.
On the distribution server, package and move the the files to the control node:
cd /tmp tar -zcvf confluent-packages.tar.gz confluent-packages scp /tmp/confluent-packages.tar.gz <control-node>:/tmp/confluent-packages.tar.gz
On the control node, decompress and install the packages:
cd /tmp tar xvzf confluent-packages.tar.gz -C /tmp
cd /tmp/confluent-packages/ yum -y localinstall createrepo-*.rpm
Create a repo, and host the control node with python SimpleHTTPServer:
createrepo .
python -m SimpleHTTPServer 8080
Step 4: Install Ansible Playbooks for Confluent Platform¶
Install Ansible on the control node, create a repo file for the rest of the Confluent Platform hosts to use, and install Confluent Platform.
On the control node, install Ansible Playbooks for Confluent Platform:
cd /tmp/confluent-packages/ yum -y localinstall ansible-*.rpm
On the control node, create a
.repo
file that each host will use. If your organization has a repository server, that file should be configured for the existing repository server.echo "[confluent-repo] name=Confluent Repo baseurl=http://<ansible-control-node>:8080 enabled=1 gpgcheck=0" \ > /tmp/confluent.repo
Configure Ansible Playbooks for Confluent Platform as described in Configure Ansible Playbooks for Confluent Platform, and add the following in the Ansible inventory file:
all: vars: repository_configuration: custom custom_yum_repofile_filepath: /tmp/confluent.repo
Configure monitoring in the Ansible inventory file:
To enable Jolokia:
jolokia_enabled: true jolokia_url_remote: false jolokia_jar_url: http://<control-node>:8080/jolokia-jvm-1.6.2-agent.jar
To enable JMX exporter:
jmxexporter_enabled: true jmxexporter_url_remote: false jmxexporter_jar_url: http://<control-node>:8080/jmx_prometheus_javaagent-0.12.0.jar
Install Ansible Playbooks for Confluent Platform as described in the Installation Guide.
Deploy using Ubuntu or Debian packages¶
Step 1: Distribute Ansible Playbooks for Confluent Platform collection¶
Build the Ansible Playbooks for Confluent Platform (cp-ansible) collection and install the collection on the control node.
Download the Ansible Playbooks for Confluent Platform code to the distribution server:
git clone https://github.com/confluentinc/cp-ansible.git cd cp-ansible git checkout 7.4.0-post
Create the collection:
ansible-galaxy collection build
The command builds a tarball of the collection in the current directory.
Upload the collection tarball to your control node:
scp <collection tarball> <control-node>:/path/to/confluent-collection
Install the collection on your control node:
ansible-galaxy collection install /path/to/confluent-collection/<collection tarball>
For more information about Ansible collection distribution, see Building your collection tarball.
Step 2: Download Confluent Platform packages¶
Add the Confluent repository key and the repository on the distribution server:
wget -qO - https://packages.confluent.io/deb/7.4/archive.key | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/7.4 stable main"
apt install software-properties-common --yes apt-add-repository --yes --update ppa:ansible/ansible apt-get update
Download the Confluent Platform packages and copy them into a directory for distribution:
# Downloads debs and their dependencies into /var/cache/apt/archives apt-get install --download-only --yes \ confluent-platform \ confluent-security \ ansible \ openjdk-8-jdk \ rsync \ ntp \ ca-certificates \ apt-transport-https \ python-apt \ dpkg-dev
mkdir /tmp/confluent-packages cp /var/cache/apt/archives/*.deb /tmp/confluent-packages
Download the required JARs if you want to enable monitoring:
curl -o /tmp/confluent-packages/jolokia-jvm-1.6.2-agent.jar \ http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar curl -o /tmp/confluent-packages/jmx_prometheus_javaagent-0.12.0.jar \ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
Step 3: Host Confluent Platform packages on control node¶
Copy the Confluent Platform packages to the control node and run a simple repository server over a Python web server.
On the distribution server, package and move the the files to the control node:
cd /tmp tar -zcvf confluent-packages.tar.gz confluent-packages scp /tmp/confluent-packages.tar.gz <control-node>:/tmp/confluent-packages.tar.gz
On the control node, decompress and install the packages:
cd /tmp tar xvzf confluent-packages.tar.gz -C /tmp cd /tmp/confluent-packages/ apt install ./dpkg-dev*.deb --yes dpkg-scanpackages ./ /dev/null | gzip > Packages.gz
On the control node, host a repository server with Python SimpleHTTPServer:
python -m SimpleHTTPServer 8080
Step 4: Install Ansible Playbooks for Confluent Platform¶
Install Ansible on the control node, create a repo file for the rest of the Confluent Platform hosts to use, and install Ansible Playbooks for Confluent Platform.
On the control node, install Ansible:
cd /tmp/confluent-packages/ apt install ./ansible-*.deb --yes
Create a
.list
file that each Confluent Platform host will use:echo "deb [trusted=yes] http://<control-node>:8080 ./" > /tmp/custom_confluent.list
Configure Ansible Playbooks for Confluent Platform as described in Configure Ansible Playbooks for Confluent Platform.
Add the following in the Ansible inventory file:
all: vars: repository_configuration: custom custom_apt_repo_filepath: /tmp/custom_confluent.list
Configure monitoring in the Ansible inventory file:
To enable Jolokia:
jolokia_enabled: true jolokia_url_remote: false jolokia_jar_url: http://<control-node>:8080/jolokia-jvm-1.6.2-agent.jar
To enable JMX exporter:
jmxexporter_enabled: true jmxexporter_url_remote: false jmxexporter_jar_url: http://<control-node>:8080/jmx_prometheus_javaagent-0.12.0.jar
Install Ansible Playbooks for Confluent Platform as described in the Installation Guide.