<a id="ansible-troubleshooting"></a>

# Troubleshoot Ansible Playbooks for Confluent Platform

Complete the following steps if Ansible Playbooks for Confluent Platform (Confluent Ansible) fails:

1. Review the error log output from Ansible itself.

   It will show the type of failure which has occurred and might indicate a
   misconfiguration in your inventory file. For example, if you set a file path
   variable to an invalid  path, the logs will say “Could not find or access “
   the file path, and you need to correct the variable and rerun the
   install.
2. Review your inventory file.

   Validate that all variables are set correctly, with proper spacing in the
   inventory file. You can review `hosts_example.yml` and the
   `sample_inventories` directory for examples.
3. Review component log and property files.

   If a component health check fails, the playbook will fetch log and property
   files back to the Ansible Control  Node inside the `error_files/`
   directory. These error logs can indicate which properties are misconfigured.
   - If Confluent Ansible  was downloaded from Ansible Galaxy or Ansible
     Automation Hub, the `error_files/` directory is under
     `~/.ansible/collections/ansible_collections/confluent/platform/`.
   - If Confluent Ansible was downloaded from GitHub, the `error_files/`
     directory is located under the root of `cp-ansible`.
4. If the log files do not provide a clear reason for the failure, use one of
   the following methods to generate more info:
   * Rerun the playbook with the `--diff` option and redirect the output to a
     file. For more information about the flag, see [Ansible Playbook Options](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#cmdoption-ansible-playbook-D).

     This outputs the differences in the playbook files and templates. With this
     option, sensitive information, such as passwords, certificates, and keys,
     are not printed in the output.
   * Rerun the playbook again with the `-vvv` option to enable debug
     logging and redirect the output to a file:
     ```bash
     ansible-playbook -vvv -i hosts.yml confluent.platform.all > failure.txt
     ```

     When debug is enabled, the information in the output cannot be suppressed,
     including sensitive information, such as passwords, certificates, and keys.
     It is not recommended to use the debug mode in production environments. For
     details, see [Logging Ansible Output](https://docs.ansible.com/ansible/latest/reference_appendices/logging.html).
   * [Generate logs using the fetch-logs playbook](#ansible-fetch-logs).
5. Open a support ticket with [Confluent Support](https://support.confluent.io) and provide the following within a
   compressed archive file:
   * Your inventory file
   * The log files generated from the `-vvv` or `--diff` option.
   * The `error_logs/` directory and its contents
   * The output of the following GIT commands as a text file. Run the command
     from the root of `cp-ansible` to show any changes made to the
     `cp-ansible` source code:
     ```none
     git status
     ```

     ```none
     git diff
     ```

<a id="ansible-support-bundle"></a>

## Support bundle

When you open a ticket with [Confluent Support](https://support.confluent.io),
you can attach a support bundle to provide all the necessary diagnostic data in
a single archive.

#### NOTE
Creating a support bundle is supported with Confluent Ansible 7.4 or later.

### What’s collected

For each component in your inventory, the support bundle collects the
following:

- Main configuration file
- Log4j configuration file
- The `systemd` service file and its override, if defined
- The client configuration file, if defined
- All log files in the component’s log directory
- Diagnostics (service status, `journalctl` output, and process information)

The support bundle covers ZooKeeper, Kafka brokers, Schema Registry, Connect, ksqlDB, Confluent Control Center (Legacy),
and the Unified Stream Manager agent.

The support bundle is a single `support_bundle_<timestamp>_<cluster_name>.tar.gz` archive file that is organized as follows:

```text
support_bundle_<timestamp>_<cluster_name>.tar.gz
├── ansible/
│   ├── inventory_sanitized.yml
│   └── ansible_logs.txt
├── bundle_manifest.yml
└── <cp-component>/
    └── <hostname>/
          ├── configs/
          ├── logs/
          └── diagnostics/
```

By default, passwords and other secrets are masked from the collected
configuration and log files before the archive is created. For details, see
[Configuration variables](#ansible-support-bundle-variables).

### Create a support bundle automatically on failure

You can configure Confluent Ansible to automatically create a support bundle
whenever a deployment playbook fails. The bundle is created before the
playbook exits, and the original error is still displayed.

To enable automatic collection, enable the callback plugin in your
`ansible.cfg` file:

```ini
[defaults]
callbacks_enabled = support_bundle_on_failure
callback_plugins = ./callback_plugins
```

Once the callback plugin is enabled, a bundle is collected automatically
because the `support_bundle_auto_collect_on_failure` variable is `true` by
default. To disable automatic collection, set this variable to `false`.

### Create a support bundle manually

- To create a support bundle from all components in your inventory:
  ```bash
  ansible-playbook -i inventory.yml confluent.platform.support_bundle
  ```
- To create a support bundle from specific components, use the `--tags` flag with the
  component names. For example, to create a bundle from Kafka brokers and Schema Registry:
  ```bash
  ansible-playbook -i inventory.yml confluent.platform.support_bundle --tags kafka_broker,schema_registry
  ```
- To exclude specific components, use the `--skip-tags` flag. For example, to
  create a bundle from all components except Confluent Control Center (Legacy):
  ```bash
  ansible-playbook -i inventory.yml confluent.platform.support_bundle --skip-tags control_center
  ```

By default, the bundle is written to the
`~/confluent-platform-support-bundles` directory. To change the output
location, set the `support_bundle_output_path` variable. For details, see
[Configuration variables](#ansible-support-bundle-variables).

To inspect a support bundle, extract the archive:

```bash
tar -xzf support_bundle_<timestamp>_<cluster_name>.tar.gz
```

<a id="ansible-support-bundle-variables"></a>

### Configuration variables

Use the following variables to control how the support bundle is created:

| Variable                                 | Default                                | Description                                                                                 |
|------------------------------------------|----------------------------------------|---------------------------------------------------------------------------------------------|
| `support_bundle_output_path`             | `~/confluent-platform-support-bundles` | Sets the local directory for the final `.tar.gz` bundle.                                    |
| `support_bundle_sanitize_configs`        | `true`                                 | Masks passwords and secrets from configuration and log files.                               |
| `support_bundle_journalctl_lines`        | `0`                                    | Sets the number of `journalctl` lines to collect per service.                               |
| `support_bundle_auto_collect_on_failure` | `true`                                 | Collects a bundle automatically when a playbook fails.                                      |
| `support_bundle_ansible_log_path`        | `""`                                   | Specifies the path to an Confluent Ansible execution log file to include<br/>in the bundle. |

<a id="ansible-fetch-logs"></a>

## Generate logs using the fetch-logs playbook

When troubleshooting, you might need to collect the service logs and config
files of all Confluent components. Instead of having to `ssh` to each host
machine and fetch the logs/files, you can run the `fetch_logs` playbook to
get all service logs/config files in a single directory on the control node.

The playbook stores the gathered log files in a separate zip file for each
component. The zip files are located in:

- `~/.ansible/collections/ansible_collections/confluent/platform/playbooks/troubleshooting`
  if Confluent Ansible  was downloaded from Ansible Galaxy or Ansible Automation
  Hub
- `<the root of cp-ansible>/playbooks/troubleshooting` if Confluent Ansible was
  downloaded from GitHub `<cp-ansible-directory>`

To gather logs and config files of all components:

```yaml
ansible-playbook -i hosts.yml confluent.platform.fetch_logs
```

To gather service logs and config files of a specific component use the
`--tags` flag with the component name. For example, to get logs and config
files for Kafka:

```yaml
ansible-playbook -i hosts.yml confluent.platform.fetch_logs --tags 'kafka_broker'
```

## Troubleshoot known issues

### Issue: TypeError: dump_all() got an unexpected keyword argument ‘sort_keys’

The error happens when using Confluent Ansible 8.0 with Python 3.6 or lower
version. Python 3.11 or later version is required for Confluent Ansible 8.0.

**Solution:** Install Python 3.11 and a compatible PyYAML version, and configure
it as the system default using the steps below:

```bash
# Install Python 3.11 and pip
dnf install -y python3.11 python3.11-pip && dnf clean all

# Install compatible PyYAML version
/usr/bin/pip3.11 install --no-cache-dir PyYAML

# Set Python 3.11 and pip3.11 as system defaults
ln -sf /usr/bin/python3.11 /usr/bin/python3
ln -sf /usr/bin/pip3.11 /usr/bin/pip3

# Register Python 3.11 as the unversioned default
alternatives --install /usr/bin/unversioned-python unversioned-python /usr/bin/python3.11 1
```

### Issue: An error, “Clusters not found”, returns after Kafka brokers restart

If keys get updated on Kafka brokers or any other Confluent Platform component, the
communication between component services and the brokers would get broken.

**Solution:** Regenerate certificates when you are updating the keys,
during an update or a redeployment of the cluster.

* To regenerate certificates along with keys in your inventory file:
  ```yaml
  regenerate_ca: true
  ```
* To only update the certificates whe keys have been already generated:
  ```yaml
  regenerate_ca: true

  regenerate_keystore_and_truststore: false
  ```

<a id="ansible-python-version-mismatch"></a>

### Issue: Python version mismatch

If you are using different versions of Python across the nodes,
for example, the Ansible control node has python 2.7 installed while the target
nodes has Python 3 set as default python, you may hit the following error:

```text
The Python 2 bindings for rpm are needed for this module. If you require
Python 3 support use the `dnf` Ansible module instead. The Python 2 yum
module is needed for this module. If you require Python 3 support use the
`dnf` Ansible module instead.
```

**Solution:** Install the same, recommended Python version on all of your
control nodes and managed nodes.

### Issue: Missing Ansible POSIX collection

If the required [Ansible POSIX collection](ansible-requirements.md#ansible-requirements-misc) is
missing in your environment, you will get an error similar to:

```text
ERROR! couldn't resolve module/action 'sysctl'
```

**Solution:** Install the Ansible POSIX collection.

```bash
ansible-galaxy collection install ansible.posix
```

### Issue: Incorrect Ansible hash behavior

If the default Ansible hash behavior is not set to `MERGE`, you will get an
error similar to:

```text
TASK [confluent.platform.common : Confirm Hash Merging Enabled]

fatal: [ip-10-0-2-212.us-west-2.compute.internal]: FAILED! => {
    "assertion": "lookup('config', 'DEFAULT_HASH_BEHAVIOUR') == 'merge'",
    "changed": false,
    "evaluated_to": false,
    "msg": "Hash Merging must be enabled in ansible.cfg"
}
```

**Solution**: Set the Ansible hash behavior to `merge`.

```bash
export ANSIBLE_HASH_BEHAVIOUR=merge
```

### Issue: Missing Ansible community general collection

If the required [Ansible community general collection](ansible-requirements.md#ansible-requirements-misc) is missing in your environment, you will get an
error similar to:

```text
TASK [confluent.platform.common : Custom Java Install]

ERROR! couldn't resolve module/action 'alternatives'.
```

**Solution**: Install the Ansible community general collection.

```bash
ansible-galaxy collection install community.general
```

### Issue: Missing Ansible community crypto collection

If the required [Ansible community crypto collection](ansible-requirements.md#ansible-requirements-misc) is missing in your environment, you will get an
error similar to:

```text
TASK [confluent.platform.ssl : Create Keystore and Truststore with Self Signed Certs]

ERROR! couldn't resolve module/action 'community.crypto.certificate_complete_chain'
```

**Solution**: Install the Ansible community crypto collection.

```bash
ansible-galaxy collection install community.crypto
```

### Issue: Corrupted master key

When your master key is corrupted, you get an error message similar to the
following:

```text
TASK [confluent.platform.common : Encrypt Properties] **************************
task path: /root/.ansible/collections/ansible_collections/confluent/platform/roles/common/tasks/secrets_protection.yml

Error! failed to unwrap the data key: invalid master key or corrupted data key
```

**Solution**: Let Confluent Ansible recreate the master key.

1. Remove the variables, `regenerate_masterkey`,
   `secrets_protection_masterkey`, and `secrets_protection_security_file`
   from your inventory file.
2. Run the following command:
   ```bash
   ansible-playbook -i <inventory.yml> confluent.platform.all --skip-tags package
   ```

### Issue: An error while setting up KRaft-based cluster with mTLS RBAC and custom user

When setting up a KRaft-based mTLS RBAC cluster with custom user (greenfield
setup or a migration), you might received the following error:

```text
ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.nio.file.AccessDeniedException: /etc/controller/server.properties
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
```

**Workaround:** Define `kafka_controller_user` and
`kafka_controller_group` variables under `all` and `kafka_broker` in
your inventory file if it is currently defined in the `kraft_controller`
section only.

```yaml
all:
  kafka_controller_user: <user>
  kafka_controller_group: <group>

kafka_broker:
  kafka_controller_user: <user>
  kafka_controller_group: <group>
```