Support Bundle and Diagnostics

Generate a support bundle and collect diagnostics for Confluent for Kubernetes using the following steps.

Support bundle

Confluent for Kubernetes (CFK) aggregates information, such as events, Kubernetes versions, the log, and Confluent APIs status in a tar.gz file for you to upload to the Confluent Support site.

You can create a support bundle to provide Confluent the required information for debugging.

Use the latest version of the Confluent plugin tool to generate a support bundle to capture the logs and configurations for the new features that are available in the new release of CFK.

The CFK support bundles aren’t currently supported on Microsoft Windows.

A support bundle includes the following information:

For the Kubernetes cluster (in the top-level directory):
- The Kubernetes events (kubectl get events)
- The Kubernetes server version (kubectl version)
For each Confluent Platform cluster component (in the /clusters-resources/component directory):
- The ConfigMaps generated by CFK for the component
- The pod logs for each pod in the component service (kubectl logs <pod-names>)
- The custom resource object for the component (kubectl get <component_cr_type> <component_name>)
- The StatefulSet object generated by CFK for the component (kubectl get sts <component-sts-name>)
For each Confluent Platform application component (in the /application-resources/component directory):
- The custom resource object for the component (kubectl get <component_cr_type> <component_name>)
For the Confluent for Kubernetes Operator deployment (in the /operator directory):
- The CFK Operator pod logs (kubectl logs confluent-operator)
- The deployment object generated for the CFK Operator (kubectl get deployment confluent-operator -oyaml)

For a sample support bundle, see Sample support bundle contents.

Create a support bundle

To create a support bundle:

Install the Confluent plugin as described in Confluent plugin.
Run the following command to create a support bundle:
```
kubectl confluent support-bundle --namespace <namespace>
```
When you have large logs with debug/trace logging enabled, you can capture the log files with the follow-logs-duration option. Specify the duration (in seconds) to follow the pod logs:
```
kubectl confluent support-bundle --namespace <namespace> --follow-logs-duration <duration in seconds to follow the logs>
```

To see other flags you can use to customize your support bundle, run:

kubectl confluent support-bundle -h

Sample support bundle contents

Extract the contents of the CFK support bundle using the following command:

tar -xzf support-bundle-ns-confluent.tar.gz

Note

If you deploy Confluent Gateway using Confluent for Kubernetes, the support bundle also includes Confluent Gateway configuration and log files.

The following is a sample list of files extracted directly into the directory where you run the above tar command (the archive has no top-level wrapper directory):

├── cluster-details.txt
├── application-resources
│   ├── clusterlink
│   │   └── democlusterlink.yaml
│   ├── confluentrolebinding
│   │   ├── internal-connect-0.yaml
│   │   ├── internal-connect-1.yaml
│   │   ├── internal-connect-2.yaml
│   │   ├── internal-controlcenter-0.yaml
│   │   ├── internal-kafkarestproxy-0.yaml
│   │   ├── internal-kafkarestproxy-1.yaml
│   │   ├── internal-ksqldb-0.yaml
│   │   ├── internal-ksqldb-1.yaml
│   │   ├── internal-ksqldb-2.yaml
│   │   ├── internal-schemaregistry-0.yaml
│   │   └── internal-schemaregistry-1.yaml
│   ├── connector
│   │   └── democonnector.yaml
│   ├── kafkarestclass
│   │   └── default.yaml
│   ├── kafkatopic
│   │   └── demotopic.yaml
│   └── kraftmigrationjob
│       └── kraft-migration.yaml
├── cluster-resources
│   ├── connect
│   │   └── connect
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── connect.yaml
│   │       ├── logs
│   │       │   ├── connect-0.log
│   │       │   └── connect-1.log
│   │       ├── pdb
│   │       │   └── pdb-connect.yaml
│   │       ├── pods
│   │       │   ├── pod-connect-0.yaml
│   │       │   └── pod-connect-1.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   ├── controlcenter
│   │   └── controlcenter
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── controlcenter.yaml
│   │       ├── pods
│   │       │   └── pod-controlcenter-0.yaml
│   │       ├── pv
│   │       │   └── pv-controlcenter-0-pvc-<pvc-uid>.yaml
│   │       ├── pvc
│   │       │   └── pvc-data0-controlcenter-0.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   ├── kafka
│   │   └── kafka
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── kafka.yaml
│   │       ├── logs
│   │       │   ├── kafka-0.log
│   │       │   ├── kafka-1.log
│   │       │   └── kafka-2.log
│   │       ├── pdb
│   │       │   └── pdb-kafka.yaml
│   │       ├── pods
│   │       │   ├── pod-kafka-0.yaml
│   │       │   ├── pod-kafka-1.yaml
│   │       │   └── pod-kafka-2.yaml
│   │       ├── pv
│   │       │   ├── pv-kafka-0-pvc-<pvc-uid>.yaml
│   │       │   ├── pv-kafka-1-pvc-<pvc-uid>.yaml
│   │       │   └── pv-kafka-2-pvc-<pvc-uid>.yaml
│   │       ├── pvc
│   │       │   ├── pvc-data0-kafka-0.yaml
│   │       │   ├── pvc-data0-kafka-1.yaml
│   │       │   └── pvc-data0-kafka-2.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   ├── kafkarestproxy
│   │   └── kafkarestproxy
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── kafkarestproxy.yaml
│   │       ├── pods
│   │       │   └── pod-kafkarestproxy-0.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   ├── ksqldb
│   │   └── ksqldb
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── ksqldb.yaml
│   │       ├── pdb
│   │       │   └── pdb-ksqldb.yaml
│   │       ├── pods
│   │       │   ├── pod-ksqldb-0.yaml
│   │       │   └── pod-ksqldb-1.yaml
│   │       ├── pv
│   │       │   ├── pv-ksqldb-0-pvc-<pvc-uid>.yaml
│   │       │   └── pv-ksqldb-1-pvc-<pvc-uid>.yaml
│   │       ├── pvc
│   │       │   ├── pvc-data0-ksqldb-0.yaml
│   │       │   └── pvc-data0-ksqldb-1.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   ├── schemaregistry
│   │   └── schemaregistry
│   │       ├── cm
│   │       │   └── configmaps.yaml
│   │       ├── pdb
│   │       │   └── pdb-schemaregistry.yaml
│   │       ├── pods
│   │       │   └── pod-schemaregistry-0.yaml
│   │       ├── schemaregistry.yaml
│   │       ├── sts
│   │       │   └── statefulset.yaml
│   │       └── svc
│   │           └── services.yaml
│   └── zookeeper
│       └── zookeeper
│           ├── cm
│           │   └── configmaps.yaml
│           ├── logs
│           │   ├── zookeeper-0.log
│           │   └── zookeeper-1.log
│           ├── pdb
│           │   └── pdb-zookeeper.yaml
│           ├── pods
│           │   ├── pod-zookeeper-0.yaml
│           │   ├── pod-zookeeper-1.yaml
│           │   └── pod-zookeeper-2.yaml
│           ├── pv
│           │   ├── pv-zookeeper-0-pvc-<pvc-uid>.yaml
│           │   └── pv-zookeeper-1-pvc-<pvc-uid>.yaml
│           ├── pvc
│           │   ├── pvc-data0-zookeeper-0.yaml
│           │   └── pvc-data0-zookeeper-1.yaml
│           ├── sts
│           │   └── statefulset.yaml
│           ├── svc
│           │   └── services.yaml
│           └── zookeeper.yaml
├── event.txt
├── event.yaml
├── misc
│   ├── kubectl-all-confluent.txt
│   ├── kubectl-controller-revisions-confluent.txt
│   └── kubectl-endpoints-confluent.txt
└── operator
    ├── deployment.yaml
    └── confluent-operator
        └── logs
            └── confluent-operator-6f587554c7-abcc7.log

Heap dump

If troubleshooting memory-related issues in CFK, such as out-of-memory errors caused by a memory leak, you can generate heap dumps.

To capture the heap dumps, you can install the confluent kubectl plugins, and use the heapdump option:

kubectl confluent cluster <cluster_name> heapdump \
   --pod-names <pod names> \
   --interval-seconds <seconds> \
   --run-count <run count> \
   --out-dir <output directory> \
   --namespace <namespace>

The command takes the following optional flags:

pod-names: The default values are all pods in the cluster.
interval-seconds: Must be 60 or greater. The default value is 60.
run-count: The default value is 1, and the limit is 10.
out-dir: The default value is ./.
namespace: The namespace where you deployed the Confluent Platform pod or cluster.

The command writes the heap dump output to:

<out-dir>/heapdump-<cluster-name>-all-ns-<namespace>.tar.gz

Example plugin commands:

To generate the heap dump for all Kafka cluster pods:

kubectl confluent cluster kafka heapdump \
  --out-dir $OUT_DIR \
  --namespace $NAMESPACE

To generate the heap dump for the kafka-0 Kafka pod in the Kafka cluster:

kubectl confluent cluster kafka heapdump \
  --pod-names kafka-0 \
  --out-dir $OUT_DIR \
  --namespace $NAMESPACE

For alternate ways to generate the heap dump using jcmd, see the Confluent Support Knowledgebase article (requires login).

Thread dump

If troubleshooting issues in CFK, such as slowness, performance, high CPU usage, or hanging issues, you can take thread dumps over time at intervals to analyze the pattern and thread progress.

To capture and debug the thread dumps, you can install the confluent kubectl plugins, and run the threaddump plugin command:

kubectl confluent cluster <cluster_name> threaddump \
   --pod-names <pod names> \
   --interval-seconds <seconds> \
   --run-count <run count> \
   --out-dir <output directory> \
   --namespace <namespace>

The command takes the following optional flags:

pod-names: The default values are all pods in the cluster.
interval-seconds: Must be 60 or greater. The default value is 60.
run-count: The default value is 1, and the limit is 10.
out-dir: The default value is ./.
namespace: The namespace where you deployed the Confluent Platform pod or cluster.

The command writes the output to:

<out-dir>/threaddump-<cluster-name>-all-ns-<namespace>.tar.gz

The following are example usages of the CFK dump plugins.

Thread dump of the CFK operator pod in the confluent namespace:
```
kubectl confluent operator threaddump --namespace confluent
```
Note

In CFK 3.3.0, this command can fail with exec: "curl": executable file not found in $PATH because the CFK Operator image doesn’t include curl. If you hit this error, use the operator pod’s kubectl logs output or a manual jstack command instead.
Thread dump of all pods in a Kafka cluster in operator namespace:
```
kubectl confluent cluster kafka threaddump
```

Thread dump of a specific pod in a Kafka cluster for five times every 60 seconds:

kubectl confluent cluster kafka threaddump \
   --pod-names kafka-0 \
   --interval-seconds 60 \
   --run-count 5 \
   --out-dir $OUT_DIR \
   --namespace $NAMESPACE

Thread dump of all Connect pods:

kubectl confluent cluster connect threaddump --namespace confluent

Thread dump of a specific pod in the Connect cluster three times at 60 seconds intervals in a custom directory:

kubectl confluent cluster connect threaddump \
  --namespace confluent \
  --out-dir /tmp \
  --interval-seconds 60 \
  --run-count 3 \
  --pod-names connect-0

Thread dump of all Connect pods three times at 60 seconds intervals in the custom directory:

kubectl confluent cluster connect threaddump \
  --namespace confluent \
  --out-dir /tmp \
  --interval-seconds 60 \
  --run-count 3

For an alternate way to generate the thread dump using jstack, see the Confluent Support Knowledgebase article (requires login).

Logs

Use the command below to view the logs for a currently running pod.

Logs are sent directly to STDOUT for each pod.

kubectl logs <pod-names> -n <namespace>

Additionally, in the component custom resources, you can change the log level to DEBUG using the Configuration overrides feature. For example, add the following in your Kafka CR to get more details in the Kafka logs:

In Confluent Platform 7.x, override Log4j configuration as below:

spec:
  configOverrides:
    log4j:
      - log4j.rootLogger=DEBUG, stdout

In Confluent Platform 8.x, override Log4j 2 configuration as below:

spec:
  configOverrides:
    log4j2:
      Configuration:
        Appenders:
          Console:
            - name: stdout
              target: SYSTEM_OUT
        Loggers:
          Root:
            - name: stdout
              level: DEBUG
              AppenderRef:
                - ref: stdout

View logs of crashed pods

If Kubernetes replaces a crashed container, you can retrieve the logs from its previous instance using the --previous (-p) flag to investigate the root cause of the pod crash.

kubectl logs --previous <pod-name>  --namespace <namespace>

Metrics

JMX metrics are available on port 7203 of each pod.
Jolokia (a REST interface for JMX metrics) is available on port 7777 of each pod.

Debug

Problems that can go wrong while using Confluent for Kubernetes (CFK) generally fall into the following categories:

A problem happens while deploying CFK.
A problem exists at the infrastructure level.

Something has gone wrong at the Kubernetes layer.
A problem exists at the application level.

The infrastructure is fine but something has gone wrong with Confluent Platform itself. Typically, how you configured Confluent Platform components causes this.

To debug deployment problems, run the Helm install command with the --set debug="true" to enable verbose output:

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --namespace <namespace> \
  --set debug="true"

Look for Kubernetes issues first, then debug Confluent Platform.

Check for potential Kubernetes errors by entering the following command:
```
kubectl get events -n <namespace>
```
Check for a specific resource issue, enter the following command (using the resource type example pods):
```
kubectl describe pods <podname> -n <namespace>
```
If everything looks okay after running the preceding commands, check the individual pod logs using the following command:
```
kubectl logs <pod name> -n <namespace>
```
Confluent Platform containers write application logs directly to STDOUT, so you can read the logs directly with this command. If there is anything wrong at the application level, such as an invalid configuration, it appears in the logs.

If Kubernetes replaced a crashed pod and you want to check the previous pod’s logs, add --previous to the end of the command above.

Your infrastructure system administrator should resolve problems caused by the datacenter infrastructure, such as virtual machine (VM) firewall rules and DNS configuration.