# Support Bundle and Diagnostics Generate a support bundle and collect diagnostics for Confluent for Kubernetes using the following steps. ## Support bundle Confluent for Kubernetes (CFK) aggregates information, such as events, Kubernetes versions, the log, and Confluent APIs status in a `tar.gz` file for you to upload to the Confluent Support site. You can create a support bundle to provide Confluent the required information for debugging. Use the latest version of the Confluent plugin tool to generate a support bundle to capture the logs and configurations for the new features that are available in the new release of CFK. The CFK support bundles aren’t currently supported on Microsoft Windows. A support bundle includes the following information: * For the Kubernetes cluster (in the top-level directory): * The Kubernetes events (`kubectl get events`) * The Kubernetes server version (`kubectl version`) * For each Confluent Platform cluster component (in the `/clusters-resources/component` directory): * The ConfigMaps generated by CFK for the component * The pod logs for each pod in the component service (`kubectl logs `) * The custom resource object for the component (`kubectl get `) * The StatefulSet object generated by CFK for the component (`kubectl get sts `) * For each Confluent Platform application component (in the `/application-resources/component` directory): * The custom resource object for the component (`kubectl get `) * For the Confluent for Kubernetes Operator deployment (in the `/operator` directory): * The CFK Operator pod logs (`kubectl logs confluent-operator`) * The deployment object generated for the CFK Operator (`kubectl get deployment confluent-operator -oyaml`) For a sample support bundle, see [Sample support bundle contents](#co-sample-support-bundle). ### Create a support bundle To create a support bundle: 1. Install the Confluent plugin as described in [Confluent plugin](co-install-plugin.md#co-install-plugin). 2. Run the following command to create a support bundle: ```bash kubectl confluent support-bundle --namespace ``` When you have large logs with debug/trace logging enabled, you can capture the log files with the `follow-logs-duration` option. Specify the duration (in seconds) to follow the pod logs: ```bash kubectl confluent support-bundle --namespace --follow-logs-duration ``` To see other flags you can use to customize your support bundle, run: ```bash kubectl confluent support-bundle -h ``` ### Sample support bundle contents Extract the contents of the CFK support bundle using the following command: ```bash tar -xzf support-bundle-ns-confluent.tar.gz ``` #### NOTE If you deploy Confluent Gateway using Confluent for Kubernetes, the support bundle also includes Confluent Gateway configuration and log files. The following is a sample list of files extracted directly into the directory where you run the above `tar` command (the archive has no top-level wrapper directory): ```text ├── cluster-details.txt ├── application-resources │ ├── clusterlink │ │ └── democlusterlink.yaml │ ├── confluentrolebinding │ │ ├── internal-connect-0.yaml │ │ ├── internal-connect-1.yaml │ │ ├── internal-connect-2.yaml │ │ ├── internal-controlcenter-0.yaml │ │ ├── internal-kafkarestproxy-0.yaml │ │ ├── internal-kafkarestproxy-1.yaml │ │ ├── internal-ksqldb-0.yaml │ │ ├── internal-ksqldb-1.yaml │ │ ├── internal-ksqldb-2.yaml │ │ ├── internal-schemaregistry-0.yaml │ │ └── internal-schemaregistry-1.yaml │ ├── connector │ │ └── democonnector.yaml │ ├── kafkarestclass │ │ └── default.yaml │ ├── kafkatopic │ │ └── demotopic.yaml │ └── kraftmigrationjob │ └── kraft-migration.yaml ├── cluster-resources │ ├── connect │ │ └── connect │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── connect.yaml │ │ ├── logs │ │ │ ├── connect-0.log │ │ │ └── connect-1.log │ │ ├── pdb │ │ │ └── pdb-connect.yaml │ │ ├── pods │ │ │ ├── pod-connect-0.yaml │ │ │ └── pod-connect-1.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ ├── controlcenter │ │ └── controlcenter │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── controlcenter.yaml │ │ ├── pods │ │ │ └── pod-controlcenter-0.yaml │ │ ├── pv │ │ │ └── pv-controlcenter-0-pvc-.yaml │ │ ├── pvc │ │ │ └── pvc-data0-controlcenter-0.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ ├── kafka │ │ └── kafka │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── kafka.yaml │ │ ├── logs │ │ │ ├── kafka-0.log │ │ │ ├── kafka-1.log │ │ │ └── kafka-2.log │ │ ├── pdb │ │ │ └── pdb-kafka.yaml │ │ ├── pods │ │ │ ├── pod-kafka-0.yaml │ │ │ ├── pod-kafka-1.yaml │ │ │ └── pod-kafka-2.yaml │ │ ├── pv │ │ │ ├── pv-kafka-0-pvc-.yaml │ │ │ ├── pv-kafka-1-pvc-.yaml │ │ │ └── pv-kafka-2-pvc-.yaml │ │ ├── pvc │ │ │ ├── pvc-data0-kafka-0.yaml │ │ │ ├── pvc-data0-kafka-1.yaml │ │ │ └── pvc-data0-kafka-2.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ ├── kafkarestproxy │ │ └── kafkarestproxy │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── kafkarestproxy.yaml │ │ ├── pods │ │ │ └── pod-kafkarestproxy-0.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ ├── ksqldb │ │ └── ksqldb │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── ksqldb.yaml │ │ ├── pdb │ │ │ └── pdb-ksqldb.yaml │ │ ├── pods │ │ │ ├── pod-ksqldb-0.yaml │ │ │ └── pod-ksqldb-1.yaml │ │ ├── pv │ │ │ ├── pv-ksqldb-0-pvc-.yaml │ │ │ └── pv-ksqldb-1-pvc-.yaml │ │ ├── pvc │ │ │ ├── pvc-data0-ksqldb-0.yaml │ │ │ └── pvc-data0-ksqldb-1.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ ├── schemaregistry │ │ └── schemaregistry │ │ ├── cm │ │ │ └── configmaps.yaml │ │ ├── pdb │ │ │ └── pdb-schemaregistry.yaml │ │ ├── pods │ │ │ └── pod-schemaregistry-0.yaml │ │ ├── schemaregistry.yaml │ │ ├── sts │ │ │ └── statefulset.yaml │ │ └── svc │ │ └── services.yaml │ └── zookeeper │ └── zookeeper │ ├── cm │ │ └── configmaps.yaml │ ├── logs │ │ ├── zookeeper-0.log │ │ └── zookeeper-1.log │ ├── pdb │ │ └── pdb-zookeeper.yaml │ ├── pods │ │ ├── pod-zookeeper-0.yaml │ │ ├── pod-zookeeper-1.yaml │ │ └── pod-zookeeper-2.yaml │ ├── pv │ │ ├── pv-zookeeper-0-pvc-.yaml │ │ └── pv-zookeeper-1-pvc-.yaml │ ├── pvc │ │ ├── pvc-data0-zookeeper-0.yaml │ │ └── pvc-data0-zookeeper-1.yaml │ ├── sts │ │ └── statefulset.yaml │ ├── svc │ │ └── services.yaml │ └── zookeeper.yaml ├── event.txt ├── event.yaml ├── misc │ ├── kubectl-all-confluent.txt │ ├── kubectl-controller-revisions-confluent.txt │ └── kubectl-endpoints-confluent.txt └── operator ├── deployment.yaml └── confluent-operator └── logs └── confluent-operator-6f587554c7-abcc7.log ``` ## Heap dump If troubleshooting memory-related issues in CFK, such as out-of-memory errors caused by a memory leak, you can generate heap dumps. To capture the heap dumps, you can install the [confluent kubectl plugins](co-install-plugin.md#co-install-plugin), and use the [heapdump](co-plugin-cli/kubectl_confluent_cluster_kafka_heapdump.md#kubectl-confluent-cluster-kafka-heapdump) option: ```bash kubectl confluent cluster heapdump \ --pod-names \ --interval-seconds \ --run-count \ --out-dir

\ --namespace ``` The command takes the following optional flags: * `pod-names`: The default values are all pods in the cluster. * `interval-seconds`: Must be `60` or greater. The default value is `60`. * `run-count`: The default value is `1`, and the limit is `10`. * `out-dir`: The default value is `./`. * `namespace`: The namespace where you deployed the Confluent Platform pod or cluster. The command writes the heap dump output to: ```text /heapdump--all-ns-.tar.gz ``` Example plugin commands: * To generate the heap dump for all Kafka cluster pods: ```bash kubectl confluent cluster kafka heapdump \ --out-dir $OUT_DIR \ --namespace $NAMESPACE ``` * To generate the heap dump for the `kafka-0` Kafka pod in the Kafka cluster: ```bash kubectl confluent cluster kafka heapdump \ --pod-names kafka-0 \ --out-dir $OUT_DIR \ --namespace $NAMESPACE ``` For alternate ways to generate the heap dump using `jcmd`, see the [Confluent Support Knowledgebase article](https://support.confluent.io/hc/en-us/articles/15576875639700-How-to-create-heap-dump-for-Confluent-Platform-component-deployed-through-Confluent-for-Kubernetes) (requires login). ## Thread dump If troubleshooting issues in CFK, such as slowness, performance, high CPU usage, or hanging issues, you can take thread dumps over time at intervals to analyze the pattern and thread progress. To capture and debug the thread dumps, you can install the [confluent kubectl plugins](co-install-plugin.md#co-install-plugin), and run the [threaddump](co-plugin-cli/kubectl_confluent_cluster_kafka_threaddump.md#kubectl-confluent-cluster-kafka-threaddump) plugin command: ```bash kubectl confluent cluster threaddump \ --pod-names \ --interval-seconds \ --run-count \ --out-dir

\ --namespace ``` The command takes the following optional flags: * `pod-names`: The default values are all pods in the cluster. * `interval-seconds`: Must be `60` or greater. The default value is `60`. * `run-count`: The default value is `1`, and the limit is `10`. * `out-dir`: The default value is `./`. * `namespace`: The namespace where you deployed the Confluent Platform pod or cluster. The command writes the output to: ```text /threaddump--all-ns-.tar.gz ``` The following are example usages of the CFK dump plugins. * Thread dump of the CFK operator pod in the `confluent` namespace: ```bash kubectl confluent operator threaddump --namespace confluent ``` #### NOTE In CFK 3.3.0, this command can fail with `exec: "curl": executable file not found in $PATH` because the CFK Operator image doesn’t include `curl`. If you hit this error, use the operator pod’s `kubectl logs` output or a manual `jstack` command instead. * Thread dump of all pods in a Kafka cluster in `operator` namespace: ```bash kubectl confluent cluster kafka threaddump ``` * Thread dump of a specific pod in a Kafka cluster for five times every 60 seconds: ```bash kubectl confluent cluster kafka threaddump \ --pod-names kafka-0 \ --interval-seconds 60 \ --run-count 5 \ --out-dir $OUT_DIR \ --namespace $NAMESPACE ``` * Thread dump of all Connect pods: ```bash kubectl confluent cluster connect threaddump --namespace confluent ``` * Thread dump of a specific pod in the Connect cluster three times at 60 seconds intervals in a custom directory: ```bash kubectl confluent cluster connect threaddump \ --namespace confluent \ --out-dir /tmp \ --interval-seconds 60 \ --run-count 3 \ --pod-names connect-0 ``` * Thread dump of all Connect pods three times at 60 seconds intervals in the custom directory: ```bash kubectl confluent cluster connect threaddump \ --namespace confluent \ --out-dir /tmp \ --interval-seconds 60 \ --run-count 3 ``` For an alternate way to generate the thread dump using `jstack`, see the [Confluent Support Knowledgebase article](https://support.confluent.io/hc/en-us/articles/15577296161044-How-to-create-thread-dump-for-Confluent-Platform-component-deployed-through-Confluent-for-Kubernetes) (requires login). ## Logs Use the command below to view the logs for a currently running pod. Logs are sent directly to STDOUT for each pod. ```none kubectl logs -n ``` Additionally, in the component custom resources, you can change the log level to `DEBUG` using the [Configuration overrides](co-configure-misc.md#co-config-overrides) feature. For example, add the following in your Kafka CR to get more details in the Kafka logs: * In Confluent Platform 7.x, override Log4j configuration as below: ```yaml spec: configOverrides: log4j: - log4j.rootLogger=DEBUG, stdout ``` * In Confluent Platform 8.x, override Log4j 2 configuration as below: ```yaml spec: configOverrides: log4j2: Configuration: Appenders: Console: - name: stdout target: SYSTEM_OUT Loggers: Root: - name: stdout level: DEBUG AppenderRef: - ref: stdout ``` ### View logs of crashed pods If Kubernetes replaces a crashed container, you can retrieve the logs from its previous instance using the `--previous` (`-p`) flag to investigate the root cause of the pod crash. ```bash kubectl logs --previous --namespace ``` ## Metrics * JMX metrics are available on port 7203 of each pod. * Jolokia (a REST interface for JMX metrics) is available on port 7777 of each pod. ## Debug Problems that can go wrong while using Confluent for Kubernetes (CFK) generally fall into the following categories: * A problem happens while deploying CFK. * A problem exists at the infrastructure level. Something has gone wrong at the Kubernetes layer. * A problem exists at the application level. The infrastructure is fine but something has gone wrong with Confluent Platform itself. Typically, how you configured Confluent Platform components causes this. To debug deployment problems, run the Helm install command with the `--set debug="true"` to enable verbose output: ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ --namespace \ --set debug="true" ``` Look for [Kubernetes issues](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/) first, then debug Confluent Platform. 1. Check for potential Kubernetes errors by entering the following command: ```bash kubectl get events -n ``` 2. Check for a specific resource issue, enter the following command (using the resource type example *pods*): ```bash kubectl describe pods -n ``` 3. If everything looks okay after running the preceding commands, check the individual pod logs using the following command: ```bash kubectl logs -n ``` Confluent Platform containers write application logs directly to STDOUT, so you can read the logs directly with this command. If there is anything wrong at the application level, such as an invalid configuration, it appears in the logs. If Kubernetes replaced a crashed pod and you want to check the previous pod’s logs, add `--previous` to the end of the command above. Your infrastructure system administrator should resolve problems caused by the datacenter infrastructure, such as virtual machine (VM) firewall rules and DNS configuration.