Configure Kafka Logs for Docker in Confluent Platform

To configure Apache Kafka® logging in a Confluent Platform Docker container, set KAFKA_LOG4J_ROOT_LOGLEVEL for the root log level and KAFKA_LOG4J_LOGGERS for individual loggers. Kafka uses the Java-based logging utility Apache Log4j 2. The same pattern applies to other Confluent Platform components.

Important

Confluent Platform and its components, version 7.9 and earlier, use Log4j 1.x and not Log4j 2.x. This affects how you configure logging and the format of the logging configuration file. For information about logging in Confluent Platform 7.9 and earlier, see Configure Confluent Platform Logging (7.8).

Loggers for Confluent Platform components

Each Confluent Platform component has a Log4j environment-variable prefix. Replace {COMPONENT} with the prefix for the component you are configuring. For example, use CONNECT_LOG4J_LOGGERS to set loggers for Kafka Connect. The following table lists each component and its prefix.

To add or override default loggers, use the {COMPONENT}_LOG4J_LOGGERS environment variable. This variable accepts comma-separated values that specify the logger and log level for that logger.

For more information on Log4j 2 settings, see Log4j 2 Configuration.

Component	{COMPONENT} or variable
Kafka	`KAFKA`
Confluent Control Center	`CONTROL_CENTER`
Schema Registry	`SCHEMA_REGISTRY`
REST Proxy	`KAFKA_REST`
Kafka Connect	`CONNECT`
ksqlDB	`KSQL`
Replicator	`REPLICATOR`
default loggers	`{COMPONENT}_LOG4J_LOGGERS`
root log level	`{COMPONENT}_LOG4J_ROOT_LOGLEVEL`
tools log level	`{COMPONENT}_LOG4J_TOOLS_LOGLEVEL`

Log levels

You might choose to increase the log level if an issue occurs and you are not getting sufficient log output to help you identify the cause. Note that increasing the log level can slow service operation due to increased I/O load. In addition, increasing the log level can use more disk space. If you have a size limit on the log files, you could lose older entries, which might be helpful for in debugging.

Following is a list of the supported log levels.

Level	Description
OFF	Turns off logging.
FATAL	Severe errors that cause premature termination.
ERROR	Other runtime errors or unexpected conditions.
WARN	Runtime situations that are undesirable or unexpected, but not necessarily wrong.
INFO	Runtime events of interest at startup and shutdown.
DEBUG	Detailed diagnostic information about events.
TRACE	Detailed diagnostic information about everything.

Default loggers for Kafka

Kafka ships with default loggers. Setting a log level on a default logger overrides the root logger level and changes the log output for that logger. The following table lists each default logger.

Logger	Description
`kafka.request.logger`	Displays all requests being served by the broker. A broker serving many requests has a high log volume when this is set to INFO level.
`kafka.controller`	Provides information on state changes in the kafka cluster and is not verbose for a healthy cluster.
`kafka.log.LogCleaner`	Describes how and when log segment cleanup is occurring for topics that are using log compaction.
`kafka.authorizer.logger`	Provided by the pluggable security authorizer. Increase verbosity of this log to help debug issues with authorization.
`state.change.logger`	Tracks the state changes in the cluster. Typically not verbose in a healthy cluster.

To see the default logger settings for the Kafka Docker image provided by Confluent, see the log4j2.yaml.template on GitHub.

Kafka log level examples

To override the log levels for the Kafka root, controller, and other default loggers, use the following code:

KAFKA_LOG4J_ROOT_LOGLEVEL: 'WARN'
KAFKA_LOG4J_LOGGERS: 'kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN'

The following command changes the log levels for Kafka running in KRaft combined mode:

docker run -d \
    --name=kafka-log-example \
    --net=host \
    -e KAFKA_PROCESS_ROLES='broker,controller' \
    -e KAFKA_NODE_ID=1 \
    -e KAFKA_LISTENERS='PLAINTEXT://localhost:29092,CONTROLLER://localhost:29093' \
    -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP='CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' \
    -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://localhost:29092' \
    -e KAFKA_INTER_BROKER_LISTENER_NAME='PLAINTEXT' \
    -e KAFKA_CONTROLLER_LISTENER_NAMES='CONTROLLER' \
    -e KAFKA_CONTROLLER_QUORUM_VOTERS='1@localhost:29093' \
    -e KAFKA_JMX_PORT=39999 \
    -e KAFKA_LOG4J_LOGGERS="kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN" \
    -e KAFKA_LOG4J_ROOT_LOGLEVEL=WARN \
    -e KAFKA_LOG4J_TOOLS_LOGLEVEL=ERROR \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    -e CLUSTER_ID='MkU3OEVBNTcwNTJENDM2Qk' \
    confluentinc/cp-kafka:8.2.1

Log4j log outputs

All log4j logs are sent to stdout by default. You can change this by extending the images.