Configure Kafka Logs for Docker in Confluent Platform

Apache Kafka® uses the Java-based logging utility Apache Log4j for logging. You can change the default Log4j logging levels or add new logging levels for a Confluent Platform component.

Loggers for Confluent Platform components

You use environment variables to set log levels for default loggers or override the default settings for the root or tools logger.

The following table lists components and the component name to use in the environment variable setting. Replace {COMPONENT} in this section with the component name you are changing the log level for. For example, to set loggers for Kafka Connect, you would use CONNECT_LOG4J_LOGGERS.

To add or override default loggers, use the {COMPONENT}_LOG4J_LOGGERS environment variable. This variable accepts comma-separated values that specify the logger and log level for that logger.

For more information on log4j settings, see Log4j Configuration.

Component	{COMPONENT} or variable
ZooKeeper	`ZOOKEEPER`
Kafka	`KAFKA`
Confluent Control Center	`CONTROL_CENTER`
Schema Registry	`SCHEMA_REGISTRY`
REST Proxy	`KAFKA_REST`
Kafka Connect	`CONNECT`
ksqlDB	`KSQL`
Replicator	`REPLICATOR`
default loggers	`{COMPONENT}_LOG4J_LOGGERS`
root log level	`{COMPONENT}_LOG4J_ROOT_LOGLEVEL`
tools log level	`{COMPONENT}_LOG4J_TOOLS_LOGLEVEL`

Log levels

You might choose to increase the log level if an issue occurs and you are not getting sufficient log output to help you identify the cause. Note that increasing the log level can slow service operation due to increased I/O load. In addition, increasing the log level can use more disk space. If you have a size limit on the log files, you could lose older entries, which might be helpful for in debugging.

Following is a list of the supported log levels.

Level	Description
OFF	Turns off logging.
FATAL	Severe errors that cause premature termination.
ERROR	Other runtime errors or unexpected conditions.
WARN	Runtime situations that are undesirable or unexpected, but not necessarily wrong.
INFO	Runtime events of interest at startup and shutdown.
DEBUG	Detailed diagnostic information about events.
TRACE	Detailed diagnostic information about everything.

Default loggers for Kafka

There are several default appenders to set for Kafka. You should specify a log level for each of these loggers to to see a noticeable change in log output because settings for these loggers override the root logger.

Logger	Description
`kafka.request.logger`	Displays all requests being served by the broker. A broker serving many requests will have a high log volume when this is set to INFO level.
`kafka.controller`	Provides information on state changes in the kafka cluster and is not verbose for a healthy cluster.
`kafka.log.LogCleaner`	Describes how and when log segment cleanup is occurring for topics that are using log compaction.
`kafka.authorizer.logger`	Provided by the pluggable security authorizer. Increase verbosity of this log to help debug issues with authorization.
`state.change.logger`	Tracks the state changes in the cluster. Typically not verbose in a healthy cluster.

To see the default logger settings for the Kafka Docker image provided by Confluent, see the GitHub log4j template.

Examples

Following are some examples of how to set log levels for Kafka.

To override the log levels for the Kafka root, controller, and other default loggers, you would use the following code:

KAFKA_LOG4J_ROOT_LOGLEVEL: 'WARN'
KAFKA_LOG4J_LOGGERS: 'kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN'

Following is example command to change the log levels for Kafka:

docker run -d \
  --name=kafka-log-example \
  --net=host \
  -e KAFKA_BROKER_ID=1 \
  -e KAFKA_ZOOKEEPER_CONNECT=localhost:32181/jmx \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:39092 \
  -e KAFKA_JMX_PORT=39999 \
  -e KAFKA_LOG4J_LOGGERS="kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN" \
  -e KAFKA_LOG4J_ROOT_LOGLEVEL=WARN \
  -e KAFKA_LOG4J_TOOLS_LOGLEVEL=ERROR \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  confluentinc/cp-kafka:7.7.6

Log4j log outputs

All log4j logs are sent to stdout by default. You can change this by extending the images.