Configure Log Redactor for Confluent Platform with Ansible Playbooks¶

To protect sensitive information in Confluent Platform logs, you can configure Confluent log redactor using Ansible Playbooks for Confluent Platform. For details about Confluent log redactor, see Confluent Log Redactor.

Log Redactor can be used only when the custom_log4j variable is set to true. The custom_log4j variable is set to true (enabled) by default in Ansible Playbooks for Confluent Platform.

Configure Log Redactor for all Confluent components¶

To configure the log redactor for all components, set the following variables in the inventory file:

all:
  vars:
    logredactor_enabled: true              --- [1]
    logredactor_rule_path_local:           --- [2]
    logredactor_rule_path:                 --- [3]
    logredactor_policy_refresh_interval:   --- [4]

[1] Required to enable the log redactor.

In Connect, the log redactor is enabled by default.
[2] Set to the path of the rule file on local/control node. Ensure that the rule file is present at this path on the control node.
[3] Set to the path where you want the rule file to be copied on the component node.
[4] Optional. Set to a millisecond value to watch for policy rule changes and updates at runtime.

Configure the log redactor for a specific Confluent component¶

To configure the log redactor for a specific component, set the following variables in the inventory file as variables of the given component, replacing <component> with the following component names:

kafka_connect
kafka_rest
ksql
schema_registry
zookeeper
control_center

<component>:
  vars:
    logredactor_enabled: true                    --- [1]
    logredactor_rule_path_local:                 --- [2]
    logredactor_rule_path:                       --- [3]
    logredactor_policy_refresh_interval:         --- [4]
    <component>_logredactor_logger_specs_list:   --- [5]
      - { logger_name: , appenderRefs: }

[1] Required to enable the log redactor.
[2] Set to the path of the rule file on the local/control node. Ensure that the rule file is present at this path on the control node.
[3] Set to the path of the rule file on the component node.
[4] Optional. Set to a millisecond value to watch for policy rule changes and updating at runtime.
[5] Optional. Set to the list of loggers to redact. Use the list to extend the log redaction for other loggers defined in the component’s log4j file. The default value can be found in the roles/<component>/defaults/main.yml file.

Specify the logger name and appenderRefs to be used in the redactor definition.

For example, to enable Kafka log redaction:

kafka_broker:
  vars:
    logredactor_enabled: true
    logredactor_rule_path_local: /my/rule/path
    logredactor_rule_path: /my/rule/path
    logredactor_policy_refresh_interval: 600
    kafka_broker_logredactor_logger_specs_list:
      - { logger_name: "log4j.rootLogger", appenderRefs: "kafkaAppender" }
      - { logger_name: "log4j.testLogger", appenderRefs: "testAppender" }

Note

The log redactor is enabled by default for Kafka Connect to redact sensitive logs. To add a rule for redaction, you can edit the rule file ${log4j.config.dir}/connect-log-redactor-rules.json on the Connect node.