<a id="log-redaction"></a>

# Redact Confluent Logs in Confluent Platform

Modern software that runs in a Java Virtual Machine (JVM) is most often built up
from hundreds of component libraries, which come from a wide variety of vendors
and open source projects. Typically, each component library creates log messages
to capture errors, warnings, informative messages, and debug information
throughout their classes and methods. In rare cases, a log statement may
inadvertently include sensitive information, unbeknownst to the component author,
and once packaged up, end users may encounter scenarios that expose sensitive
information in application logs. Of course, this can potentially lead to
security concerns, and should be reported to the software provider. Minimally,
you should reach out to the provider to request a fix, if one is available.

Regardless of the availability of such fixes for your component libraries, you
can use Confluent Log Redactor plugin for [Apache Log4j 2](https://logging.apache.org/log4j/2.12.x/) to configure regular expression
patterns (redaction rules) to identify and redact specific patterns of sensitive
information from your logs, before they are delegated to other appenders and emitted.
You can configure Log Redactor for a component (such as Kafka, or Connect) by
updating its Log4j 2 configuration file.

For example, you might see that an HTTP `Authorization` header appears in your
logs, and file a fix request ticket with the component provider. You can also
configure Confluent Log Redaction to create a simple rule to match
`Authorization: Basic [0-9a-zA-Z\+\=\/]+` and replace it with
`Authorization: Basic *****`.

<a id="log-redactor-install"></a>

## Install the Confluent Log Redactor plugin

The Confluent Log Redactor plugin is installed by default (for fresh installs) for
Connect.

## Configure the Log Redactor plugin

To configure the Log Redactor plugin, you need to:

1. Back up your configuration file changes.
2. Ensure that the JAR file is in the classpath for Connect.
3. Update the `log4j2.yaml` files.
4. Reference the Log Redactor class.

Configuration steps are covered in the following subsections.

### Set up redaction rules

The redaction rules are specified in JSON format.
The file content looks like the following example:

```text
{
  "version": "1",
  "rules": [
    {
      "description": "This is the first rule",
      "trigger": "triggerstring 1",
      "search": "regex 1",
      "replace": "replace 1"
    },
    {
      "description": "This is the second rule",
      "trigger": "triggerstring 2",
      "caseSensitive": false,
      "search": "regex 2",
      "replace": "replace 2"
    }
  ]
}
```

If the log message matches a rule, the `search` field is used to search the
message, and all occurrences will be replaced with `replace`.

| Field         | Description                                                                                                                                                                                                                                                              | Required?   |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
| trigger       | A simple string compare. May be used to provide a performance hint;<br/>a simple string compare may be faster than a regular expression.<br/>If it does not exist, the message will always apply `search`.<br/>Note that in future versions, this hint might be ignored. | No          |
| caseSensitive | A boolean indicating if the `trigger` and `search` are to be used<br/>in case-sensitive or case-insensitive matching. Defaults to `true`<br/>(case-sensitive matching).                                                                                                  | No          |
| search        | A regular expression. Make sure that proper escaping is used.                                                                                                                                                                                                            | Yes         |
| replace       | A simple string. In practice, it usually looks something like `XXXXXXX`.<br/>If missing, the rule will be detect-only and not redact.                                                                                                                                    | No          |
| description   | Intended for self-documentation purposes.                                                                                                                                                                                                                                | No          |

In the current implementation, the ordering of the rules is significant.
The rules are evaluated strictly in the order given. Thus, in theory later
rules might process the output of earlier rules (`aaa->bbb`, `bbb->ccc`).
The use of rules that depend on this behavior is discouraged as Confluent
offers no guarantee that this behavior will be maintained in future versions.

#### Example of a rules.json file

Here is an example of a `rules.json` file:

```text
{
  "version": 1,
  "rules": [
    {
      "description": "No more vowels",
      "search": "[aeiou]",
      "replace": "x"
    },
    {
      "description": "Passwords",
      "trigger": "password",
      "search": "password=.*",
      "replace": "password=xxxxx"
    }
  ]
}
```

This example has two rules. The first rule banishes lowercase vowels from all
log messages and replaces them with `x`’s. The second rule looks for lines
containing `password` and replaces any `password=...` occurrences with
`password=xxxxx`.

### Watch for policy rule changes and updating at runtime

Our log redactor can redact log content against dynamically changing redaction
rules found in a file on the filesystem.

To set up the log redactor, use the following configuration in the
`log4j2.yaml` file:

```none
 Rewrite:
- name: RedactorAppender
  RedactorPolicy:
    name: "io.confluent.log4j2.redactor.RedactorPolicy"
    rules: "path/to/log-redactor-rules.json"
    refreshInterval: 60000
  AppenderRef:
    - ref: STDOUT
    - ref: ConnectAppender
```

`refreshInterval`
: (Optional) Specifies the interval in milliseconds for how often the file system is checked
  for changes. When a change is detected, the redactor automatically adopts
  the changes. If unspecified, the policy rules are only
  read once at startup.

## Configure Log Redactor for Metadata Service (MDS)

The Confluent Log Redactor is included in Confluent Platform and can be configured for
the Metadata Service (MDS) by updating the Log4j 2 configuration file for Confluent Platform, which provides
logging configuration for Confluent Platform components.

Here is an excerpt of an updated `log4j2.yaml` file that shows a redactor
configuration using MDS redactor rules:

```text
Redactor:
  name: redactor
  appenderRefs: stdout,file
  policy: io.confluent.log4j.redactor.RedactorPolicy
  policyRules: path/to/metadata-log-redactor-rules.json

Loggers:
  Root:
    level: ERROR
    AppenderRef:
      ref: redactor
```

## Related content

- [Configure Confluent Platform Logging](../../monitor/cp-logging.md#cp-logging)
- [Configure Kafka Logs for Docker in Confluent Platform](../../installation/docker/operations/logging.md#docker-operations-logging)
- [Manage Control Center Logs for Confluent Platform](https://docs.confluent.io/control-center/current/installation/logging.html)
- [Kafka Connect Logging for Confluent Platform](../../connect/logging.md#connect-logging)
- [Apache Log4j 2](https://logging.apache.org/log4j/2.12.x/)