# Configure Audit Logs in Confluent Platform Using Confluent CLI
You can use [Confluent CLI](https://docs.confluent.io/confluent-cli/current/command-reference/audit-log/index.html)
commands to dynamically update your audit log configurations. The Confluent CLI
is the recommended tool for scripted or command line interactions with the
[MDS API Audit Log Configuration endpoints](../../authorization/rbac/mds-api.md#mds-api). Changes made using
Confluent CLI are pushed from the MDS (metadata service) out to all registered clusters,
allowing for centralized management of the audit log configuration.
## Prerequisites
- Before configuring audit logs using the Confluent CLI, you must:
[migrate legacy audit log configurations](#migrate-kafka-cluster-audit-log-configs)
from all of your Kafka clusters into a combined JSON policy.
#### IMPORTANT
You must satisfy this prerequisite prior to registering
all of the Kafka clusters.
- Configure the [Confluent Server Authorizer](../../csa-introduction.md#confluent-server-authorizer) on all Kafka clusters.
- [Register](../../cluster-registry.md#cluster-registry-registering) all of your Kafka clusters in the [Cluster Registry in Confluent Platform](../../cluster-registry.md#cluster-registry).
- [Configure MDS to manage audit logging](mds-config-for-centralized-audit-logs.md#mds-config-for-centralized-audit-logs).
- Grant the [AuditAdmin](../../authorization/rbac/rbac-predefined-roles.md#rbac-predefined-roles) role to the same principal on every registered Kafka cluster, including MDS.
## Migrate individual Kafka cluster audit log configurations
You can expand audit logging capabilities to use a
[centralized (MDS) configuration](mds-config-for-centralized-audit-logs.md#mds-config-for-centralized-audit-logs)
that spans across multiple registered Kafka clusters.
If you wish to preserve an existing audit log configuration to leverage
the audit log routing configurations already specified for each Kafka cluster, you
must migrate the existing configuration files and combine them in a common file to
be used in the newly-centralized configuration.
#### NOTE
If you use the [Confluent CLI](https://docs.confluent.io/confluent-cli/current/command-reference/audit-log/config/index.html) or the
[MDS API](../../authorization/rbac/mds-api.md#mds-api) without registering any clusters in the cluster
registry, then any changes you make to the audit log configuration through
the API using the Confluent CLI will only affect the audit log
configuration of the Kafka cluster where MDS is running, and will not impact
any other Kafka clusters.
### How the audit log migration tool works
The audit log migration tool performs the following tasks:
- Sets the output bootstrap servers to the value specified (when specified).
Note that output bootstrap servers are empty by default.
- Combines the input audit log destination topics. For topics that appear in more
than one Kafka cluster configuration, the migration tool uses the maximum
[retention time](audit-logs-concepts.md#audit-log-retention) specified in the configuration.
- Sets the default audit log topic as `confluent-audit-log-events`. If necessary, the
migration tool will add this topic to the set of destination topics (in which
case, it specifies a retention period of 7776000000 milliseconds).
- Combines the set of all [excluded principals](audit-logs-concepts.md#audit-logs-excluded-principals).
- Replaces the `/kafka=*/` part of each [Confluent Resource Name (CRN)](audit-logs-concepts.md#confluent-resource-name) pattern
using the cluster ID of the contributing Kafka cluster. For example, a route in
the configuration from `cluster1` with a CRN like
`crn:///kafka=*/topic=accounting-*` will be transformed to
`crn:///kafka=cluster1/topic=accounting-*`.
For routes that have a CRN that uses something other than `/kafka=*/`, the
migration tool will not replace the Kafka cluster ID. For example, if a route specifies
`kafka=pkc-123` and the cluster ID is `pkc-abc` then the tool will leave
it untouched and return the warning:
```none
Mismatched Kafka Cluster Warning: Routes from one Kafka cluster ID on a
completely different cluster ID are unexpected, but not necessarily wrong.
For example, this message might be returned if you attempt to reuse the
same routing configuration on multiple clusters.
```
- For any incoming audit log router configurations that have default topics other
than `confluent-audit-log-events`, the script will add extra routes
for the following CRN patterns (if they do not already exist):
| Topic Route | Event Category Type |
|--------------------------------------------------------------------------|-----------------------|
| `crn:///kafka=` | AUTHORIZE, MANAGEMENT |
| `crn:///kafka=/topic=*` | AUTHORIZE, MANAGEMENT |
| `crn:///kafka=/control-center-broker-metrics=*` | AUTHORIZE |
| `crn:///kafka=/control-center-alerts=*` | AUTHORIZE |
| `crn:///kafka=/delegation-token=*` | AUTHORIZE |
| `crn:///kafka=/control-center-broker-metrics=*` | AUTHORIZE |
| `crn:///kafka=/control-center-alerts=*` | AUTHORIZE |
| `crn:///kafka=/cluster-registry=*` | AUTHORIZE |
| `crn:///kafka=/security-metadata=*` | AUTHORIZE |
| `crn:///kafka=/all=*` | AUTHORIZE |
| `crn:///kafka=/connect=` | AUTHORIZE |
| `crn:///kafka=/connect=/connector=*` | AUTHORIZE |
| `crn:///kafka=/connect=/secret=*` | AUTHORIZE |
| `crn:///kafka=/connect=/all=*` | AUTHORIZE |
| `crn:///kafka=/schema-registry=` | AUTHORIZE |
| `crn:///kafka=/schema-registry=/subject=*` | AUTHORIZE |
| `crn:///kafka=/schema-registry=/all=*` | AUTHORIZE |
| `crn:///kafka=/ksql=` | AUTHORIZE |
| `crn:///kafka=/ksql=/ksql-cluster=*` | AUTHORIZE |
| `crn:///kafka=/ksql=/all=*` | AUTHORIZE |
#### NOTE
If you do not want the routes listed above added in your newly-migrated
audit log configuration, then edit your input `server.properties` files
to only use `confluent-audit-log-events` in the `default_topics`
before migrating.
### Default audit log configuration
The default centralized audit log configuration writes event messages to the
topic `confluent-audit-log-events` on the Confluent Platform cluster.
## Viewing the Audit Log Configuration
There are a variety of scenarios in which you may want to view the current audit
log configuration. For example, if you want to identify the target
cluster currently being used for audit logging, or you want to see the
target topic for specific audit log operations, you can view the audit
log configuration using the `confluent audit-log config describe` command. This
command returns the entire JSON configuration to standard output. If you want to
[update](#audit-log-config-update-cli) the audit log configuration
rules, use this command to capture the current configuration before opening it
in your text editor. This method is easier than starting with an empty file.
```none
confluent audit-log config describe > /tmp/audit-log-config.json
cat /tmp/audit-log-config.json
{
"destinations": {
"bootstrap_servers": [
"localhost:9091",
"localhost:9093"
],
"topics": {
"confluent-audit-log-events": {
"retention_ms": 259200000
}
}
},
"excluded_principals": [
"User:Alice",
"User:service_account"
],
"default_topics": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
},
"metadata": {
"resource_version": "FElOq8fl5Mp4imIaRbX4iA",
"updated_at": "2020-08-06T18:50:01Z"
}
}
```
## Update the audit log configuration
You can use the `confluent audit-log config update` command to dynamically
replace the existing audit log configuration. The `update` option pushes the
updated JSON configuration to the [MDS API Audit Log Configuration](../../authorization/rbac/mds-api.md#mds-api).
The input to the `confluent audit-log config update` command is read from
standard input by default. You can use the `--file` argument to specify the
path to an input file instead. For example, in most shells, all of the following
approaches are equivalent:
```bash
cat /path/to/my/file.json | confluent audit-log config update
confluent audit-log config update < /path/to/my/file.json
confluent audit-log config update --file /path/to/my/file.json
```
Following is a typical workflow for dynamically updating the audit log
configuration:
1. Run `confluent audit-log config describe > tempfile` to capture the existing
configuration.
2. Edit the `tempfile`.
3. Run `confluent audit-log config update < tempfile` to write changes back to
the MDS API Audit Log Configuration.
Alternatively, your organization may store all configurations in a source control
system, and push out configuration updates automatically using hooks or triggers
as source code is updated. In this use case, your “golden” JSON configuration is
stored in source control, so your triggers should always overwrite the JSON
configuration stored in the MDS API Audit Log Configuration.
Following is a typical “infrastructure as code” workflow:
1. Check out the version-controlled configuration file. Edit the file and create
a merge request.
2. Your peers review and approve, and you merge the changes back into the
source control system.
3. Your control system’s hooks call your organization’s automated scripts, which
check out the “golden” JSON configuration to a file at
`` and call
`confluent audit-log config update --force --file `,
overwriting the previous audit log configuration known to the MDS API with the
newly-updated configuration.
### Overwrite concurrent modifications using the `--force` option
Use the `--force` option when you want to overwrite any concurrent modifications.
For example, in a scenario where administrator A starts editing the configuration,
and administrator B makes a concurrent edit, when administrator A completes her
edit and attempts to upload the new configuration, she gets an error. This
error occurs because both administrators are trying to update the
old version of the configuration concurrently. In this case, after
inspecting the new version of the configuration, administrator A
may decide that it makes sense to use the `--force` option to overwrite
administrator B’s changes.
Note that the MDS API checks the `resource_version` in the JSON updates. If it
does not match the latest version, the API returns a concurrent update error:
```none
Error: Metadata Service backend error: 409 Conflict: {"destinations":{"topics":{"confluent-audit-log-events":{"retention_ms":1000000000}}},"default_topics":{"allowed":"confluent-audit-log-events","denied":"confluent-audit-log-events"},"metadata":{"resource_version":"4Ecnf-3erIWXjqgbjLXauw","updated_at":"2020-07-15T22:30:12Z"}}
```
Using the `--force` option in this scenario allows the `resource_version` in
your updated JSON to deviate from the current one.
In the following configuration update example, the original configuration is being
replaced by the configuration in the file `acme-audit-log-config-hr`. Note the
use of the `--force` option, which is required because the `resource_version`
here differs from the one in the existing configuration:
```bash
confluent audit-log config update --force --file acme-audit-log-config-hr
```
Upon the successful completion of the update, the new configuration is returned:
```json
{
"destinations": {
"bootstrap_servers": [
"localhost:9091",
"localhost:9093"
],
"topics": {
"confluent-audit-log-events": {
"retention_ms": 259200000
},
"confluent-audit-log-events_hr": {
"retention_ms": 604800000
}
}
},
"excluded_principals": [
"User:Alice",
"User:service_account"
],
"default_topics": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
},
"routes": {
"crn:///kafka=*/topic=hr-*": {
"management": {
"allowed": null,
"denied": null
},
"authorize": {
"allowed": null,
"denied": null
},
"produce": {
"allowed": "confluent-audit-log-events_hr",
"denied": "confluent-audit-log-events_hr"
},
"consume": {
"allowed": "confluent-audit-log-events_hr",
"denied": "confluent-audit-log-events_hr"
},
"describe": {
"allowed": "",
"denied": ""
}
}
},
"metadata": {
"resource_version": "3N-cNxcdIRtVr-4Jvkbg5w",
"updated_at": "2020-08-06T19:01:32Z"
}
}
```
## Edit the audit log configuration
You can modify the existing audit log configuration without creating an intermediate
file (as you must do when using `confluent audit-log config update`).
When you run the `confluent audit-log config edit` command, your default editor
($EDITOR) opens a temporary file with the contents of the latest audit log
configuration known to the MDS server. Enter your configuration updates in the editor.
After you finish your configuration updates and close the editor, the Confluent CLI
completes the updates by sending the temporary file to the MDS server.
## Troubleshoot audit log routes
The [routing rules](audit-logs-properties-config.md#topic-routing-use-cases) for audit logs can be complex.
The CLI provides two commands to help clarify and troubleshoot the audit log
routing rules, given a specific resource [CRN](audit-logs-concepts.md#confluent-resource-name):
- Use `confluent audit-log route list` to list all of the audit log routes that
match the given resource or any of its sub-resources.
- Use `confluent audit-log route lookup` to determine which of those routes is
selected by the “longest common prefix” precedence rule for the given resource.
#### NOTE
The [MDS API Audit Log Configuration](../../authorization/rbac/mds-api.md#mds-api) route lookup requires a
CRN for a single resource, not a pattern using the wildcard character (`*`).
If you include a wildcard in your search, you will get an error.
To view all of the audit log routes related to the Kafka cluster identified by
the CRN `crn://mds1.example.com/kafka=test-cluster-1`, use the following command:
```bash
confluent audit-log route list -r "crn://mds1.example.com/kafka=test-cluster-1"
```
The following routes are returned, with all defaults populated:
```json
{
"default_topics": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
},
"routes": {
"crn://mds1.example.com/kafka=*/topic=billing-*": {
"management": {
"allowed": "confluent-audit-log-events_billing",
"denied": "confluent-audit-log-events_other"
},
"authorize": {
"allowed": null,
"denied": null
},
"produce": {
"allowed": "confluent-audit-log-events_billing",
"denied": "confluent-audit-log-events_other"
},
"consume": {
"allowed": "confluent-audit-log-events_billing",
"denied": "confluent-audit-log-events_other"
}
},
"crn://mds1.example.com/kafka=*/topic=payroll-*": {
"management": {
"allowed": "confluent-audit-log-events_payroll",
"denied": "confluent-audit-log-events_other"
},
"authorize": {
"allowed": null,
"denied": null
},
"produce": {
"allowed": "confluent-audit-log-events_payroll",
"denied": "confluent-audit-log-events_other"
},
"consume": {
"allowed": "confluent-audit-log-events_payroll",
"denied": "confluent-audit-log-events_other"
}
},
"crn://mds1.example.com/kafka=test-cluster-1": {
"management": {
"allowed": "confluent-audit-log-events_other",
"denied": "confluent-audit-log-events_other"
},
"authorize": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
}
}
}
}
```
To find out where the audit log messages for the resource
`"crn://mds1.example.com/kafka=test-cluster-1"` are being routed:
```bash
confluent audit-log route lookup "crn://mds1.example.com/kafka=test-cluster-1"
```
The single route showing where audit log messages using this CRN is routed is returned,
with all defaults populated:
```json
{
"route": "crn://mds1.example.com/kafka=test-cluster-1",
"categories": {
"management": {
"allowed": "confluent-audit-log-events_other",
"denied": "confluent-audit-log-events_other"
},
"authorize": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
},
"produce": {
"allowed": "",
"denied": ""
},
"consume": {
"allowed": "",
"denied": ""
},
"interbroker": {
"allowed": "",
"denied": ""
},
"heartbeat": {
"allowed": "",
"denied": ""
},
"describe": {
"allowed": "",
"denied": ""
}
}
}
```
## Related content
- [Audit Log Concepts in Confluent Platform](audit-logs-concepts.md#audit-logs-concepts)
- [Configure MDS to Manage Centralized Audit Logs](mds-config-for-centralized-audit-logs.md#mds-config-for-centralized-audit-logs)
- [Use Properties Files to Configure Audit Logs in Confluent Platform](audit-logs-properties-config.md#audit-logs-properties-config)
- [Auditable Events in Confluent Platform](auditable-events.md#auditable-events)