Access and Consume Audit Logs on Confluent Cloud

Audit logs are automatically enabled for each Standard, Enterprise, Dedicated, or Freight cluster. You can typically access your audit logs within five minutes of successfully provisioning your cluster. Basic clusters do not include audit logs.

You can filter audit logs when you view them with the Confluent Cloud Console or set them up for consumption.

Live audit log records are automatically deleted after seven days. To retain audit logs beyond this period, replicate them to another Kafka cluster or an external system using Cluster Linking or a Kafka client. For details, see Retain Audit Log Records on Confluent Cloud.

Tip

You can also use Cluster Linking to sync your audit logs to your own Dedicated Confluent Cloud clusters, which enables you to consume audit logs with your existing consumer configurations, or use audit logs with fully-managed Connect and ksqlDB. For a step-by-step guide, see Use Cluster Linking to Manage Audit Logs.

Prerequisites

Cluster type

You must have a Standard, Enterprise, Dedicated, or Freight Kafka cluster, as audit logs for these types are enabled by default. Basic clusters do not include audit logs.

Kafka client or Confluent Cloud Console

You can view audit log events in the Cloud Console or use any Kafka client, such as the Confluent CLI, C/C++, or Java to consume data from the Confluent Cloud audit log topic as long as the client supports SASL authentication.

The Cloud Console provides a configuration you can copy and paste into your Kafka client of choice. This configuration allows you to connect to the audit log cluster and consume from the audit log topic.

Permissions

To view audit logs in the Confluent Cloud Console, you must have the role permission to manage your organization. To consume audit logs using any Kafka client, you must have an API key and secret for the audit log cluster. An OrganizationAdmin role must create the API key and secret for the audit log cluster. Once created, the API key can be used by any user to consume audit logs. The API key is bound only to the audit log cluster.

View your audit logs

You can view and access your audit logs in the Confluent Cloud Console. To do this, you must have the role permission to manage your organization, and your organization must have at least one Standard, Enterprise, Dedicated, or Freight Kafka cluster. For more information about the events that display in the console, see Displayed events.

In Cloud Console, open the Administration menu. Below the current user name, select Audit logs.
Filter the audit logs to find the information you need:
- Principal, subject, or IP address
- Period of time, by minutes, hours, or days, up to the last seven days
- Method, such as authentication
- Authorization, filtered by enabled or denied
You can also select a single row to see the log content and fields for a specific event.
You can export audit events or logs that are displayed in the Cloud Console.
- To export a log, click the Actions dropdown menu. Download the audit log as a CSV or JSON file.
- To export an event, select it in the Timestamp column. In the panel that opens, click Download to download the audit log event as a CSV or JSON file.

Alternatively, you can consume your audit log using these tools:

Confluent CLI
Java
C/C++

Consume with Confluent CLI

An OrganizationAdmin must create the API key and secret for the audit log cluster. Once created, the API key can be used by any user to consume audit logs. The API key is bound only to the audit log cluster.

Open a terminal and log in to your Confluent Cloud organization.
```
confluent login
```
Run the confluent audit-log describe command to identify which resources to use.
The following example shows the audit log information provided for a sample cluster.
```
confluent audit-log describe
 +-----------------+----------------------------+
 | Cluster         | lkc-yokxv6                 |
 | Environment     | env-abc123                 |
 | Service Account | sa-ymnkzp                  |
 | Topic Name      | confluent-audit-log-events |
 +-----------------+----------------------------+
```
Note
The topic confluent-audit-log-events cannot be viewed in the Confluent Cloud Console, but you can consume it using the Confluent CLI or any Kafka client.
The Service Account in the output is only for information. The Service Account does not need to be used when creating API keys for the audit log cluster.
Specify the environment and cluster to use by running the confluent environment use and confluent kafka cluster use commands. The environment and cluster names are available from the data retrieved in the previous step.
```
confluent environment use env-abc123
confluent kafka cluster use lkc-yokxv6
```
If you have an existing API key and secret for your audit log cluster, you can store it locally using the confluent api-key store command.
```
confluent api-key store <API-key> --resource lkc-yokxv6
```
To view the existing API keys for your audit log cluster, run the confluent api-key list command using the --resource option.
```
confluent api-key list --resource lkc-yokxv6
```
Note
To consume data from the audit log topic, you must have an API key and secret.
If you need to create a new API key and secret for your audit log cluster, run the confluent api-key create command with the --resource flag. Be sure to save the API key and secret. The secret is not retrievable later.
```
confluent api-key create --resource lkc-yokxv6
```
Important
To get the identifier for the audit log cluster, run the Confluent CLI confluent audit-log describe command.
There is a limit of two API keys per audit log cluster. If you need to delete an existing API key for your audit log cluster, run the following command:
```
confluent api-key delete <API-key>
```
After creating your API key and secret, copy the API key and paste it into the following command:
```
confluent api-key use <API-key> --resource lkc-yokxv6
```
Consume audit log event messages from the audit log topic.
You can use the Confluent CLI to consume audit log events from the audit log cluster by running the following confluent kafka topic consume command:
```
confluent kafka topic consume -b confluent-audit-log-events
```
For details about the options you can use with this command, see confluent kafka topic consume.

New connections using a deleted API key are not allowed. You can rotate keys by creating a new key, configuring your clients to use the new key, and then deleting the old one.

To watch events “live” as they’re being used to authenticate, and also view which Kafka cluster API keys are being used:

confluent kafka topic consume confluent-audit-log-events > audit-events.json &

tail -f audit-events.json \
| grep 'kafka.Authentication' \
| jq .data.authenticationInfo.metadata.identifier \
> recently-used-api-keys.txt &

tail -f recently-used-api-keys.txt

Note

The jq command-line JSON processor is third-party software that is not included or installed in Confluent Cloud. If you wish to use it, you must download and install it yourself.

Consume with Java

An OrganizationAdmin role must create the API key and secret for the audit log cluster. Once created, the API key can be used by any user to consume audit logs. The API key is bound only to the audit log cluster.

Sign in to Confluent Cloud Console at https://confluent.cloud.
From the Administration menu, select Audit logs.
On the Audit logs page, click Consume Logs. Then, in the Consume Logs panel, select Consume with Java. Then, in the Consume Logs panel, select the Consume with Java tab.
Copy and paste the provided configuration into your client.
If necessary, click Create Kafka cluster API key & secret to create a key/secret pair for your Kafka cluster.
Start and connect to the Java client.

Consume with C/C++

Sign in to Confluent Cloud Console at https://confluent.cloud.
From the Administration menu, select Audit logs.
On the Audit logs page, click the Consume Logs button. Then, in the Consume Logs panel, click the Consume with C/C++ tab.
Copy and paste the provided configuration into your client.
If necessary, click Create Kafka cluster API key & secret to create a an API key and secret for your Kafka cluster.
Start and connect to the C/C++ client.

Securing Confluent Cloud audit logs

The Confluent Cloud audit log destination cluster is located on a public cluster in AWS us-west-2 with encrypted storage and a default KMS-managed key rotation schedule of every three years. API keys specific to the audit log cluster are the only means of gaining read-only access.

Secure your organization’s audit logs by protecting the API keys used to read them.

As mentioned previously, Confluent Cloud supports up to two active audit log cluster API keys, and allows live key rotation. For API key best practices, see Best Practices for Using API Keys on Confluent Cloud.

Accessing audit log messages from behind a firewall

Like other Confluent Cloud public clusters, audit log clusters do not have a static IP address. For information about configuring firewall access to public clusters, see Are Confluent Cloud IP addresses and hostnames static?.