Managing Audit Logs for Confluent Cloud

Audit logs provide a way to capture, protect, and preserve Kafka authentication and authorization activity into topics in Kafka clusters on Confluent Cloud. Specifically, audit logs record the runtime decisions of the permission checks that occur as users and service accounts connect to clusters and attempt to take actions that are protected by ACLs.

Each auditable event includes information about who tried to do what, when they tried, and whether or not the system gave permission to proceed.

The primary value of audit logs is that they provide data you can use to assess security risks in your Confluent Cloud clusters. They contain all of the information necessary to follow a user’s interaction with your Confluent Cloud clusters, and provide a way to:

  • Track user and application access
  • Identify abnormal behavior and anomalies
  • Proactively monitor and resolve security risks

You can use Splunk, S3, and other sink connectors to move your audit log data to a target platform for analysis.

Within Confluent Cloud, all audit log messages from your clusters are retained for seven days on an independent cluster. Users cannot modify, delete, nor produce messages directly to the audit log topic, and to consume the messages, users must have an API key specific to the audit log cluster.

Audit logs are emitted at the time of event occurrence; however, short delays or gaps during operational maintenance are possible, although rare.

Prerequisites

Kafka client
You can use any Kafka Clients (for example, Confluent CLI, C/C++, or Java) to consume from the Confluent Cloud audit log topic as long as it supports SASL authentication. Thus, any prerequisites are specific to the Kafka client you use. For details about configuring Confluent Cloud clients, see Configure Confluent Cloud Clients.
Consume configuration
Confluent Cloud provides a configuration you can copy and paste into your Kafka client of choice. This configuration allows you to connect to the audit log cluster and consume from the audit log topic.
Cluster type
Both Standard and Dedicated cluster types support audit logs.

Auditable events

Confluent Cloud audit logs include two kinds of events: authentication events that are sent when a client connects to a Kafka cluster, and authorization events that are sent when the Kafka cluster checks to verify whether or not a client is allowed to take an action.

Note

Users may attempt to authorize a task solely to find out if they can perform the task, but not follow through with it. In these instances, the authorization is still captured in the audit log.

Confluent Cloud audit logs capture the following events:

Event name Description
kafka.AlterConfigs A Kafka configuration is being altered/updated.
kafka.AlterMirrors The properties of a mirror topic that exists on a Cluster Link to this cluster are being altered.
kafka.Authentication A client has connected to the Kafka cluster using an API key or token.
kafka.CreateAcls A Kafka broker ACL is being created.
kafka.CreateClusterLinks A cluster link is being created between this cluster and another cluster.
kafka.CreatePartitions Partitions are being added to a topic.
kafka.CreateTopics A topic is being created.
kafka.DeleteAcls A Kafka broker ACL is being deleted.
kafka.DeleteClusterLinks A cluster link is being deleted.
kafka.DeleteGroups A Kafka group is being deleted.
kafka.DeleteRecords A Kafka record is being deleted. Commonly seen on ksqlDB internal topics for repartitioning
kafka.DeleteTopics A Kafka topic is being deleted.
kafka.IncrementalAlterConfigs A dynamic configuration of a Kafka broker is being altered.
kafka.OffsetDelete A committed offset for a partition in a consumer group is being deleted.

All Confluent Cloud audit log messages are captured in the audit log topic, confluent-audit-log-events.

The following example shows an authentication event that was sent when service account 306343 used the API key MAIDSRFG53RXYTKR to connect to the Kafka cluster lkc-6k8r8q:

{
    "id": "29ca0e51-fdcd-44bd-a393-43193432b614",
    "source": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "specversion": "1.0",
    "type": "io.confluent.kafka.server/authentication",
    "datacontenttype": "application/json",
    "subject": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "time": "2020-12-28T22:41:43.395Z",
    "data": {
        "serviceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "methodName": "kafka.Authentication",
        "resourceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "authenticationInfo": {
            "principal": "User:306343",
            "metadata": {
                "mechanism": "SASL_SSL/PLAIN",
                "identifier": "MAIDSRFG53RXYTKR"
            }
        },
        "result": {
            "status": "SUCCESS",
            "message": ""
        }
    }
}

See also: Audit Log Event Schema.

Audit log content

The following example shows the content of an audit log message that was triggered by an authorization event when user 269915 tried to create an ACL on Kafka cluster lkc-6k8r8q, and was allowed to do so because as a super user, they had the Alter Cluster permission:

{
    "id": "7908ccba-dfc0-42cc-825a-efb329b8c40c",
    "source": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "specversion": "1.0",
    "type": "io.confluent.kafka.server/authorization",
    "datacontenttype": "application/json",
    "subject": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "time": "2020-12-28T22:39:13.680Z",
    "data": {
        "serviceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "methodName": "kafka.CreateAcls",
        "resourceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "authenticationInfo": {
            "principal": "User:269915"
        },
        "authorizationInfo": {
            "granted": true,
            "operation": "Alter",
            "resourceType": "Cluster",
            "resourceName": "kafka-cluster",
            "patternType": "LITERAL",
            "superUserAuthorization": true
        }
    }
}

Configuring Confluent Cloud audit logging

Because audit logging is enabled by default (for Standard and Dedicated clusters only), the only configuration task required is to consume from the audit log topic.

Note

The audit log topic is created when the first auditable event occurs.

Access the audit log user interface

After logging in to Confluent Cloud, to access the audit log user interface, click the hamburger menu (☰) in the upper-right corner. If your role gives you permission to manage your organization, and your organization has at least one Standard or Dedicated cluster, there will be an Audit log menu entry. Enablement is automatic, and typically takes place within five minutes of successfully provisioning your first Standard or Dedicated cluster.

Consume with CLI

  1. Log in to your Confluent Cloud organization using the Confluent Cloud CLI.

    ccloud login
    
  2. Run audit-log describe to identify which resources to use.

    ccloud audit-log describe
     +-----------------+----------------------------+
     | Cluster         | lkc-yokxv6                 |
     | Environment     | env-x11xz                  |
     | Service Account |                     292163 |
     | Topic Name      | confluent-audit-log-events |
     +-----------------+----------------------------+
    
  3. Specify the environment and cluster to use (using the data that you retrieved in the previous step).

    ccloud environment use env-x11xz
    ccloud kafka cluster use lkc-yokxv6
    
  4. If you have an existing API key and secret for audit logs, you can use it as shown here:

    ccloud api-key store <API-key> --resource lkc-yokxv6
    

    To view the existing API keys for your audit log cluster:

    ccloud api-key list --resource lkc-yokxv6
    

    Note

    You must have an API key and secret to consume from the audit log topic.

  5. If you need to create a new API key and secret for your audit log cluster, run the following Confluent CLI command:

    ccloud api-key create --service-account 292163 --resource lkc-yokxv6
    

    Note

    Be sure to save the API key and secret. The secret is not retrievable later.

    Important

    There is a limit of 2 API keys per audit log cluster. If you need to delete an existing API key for your audit log cluster, run the following command:

    ccloud api-key delete <API-key>
    
  6. After creating your API key pair, copy the API key and paste it into the following command:

    ccloud api-key use <API-key> --resource lkc-yokxv6
    
  7. Consume audit log events from the audit log topic.

    ccloud kafka topic consume -b confluent-audit-log-events
    

    Refer to ccloud kafka topic consume for details about the flags you can use with this command.

New connections using a deleted API key are not allowed. You can rotate keys by creating a new key, configuring your clients to use the new key, and then deleting the old one.

To watch events “live” as they’re being used to authenticate, and also view which Kafka cluster API keys are being used:

ccloud kafka topic consume confluent-audit-log-events > audit-events.json &

tail -f audit-events.json \
| grep 'kafka.Authentication' \
| jq .data.authenticationInfo.metadata.identifier \
> recently-used-api-keys.txt &

tail -f recently-used-api-keys.txt

Consume with Java

  1. Sign in to Confluent Cloud at https://confluent.cloud.

  2. Navigate to ADMINISTRATION -> Audit log, which you can find in the top-right hamburger menu in the Confluent Cloud user interface.

  3. On the Audit log page, click the Consume with Java tab.

    ../../_images/ccloud-audit-log-consume-java.png
  4. Copy and paste the provided configuration into your client.

  5. If necessary, click Create Kafka cluster API key & secret to create a key/secret pair for your Kafka cluster.

  6. Start and connect to the Java client.

Consume with C/C++

  1. Sign in to Confluent Cloud at https://confluent.cloud.

  2. Navigate to ADMINISTRATION -> Audit log.

  3. On the Audit log page, click the Consume with C/C++ tab.

    ../../_images/ccloud-audit-log-consume-c++.png
  4. Copy and paste the provided configuration into your client.

  5. If necessary, click Create Kafka cluster API key & secret to create a key/secret pair for your Kafka cluster.

  6. Start and connect to the C/C++ client.

Securing Confluent Cloud audit logs

The Confluent Cloud audit log destination cluster is located on a public cluster in AWS us-west-2 with encrypted storage and a default KMS-managed key rotation schedule of every three years. API keys specific to the audit log cluster are the only means of gaining read-only access.

Secure your organization’s audit logs by protecting the API keys used to read them.

As mentioned previously, Confluent Cloud supports up to two active audit log cluster API keys, and allows live key rotation.

To delete an existing API key for your audit log cluster, run the following command:

ccloud api-key delete <API-key>

Accessing audit log messages from behind a firewall

Like other Confluent Cloud public clusters, audit log clusters do not have a static IP address. To access your audit log messages, you must move your consumer outside the firewall. For details about configuring access to the Confluent Cloud web UI, see Configure Confluent Cloud UI access for AWS PrivateLink and Azure Private Link.

Troubleshooting Confluent Cloud audit logging

This section provides tips to help you troubleshoot audit logging issues.

Cluster not sending audit log events

Audit logs are available on Standard and Dedicated clusters only. If using a Basic cluster type, consider upgrading.

Topic altered without authorization does not appear in audit log messages

If you attempted to alter a topic on a Dedicated cluster, but the topic does not appear subsequently in the audit logs messages, it could be because some common administration tools attempt to describe a resource before attempting to create, alter, or delete it. In cases where the describe request fails or is rejected, the tool may not attempt to send a second request. In such cases, the audit logs will include an authentication event, but will not include any additional authorization checks, because the create, alter, and/or delete request was never made.

Newly-created topic does not appear in audit log messages

When the API checks a user’s permission to create a topic, it first attempts to confirm that the user has cluster-level permission to create any topic. If so, access is granted. If not, the API performs a secondary check to see if the user has permission to create the specific topic name (or a prefix using that name). If this cluster-level check succeeds, then the audit log event will include the ID of the cluster without any reference to the specific topic, because the topic name is not used in the cluster-level permission check.

Unfamiliar names appear in authorization checks

You may find several CreateTopics authorization checks with topic names that you don’t recognize or that don’t exist. When logged in to the Confluent Cloud web interface, it is common for certain pages to perform “dry-run” permission checks in the background to check permissions and show only the controls that are relevant to the logged in user. The authentication and authorization checks for these dry-run requests are logged. They typically have a resourceName that looks like crn:///kafka=lkc-abcde/topic=341e9e2e-f734-439d-8469-4433ce7f627c.

Authentication failures do not appear in audit log messages

If an authentication failure does not appear in your audit log messages, it could be because Confluent Cloud logs all authentication failures internally, but only passes them on to your audit logs when the connection tries to use one of your valid, active API keys on the cluster, but with an incorrect secret.

Audit log messages include identifiers with no descriptions

If you come across audit log messages that use identifiers with no descriptions, for example, User:12345, run either of the following commands:

ccloud admin user list
ccloud api-key list

Describe command output is empty

It is possible to run the ccloud audit-log describe command and not see any output, even when your organization has audit logging enabled. In such cases, the Confluent Cloud CLI may have cached your organization’s information prior to when audit logging was enabled. To refresh the cache, run the following command:

ccloud login --prompt

Audit log messages are not being generated for producer and consumer requests

Audit log messages include the authentication events from when the producers and consumers connect, but not the produce and consume requests themselves.