Use Cluster Linking to Manage Audit Logs

Looking for Confluent Platform Cluster Linking docs? You are currently viewing Confluent Cloud documentation. If you are looking for Confluent Platform docs, check out Cluster Linking on Confluent Platform.

Confluent Audit Logs provide a view into the activity in a Confluent Cloud organization. With Cluster Linking, audit logs can be synced in real time to any Dedicated Confluent Cloud cluster. This enables you to:

  • Consume the audit logs directly from your cluster so that you can reuse your existing consumer configurations and use Stream Governance with the audit logs topic.
  • Manipulate the audit logs and build stream processing applications in ksqlDB.
  • Sync the audit logs into an external data store like Amazon S3, Snowflake, or Splunk with Connect.

Setup

  • Create new or identify any existing Dedicated cluster to serve as the Destination (target) cluster.
  • The audit logs cluster will serve as the Source cluster. Details on how to access this cluster are described in the steps below.
  • Use any cloud provider and networking type. This tutorial uses the public internet, and automatic encryption options.

If you need more guidance on this, see the first steps in the Quick Start for Confluent Cloud about how to create a cluster and choose networking.

Configure Permissions to Consume Audit Logs

Log on to Confluent CLI and select your provisioning environment

confluent environment list
confluent environment use <env id>
Now using "<env id>" as the default (active) environment.

Connect to your cluster as soon as it is available

confluent kafka cluster use <cluster id>
Set Kafka cluster "<cluster id>" as the active cluster for environment "<env id>"

Get the details of your cluster

confluent kafka cluster describe <cluster id>
+------------------+---------------------------------------------------------+
| ID               | <cluster id>                                            |
| Name             | abcdefg-xyz-auditlog-linking                            |
| Type             | DEDICATED                                               |
| Ingress          | 50                                                      |
| Egress           | 150                                                     |
| Storage          | Infinite                                                |
| Provider         | aws                                                     |
| Availability     | single-zone                                             |
| Region           | eu-west-2                                               |
| Status           | UP                                                      |
| Endpoint         | SASL_SSL://pkc-nnnnn.eu-west-2.aws.confluent.cloud:9092 |
| REST Endpoint    | https://pkc-nnnnn.eu-west-2.aws.confluent.cloud:443     |
| Cluster Size     | 1                                                       |
+---------------+------------------------------------------------------------+

Check for Audit Logs

confluent audit-log describe

This will give you the IDs you need to access the audit logs.

+-----------------+-----------------------------+
| Cluster         | <cluster id>                |
| Environment     | <env id>                    |
| Service Account | <sa id>                     |
| Topic Name      | confluent-audit-log-events  |
+-----------------+-----------------------------+

Create a service account using resource IDs and details from previous commands

confluent environment use <env id>
confluent kafka cluster use <cluster id>
confluent api-key create --service-account <service-account-id> --resource <cluster-id>

(Optional) Look up an existing API keys

This is optional. If you already have an API key, you can use the following command to look it up.

confluent api-key list --resource <cluster id>

You will see your keys in the list. If you plan to reuse any of these keys, note that you must know the secret.

Associate your API key with the Audit Log resource

confluent api-key use <api key> --resource <cluster id>
Set API Key "<api key>" as the active API key for "<cluster id>".

Confirm the Audit Log can be consumed

confluent kafka topic consume -b confluent-audit-log-events

Use Ctrl+C to exit the consumer.

Set up Cluster Linking

The Dedicated cluster you created in previous steps will host the mirrored auditlog topic.

In these next steps, you will establish a link on that cluster (the target cluster) and link it to the source (the Audit Log cluster).

  1. Make sure that you are working on the destination cluster (the Dedicated cluster).

    You can use these commands to list environments and clusters, and switch to the appropriate ones.

    confluent environment list
    
    confluent environment use <env id>
    
    confluent kafka cluster list
    
    confluent kafka cluster use <cluster id>
    
  2. Run the following command to create a link called my-link.

    confluent kafka link create my-link --cluster <destination id> \
        --source-cluster-id  <source-id> \
        --source-bootstrap-server <source bootstrap server> \
        --source-api-key <key> --source-api-secret <secret>
    
    • The source-bootstrap-server can be obtained by running the following to get the environment and cluster IDs:

      confluent audit-log describe
      

      Your output will be similar to the following:

      +-----------------+----------------------------+
      | Cluster         | <cluster-id>               |
      | Environment     | env-33r92                  |
      | Service Account | sa-43kzk2                  |
      | Topic Name      | confluent-audit-log-events |
      +-----------------+----------------------------+
      
    • Then run the following commands to locate and describe the Audit Logs source cluster:

      confluent environment use <audit-logs-env-id>
      
      confluent kafka cluster list
      

      Your output should resemble

            Id         |             Name             |   Type   | Provider |  Region   | Availability | Status
      -----------------+------------------------------+----------+----------+-----------+--------------+---------
        * <cluster-id> | _confluent_audit_log_cluster | STANDARD | aws      | us-west-2 | single-zone  | UP
      
    • Run this command to get the details for the Audit Logs source cluster:

      confluent kafka cluster describe <audit-logs-cluster-id>
      

      Your output should resemble:

      +--------------+---------------------------------------------------------+
      | Id           | <cluster-id>                                            |
      | Name         | _confluent_audit_log_cluster                            |
      | Type         | DEDICATED                                               |
      | Ingress      |                                                     100 |
      | Egress       |                                                     100 |
      | Storage      | Infinite                                                |
      | Provider     | aws                                                     |
      | Availability | single-zone                                             |
      | Region       | us-west-2                                               |
      | Status       | UP                                                      |
      | Endpoint     | SASL_SSL://<pkc-id>.us-west-2.aws.confluent.cloud:9092  |
      | RestEndpoint | https://<pkc-id>.us-west-2.aws.confluent.cloud:443      |
      +--------------+---------------------------------------------------------+
      
    • Look under Endpoint for the bootstrap server for your source cluster.

Mirror the Audit Log

Click through to view the properties of the link and use the Add mirror topic button to add the mirror. For the topic name, enter confluent-audit-log-events. The link and mirror topic should be displayed, as shown below:

../../_images/cluster-links-audit-logs-mirror.png

View the Mirror Topic from the Dedicated Cluster

Follow these steps to view the Audit Logs mirror topic on the dedicated cluster (destination).

  1. In Confluent Cloud, select your environment and your dedicated cluster instance (the URI for this resource will be something like confluent.cloud/environments/your_env_id/clusters/your_cluster_id/).

  2. Click through to view the messages on your dedicated cluster.

    ../../_images/cluster-links-audit-logs-view-mirrot-topic-messages-on-dest.png
  3. Click on the See in Stream Lineage in the top right to see which consumer groups are consuming the audit logs topic.

    ../../_images/cluster-links-audit-log-consumers-stream-lineage.png