Get Started with Lightning Tables in Confluent Cloud

Enable a Lightning Table for an Apache Kafka® topic and then query it with SQL.

Prerequisites

Before you begin, make sure that you have the following:

  • A Basic, Standard, Enterprise, or Dedicated Kafka cluster on Confluent Cloud. The cloud provider must be AWS, and the cluster must be hosted in a supported region.

  • A Kafka topic.

  • A schema for the topic (Avro, Protobuf, or JSON Schema). If the topic doesn’t have one, the Confluent Cloud Console prompts you to create one when you enable the Lightning Table. For more information, see Enable a Lightning Table for a topic without a schema.

  • The following roles, at minimum:

    • CloudClusterAdmin or ResourceOwner (on the topic) to enable a Lightning Table for a topic.

    • DeveloperRead (on the Kafka cluster) to query data.

    • DeveloperRead (on the Schema Registry cluster) to list topics and get metadata.

Note

Upsert mode requirements

If your topic’s cleanup.policy configuration includes compact, Kafka retains only the latest record for each key. To match, the Lightning Table automatically uses upsert mode. In upsert mode, the Lightning Table uses the Kafka record key as the primary key. The key can be raw bytes, or it can use a schema in Avro, Protobuf, or JSON Schema format.

For more information, see Upsert mode.

If you need to create a cluster and topic, see Quick Start. When you create the cluster, choose AWS and a supported region.

For full details on roles and permissions, see Manage Access to Lightning Tables in Confluent Cloud.

Enable a Lightning Table for a topic

When you enable a Lightning Table for a topic, Confluent Cloud makes the topic data available for low-latency SQL queries.

To enable a Lightning Table, complete the following steps.

  1. Sign in to the Confluent Cloud Console at https://confluent.cloud/.

  2. From the navigation menu, select Environments.

  3. Select your environment.

  4. From the navigation menu, select Clusters.

  5. Select your cluster.

  6. From the navigation menu, select Lightning Tables.

  7. On the Lightning Tables page, click Add Lightning Table.

    If the cluster doesn’t have any Lightning Tables yet, click Enable Lightning Table instead.

    A dialog opens that lists the topics in your cluster. The list shows only topics that have a schema. To show topics without a schema, clear the Has schema checkbox.

    Dialog that lists the topics in a cluster, with an Enable link for each topic in the Lightning Table column.

    Topic list for enabling a Lightning Table

  8. In the Lightning Table column for your topic, click Enable.

  9. In the confirmation dialog, click Enable.

Enabling a Lightning Table takes several minutes. You can close the dialog and check back later.

Use the following command to enable a Lightning Table for an existing topic:

confluent rtce rtce-topic create \
  --cloud <cloud> \
  --region <region> \
  --topic-name <topic_name> \
  --description "<topic_description>"

The command accepts the following flags:

  • --cloud: Required. The cloud provider. Set to aws.

  • --region: Required. The AWS region of the cluster.

  • --topic-name: Required. The name of the Kafka topic to enable.

  • --description: Optional. A description of the topic data.

The following example enables a Lightning Table for a topic:

confluent rtce rtce-topic create \
  --cloud aws \
  --region us-west-2 \
  --topic-name orders_topic \
  --description "Customer orders table"

Send a POST request to enable a Lightning Table for a topic:

curl -X POST \
  https://api.confluent.cloud/rtce/v1/rtce-topics \
  -H "Content-Type: application/json" \
  -u "<api_key>:<api_secret>" \
  -d '{
    "spec": {
      "cloud": "<cloud>",
      "description": "<topic_description>",
      "environment": {"id": "<env_id>"},
      "kafka_cluster": {"id": "<lkc_id>"},
      "region": "<region>",
      "topic_name": "<topic_name>"
    }
  }'

The request body accepts the following parameters:

  • cloud: Required. The cloud provider. Set to AWS.

  • region: Required. The AWS region of the cluster.

  • topic_name: Required. The name of the Kafka topic to enable.

  • environment.id: Required. The ID of the Confluent Cloud environment.

  • kafka_cluster.id: Required. The ID of the Kafka cluster.

  • description: Optional. A description of the topic data.

The following example enables a Lightning Table for a topic:

curl -X POST \
  https://api.confluent.cloud/rtce/v1/rtce-topics \
  -H "Content-Type: application/json" \
  -u "$KEY:$SECRET" \
  -d '{
    "spec": {
      "cloud": "AWS",
      "description": "Customer orders table",
      "environment": {"id": "env-abc123"},
      "kafka_cluster": {"id": "lkc-abc123"},
      "region": "us-west-2",
      "topic_name": "orders_topic"
    }
  }'

To verify that the Lightning Table is enabled, send a GET request:

curl -X GET \
  "https://api.confluent.cloud/rtce/v1/rtce-topics?environment=<env_id>&spec.kafka_cluster=<lkc_id>" \
  -u "$KEY:$SECRET"

Use the confluent_rtce_topic resource to enable a Lightning Table for a topic:

resource "confluent_rtce_topic" "example" {
  cloud = "<cloud>"
  description = "<topic_description>"
  environment {
    id = "<env_id>"
  }
  kafka_cluster {
    id = "<lkc_id>"
  }
  region = "<region>"
  topic_name = "<topic_name>"
}

The resource accepts the following arguments:

  • cloud: Required. The cloud provider. Set to "AWS".

  • region: Required. The AWS region of the cluster.

  • topic_name: Required. The name of the Kafka topic to enable.

  • environment.id: Required. The ID of the Confluent Cloud environment.

  • kafka_cluster.id: Required. The ID of the Kafka cluster.

  • description: Optional. A description of the topic data.

The following example enables a Lightning Table for a topic:

resource "confluent_rtce_topic" "example" {
  cloud = "AWS"
  description = "Customer orders table"
  environment {
    id = "env-abc123"
  }
  kafka_cluster {
    id = "lkc-abc123"
  }
  region = "us-west-2"
  topic_name = "orders_topic"
}

For more information about enabling and managing Lightning Tables, see Manage Lightning Tables in Confluent Cloud.

Grant query permissions

To query a Lightning Table, the user or service account that runs the query needs permission to query data. For example, the DeveloperRead role on a topic lets you query that topic’s Lightning Table. To list topics and get topic metadata, you also need the DeveloperRead role on the cluster or on the Schema Registry cluster.

For every role that grants these permissions, see Roles and permissions. To grant a role, see Manage RBAC role bindings on Confluent Cloud.

Run your first query

If you enabled the Lightning Table in the Cloud Console, a dialog opens when setup completes. The dialog confirms that the Lightning Table is enabled and lets you query it with SQL. If you used the Confluent CLI, the API, or Terraform, see Query a Lightning Table in Confluent Cloud to choose a query interface.

Dialog that confirms a Lightning Table is enabled, with SQL, API, and MCP tabs and a prefilled SQL query.

Prefilled SQL query for a new Lightning Table

To run your first query, complete the following steps.

  1. On the SQL tab, review the prefilled query.

    The query returns up to 100 rows from your topic:

    SELECT * FROM `<topic_name>` LIMIT 100;
    
  2. Click Run.

    The dialog closes, and the Preview Lightning Table pane opens and shows the results. To explore your data further, you can edit the query in the pane and run it again.

    Preview Lightning Table pane that shows a completed SQL query and a grid of results.

    Query results in the Preview Lightning Table pane

  3. When you finish querying, close the pane.

In addition to the SQL tab, the pane has the following tabs with sample code that you can copy:

  • API: A curl command that sends a POST request to query the Lightning Table.

  • MCP: An npx command that opens the MCP Inspector, so you can try the Model Context Protocol (MCP) server for the Lightning Table.

To run the API or MCP sample code, or to query with Java Database Connectivity (JDBC), create a Global API key. For steps, see Create a Global API key.

To open the pane again later, see Query a Lightning Table in the Cloud Console.

If you access the Lightning Table over a private network, you can’t run queries in the Cloud Console. Use the API, MCP, or JDBC with the associated private endpoint instead. For more information, see Private Networking for Lightning Tables in Confluent Cloud.

Next steps

To learn more about working with Lightning Tables, see the following pages:

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.