# Tableflow Quick Start with Delta Lake Tables in Confluent Cloud Confluent Tableflow enables exposing Apache Kafka® topics as Delta Lake tables. Delta Lake is an open table format that adds ACID transactions and schema enforcement to data in object storage. Complete these steps to materialize an Kafka topic as a Delta Lake table and publish it to Unity Catalog: - [Step 1: Create a topic and publish data](#cloud-tableflow-quick-start-create-topic-delta-lake) - [Step 2: Configure your S3 bucket and provider integration](#cloud-tableflow-quick-start-configure-bucket-delta-lake) - [Step 3: Enable Tableflow on your topic](#cloud-tableflow-quick-start-enable-tableflow-delta-lake) - [Step 4: Create a read-only external location in Databricks](#cloud-tableflow-quick-start-create-external-location) - [Step 5: Automatically publish the table as an external Delta Lake table to Unity Catalog](#cloud-tableflow-quick-start-publish-table-to-unity-catalog) ## Prerequisites - DeveloperWrite access on all schema subjects - CloudClusterAdmin access on your Kafka cluster - Assigner access on all provider integrations - Access to a Databricks workspace. For more information, see [Grant Role-Based Access for Tableflow in Confluent Cloud](../operate/tableflow-rbac.md#tableflow-rbac). ## Step 1: Create a topic and publish data In this step, you create a `stock-trades` topic by using Confluent Cloud Console. Click **Create topic**, provide the topic name, and create it with default settings. You can skip defining a contract. Publish data to the `stock-trades` topic by using the Datagen Source connector with the `Stock Trades` data set. When you configure the Datagen connector, click **Additional configuration** and proceed through the provisioning workflow. When you reach the **Configuration** step, in the **Select output record value format**, select **Avro**. Click **Continue** and keep the default settings. For more information, see [Datagen Source Connector Quick Start](../../../connectors/cc-datagen-source.md#cc-datagen-source-st-qs). ## Step 2: Configure your S3 bucket and provider integration Configure the storage bucket that holds the table data before you materialize your Kafka topic as a table. Create a Confluent Cloud [provider integration](../../../integrations/provider-integrations/overview.md#provider-integrations-overview) to access your Amazon S3 bucket and write materialized data into it. A provider integration is a Confluent Cloud resource that grants access to your AWS account. 1. In the AWS Management Console, [create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html#creating-bucket) in your preferred AWS account. Name the bucket `tableflow-quickstart-storage`. 2. In your Confluent Cloud environment, navigate to the **Provider integrations** tab to create a provider integration and grant Confluent Cloud access to your S3 bucket. 3. Click **Add integration**. The **Add integration details** page opens. 4. Select **AWS IAM role** and click **Continue**. The **Configure role in AWS** page opens. 5. Select **New role** and click **Continue**. 6. Select **Tableflow S3 Bucket** and copy the AWS Identity and Access Management (IAM) policy template. ![AWS IAM policy template selection for a Tableflow S3 bucket.](topics/tableflow/images/tableflow-aws-iam-permission-policy.png) 7. In the AWS Management Console, navigate to IAM. 8. In the **Access Management** section, click **Policies**, and on the **Policies** page, click **Create Policy**. As a best practice, create a designated IAM policy that grants Confluent Cloud access to your S3 bucket. 9. Paste the IAM policy template that you copied earlier. Update it with the name of your S3 bucket, for example, `tableflow-quickstart-storage`, and create a new AWS IAM policy. ![AWS IAM policy editor with the pasted Tableflow S3 bucket policy.](topics/tableflow/images/tableflow-amazon-create-policy.png) #### NOTE If your S3 bucket uses AWS Key Management Service (KMS) server-side encryption (SSE-KMS or DSSE-KMS), you must add a statement to the KMS key policy that allows Confluent Cloud to use the key. If the KMS key is in a different AWS account from your S3 bucket, you must also grant KMS actions to the IAM role in your permission policy. For more information, see [Amazon S3 storage for Tableflow](../how-to-guides/configure-storage.md#cloud-tableflow-storage-byos-s3) and [Use self-managed encryption keys](../../../security/encrypt/byok/tableflow-byok.md#tableflow-byok-integration). 10. In AWS IAM, navigate to **Roles** and click **Create Role**. 11. For the **Trusted entity type**, select **Custom trust policy**. 12. From the Tableflow UI in Cloud Console, copy the `Trust-policy.json` JSON file and paste it into the policy editor in the AWS Management Console. 13. Attach the permission policy that you created earlier and save your new IAM role, for example, `tableflow-quickstart-role`. 14. Copy the role Amazon Resource Name (ARN), for example, `arn:aws:iam:::role/tableflow-quickstart-role`. 15. In the Cloud Console, locate the **Map the role in Confluent** section on the **Provider integrations** tab. In the **AWS ARN** section, paste the ARN that you copied and click **Continue**. ![The AWS ARN field where you paste the IAM role ARN in the Confluent Cloud Console.](topics/tableflow/images/tableflow-map-role-aws-arn.png) 16. After creating the provider integration, update the trust policy of the AWS IAM role, for example `tableflow-quickstart-role`, by using the policy displayed in Cloud Console. ![AWS IAM role trust policy displayed in the Confluent Cloud Console.](topics/tableflow/images/tableflow-update-trust-policy.png) ## Step 3: Enable Tableflow on your topic With the provider integration configured, you can enable Tableflow on your Kafka topic to materialize it as a table in the storage bucket that you created in [Step 2](quick-start-custom-storage-glue.md#cloud-tableflow-quick-start-configure-bucket). 1. Navigate to your **stock-trades** topic and click **Enable Tableflow**. 2. In the **Enable Tableflow** dialog, select **Delta** as the table format. ![Enable Tableflow dialog in Confluent Cloud Console](topics/tableflow/images/tableflow-enable-dialog-delta.png) 3. Click **Configure custom storage**. 4. In the **Choose where to store your Tableflow data** section, click **Store in your bucket**. 5. In the **Provider integration** dropdown, select the provider integration that you created in [Step 2](quick-start-custom-storage-glue.md#cloud-tableflow-quick-start-configure-bucket). Provide the name of the storage bucket that you created, which in this guide is *tableflow-quickstart-storage*. 6. Click **Continue** to review the configuration and launch Tableflow. Materializing a newly created topic as a table can take a few minutes. 7. After enabling Tableflow, in the **Monitor** section, copy the storage location of the table. ![Tableflow storage location in Cloud Console](topics/tableflow/images/tableflow-storage-location.png) For low-throughput topics in which Kafka segments have not been filled, Tableflow tries optimistically to publish data every 5 minutes. This is best-effort and not guaranteed. ## Step 4: Create a read-only external location in Databricks You can query Delta tables materialized by Tableflow in Databricks as [external tables](https://docs.databricks.com/aws/en/tables/external). To create an external table in Databricks, you must first create an *External Location*. 1. Log in to the Databricks workspace that you use to query Delta Lake tables. 2. Click **Catalog** to open Catalog Explorer. 3. On the **Quick access** page, click **External data**, and in the **External locations** tab, and click **Create external location**. 4. In the **Create a new external location** dialog, select **Manual** and click **Next**. 5. Create a new external location by providing the external location name, the URL of the S3 bucket where Delta Tables are stored, and create a new storage credential by following [these steps](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials). Check **Limit to read-only use** to ensure that the external location is read-only. ![Screenshot of the new external location dialog in Databricks](topics/tableflow/images/tableflow-databricks-new-external-location.png) ## Step 5: Automatically publish the table as an external Delta Lake table to Unity Catalog Once Tableflow is enabled on your Kafka topic, you can set up Unity Catalog Integration to publish Delta Lake tables to Unity Catalog automatically by following these instructions: [Integrate Tableflow with Unity Catalog](../how-to-guides/catalog-integration/integrate-with-unity-catalog.md#cloud-tableflow-integrate-with-unity-catalog). ## Related content - [Tableflow Quick Start with Managed Storage](quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage) - [Tableflow Quick Start Using Your Storage and AWS Glue](quick-start-custom-storage-glue.md#cloud-tableflow-quick-start) - [Stream Processing with Confluent Cloud for Apache Flink](../../../flink/overview.md#ccloud-flink) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).