# Integrate Tableflow with Unity Catalog in Confluent Cloud Unity Catalog provides a fine-grained, unified governance solution for all data and AI assets on the Databricks Data Intelligence Platform. Tableflow’s Unity Catalog integration enables you to publish materialized tables to Unity Catalog, making them accessible as external Delta Lake tables in Databricks. ![Tableflow integration with Unity Catalog](topics/tableflow/images/tableflow-unity-catalog-integration.png) Databricks Unity Catalog integrates with Tableflow at the cluster level, enabling the automatic publication of all Delta Lake-enabled topics as tables within Unity Catalog. #### NOTE Tableflow Unity Catalog integration uses Databricks Unity Catalog Open Preview for creating external tables by using Unity Catalog open APIs. For more information, see [Create external Delta tables from external clients](https://docs.databricks.com/aws/en/external-access/create-external-tables). #### IMPORTANT Tableflow does not support integrating with Unity Catalogs that are deployed on private networks. The Databricks workspace used for Unity Catalog integration must be accessible over the public internet. If your Unity Catalog is on a private network, you can manually register Tableflow tables in Unity Catalog as a workaround. For more information, see [Using Tableflow with Databricks workspaces on private networks](#cloud-tableflow-manual-unity-catalog-registration). Tableflow maps and creates a new schema in Unity Catalog using your Kafka cluster ID. Then it publishes all the topics that have Delta Lake format enabled as external Delta Lake tables in Unity Catalog. ## Prerequisites - DeveloperWrite access on all schema subjects. - CloudClusterAdmin access on your Kafka cluster. - Assigner access on all provider integrations. - Access to a Databricks workspace. - An external Delta Lake table, like the one created in [Quick Start with Delta Lake Tables](../../get-started/quick-start-delta-lake.md#cloud-tableflow-quick-start-delta-lake). ## Configure Unity Catalog Follow these steps to enable Databricks Unity Catalog sync for Delta Lake tables in Tableflow. 1. Log in to your Databricks workspace and create or select the catalog that you want to use to sync Delta Lake tables created by Tableflow. For this guide, the catalog is named *tableflow-quick-start-catalog*. 2. Navigate to the Databricks workspace settings and click **Identity and Access**. 3. Create a new Service Principal named *tableflow-quick-start-sp* and generate a secret. Store the client ID and secret securely, because you will need them later to configure the Unity Catalog integration in Tableflow. ![Add a service principal in the Databricks workspace](topics/tableflow/images/tableflow-databricks-add-service-principal.png)![Service principal details](topics/tableflow/images/tableflow-databricks-service-principal-details.png) 4. Navigate to the **tableflow-quick-start-catalog** and grant the **tableflow-quick-start-sp** service principal Data Editor access to the catalog. In addition to the Data Editor preset, select **EXTERNAL USE SCHEMA**. #### IMPORTANT The **EXTERNAL USE SCHEMA** privilege is required. Without it, creating the Unity Catalog integration in Tableflow fails, because this privilege allows an external engine to access objects in the schema. ![Grant Data Editor and External Use Schema access to the tableflow-quick-start-sp service principal](topics/tableflow/images/tableflow-databricks-grant-data-editor-privileges.png) 5. Navigate to the External Location you created previously and grant the following permissions to **tableflow-quick-start-sp**: - CREATE EXTERNAL TABLE - READ FILES - WRITE FILES ![Grant privileges to the tableflow-quick-start-sp service principal](topics/tableflow/images/tableflow-databricks-grant-privileges.png) ## Configure Unity Catalog integration with Tableflow 1. Log in to Confluent Cloud Console. 2. Navigate to the Kafka cluster that has the topics you want to sync, and click **Tableflow**. 3. In the **External Catalog Integrations** section, click **Add Integration**. 4. In the **Select integration type** section, select **Unity Catalog**. 5. Provide a name to identify your catalog integration and click **Continue**. 6. On the Add connection details page, provide the following inputs and click **Continue**. - Databricks Workspace URL. - Client ID and secret of the service principal that you obtained earlier. - Name of the Unity Catalog to integrate with. ![Add connection details](topics/tableflow/images/tableflow-add-databricks-unity-integration.png) 7. Return to your Databricks workspace, and in **Catalog Explorer**, confirm that the Delta table is published within the catalog, categorized under the schema name corresponding to your cluster ID. ![Delta table published in Databricks Unity Catalog](topics/tableflow/images/tableflow-databricks-catalog-explorer.png) 8. Once the table is available in Databricks, you can perform queries on it and use it for your data analytics tasks. Also, you can govern this table in Unity Catalog like any other asset. ![Query a Delta table in Databricks Unity Catalog](topics/tableflow/images/tableflow-databricks-unity-catalog-query.png) ## Using Tableflow with Databricks workspaces on private networks If your Databricks workspace or Unity Catalog is deployed on a private network, Tableflow cannot automatically sync table metadata. As an alternative, you can perform a one-time manual table registration in Unity Catalog by referencing the storage location where Tableflow writes Delta tables. With this approach, Tableflow materializes Delta tables into your configured object storage, and you manually register them in Unity Catalog. The tables continue to receive updates from Tableflow automatically. ### Prerequisites Before you begin, ensure you have the following: - A Kafka topic with streaming data. - Tableflow enabled on your topic, with a Delta table materialized in your cloud object storage. For help setting up these resources, see the [Quick Start with Delta Lake Tables](../../get-started/quick-start-delta-lake.md#cloud-tableflow-quick-start-delta-lake). - Unity Catalog enabled in your Databricks workspace. - `CREATE TABLE` permissions in the target catalog and schema. - A storage credential configured for the Tableflow bucket or container. - An external location configured for the storage path. ### Register a Tableflow table in Unity Catalog 1. Get the cloud storage location where Tableflow stores the Delta table. ### Confluent Cloud console 1. Go to your cluster, click **Topics**, and select your topic. 2. Click the **Tableflow** tab. 3. Copy the **Table Location** to save as the cloud storage path for the next step. ### Confluent Cloud CLI Run the following command to get the storage location: ```bash confluent kafka topic tableflow describe \ --cluster \ --environment \ --output json | jq -r '.storage_location' ``` Example outputs by cloud provider: - **AWS S3:** `s3://my-bucket/tableflow/env-xxxxx/lkc-xxxxx/v1/abc123.../` - **Azure ADLS Gen2:** `abfss://container@account.dfs.core.windows.net/tableflow/env-xxxxx/lkc-xxxxx/v1/abc123.../` 2. Register the Tableflow Delta table in Unity Catalog as an external table. Run the following SQL in your Databricks workspace: ```sql CREATE EXTERNAL TABLE IF NOT EXISTS ``.``.`` USING DELTA LOCATION ''; ``` The following examples show the command for each cloud provider: ```sql -- AWS S3 CREATE EXTERNAL TABLE IF NOT EXISTS `my_catalog`.`default`.`stock-trades` USING DELTA LOCATION 's3://my-bucket/tableflow/env-xxxxx/lkc-xxxxx/v1/abc123.../'; -- Azure ADLS Gen2 CREATE EXTERNAL TABLE IF NOT EXISTS `my_catalog`.`default`.`stock-trades` USING DELTA LOCATION 'abfss://container@account.dfs.core.windows.net/tableflow/env-xxxxx/lkc-xxxxx/v1/abc123.../'; ``` 3. Verify the registration by querying the table: ```sql SELECT * FROM `my_catalog`.`default`.`stock-trades` LIMIT 100; ``` ## Related content - [Quick Start with Delta Lake Tables](../../get-started/quick-start-delta-lake.md#cloud-tableflow-quick-start-delta-lake) - [Integrate Tableflow with the AWS Glue Catalog](integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog) - [Integrate Tableflow with the Snowflake Open Catalog](integrate-with-snowflake-open-catalog-or-apache-polaris.md#cloud-tableflow-integrate-with-snowflake-open-catalog) - [Integrate with Cloud Service Providers in Confluent Cloud](../../../../integrations/provider-integrations/overview.md#provider-integrations-overview) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).