Integrate Tableflow with Snowflake Open Catalog or Apache Polaris in Confluent Cloud

Snowflake Open Catalog is a catalog solution designed for Apache Iceberg™ tables, leveraging the open-source Iceberg REST protocol. Tableflow seamlessly connects with Snowflake Open Catalog or other Apache Polaris catalog flavors by using catalog integration.

Tableflow integration with Apache Polaris catalog

Snowflake Open Catalog integrates with Tableflow at the cluster level, enabling the automatic publication of all Tableflow-enabled topics as tables within the Open Catalog.

Configure Snowflake Open Catalog integration

Follow these steps to enable Snowflake Open Catalog integration at the cluster level. Start by creating an external catalog in Snowflake Open Catalog.

  1. Create an IAM policy that grants access to your bucket, for example, my_snow_table_bucket, where you plan to materialize Iceberg tables by following the steps in Create an IAM policy that grants access to your S3 location.

  2. Create an IAM role in AWS by following the steps in Create an IAM role to grant privileges on your S3 bucket. Attach the IAM policy you created in the previous step.

  3. In Snowflake Open Catalog, create an external catalog, for example, my_tableflow_ext_catalog, using a storage bucket, for example, my_table_bucket, by following the steps in Create a catalog in Open Catalog.

    Screenshot showing catalog creation in Snowflake for Tableflow integration

  4. Retrieve the Snowflake Open catalog arn and external ID by following the steps in Retrieve the AWS IAM user for your Open Catalog account.

  5. Grant the IAM user access to your bucket following the steps in Grant the IAM user permissions to access bucket objects.

  6. Configure a new service connection for Tableflow and obtain the client ID and client secret by following the steps in Configure a service connection.

    Screenshot showing service connection configuration in Snowflake Open Catalog for Tableflow integration

  7. Set up catalog privileges for the connection by following the steps in Set up catalog privileges.

  8. In Confluent Cloud Console, create a catalog integration in Tableflow by providing the following inputs.

    • Snowflake Open Catalog URL: https://<polaris account id>.snowflakecomputing.com
    • Warehouse name: <my_tableflow_ext_catalog>
    • Client ID: <client ID of the service connection>
    • Client secret: <client secret of the service connection>
    • Allowed scope: <PRINCIPAL_ROLE:your_polaris_principal_role>

  9. After connecting Tableflow with the Snowflake Polaris catalog, navigate to the external catalog that you created and verify that the Iceberg tables are published to the catalog.

Related content

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.