Tableflow Quick Start with Iceberg Tables Using Your Storage and AWS Glue in Confluent Cloud
Confluent Tableflow exposes Apache Kafka® topics as Apache Iceberg™ tables. Iceberg is an open table format for large analytic datasets in object storage.
Complete these steps to materialize a Kafka topic as an Iceberg table using your own Amazon S3 storage and the AWS Glue Data Catalog:
Prerequisites
DeveloperReadaccess on all schema subjects.CloudClusterAdminaccess on your Kafka cluster.Assigneraccess on all provider integrations.
For more information, see Grant Role-Based Access for Tableflow in Confluent Cloud.
Step 1: Create a topic and publish data
In this step, you create a stock-trades topic in Confluent Cloud Console.
Click Create topic, provide the topic name, and create it with default settings. You can skip defining a contract.
Publish data to the
stock-tradestopic with the Datagen Source connector and the Stock Trades dataset.When you configure the Datagen Source connector, click Additional configuration and proceed through the provisioning workflow.
When you reach the Configuration step, in the Select output record value format field, select Avro. Click Continue and keep the default settings.
For more information, see Datagen Source Connector Quick Start.
Step 2: Configure your S3 bucket and provider integration
Configure the storage bucket that holds the table data before you materialize your Kafka topic as an Iceberg table.
Create a Confluent Cloud provider integration to access your Amazon S3 bucket and write materialized data into it. A provider integration is a Confluent Cloud resource that grants access to your AWS account.
In the AWS Management Console, create an S3 bucket in your preferred AWS account. Name the bucket
tableflow-quickstart-storage.In your Confluent Cloud environment, navigate to the Provider integrations tab to create a provider integration and grant Confluent Cloud access to your S3 bucket.
Click Add integration.
The Add integration details page opens.
Select AWS IAM role and click Continue.
The Configure role in AWS page opens.
Select New role and click Continue.
Select Tableflow S3 Bucket and copy the AWS Identity and Access Management (IAM) policy template.
In the AWS Management Console, navigate to IAM.
In the Access Management section, click Policies, and on the Policies page, click Create Policy.
As a best practice, create a designated IAM policy that grants Confluent Cloud access to your S3 location.
Paste the IAM policy template you copied earlier. Update it with the name of your S3 bucket, for example,
tableflow-quickstart-storage, and create a new AWS IAM policy.
Note
If your S3 bucket uses AWS Key Management Service (KMS) server-side encryption (SSE-KMS or DSSE-KMS), you must add a statement to the KMS key policy that allows Confluent Cloud to use the key. If the KMS key is in a different AWS account from your S3 bucket, you must also grant KMS actions to the IAM role in your permission policy. For more information, see Amazon S3 storage for Tableflow and Use self-managed encryption keys.
Navigate to AWS IAM Roles and click Create Role.
For the Trusted entity type, select Custom trust policy.
From the Tableflow UI in Cloud Console, copy the
Trust-policy.jsonJSON file and paste it into the policy editor in the AWS Management Console.Attach the permission policy you created earlier and save your new IAM role, for example,
tableflow-quickstart-role.Copy the role Amazon Resource Name (ARN), for example,
arn:aws:iam::<account_id>:role/tableflow-quickstart-role.In the Cloud Console, locate the Map the role in Confluent section on the Provider integrations tab. In the AWS ARN section, paste the ARN you copied and click Continue.
After creating the provider integration, update the trust policy of the AWS IAM role, for example
tableflow-quickstart-role, by using the policy displayed in Cloud Console.
Step 3: Enable Tableflow on your topic
You can now enable Tableflow on your Kafka topic to materialize it as an Iceberg table in the storage bucket that you created in Step 2.
Navigate to your
stock-tradestopic and click Enable Tableflow.In the Enable Tableflow dialog, click Configure custom storage.
In the Choose where to store your Tableflow data section, click Store in your bucket.
In the Provider integration list, select the provider integration that you created in Step 2. Enter the name of the storage bucket that you created, which in this example is
tableflow-quickstart-storage.Click Continue to review the configuration and launch Tableflow.
Materializing a newly created topic as an Iceberg table can take a few minutes.
Tableflow optimistically tries to publish data every 15 minutes for low-throughput topics whose Kafka segments aren’t full. Tableflow makes this attempt on a best-effort basis without a delivery guarantee.
Step 4: Configure AWS Glue Data Catalog integration
Follow the steps in Integrate Tableflow with the AWS Glue Catalog to configure the AWS Glue Data Catalog as a catalog integration.
Confluent Cloud automatically creates the stock-trades Iceberg table and a database in the AWS Glue Data Catalog after you configure the catalog integration. Tableflow bases the database name on your Kafka cluster ID.
It can take a few minutes for Tableflow to publish Iceberg tables to the AWS Glue Data Catalog.
Tip
You can monitor each topic’s catalog sync status from the Tableflow page in Cloud Console after you configure the catalog integration. For more information, see Topic catalog sync status.
Step 5: Query Iceberg tables
You can use any AWS Glue and Iceberg-compatible compute engine to query the table. This example uses Amazon Athena SQL.
Follow the steps in Amazon Athena SQL to start writing queries for the stock-trades table.
Step 6: Query data with other analytics engines (optional)
Explore other integration options for using Tableflow with analytics engines: