# Query Iceberg Tables with AWS Analytics Services and Tableflow in Confluent Cloud Tableflow integrates seamlessly with Amazon Athena by using the AWS Glue Data Catalog or the Tableflow Apache Iceberg™ REST Catalog, enabling you to query Apache Iceberg™ tables materialized by Tableflow as a *read-only table*. Amazon Athena efficiently accesses and analyzes real-time streaming data managed by Tableflow, enabling powerful, serverless analytics at scale. ## Use Amazon Athena with AWS Glue Data Catalog To use Amazon Athena with AWS Glue Data Catalog, you must [configure AWS Glue Catalog integration in Tableflow](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog). When the catalog integration is complete, all materialized Iceberg tables of the Apache Kafka® cluster are discoverable in the AWS Glue Data Catalog. With Amazon Athena, you can use Athena SQL or Amazon Athena for Spark when performing data analytics. ### Amazon Athena SQL You can use Athena SQL to query your data directly in Amazon S3 by using the AWS Glue Data Catalog. #### Prerequisites - Configure [AWS Glue Catalog integration](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog) in Tableflow. - Verify that Tableflow is enabled for the topic you want to materialize into a table by completing steps 1-3 in the [Tableflow Quick Start](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage). - Ensure you have the required permissions for Amazon Athena to access the table’s storage bucket. - Ensure AWS Athena has read-only access to the Glue catalog and the storage bucket. After you configure the AWS Glue Data Catalog in Tableflow, follow these steps to query your materialized tables. 1. Log in to Amazon Athena and select **Query your data with Trino SQL**. ![Screenshot showing the Amazon Athena Get Started dialog](topics/tableflow/images/tableflow-amazon-athena-get-started-with-trino.png) 2. Start the Athena SQL query editor. ![Screenshot showing the Athena SQL query editor](topics/tableflow/images/tableflow-amazon-athena-sql-query-editor.png) 3. In the **Data source** dropdown, select **AwsDataCatalog**, which is the AWS Glue Data Catalog. 4. In the **Database** dropdown, select the database created based on the Kafka cluster ID in the Athena SQL Query Editor. 5. In the query editor, write an Athena SQL query to consume data from an Iceberg table that’s materialized by Tableflow, as shown in the following example. ```sql SELECT orderitemid, orderid, productid, quantity FROM "AwsDataCatalog"."lkc-a1b2c3"."retail_cdc.public.order_items" LIMIT 10; ``` ### Amazon Athena for Spark (PySpark) with Tableflow Catalog You can use Amazon Athena for Apache Spark to query Tableflow’s Iceberg tables by using AWS Glue Data Catalog as the Iceberg catalog. #### Prerequisites - Configure [AWS Glue Catalog integration](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog) in Tableflow. - Verify that Tableflow is enabled for the topic you want to materialize into a table by completing steps 1-3 in the [Tableflow Quick Start](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage). - Ensure you have the required permissions for Amazon Athena to access the storage bucket of the table. - Ensure AWS Athena has read-only access to the Glue catalog and the storage bucket. 1. Ensure that you have the values from [Step 3](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage-credentials) of the quick start. - Tableflow REST Catalog endpoint - Credentials to access the Tableflow REST Catalog 2. Log in to Amazon Athena and select **Analyze your data using PySpark and Spark SQL**. ![Screenshot showing the Amazon Athena Get Started dialog](topics/tableflow/images/tableflow-amazon-athena-get-started-with-pyspark.png) 3. Under the Athena workgroup, create a new notebook. 4. In the **Workgroup** textbox, enter a name, for example, “tableflow-wg”. 5. In the **Apache Spark properties** section, select **Custom**, which enables providing Spark properties in JSON format. ![Screenshot showing the Amazon Athena Create Notebook dialog](topics/tableflow/images/tableflow-amazon-athena-create-notebook.png) 6. Copy the following Spark properties configuration to Athena. ### Tableflow Catalog Replace *spark.sql.catalog.tableflow-cluster.uri* and *spark.sql.catalog.tableflow-cluster.credential* with the values from [Step 3](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage-credentials) of the quick start. ```json { "spark.sql.catalog.tableflow-cluster": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.tableflow-cluster.catalog-impl": "org.apache.iceberg.rest.RESTCatalog", "spark.sql.catalog.tableflow-cluster.credential": ":", "spark.sql.catalog.tableflow-cluster.uri": "", "spark.sql.catalog.tableflowdemo.s3.remote-signing-enabled": "true", "spark.sql.defaultCatalog": "tableflow-cluster", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" } ``` ### AWS Glue Data Catalog Replace *your-glue-catalog-bucket* with your value. ```json { "spark.sql.catalog.glue": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.glue.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", "spark.sql.catalog.glue.warehouse": "s3:///", "spark.sql.catalog.glue.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "spark.sql.catalog.glue_catalog.glue.skip-name-validation": "true", "spark.sql.defaultCatalog": "glue", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" } ``` 7. In Athena PySpark, run the following query to see the Iceberg tables in the Tableflow Catalog. ```sql %%sql SHOW TABLES in ``; ``` 8. Select a table of your choice and query data with the following PySpark query. ```sql %%sql SELECT * FROM ``.``; ``` ## Amazon Redshift You can use Amazon Redshift Spectrum or Redshift Serverless to query Iceberg tables registered in the AWS Glue Data Catalog as read-only tables. ### Using Amazon Redshift with AWS Glue Data Catalog #### Prerequisites - Configure [AWS Glue Catalog integration](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog) in Tableflow. - Verify that Tableflow is enabled for the topic you want to materialize into a table by completing steps 1-3 in the [Tableflow Quick Start](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage). To use Amazon Redshift with AWS Glue Data Catalog, you must [configure AWS Glue Catalog integration in Tableflow](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog). When the catalog integration is complete, all materialized Iceberg tables of the Kafka cluster are discoverable in the AWS Glue Data Catalog. 1. Create a new Amazon Redshift cluster and navigate to the Query Editor V2. 2. Navigate to the Redshift query editor and select **awsdatacatalog** as the catalog. Because Iceberg tables have already been published in the AWS Glue Data Catalog, you can discover and query them through **awsdatacalog**. ![Screenshot showing the Athena SQL query editor](topics/tableflow/images/tableflow-amazon-redshift-sql-query-editor.png) ## Amazon EMR ### Using Amazon EMR with AWS Glue Data Catalog #### Prerequisites - Configure [AWS Glue Catalog integration](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog) in Tableflow. - Verify that Tableflow is enabled for the topic you want to materialize into a table by completing steps 1-3 in the [Tableflow Quick Start](../../get-started/quick-start-managed-storage.md#cloud-tableflow-quick-start-managed-storage). - Ensure you have the required permissions for Amazon Athena to access the storage bucket of the table. - Ensure AWS Athena has read-only access to the Glue catalog and the storage bucket. To use Amazon EMR with AWS Glue Data Catalog, you must [configure AWS Glue Catalog integration in Tableflow](../catalog-integration/integrate-with-aws-glue-catalog.md#cloud-tableflow-integrate-with-aws-glue-catalog). 1. Log in to Amazon EMR Studio, and [create a new Studio and a Spark workspace](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-studio.html). 2. In Amazon EMR, run the `configure -f` command with the following configuration. Replace *your-warehouse* with the value for your warehouse. 3. Configure Spark to use the AWS Glue Data Catalog by providing the catalog implementation, `org.apache.iceberg.aws.glue.GlueCatalog`. Replace *your-warehouse* with the value for your warehouse. ```none %%configure -f { "conf": { "spark.sql.catalog.glue_demo_catalog": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.glue_demo_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", "spark.sql.catalog.glue_demo_catalog.warehouse": "s3://", "spark.sql.extensions":"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.glue_demo_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "spark.sql.catalog.glue_demo_catalog.glue.skip-name-validation":"true" } } ``` 4. Run the following Spark queries to consume the Iceberg tables created by Tableflow. ```sql spark.sql("SHOW TABLES IN glue_demo_catalog.`lkc-devcrmn56p`").show() spark.sql("SELECT orderitemid, orderid, productid, quantity FROM glue_demo_catalog.`lkc-devcrmn56p`.`retail_cdc.public.order_items`").show() ``` Your output should resemble: ![Screenshot showing example Spark query results on Iceberg tables materialized with Confluent Tableflow](topics/tableflow/images/tableflow-spark-query-results.png) ## Related content - [Query with Flink](query-with-flink.md#cloud-tableflow-query-with-flink) - [Query with DuckDB](query-with-duckdb.md#cloud-tableflow-query-with-duckdb) - [Query with Snowflake](query-with-snowflake.md#cloud-tableflow-query-with-snowflake) - [Query with Trino](query-with-trino.md#cloud-tableflow-query-with-trino) - [Stream Processing with Confluent Cloud for Apache Flink](../../../../flink/overview.md#ccloud-flink) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).