Access the Tableflow Iceberg REST Catalog over AWS PrivateLink

The Tableflow Apache Iceberg™ REST Catalog (IRC) is accessed by query engines, such as Apache Spark and Trino, to query data. External catalogs, such as AWS Glue, access it for catalog federation. If your analytics run in VPCs without internet access, or your security policies require traffic to the IRC to remain on a private network, you can reach the catalog over AWS PrivateLink.

This topic shows how to make the IRC reachable from your AWS VPC over AWS PrivateLink, so that query engines and federated catalogs in that VPC can use Tableflow tables without a public endpoint.

Note

Ingress private network connectivity to the Tableflow catalog is available on AWS only. Azure and Google Cloud aren’t supported. For an overview and limitations, see Use Private Networking with Tableflow in Confluent Cloud.

How it works

Private access to the Tableflow IRC is established through an ingress PrivateLink Gateway in Confluent Cloud, an interface VPC endpoint, and a private DNS name. Query engines and AWS Glue use an endpoint in your VPC and a DNS name that resolves only within that VPC. They connect to the private hostname as they would to the public catalog endpoint, while data files are read directly from your storage bucket.

At a high level, you set up private access with the following steps:

1. Create an ingress PrivateLink Gateway in your Confluent Cloud environment, in the AWS Region of your VPC. Confluent Cloud publishes the Tableflow catalog as an AWS PrivateLink endpoint service.

2. Create an interface VPC endpoint in your VPC that connects to that service. The endpoint gets one private IP address in each subnet you select.

3. Create an access point on the gateway that registers your VPC endpoint. Confluent Cloud accepts the endpoint connection and assigns the access point a DNS domain.

4. Create a Route 53 private hosted zone attached to your VPC, named after the DNS domain, with a record that resolves the private catalog hostname to your VPC endpoint.

5. Point your query engine or federated catalog at the private catalog hostname, tableflow-<accesspointId>.<region>.aws.accesspoint.glb.confluent.cloud, where <accesspointId> is your access point ID with the hyphen removed. For example, ap-123456 becomes ap123456. The gateway routes requests for this hostname to the Tableflow IRC.

Note

Tableflow tables are read-only for query engines, and all writes go through Tableflow.

Prerequisites

Before you begin, ensure that you have the following:

• A Confluent Cloud cluster in AWS with at least one Tableflow-enabled topic that uses your own storage in Iceberg format and is in the Running phase. For steps, see Tableflow Quick Start with Iceberg Tables Using Your Storage and AWS Glue in Confluent Cloud.

• An AWS VPC in the region where you want the gateway, with a private subnet in each availability zone you use, and with DNS hostnames and DNS resolution enabled.

• Permissions in that AWS account to create interface VPC endpoints and Route 53 private hosted zones.

• A client host in a private subnet of the VPC, for verification and for Spark. For example, use an EC2 instance that you manage through AWS Systems Manager Session Manager.

• A Tableflow API key and secret for the catalog. For steps, see Tableflow Quick Start with Iceberg Tables Using Your Storage and AWS Glue in Confluent Cloud.

• Your Confluent Cloud organization ID and environment ID, which form part of the catalog URI.

• For the CLI instructions, the Confluent CLI, logged in, and the AWS CLI.

Step 1: Create the ingress PrivateLink Gateway

Create the gateway in the environment that contains your cluster. Each step in this procedure includes the Confluent Cloud Console or AWS console instructions and the matching CLI commands.

Confluent Cloud Console

1. In the environment that contains your cluster, click Network management.

2. On the For serverless products tab, click + Add gateway configuration. The ingress gateway is managed on this tab for every cluster type.

3. Keep PrivateLink selected as the gateway type. In the Set up connections to/from Confluent Cloud section, find the From your VPC or VNet to Confluent Cloud card and click + Create configuration. The Configure gateway pane opens.

4. On the Gateway tab, enter a Gateway name, keep aws as the cloud provider, and select the Region of your VPC in Region.

   

5. Click Submit. The pane moves to the Access point tab and the gateway status is Created.

6. Copy the PrivateLink Service ID shown in the pane. You use it in Step 2.

   

Confluent CLI

confluent network gateway create my-ingress-gateway \
  --cloud aws --region <region> --type ingress-privatelink \
  --environment <env-id>

confluent network gateway describe <gateway-id> --environment <env-id>
# note the AWS VPC Endpoint Service Name

Step 2: Create the interface VPC endpoint

Each gateway needs its own interface VPC endpoint in your VPC, created against the PrivateLink Service ID from Step 1. The service ID is the same for every gateway in an environment and Region. Leave private DNS disabled, because you create the DNS records yourself in Step 4. The endpoint stays in Pending acceptance until you create the access point in Step 3.

AWS console

1. Open VPC, then PrivateLink and Lattice, then Endpoints, and click Create endpoint.

2. Under Service settings, enter the PrivateLink Service ID in Service name and click Verify service.

3. Select your VPC and one private subnet in each availability zone you use.

4. Select a security group that allows inbound TCP 443 from your client hosts.

5. Clear Enable DNS name if it is selected, then click Create endpoint.

6. Copy the VPC endpoint ID. After the endpoint becomes Available in Step 3, copy the first entry under DNS names on the endpoint’s Details tab. You use it in Step 4.

AWS CLI

aws ec2 create-vpc-endpoint \
  --vpc-id <vpc-id> --vpc-endpoint-type Interface \
  --service-name <privatelink-service-id> \
  --subnet-ids <private-subnet-ids> \
  --security-group-ids <sg-allowing-443-from-clients> \
  --no-private-dns-enabled

Step 3: Create the access point

Create an access point on the gateway that registers your VPC endpoint.

Confluent Cloud Console

1. Continue on the Access point tab of the Configure gateway pane. To return to it later, open Network management, and on the For serverless products tab, select your gateway, open the Access points tab, and click Create access point.

2. Paste the endpoint ID from Step 2 in VPC Endpoint ID from AWS.

3. Enter an Access point name and click Create access point.

   

4. Wait for Access point successfully created. The access point shows Ready, and the pane shows the DNS Domain, for example ap123456.us-west-2.aws.accesspoint.confluent.cloud. Copy it for Step 4.

   

5. In the AWS console, confirm that the VPC endpoint status has changed from Pending acceptance to Available. This takes a minute or two.

6. Click Finish. On Network management, the gateway status moves from Provisioning to Ready.

   

Confluent CLI

confluent network access-point private-link ingress-endpoint create \
  --cloud aws --gateway <gateway-id> --vpc-endpoint-id <vpce-id> \
  --environment <env-id>

confluent network access-point private-link ingress-endpoint describe \
  <access-point-id> --environment <env-id>
# note the DNS Domain

Determine your catalog endpoint

Every client setting in the remaining steps uses values derived from the access point. Build them once and keep them at hand.

Value	Format	Example
Access point ID (Step 3)	ap-xxxxxx	ap-123456
DNS domain (Step 3)	<accesspointId>.<region>.aws.accesspoint.confluent.cloud, where <accesspointId> is your access point ID with the hyphen removed. For example, ap-123456 becomes ap123456.	ap123456.us-west-2.aws.accesspoint.confluent.cloud
<catalog-hostname>	tableflow-<accesspointId>.<region>.aws.accesspoint.glb.confluent.cloud. This is the DNS domain with tableflow- in front (hyphen, not dot) and glb. inserted before confluent.cloud.	tableflow-ap123456.us-west-2.aws.accesspoint.glb.confluent.cloud
<catalog-uri>, the Iceberg REST base URL	https://<catalog-hostname>/iceberg/catalog/organizations/<org-id>/environments/<env-id>	https://tableflow-ap123456.us-west-2.aws.accesspoint.glb.confluent.cloud/iceberg/catalog/organizations/<org-id>/environments/<env-id>

The private catalog hostname is published in public DNS only as a pointer to your DNS domain. It resolves to an address only inside VPCs associated with your private hosted zone, so the catalog is only reachable over PrivateLink.

The same hostname and URI serve every Tableflow cluster in the environment. The cluster ID appears only as the namespace in catalog calls.

Step 4: Create the private hosted zone

In Route 53, the AWS DNS service, create a private hosted zone named exactly as the DNS domain, and add one tableflow CNAME record in it. The zone is needed because public DNS answers <catalog-hostname> with a CNAME to tableflow.<accesspointId>.<region>.aws.accesspoint.confluent.cloud, a name that has no public records. The private zone resolves that name only inside your VPC, and it returns the DNS name of your VPC endpoint. Keep the VPC on the Amazon-provided resolver so that the first, public lookup still works.

AWS console

1. Open Route 53, then Hosted zones, and click Create hosted zone.

2. Set Domain name to the DNS domain from Step 3, for example ap123456.us-west-2.aws.accesspoint.confluent.cloud.

3. Set Type to Private hosted zone, select the Region and VPC ID of the VPC that contains the endpoint, and click Create hosted zone.

4. Click Create record. Set Record name to tableflow, Record type to CNAME, and Value to the VPC endpoint DNS name from Step 2. Set TTL to 60 seconds and click Create records.

AWS CLI

ZONE=$(aws route53 create-hosted-zone --name <dns-domain> \
  --caller-reference "tf-irc-$(date +%s)" \
  --vpc VPCRegion=<region>,VPCId=<vpc-id> \
  --hosted-zone-config PrivateZone=true \
  --query HostedZone.Id --output text)

aws route53 change-resource-record-sets --hosted-zone-id $ZONE \
  --change-batch '{
  "Changes": [{"Action": "UPSERT", "ResourceRecordSet": {
    "Name": "tableflow.<dns-domain>", "Type": "CNAME", "TTL": 60,
    "ResourceRecords": [{"Value": "<vpce-dns-name>"}]}}]}'

The Confluent Cloud networking documentation uses a wildcard record (*) in this zone to cover every service behind the access point. A tableflow record covers the Tableflow IRC only.

Step 5: Verify the connection from your VPC

Run the following commands from your client host in the VPC. The first command checks DNS. The second command requests a token. The third command lists the catalog namespaces through the private path.

getent hosts <catalog-hostname>

TOKEN=$(curl -s -X POST <catalog-uri>/v1/oauth/tokens \
  -d grant_type=client_credentials -d scope=catalog \
  -d client_id=<tableflow-api-key> \
  --data-urlencode client_secret=<tableflow-api-secret> \
  | jq -r .access_token)

curl -s -H "Authorization: Bearer $TOKEN" <catalog-uri>/v1/namespaces

The first command prints private IP addresses from your subnets, one for each subnet of the endpoint. The third command prints a JSON list of namespaces that includes your cluster ID. If you run the first command from a host outside the VPC, it returns nothing, because public DNS answers the hostname only with the CNAME to the access point domain, so the catalog has no public address.

Query with Spark

On your client host, use the Iceberg Spark runtime and AWS bundle, for example iceberg-spark-runtime-3.5_2.12 and iceberg-aws-bundle version 1.8.1. Place the JAR files in $SPARK_HOME/jars, because a VPC without an internet path can’t download them through --packages. The IAM role of the host needs s3:GetObject, s3:ListBucket, and s3:GetBucketLocation on the storage bucket.

spark-sql \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.tf=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.tf.type=rest \
  --conf spark.sql.catalog.tf.uri=<catalog-uri> \
  --conf spark.sql.catalog.tf.credential=<tableflow-api-key>:<tableflow-api-secret> \
  --conf spark.sql.catalog.tf.oauth2-server-uri=<catalog-uri>/v1/oauth/tokens \
  --conf spark.sql.catalog.tf.scope=catalog \
  --conf spark.sql.catalog.tf.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.tf.client.region=<region>

Then run queries such as the following:

SHOW NAMESPACES IN tf;
SELECT * FROM tf.`<cluster-id>`.<topic-name> LIMIT 10;

AWS Glue catalog federation over PrivateLink

Catalog federation through AWS Glue and AWS Lake Formation uses the private path when the Glue connection is attached to a private subnet in your VPC, because Glue then calls the catalog from network interfaces inside the VPC. Follow the catalog federation section of Integrate Tableflow with the AWS Glue Data Catalog with the following changes.

1. Prepare the VPC for the Glue connection. Glue places network interfaces in the subnet you choose and calls the catalog from them. Those interfaces must reach the catalog endpoint, AWS Secrets Manager, AWS STS, and AWS KMS.

   • If the subnet has no NAT gateway, create interface VPC endpoints for secretsmanager, sts, and kms with private DNS enabled.

   • Create a security group for the Glue interfaces with a self-referencing inbound rule.

   • On the security group of the catalog VPC endpoint from Step 2, allow inbound TCP 443 from the Glue security group.

2. Extend the IAM role. In addition to the permissions in the Glue guide, the role needs the following permissions:

   • EC2 permissions to place its network interfaces in your VPC: ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeNetworkInterfaces, ec2:ModifyNetworkInterfaceAttribute, ec2:AttachNetworkInterface, ec2:DetachNetworkInterface, ec2:DescribeSubnets, ec2:DescribeVpcs, ec2:DescribeVpcEndpoints, ec2:DescribeSecurityGroups, ec2:DescribeRouteTables, ec2:DescribeAvailabilityZones, and ec2:CreateTags

   • KMS permissions for the key that protects the OAuth secret: kms:Decrypt, kms:DescribeKey, and kms:GenerateDataKey

   • Secrets Manager permission to write refreshed tokens back to the secret: secretsmanager:UpdateSecret, in addition to secretsmanager:PutSecretValue

3. Create the connection with the AWS CLI and attach it to your VPC. Use <catalog-uri> as the endpoint, and set the subnet and security group from the first change. Wait until the connection status is READY.

   aws glue create-connection --connection-input '{
     "Name": "tableflow-irc-conn",
     "ConnectionType": "ICEBERGRESTCATALOG",
     "ConnectionProperties": {
       "INSTANCE_URL": "<catalog-uri>",
       "ROLE_ARN": "<role-arn>",
       "CATALOG_CASING_FILTER": "LOWERCASE_ONLY"
     },
     "AuthenticationConfiguration": {
       "AuthenticationType": "OAUTH2",
       "OAuth2Properties": {
         "OAuth2GrantType": "CLIENT_CREDENTIALS",
         "TokenUrl": "<catalog-uri>/v1/oauth/tokens",
         "OAuth2ClientApplication": {
           "UserManagedClientApplicationClientId": "<tableflow-api-key>"},
         "TokenUrlParametersMap": {"scope": "catalog"}
       },
       "SecretArn": "<secret-arn>"
     },
     "PhysicalConnectionRequirements": {
       "SubnetId": "<private-subnet-id>",
       "SecurityGroupIdList": ["<glue-sg-id>"],
       "AvailabilityZone": "<subnet-az>"
     }
   }'
   
   aws glue get-connection --name tableflow-irc-conn \
     --query Connection.Status
   

4. Create the federated catalog on this connection. In the Lake Formation wizard from the Glue guide, select the connection you created under Connector object instead of entering a new one. Continue with the Glue guide to create the catalog and query your tables from Athena. Each Tableflow cluster appears as a database and each Tableflow-enabled topic appears as a table.

Snowflake Horizon Catalog federation over PrivateLink

This path requires Snowflake Business Critical edition or higher, and the Snowflake account must run on AWS.

Snowflake Horizon Catalog federates to the Tableflow catalog through a catalog-linked database. Follow Integrate Tableflow with Snowflake Horizon Catalog for the external volume, the IAM role, and the catalog-linked database. To reach the catalog over the ingress PrivateLink Gateway instead of the public endpoint, use Snowflake outbound private connectivity with the following changes.

1. Provision the Snowflake private endpoint. As ACCOUNTADMIN, provision an endpoint against the PrivateLink Service ID from Step 1 and the <catalog-hostname> from “Determine your catalog endpoint”. Note the endpoint ID that Snowflake reports.

   SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT(
     '<privatelink-service-id>', '<catalog-hostname>');
   SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO();
   

2. Create an access point for the Snowflake endpoint. Repeat Step 3 on the same gateway. Enter the endpoint ID from the previous step. You don’t create a VPC endpoint or a Route 53 hosted zone in your account for this path, because the endpoint is in the Snowflake VPC.

3. Create the catalog integration with the private API type. Set CATALOG_API_TYPE = PRIVATE in REST_CONFIG. Use <catalog-uri> as the catalog URI and <catalog-uri>/v1/oauth/tokens as the token URI. All other settings are the same as in the Snowflake Horizon Catalog guide.

   CREATE CATALOG INTEGRATION tableflow_horizon_int
     CATALOG_SOURCE = ICEBERG_REST
     TABLE_FORMAT = ICEBERG
     CATALOG_NAMESPACE = '<kafka-cluster-id>'
     REST_CONFIG = (
       CATALOG_URI = '<catalog-uri>',
       CATALOG_API_TYPE = PRIVATE,
       ACCESS_DELEGATION_MODE = EXTERNAL_VOLUME_CREDENTIALS
     )
     REST_AUTHENTICATION = (
       TYPE = OAUTH,
       OAUTH_TOKEN_URI = '<catalog-uri>/v1/oauth/tokens',
       OAUTH_CLIENT_ID = '<tableflow-api-key>',
       OAUTH_CLIENT_SECRET = '<tableflow-api-secret>',
       OAUTH_ALLOWED_SCOPES = ('catalog')
     )
     ENABLED = TRUE;
   

4. Create the catalog-linked database on this integration, as in the Snowflake Horizon Catalog guide.

Related content

• Use Private Networking with Tableflow

• Integrate with AWS Glue Catalog

• Integrate with Snowflake Horizon Catalog

• Use AWS PrivateLink for Serverless Products on Confluent Cloud

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.