Use Private Networking with Tableflow in Confluent Cloud

You can use Tableflow on Confluent Cloud clusters that are deployed in private networks, ensuring that no data is transmitted over the public internet.

Private networks on AWS

You can use Tableflow on Confluent Cloud clusters that are located in any of the following private network options.

The traffic between Tableflow and your S3 storage is transmitted securely through Gateway VPC Endpoints, ensuring that data remains within the AWS private network and does not traverse the public internet.

AWS Glue Catalog integration is enabled through Interface VPC Endpoints, ensuring that metadata communication occurs securely within private networks without exposure to the public internet.

Tableflow Iceberg REST Catalog access

To query Tableflow tables, or to federate them into an external catalog, from a network with no public internet path, use AWS PrivateLink or PNI. For PrivateLink, create an ingress PrivateLink Gateway and access point for your environment and Region. For PNI, see Access the REST Catalog over PNI. An ingress PrivateLink Gateway is a Confluent Cloud resource that lets clients in your VPC reach Confluent Cloud services over AWS PrivateLink. An access point registers your VPC endpoint on the gateway.

Query engines, such as Apache Spark, and catalogs that federate the Tableflow Apache Iceberg™ REST Catalog, such as AWS Glue, can reach the catalog over AWS PrivateLink. You configure the gateway per environment and Region, separately from the networking of the cluster.

For query engines and AWS Glue, private access uses the following components:

  • An ingress PrivateLink Gateway in Confluent Cloud, in the AWS Region of your VPC

  • An interface VPC endpoint in your VPC that connects to the gateway

  • An access point on the gateway that registers your VPC endpoint

  • A Route 53 private hosted zone that resolves the private catalog hostname to your VPC endpoint

For steps, see Access the Tableflow Iceberg REST Catalog over AWS PrivateLink.

Access the REST Catalog over PNI

If you use Private Network Interface (PNI) to connect to Enterprise or Freight clusters, query engines and AWS Glue can reach the catalog over the same PNI connection. You don’t need a separate ingress PrivateLink Gateway, VPC endpoint, or private hosted zone. After you set up a PNI gateway and access point in an environment, Confluent Cloud creates the Tableflow catalog endpoint in that environment automatically. You don’t configure a domain name.

To set up a PNI gateway and access point, follow the steps in Use Private Network Interface on Confluent Cloud.

The catalog endpoint is scoped to the access point and uses the following format:

tableflow-<access_point_id>.<region>.aws.accesspoint.glb.confluent.cloud

For <access_point_id>, use the ID of your access point with the hyphen removed. For example:

tableflow-apxyz789.us-west-2.aws.accesspoint.glb.confluent.cloud

Use the endpoint as the host in the catalog URI, as in the following example:

https://tableflow-apxyz789.us-west-2.aws.accesspoint.glb.confluent.cloud/iceberg/catalog/organizations/<org_id>/environments/<env_id>

The endpoint resolves through public DNS to the Elastic Network Interface (ENI) addresses in your VPC. For the port, security group, and DNS considerations that apply to all services over PNI, see Access services over PNI.

Limitations

  • The built-in Apache Iceberg™ REST Catalog is available privately only on AWS, through an ingress PrivateLink Gateway or PNI.

  • You can’t use Confluent-managed storage in Private Network-enabled clusters.

  • The catalog integrations, like Unity Catalog and Snowflake Open Catalog/Apache Polaris, are not supported via private networking.

  • Each ingress PrivateLink Gateway supports up to 10 VPC endpoints.

  • An ingress PrivateLink Gateway doesn’t support connections to a different cloud Region or to resources in a different environment.

For more information, see Tableflow Quick Start Using Your Storage and AWS Glue.

Private networks on Azure

You can use Tableflow on Confluent Cloud clusters that are located in Azure private networks, ensuring that no data is transmitted over the public internet. For more information, see Use Azure Private Networking with Tableflow.