Snowflake Sink Connector for Confluent Cloud with Azure Egress Access Point

This topic presents the steps for configuring the Snowflake Sink connector in Confluent Cloud with Azure Private Link and Egress Access Point.

Prerequisites

The following is a list of prerequisites for configuring the Snowflake Sink connector with an Egress Access Point:

• A Confluent Cloud Dedicated cluster was set up and is running within an Azure Private Link network.
• A source topic was created to sink data into the Snowflake database.
• Snowflake instance was created to sink data into and is running within the same region and cloud as the Confluent Cloud cluster.
• A database and a schema were created in Snowflake to sink data into.
• Snowflake imposes restrictions on which DNS hostnames can be used to connect. Be sure to use the hostnames mentioned in Azure Private Link and Snowflake.

Note

For added security, you can set up a Network rule within Snowflake to restrict incoming traffic to the specific Private Endpoint setup as part of Egress Access Points.

Step 1. Request the Private Link Service ID from Snowflake

Open a support case with Snowflake and request the Private Link Service ID associated with your Snowflake account. The ID should be in the following format:

/subscriptions/<subscription_ID>/resourceGroups/azure<region>-privatelink/providers/Microsoft.Network/privateLinkServices/sf-pvlinksvc-azure<region>

Note that you cannot use the Resource Alias for setting up an Egress Access Point.

Leave the support ticket open as it will be required in a later step for Snowflake to allowlist Confluent’s Private Link Endpoint Resource ID.

Step 2. Create an Egress Access Point

1. In the Confluent Cloud Console, go to Environment → Network, and select the associated Private link network you want to use.

2. In the Egress Access Points tab, click Create access point.

3. Specify the following, and click Save.

   • Name: The name for the Egress Access point.
   • Azure Resource ID: Your Snowflake Private Link Resource ID you received from Snowflake support from previous step.
   • Sub-resource name: Leave it blank as it is only applicable for the 1st party Azure services.
   

Step 3. Request Snowflake support to allow Confluent’s Private Link Endpoint Resource ID

1. When the created Access Point created in the previous step transitions from “Provisioning” to “Pending Accept”, request that Snowflake allowlist Confluent’s Resource ID using the earlier support ticket opened with Snowflake.
2. When Snowflake support has approved this request, the status of the Access Point with transition from “Pending Accept” to “Ready”.

Step 4. Create a DNS record

1. Obtain the required Domain by running the following within query within Snowflake.

   USE ROLE ACCOUNTADMIN;
   SELECT KEY, VALUE::VARCHAR HOST
   FROM TABLE(FLATTEN(INPUT=>PARSE_JSON(SYSTEM$GET_PRIVATELINK_CONFIG())));
   

2. Note down the HOST values for the privatelink-account-url and the regionless-privatelink-account-url KEY values. These are required based on how you want to connect:

   • Connect using Snowflake’s Account Locator (Legacy) URL (<account_name>.<region_id>.privatelink.snowflakecomputing.com)
   • Connect using Account Name URL (<org_name>-<account_name>.privatelink.snowflakecomputing.com)

3. In the Confluent Cloud Console, in the DNS tab, click Create record on the associated Access Point.

   

4. Specify the following:

   • Ensure that the correct Access Point is selected and put in the applicable Private Link Snowflake Domain.
   • Access point: Select the Access Point you created in Step #2.
   • Domain: Specify the Domain you retrieved in Snowflake, using one of the below formats.
     • Account Locator (Legacy): <account_name>.<region_id>.privatelink.snowflakecomputing.com
     • Account Name: <org_name>-<account_name>.privatelink.snowflakecomputing.com
   

5. Click Save to create the record.

Step 5. Create the Snowflake Sink connector

1. In the Confluent Cloud Console, when the DNS record status becomes “Ready”, go to your associated Dedicated cluster.

2. In the Connectors tab, click Snowflake Sink.

3. Select the source topic.

4. Specify the Kafka authentication mechanism.

5. Specify the authentication details for Snowflake.

   1. For Connection URL, specify the Snowflake’s private endpoint URL in one of the two possible formats:

      • Account Locator URL (Legacy): https://<account_name>.<region_id>.privatelink.snowflakecomputing.com
      • Account Name URL : https://<org_name>-<account_name>.privatelink.snowflakecomputing.com
      

6. Specify configuration details for the connector.

7. Specify sizing (number of tasks) for the connector.

8. Review and launch the connector.

When the connector is successfully launched, the connector status becomes “Running”.