Use VNet Peering Connections with Confluent Cloud on Azure

A VNet peering connection is a networking connection between your VNet and Confluent Cloud that enables you to route traffic using private IPv4 addresses. VNets can communicate with each other as if they are within the same network.

For more information about VNet peering with Azure, see Virtual Network Peering.

Managed connectors created in a VPC-peered cluster can access data sources and sinks hosted in all peered VPCs, if the firewall rules allow connector traffic to and from the peered VPCs.

Requirements and considerations

  • A Confluent Cloud network of type PEERING in Azure.

    If a network does not exist, see Confluent Cloud Network on Azure.

  • Transitive VNet peering is not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud, applications running in Network A will not be able to access Confluent Cloud.

    Confluent Cloud does not support the following features that Azure provides to achieve transitive peering, namely:

    • Azure Gateway Transit
    • User Defined Routes (UDRs)

  • You can colocate multiple Confluent Cloud Dedicated clusters in the same Confluent Cloud network, but this is limited by the expected number and size of these clusters. The applicable limits are specified in Network Quotas.

  • Cross-region peering is not supported through the Confluent Cloud Console. Contact Confluent Support to see if your regions are supported and to request configuration.

Create a VNet peering connection

Confluent provides a Terraform configuration for creating a VNet peering connection. This configuration automates the manual steps described below.

A peering connection has to be created from your VNet to the Confluent Cloud network in order to access Confluent Cloud clusters and services in a Confluent Cloud network.

Follow this procedure to create a VNet network peering connection for a Confluent Cloud cluster on Azure. You can have multiple VNet peering connections. For information about limits, see Kafka cluster quotas.

  1. In the Confluent Cloud Console, go to your Confluent Cloud network resource and click + VNet Peering.

  2. Enter Name, Azure tenant ID, Azure subscription ID, Azure VNet Resource Group Name, and Azure VNet Name.

    Azure Tenant ID

    Represents an organization in Azure Active Directory. You can find this value in the Azure Portal under Azure Active Directory.

    Azure Subscription ID

    The unique identifier for your Azure subscription. You can find this in the Azure Portal on the Overview section of your |az| Virtual Network.

    Azure VNet resource group name

    The identifier for the Azure resource group that the virtual network belongs to. You can find this in the Azure Portal on the Overview section of your Azure Virtual Network.

    Azure VNet Name

    The name of your Azure virtual network. You can find this in the Azure Portal on the Overview section of your Azure Virtual Network.

  3. Click Add. Your peering connection status will transition from “Pending” to “Error” in the Confluent Cloud Console. You must grant Confluent Cloud access to your Azure AD Tenant in the next step.

  4. Grant access to your Azure AD Tenant.

    1. Go to the following URL using your AD tenant ID (<tenant-id>) and approve:

      https://login.microsoftonline.com/<tenant-id>/oauth2/authorize?client_id=f0955e3a-9013-4cf4-a1ea-21587621c9cc&response_type=code

    2. Run the following command with your subscription ID (<subscription-id>) to create a new role.

      If you have more than one subscription ID, you must update the AssignableScopes.

      az role definition create --output none --role-definition "{
\"Name\": \"Confluent Cloud Peering Creator\",
\"Description\": \"Perform cross-tenant network peering.\",
\"Actions\": [
    \"Microsoft.Network/virtualNetworks/read\",
    \"Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read\",
    \"Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write\",
    \"Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete\",
    \"Microsoft.Network/virtualNetworks/peer/action\"
],
\"AssignableScopes\": [
    \"/subscriptions/<subscription-id>/\",
]
}"

    3. Run this command with your subscription ID (<subscription-id>), VNet resource group name (<resource-group-name>), and VNet Name (<vnet-name>) specified to assign the role to the service principal:

      az role assignment create \
  --role "Confluent Cloud Peering Creator" \
  --assignee "$(az ad sp list --filter "appId eq 'f0955e3a-9013-4cf4-a1ea-21587621c9cc'" \
  --output tsv --query '[0].id')"  \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>"

      Note

      Starting with Azure CLI v. 2.37.0, the objectId property in the output JSON of a Graph object is replaced by id. If you use an earlier version, use objectId in the --output line.

    4. Click Continue. You are prompted to confirm that Confluent Cloud is present in your Azure AD Tenant.

    5. Click Create connection to finish creating the peering connection.

When you are finished, the VPC peering status should display “Active” in the Confluent Cloud Console.

Configure DNS forwarding

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VPC or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VPC.

The DNS Forwarder requires VNet peering where there is bi-directional network access between your network and Confluent Cloud clusters.

Step 1: Get DNS resolver IP addresses

To use the DNS forwarding feature with your Azure VPC, you can set up Azure Inbound Endpoints or use your own DNS server:

Create an inbound endpoint for a private DNS zone.

  • If you wish to forward DNS requests from Confluent Cloud to a private DNS zone, create Inbound Endpoints for Confluent Cloud network to access your DNS servers.

    Azure recommends deploying multiple endpoints in different availability zones.

    For details, see Configuring inbound endpoints.

    Once the endpoints are created, as described in the next step, in Confluent Cloud, input the IP addresses of the Inbound Endpoints to which to forward requests.

  • If you want to use your self-hosted DNS server, use the IP address of that DNS server in Confluent Cloud in the next step.

Step 2: Create a DNS Forwarder in Confluent Cloud

Set up DNS forwarding in Confluent Cloud.

  1. In Confluent Cloud, navigate to the DNS Forwarding tab in the Network Detail page.
  2. Input the following information:
    • DNS server IPs: One or more IP addresses of you DNS servers to which we should forward DNS request.
    • Domain list: One or more domains to which you wish to route the DNS requests.
  3. Wait until provisioning is complete and DNS is propagated.