Use Azure VNet Peering with Confluent Cloud¶

Azure Virtual Network (VNet) peering enables you to route traffic using private IPv4 addresses between a VNet and Confluent Cloud. Your VNet and Confluent Cloud can communicate with each other as if they are within the same network.

For more information about VNet peering with Azure, see Virtual Network Peering.

Managed connectors created in a VNet-peered cluster can access data sources and sinks hosted in all peered VNets, if the firewall rules allow connector traffic to and from the peered VNets.

Requirements and considerations¶

A Confluent Cloud network of the “VNet Peering” type and the “Microsoft Azure” provider.

If a network does not exist, see Create Confluent Cloud Network on Azure.
If the peered VNet’s address space contains any addresses outside of RFC 1918 (10/8, 172.16/12, or 192.168/16 prefixes) or RFC 6598 (100.64/10 prefix) CIDR ranges, the routes to these addresses will be rejected.
Transitive VNet peering is not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud, applications running in Network A will not be able to access Confluent Cloud.

Confluent Cloud does not support the following features that Azure provides to achieve transitive peering, namely:
- Azure Gateway Transit
- User Defined Routes (UDRs)
You can colocate multiple Confluent Cloud Dedicated clusters in the same Confluent Cloud network, but this is limited by the expected number and size of these clusters. The applicable limits are specified in Network Quotas.
Cross-region peering is not supported through the Confluent Cloud Console. Contact Confluent Support to see if your regions are supported and to request configuration.

Create a VNet peering connection¶

A peering connection needs to be created in order to access Confluent Cloud clusters and services in a Confluent Cloud network using VNet peering.

The high level workflow to create a VNet peering connection is:

Grant access to Azure Active Directory Tenant.

If you are using the Confluent Cloud Console to configure peering, this step can be skipped and performed in the Grant Access stage in the third step below, Create a VNet peering connection in Confluent Cloud.
Add a tag in Azure.

If you are using the Confluent Cloud Console to configure peering, this step can be skipped and performed in the Grant Access stage in the third step below, Create a VNet peering connection in Confluent Cloud.
Create a VNet peering connection in Confluent Cloud.

You need to gather the following information from the Microsoft Azure portal:

Azure Tenant ID: Your organization ID in Azure Active Directory.

You can find this value on the Azure Active Directory Tenant Overview page.
Azure Subscription ID: The identifier for your Azure subscription.

You can find this on the Overview section of your Virtual network.
Azure VNet resource group name: The identifier for the Azure resource group that the virtual network belongs to.

You can find this on the Overview section of your Virtual network.
Azure VNet Name: The name of your Azure virtual network.

You can find this on the Overview section of your Virtual network.

Grant access to Azure Active Directory Tenant¶

Grant access to your Azure Active Directory (AD) Tenant before you can create a peering connection.

Go to the following URL using your Azure AD Tenant ID (<tenant-id>) and click Accept to approve the peering connection:
```
https://login.microsoftonline.com/<tenant-id>/oauth2/authorize?client_id=f0955e3a-9013-4cf4-a1ea-21587621c9cc&response_type=code
```
The client_id in the URL is the Confluent Cloud application client ID in Azure.

Using an Azure command line tool, such as Azure Cloud Shell, run the following command to create a new role.

If you have more than one subscription ID, update the AssignableScopes accordingly.

az role definition create --output none --role-definition '{
   "Name": "Confluent Cloud Peering Creator",
   "Description": "Perform cross-tenant network peering.",
   "Actions": [
       "Microsoft.Network/virtualNetworks/read",
       "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read",
       "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
       "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete",
       "Microsoft.Network/virtualNetworks/peer/action"
   ],
   "AssignableScopes": [
       "subscriptions/<subscription-id>",
   ]
}'

Using an Azure command line tool, run the following command with your subscription ID (<subscription-id>), VNet resource group name (<resource-group-name>), and VNet name (<vnet-name>) to assign the role to the service principal:
```
az role assignment create `
  --role "Confluent Cloud Peering Creator" `
  --assignee "$(az ad sp list --filter "appId eq 'f0955e3a-9013-4cf4-a1ea-21587621c9cc'" `
  --output tsv --query '[0].id')"  `
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>"
```
Note

Starting with Azure CLI v. 2.37.0, the objectId property in the output JSON of a Graph object is replaced by id. If you use an earlier version, use objectId in the --output line.

Add a tag in Azure¶

In your Azure AD tenant, add a tag to the VNet you want to peer with Confluent Cloud network.

For the complete steps, see Apply tags with Azure portal.

In the Azure portal, navigate to the VNet you want to peer with Confluent Cloud network.
Click Tags.
Specify the tag key and the value:
- Name: ConfluentEnvIDs.
- Value: The Confluent Cloud environment ID where the Confluent Cloud networks you’d like your VNet to be peered with reside.
To peer your VNet with multiple Confluent Cloud networks that belong to different Confluent Cloud environments, specify a comma-separated list of the environment IDs. For example:
```
ConfluentEnvIDs: <CCloud-env-id-1>, <CCloud-env-id-2>
```
Click Apply.

Create a VNet peering connection in Confluent Cloud¶

Follow the steps to create an Azure VNet network peering connection.

You can have multiple VNet peering connections. For information about limits, see Kafka cluster quotas.

In the Network Management tab of the desired Confluent Cloud environment, click the For dedicated cluster tab.
Click the Confluent Cloud network to which you want to add the peering connection.
In the Ingress connections tab, click +VNet Peering.
Specify the following field values in the Configure VNet Peering phase.
- Azure Tenant ID: Your organization in Azure Active Directory.
- Azure Subscription ID: The Azure subscription ID.
- Azure VNet resource group name: The resource group ID that the VNet belongs to.
- Azure VNet Name: The name of your Azure Virtual network.
Click Add.
Grant access to your Azure AD Tenant in the Grant Access stage.
1. Go to the given URL and click Accept to approve the peering connection.
  
  The URL is pre-populated with your Tenant ID you provided in the previous Configure VNet Peering phase, and the client_id is the Confluent Cloud application client ID in Azure.
2. Using an Azure command line tool, such as Azure Cloud Shell, run the following command to create a new role.
  
  The command is pre-populated with the information you provided in the previous Configure VNet Peering phase, namely, your subscription ID (<subscription-id>).
  
  You might have to update the command shown on the page with the latest command syntax as shown below.
  
  If you have more than one subscription ID, update the AssignableScopes.
```
az role definition create --output none --role-definition '{
"Name": "Confluent Cloud Peering Creator",
"Description": "Perform cross-tenant network peering.",
"Actions": [
    "Microsoft.Network/virtualNetworks/read",
    "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read",
    "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
    "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete",
    "Microsoft.Network/virtualNetworks/peer/action"
],
"AssignableScopes": [
    "/subscriptions/<subscription-id>/",
]
}'
```
3. Run this command to assign the role to the service principal.
  
  The command is pre-populated with the information you provided in the previous Configure VNet Peering phase, namely, your subscription ID (<subscription-id>), VNet resource group name (<resource-group-name>), and VNet Name (<vnet-name>). = You might have to update the command shown on the page with the latest command syntax as shown below.
```
az role assignment create `
  --role "Confluent Cloud Peering Creator" `
  --assignee "$(az ad sp list --filter "appId eq 'f0955e3a-9013-4cf4-a1ea-21587621c9cc'" `
  --output tsv --query '[0].id')"  `
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>"
```
4. Add a tag in Azure.
5. Click Continue. You are prompted to confirm that Confluent Cloud is present in your Azure AD Tenant.
6. Click Create connection to finish creating the peering connection.

Grant access to Azure Active Directory Tenant.
Add a tag in Azure.

Create a VNet peering connection.

Now that you granted access to the Azure AD tenant, you can create the VNet peering connection using an HTTP request that resembles the following REST API example:

HTTP POST request

POST https://api.confluent.cloud/networking/v1/peerings

Authentication

See Authentication.

Request specification

{
   "spec":{
      "cloud":{
         "kind":"AzurePeering",
         "tenant":"<Azure AD Tenant ID in which you are peering with Confluent Cloud>",
         "vnet":"/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>",
         "customer_region":"<VNet region>"
      },
      "display_name":"<connection name>",
      "environment":{
         "id":"<Confluent Cloud environment id>"
      },
      "network":{
         "id":"<Confluent Cloud network id>"
      }
   }
}

Grant access to Azure Active Directory Tenant.
Add a tag in Azure.
Use the confluent network peering create Confluent CLI command to create a peering connection:
```
confluent network peering create azure-peering <flags>
```
The following command-specific flags are supported:
- --network: Required. Confluent Cloud network ID.
- --cloud: Required. The cloud provider. Set to azure.
- --cloud-account: Required. Azure Tenant ID in which your Azure Subscription exists and you are peering with Confluent Cloud.
- --virtual-network: Required. The value should be in the following pattern:
```
/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>
```
- --customer-region Cloud region ID of the Azure VNet that you are peering with Confluent Cloud network.
You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a VNet peering:
```
confluent network peering create azure-peering \
  --network n-123456 \
  --cloud azure \
  --cloud-account 1111tttt-1111-1111-1111-111111tttttt \
  --virtual-network /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Network/virtualNetworks/my-vnet \
  --customer-region centralus
```

As a prerequisite, Terraform Provider for Confluent should be installed and has access to a Confluent Cloud administrator account.

Confluent provides a Terraform configuration for creating a VNet peering connection. This configuration automates the manual steps described below.

Grant access to Azure Active Directory Tenant.
Add a tag in Azure.

Use the confluent_peering resource to create a peering connection.

To create a peering connection with Terraform Provider for Confluent use the following snippet of Terraform configuration:

# Configure the Confluent Provider
terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
    }
  }
}
provider "confluent" {
  cloud_api_key    = var.confluent_cloud_api_key    # optionally use CONFLUENT_CLOUD_API_KEY env var
  cloud_api_secret = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var
}
...
resource "confluent_peering" "azure" {
  display_name = "Azure Peering"
  azure {
    tenant          = "1111tttt-1111-1111-1111-111111tttttt"
    vnet            = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Network/virtualNetworks/my-vnet"
    customer_region = "centralus"
  }
  environment {
    id = confluent_environment.development.id
  }
  network {
    id = confluent_network.azure-peering.id
  }

  lifecycle {
    prevent_destroy = true
  }
}
# Create more resources ...

You must provide appropriate Confluent Cloud credentials to use the provider.

For the full confluent_peering resource reference, see the Confluent Terraform Provider documentation in the Terraform registry.

When you are finished, the peering status should display “Ready” in the Confluent Cloud Console.

Configure DNS forwarding¶

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VNet or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VNet.

The DNS Forwarder requires VNet peering where there is bi-directional network access between your network and Confluent Cloud clusters.

Step 1: Get DNS resolver IP addresses¶

To use the DNS forwarding feature with your Azure VNet, you can set up Azure Inbound Endpoints or use your own DNS server:

Create an inbound endpoint for a private DNS zone.

If you wish to forward DNS requests from Confluent Cloud to a private DNS zone, create Inbound Endpoints for Confluent Cloud network to access your DNS servers.

Azure recommends deploying multiple endpoints in different availability zones.

For details, see Configuring inbound endpoints.

Once the endpoints are created, input the IP addresses of the Inbound Endpoints to which to forward requests as described in the next step.
If you want to use your self-hosted DNS server, use the IP address of that DNS server in Confluent Cloud in the next step.

Step 2: Create a DNS Forwarder in Confluent Cloud¶

Set up DNS forwarding in Confluent Cloud.

In Confluent Cloud, navigate to the DNS Forwarding tab in the Network Detail page.
Input the following information:
- DNS server IPs: One or more IP addresses of you DNS servers to which we should forward DNS request.
- Domain list: One or more domains to which you wish to route the DNS requests.
Wait until provisioning is complete and DNS is propagated.

Send a request to create a DNS Forwarder resource:

REST request

POST https://api.confluent.cloud/networking/v1/dns-forwarders

REST request body

{
  "spec":
  {
    "display_name": "<The Custom name for the DNS Resolver>",
    "environment":
    {
      "id": "<The Environment ID where the DNS Resolver belongs to>"
    },
    "config":
    {
      "kind": "ForwardViaIp",
      "dns_server_ips": "<A list of IP address(es), up to 3, of DNS server(s) from your VNet>"
    },
    "domains": "<A list of domains, up to 10, for the DNS forwarder to use>",
    "gateway":
    {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

Use the confluent network dns forwarder create Confluent CLI command to set up a DNS forwarder:

confluent network dns forwarder create <dns-forwarder-name> <flags>

The following command-specific flags are supported:

--dns-server-ip: Required. A comma-separated list of IP addresses for the DNS server.
--gateway: Required. Gateway ID. To get the gateway id, run the following CLI command:
```
confluent network describe
```
--domains: A comma-separated list of domains for the DNS forwarder to use.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment and --output.

The following is an example Confluent CLI command to create a DNS forwarder:

confluent network dns forwarder create \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

The following is an example Confluent CLI command to create a named DNS forwarder:

confluent network dns forwarder create my-dns-forwarder \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

Use the confluent_dns_forwarder Confluent Terraform Provider resource to set up a DNS forwarder.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_forwarder" "main" {
  display_name = "dns_forwarder"
  environment {
    id = confluent_environment.development.id
  }
  domains = ["example.com", "domainname.com"]
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  forward_via_ip {
    dns_server_ips = ["10.200.0.0", "10.200.0.1"]
  }
}