Configure Azure Private Networking for USM Agents

This page describes how to set up Azure Private Link connectivity for the USM Agent to send Confluent Platform metadata to Confluent Cloud. You configure this connectivity by creating a gateway and an access point in your Confluent Cloud environment. For general Confluent Cloud PrivateLink setup that’s not specific to USM, see Use Azure Private Link for Serverless Products on Confluent Cloud.

Unified Stream Manager (USM) requires a secure, private network connection between your Confluent Platform cluster and Confluent Cloud that uses Azure Private Link. With Azure Private Link, your Confluent Cloud resources are only accessible from private endpoints in your VNet, so all metadata is transmitted over a private channel without exposure to the public internet.

You can create a new private network or use an existing one.

Tip

If your Confluent Cloud environment is on AWS instead of Azure, see Configure AWS Private Networking for USM Agents.

Private networking resources

Confluent Cloud uses the following private networking resources for registering Confluent Platform clusters. These resources are regional and do not map to specific availability zones.

Ingress PrivateLink Gateway: A reservation to establish a Private Link connection from your virtual network (VNet) to regional services in a Confluent Cloud environment.
Ingress PrivateLink Access Point: A registration of a private endpoint that’s allowed to connect to a Confluent Cloud environment. An access point belongs to a specific gateway.

You can create the gateway and access point by using the Confluent Cloud Console, Confluent REST API, Confluent CLI, or Terraform.

The process consists of the following steps:

In Confluent Cloud, create a gateway.
In Azure, create a private endpoint that connects to the gateway’s service.
In Confluent Cloud, create an access point for your endpoint.
Set up your DNS resolution.

The setup wizard in the Confluent Cloud guides you through this process.

If you have an existing Azure Private Link connection for your Confluent Cloud account and have configured a private network using these Azure Private Link instructions, you can select that existing network from the wizard and proceed to the next section.

Requirements and considerations

You can connect to several Confluent Cloud environments in the same region from a single VNet by creating a separate gateway for each environment.
Cross-region Azure access point resources are not supported.

Create an Ingress PrivateLink Gateway in Confluent Cloud

To establish a private network connection, first create a gateway in the Confluent Cloud wizard. This process generates the unique service name that you need to create the private endpoint in your Azure account.

Confluent Cloud Console

On the Select network configuration page of the wizard, click Add network configuration.
In the Configure gateway panel, enter the following information:
- Gateway name: A descriptive name for your network connection.
- Cloud provider: Select Microsoft Azure.
- Region: Select the Azure region that matches your VNet.
Click Continue.
The next panel displays the Private Link Service ID or Private Link Service Alias. Copy this value to use it in the next step in the Azure portal.
Keep this Confluent Cloud browser tab open. Confluent Cloud provisions the gateway, and its state changes to CREATED.

A gateway can have one of the following states:
- CREATED: You provisioned the gateway and can now add an access point.
- READY: You created the access point, and it can accept connections.
- EXPIRED: You didn’t provision a valid access point in the allotted time. Create a new gateway to continue.

Confluent REST API

Send a request to create a gateway resource:

REST request

POST https://api.confluent.cloud/networking/v1/gateways

REST request body

{
  "spec": {
    "display_name": "<A custom name for the gateway>",
    "config": {
      "kind": "AzureIngressPrivateLinkGatewaySpec",
      "region": "<Azure region of the gateway>"
    },
    "environment": {
      "id": "<The ID of the environment to add the gateway to>"
    }
  }
}

For more information, see Gateways in the Confluent REST API reference.

Confluent CLI

Use the following Confluent CLI command to create an Azure gateway:

confluent network gateway create <gateway-name> <flags>

The following command-specific flags are supported:

--cloud: Required. The cloud provider. Set to azure.
--region: Required. The Azure region of the gateway.
--type: Required. The type of gateway configuration. Set to ingress-privatelink.

For more information, see confluent network gateway create.

The following is an example Confluent CLI command to create a gateway:

confluent network gateway create my-ingress-gateway \
  --cloud azure \
  --region centralus \
  --type ingress-privatelink

Terraform

Use the confluent_gateway resource from the Confluent Terraform Provider to create a gateway.

An example snippet of Terraform configuration for a gateway:

resource "confluent_gateway" "azure_ingress" {
  display_name = "my-gateway"
  environment {
    id = "env-123abc"
  }
  azure_ingress_private_link_gateway {
    region = "centralus"
  }
}

Create a private endpoint in Azure

In a new browser tab, log in to your Azure portal. Use the Private Link Service ID or Private Link Service Alias that you copied in Create an Ingress PrivateLink Gateway in Confluent Cloud to create and configure the private endpoint.

Azure Portal

On the Private Endpoint page in the Azure portal, click + Create.
In the Basics pane, specify the following:
- Subscription: The subscription that contains the VNet you want to connect from.
- Resource group: The resource group for this VNet.
- Name: A name for the private endpoint.
- Network interface name: A network interface name.
- Region: The same region as the gateway you created in Create an Ingress PrivateLink Gateway in Confluent Cloud.
Click Next: Resource, and specify the following:
- Connection method: Select Connect to an Azure resource by resource ID or alias.
- Resource ID or alias: Paste in the Private Link Service ID or Private Link Service Alias from Create an Ingress PrivateLink Gateway in Confluent Cloud.
Click Next: Virtual Network, and specify the following:
- Virtual network: The VNet where the private endpoint is created.
- Subnet: The subnet where the private endpoint is created.
- Network policy for private endpoints: Select the organization-approved or mandated policy. The default is Disabled.
- Private IP configuration: Select Dynamically allocate IP address.
Click Next: DNS and accept the default values.
Click Next: Tags, and optionally, add tags.
Click Next: Review + create. Review the details and click Create.
Wait for the Azure deployment to complete, and then note the private endpoint resource ID. You use this value in the next step.

Azure CLI

az network private-endpoint create \
  --connection-name <connection name> \
  --name <endpoint name> \
  --private-connection-resource-id <resource ID or alias from the gateway> \
  --resource-group <resource group name> \
  --vnet-name <VNet name> \
  --subnet <subnet name> \
  --manual-request true

Specify both --vnet-name and --subnet, or pass the full subnet resource ID as --subnet. A bare subnet name is not sufficient for the Azure CLI to resolve which VNet it belongs to.

Include --manual-request true. Without it, the Azure CLI attempts to auto-approve the connection against the Private Link Service, which requires permissions on the Confluent’s subscription that you do not have, causing the command to fail. With this flag, the connection is created in a PENDING state. Creating the access point in the next step accepts the connection and transitions both the gateway and the access point to READY.

From the command output, note the private endpoint resource ID. Use this value in the next step.

Azure Terraform

Use the azurerm_private_endpoint Azure Terraform Provider resource to create a private endpoint in Azure.

Create an Ingress PrivateLink Access Point

Return to Confluent Cloud and create an access point resource. An access point represents a private endpoint in your VNet.

You don’t need to provide the Private Link Service name. Confluent checks which Private Link Service is associated with the gateway that has a pending private endpoint with the given ID.

Confluent Cloud Console

In the Access points tab, click Create access point.
1. Create a private endpoint in Azure using the Private Link Service ID or Alias shown in step 4 on the sliding panel.
2. In Private Endpoint ID, specify the Azure resource ID of the private endpoint you created in the previous step.
3. In the Access point name field, enter a name for the access point.
4. Click Create access point.
  
  After the private endpoint connection is accepted, the statuses for the gateway and the access point change to READY.

Confluent REST API

Send a request to create an access point resource:

REST request

POST https://api.confluent.cloud/networking/v1/access-points

REST request body

{
  "spec": {
    "display_name": "<A custom name for the access point>",
    "config": {
      "kind": "AzureIngressPrivateLinkEndpoint",
      "private_endpoint_resource_id": "<The Azure resource ID of your private endpoint>"
    },
    "environment": {
      "id": "<The ID of the environment that has the gateway for this access point>"
    },
    "gateway": {
      "id": "<The ID of the gateway to add the access point to>"
    }
  }
}

For more information, see Access Points in the Confluent REST API reference.

Confluent CLI

Use the following Confluent CLI command to create an access point:

confluent network access-point private-link ingress-endpoint create <access-point-name> <flags>

The following command-specific flags are supported:

--cloud: Required. The cloud provider. Set to azure.
--gateway: Required. The ID of the gateway to add the access point to.
--private-endpoint-resource-id: Required. The Azure resource ID of your private endpoint.

For more information, see confluent network access-point private-link ingress-endpoint create.

The following is an example Confluent CLI command to create an access point:

confluent network access-point private-link ingress-endpoint create my-ingress-access-point \
  --cloud azure \
  --gateway gw-123abc \
  --private-endpoint-resource-id /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Network/privateEndpoints/my-endpoint

Terraform

Use the confluent_access_point Confluent Terraform Provider resource to create an access point.

Set up DNS resolution

Confluent Cloud requires that you set up private DNS records for each access point pointing its DNS domain to the private endpoint you created.

For access points, when connecting to Confluent Cloud using access-point-specific hostnames, you must allow public DNS resolution from your network or VNet. Confluent Cloud advertises these hostnames in the public DNS resolver, and these hostnames then redirect to match the domains that you input to your private DNS resolver.

DNS resolution happens in two steps:

The Confluent Cloud Global DNS Resolver returns a CNAME for your hostnames, removing the glb subdomain and converting your access point ID to be a subdomain.

For example, with the given hostname:
```
api-<accessPointId>.<region>.azure.accesspoint.glb.confluent.cloud
```
The CNAME returned is:
```
api.<accessPointId>.<region>.azure.accesspoint.confluent.cloud
```
The CNAME then resolves to your private endpoint based on the private DNS configuration.

To use an Azure Private DNS Zone as your private DNS resolver:

In Confluent Cloud, verify that the status of the access point is READY.
In Confluent Cloud, open the newly created gateway to get the DNS domain value of Confluent Cloud.

Take the FrontDoor URL shown in the Confluent Cloud Console and remove .glb and the api- prefix. Use the resulting <accessPointId>.<region>.azure.accesspoint.confluent.cloud as the Name for your Azure Private DNS Zone.

For example, if the FrontDoor URL is api-ap123.centralus.azure.accesspoint.glb.confluent.cloud, the Private DNS Zone name is ap123.centralus.azure.accesspoint.confluent.cloud.
In the Private DNS Zones page in the Azure portal, create a Private DNS Zone with the following settings:
- Subscription and Resource group: The subscription and resource group that you used when you created the VNet.
- Name: Enter the Confluent Cloud DNS domain value that you copied in the previous step.
Click Review + create, and then click Create.
Create a DNS record for the Private DNS Zone you created in the previous step.

This record is regional DNS and is used for all the target Confluent Cloud resources in the region.
1. Go to the Private DNS Zone resource you created, and click + Record Set.
2. Specify the following values:
  - Name: *
    
    Enter * as the subdomain name.
    
    The record name consists of the subdomain and the DNS domain name. The DNS domain name is filled in with the Confluent Cloud DNS domain value you specified when you created the Private DNS Zone in the previous step.
    
    If you are creating DNS resolution for Schema Registry for a single VNet connecting to several Schema Registry clusters in the same region across different environments, enter the id of the Schema Registry, lsrc-xxxxx in the Name field to connect to a specific Schema Registry.
  - Type: Select A.
  - TTL and TTL unit: 1 Minute.
  - IP address: Enter the IP address of the private endpoint that you created in Create a private endpoint in Azure.
    
    You can find this value under the private endpoint’s associated Network interface in the Azure portal.
  Note
  
  In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.
3. Click OK.
Attach the Private DNS Zone to the VNets where clients or applications are present.
1. Go to the Private DNS Zone resource and click Virtual network links under Settings.
2. Click + Add.
3. Specify the required values and click OK to create a virtual network link.

What’s next

Configure a Service Account for the USM Agent