Use CSFLE with Azure Key Vault over Azure Private Link
By default, when Schema Registry uses an Azure Key Vault key for Client-Side Field Level Encryption (CSFLE), it reaches your vault over the public internet. This guide shows you how to configure Schema Registry to access your Azure Key Vault over a private network connection instead, by registering the key with Confluent Cloud as a self-managed encryption key.
Note
CSFLE with Azure Key Vault over Azure Private Link is a Limited Availability feature in Confluent Cloud.
Your Confluent account team must enable Private Link CSFLE for your Confluent Cloud organization before you can use this feature. If you want to participate in the Limited Availability Program, contact your Confluent account team and provide your Confluent Cloud organization ID.
To set up CSFLE with a private Azure Key Vault, complete these steps:
Create your Azure Key Vault and key. Create a vault with purge protection and RBAC authorization enabled, and then create an RSA key.
Register your key with Confluent and grant access. You tell Confluent Cloud which Azure Key Vault key to use, and Confluent Cloud returns a BYOK key ID as
cck-...and an Azure application ID. Then you grant that application the required roles on your vault. Confluent Cloud is authorized to use your key only after both sides approve. For more about BYOK on Azure, see Use Self-Managed Encryption Keys in Confluent Cloud on Azure.Set up a private connection to your Azure Key Vault. This connection routes Schema Registry traffic to your vault through your private network instead of the public internet. Without this connection, Confluent Cloud still reaches your vault over the public internet instead of the private network.
Create the Schema Registry encryption key and encrypt a field. Create a CSFLE key encryption key (KEK) that references your BYOK key so that Schema Registry encryption calls for that KEK go through your private connection.
Lock your vault to private-only access and verify. Disable public network access to your vault and confirm that Schema Registry still reaches your key.
Prerequisites
Before setting up your private link, ensure you have met these prerequisites:
Ensure your account team has enabled the CSFLE Private Link feature for your Confluent Cloud organization.
Ensure you have permission to create an Egress Private Link gateway and endpoint in the Confluent Cloud environment that contains your Schema Registry cluster, in the same Azure region as your Schema Registry cluster. You don’t need a Dedicated or Enterprise cluster.
Authenticate the Confluent CLI and Azure CLI.
Set these environment variables, ensuring your Azure Key Vault and Schema Registry region match:
# --- Azure --- export AZ_SUB="<your-azure-subscription-id>" export AZ_TENANT="<your-azure-tenant-id>" export RG="<your-resource-group>" export VAULT="<your-vault-name>" export KEY="csfle-key" export LOCATION="<your-region>" # --- Confluent --- export CC_ENV="<env-xxxxx>" # your Confluent environment
Create your Azure Key Vault and key
Schema Registry uses an RSA key stored in your Azure Key Vault to protect the data encryption keys for your encrypted fields. If you already have an Azure Key Vault with purge protection and RSA key with RBAC authorization enabled, skip these steps.
Log in and select your subscription:
az login --tenant "$AZ_TENANT" az account set --subscription "$AZ_SUB"
Create a vault with purge protection and RBAC authorization enabled:
az keyvault create --name "$VAULT" -g "$RG" -l "$LOCATION" \ --enable-purge-protection true \ --enable-rbac-authorization true
Grant yourself permission to create keys:
VAULT_ID="$(az keyvault show --name "$VAULT" --query id -o tsv)" az role assignment create --role "Key Vault Crypto Officer" \ --assignee-object-id "$(az ad signed-in-user show --query id -o tsv)" \ --assignee-principal-type User --scope "$VAULT_ID" # Wait approximately 1–2 minutes for the role to take effect.
Create an RSA key:
az keyvault key create --vault-name "$VAULT" --name "$KEY" --kty RSA --size 2048
Register your key with Confluent and grant access
Register your Azure Key Vault key with Confluent as a self-managed encryption key, and then grant the Confluent identity permission to use it.
In the Confluent Cloud Console, click the Administration menu icon in the upper-right corner, and then click Encryption keys.
Click Add key.
On the Cloud provider step, select Microsoft Azure, and then click Continue.
On the Key details step, enter these details:
Name (alias): Optional. A name that helps you identify the key, such as
csfle-key.Key vault tenant ID: Your Azure tenant ID.
Key vault resource ID: The resource ID of your vault, as
/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.KeyVault/vaults/<vault-name>.Key ID without version: The key identifier without the version suffix, as
https://<vault-name>.vault.azure.net/keys/<key-name>.
Click Register key.
On the Key configuration step, copy the Azure CLI commands that grant the Confluent application the Key Vault Crypto Service Encryption User and Key Vault Reader roles on your vault. Run these commands in your terminal.
Click Finish.
The key shows a status of Initializing while Confluent Cloud validates access, which can take up to five minutes.
On the Encryption keys page, confirm that the key is valid, and save its key ID, which is
cck-.... Use this ID to create the Schema Registry KEK:export CCK_ID="<cck-... from the Encryption keys page>"
Register the key as a self-managed encryption key:
confluent byok create "https://$VAULT.vault.azure.net/keys/$KEY" \ --tenant "$AZ_TENANT" \ --key-vault "/subscriptions/$AZ_SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$VAULT"
From the output, save the
IDvalue of your key, which iscck-..., and the application ID from theRolesfield. The application ID is the Confluent identity that accesses your key:export CCK_ID="<cck-... from output>" export APP_ID="<application id from Roles>"
Grant your Confluent identity access to your vault. To see the exact commands for your key, run
confluent byok describe "$CCK_ID". The commands look similar to the following:VAULT_ID="$(az keyvault show --name "$VAULT" --query id -o tsv)" SP_ID="$(az ad sp show --id "$APP_ID" --query id -o tsv 2>/dev/null || \ az ad sp create --id "$APP_ID" --query id -o tsv)" # Key Vault Crypto Service Encryption User az role assignment create --role "e147488a-f6f5-4113-8e2d-b22465e65bf6" \ --scope "$VAULT_ID" \ --assignee-object-id "$SP_ID" \ --assignee-principal-type ServicePrincipal # Key Vault Reader az role assignment create --role "21090545-7ca7-4776-b22c-e363652d74d2" \ --scope "$VAULT_ID" \ --assignee-object-id "$SP_ID" \ --assignee-principal-type ServicePrincipal
Confirm that the key is valid:
confluent byok describe "$CCK_ID"
In the output, confirm that
Validation PhaseisVALID.
Set up a private connection to your Azure Key Vault
Create an Egress Private Link Endpoint in Confluent Cloud that connects to your Azure Key Vault. This connection routes Schema Registry traffic to your vault through your private network instead of the public internet.
Schema Registry doesn’t run on a Kafka cluster, so you don’t need a Dedicated or Enterprise cluster for this connection. Instead, create the endpoint on an Egress Private Link gateway in the Confluent Cloud environment that contains your Schema Registry cluster, in the same Azure region as your Schema Registry cluster. Each environment supports one gateway per region. If a gateway already exists in that region, use it.
To create the connection, complete these steps:
Create an Egress Private Link gateway in your Schema Registry environment and region, if one doesn’t already exist. For details, see Create a gateway for outbound connectivity in Confluent Cloud.
Create an Egress Private Link Endpoint on the gateway. Use your vault’s resource ID as the target service resource ID. For details, see Create an Egress Private Link Endpoint in Confluent Cloud.
Note
When creating your access point in Cloud Console, select Other as the service.
In the Azure portal, approve the private endpoint connection on your vault.
Create the Schema Registry encryption key and encrypt a field
Create a Schema Registry KEK that uses your self-managed encryption key, and then add an encryption rule to your schema to encrypt the fields you want to protect. KEKs that reference a BYOK key must be shared with Confluent Cloud.
In the Confluent Cloud Console, select your environment, and then click Stream Governance > Schema Registry > Encryption keys.
Click Add encryption key.
In the Name field, enter
csfle-kek.In the Key management system provider field, select Azure, and then enter the key ID of your Azure Key Vault key, as
https://<vault-name>.vault.azure.net/keys/<key-name>.Enable Share key access with Confluent Cloud, and then select the self-managed encryption key that you registered in Register your key with Confluent and grant access.
Click Add.
Click Schemas, and then select the schema that contains the fields you want to protect.
Click Evolve > Rules > Add rules.
In the Add rules panel, enter the following information:
Category: Select Data encryption rule.
Rule name: Enter a name that’s unique within the subject, such as
encryptPII.Encrypt fields with: Select the tags for the fields to encrypt, such as
PII, and then selectcsfle-kekas the Encryption key.
Click Add. Save the schema.
For details about the rule options, see Add an encryption rule.
Create a KEK that references your encryption key. The
cckIdproperty links the KEK to your registered key:confluent schema-registry kek create csfle-kek \ --kms-type azure-kms \ --kms-key "https://$VAULT.vault.azure.net/keys/$KEY" \ --kms-properties "cckId=$CCK_ID" \ --shared \ --environment "$CC_ENV"
Reference
csfle-kekin the encryption rule when you register the schema by using the Schema Registry API. Because you already created the KEK, the rule needs only theencrypt.kek.nameparameter. For example, this rule encrypts every field taggedPII:{ "schema": "...", "metadata": {...}, "ruleSet": { "domainRules": [ { "name": "encryptPII", "kind": "TRANSFORM", "type": "ENCRYPT", "mode": "WRITEREAD", "tags": ["PII"], "params": { "encrypt.kek.name": "csfle-kek" }, "onFailure": "ERROR,NONE" } ] } }
For the full rule format and client configuration, see Configure client-side field level encryption.
Lock your vault to private-only and verify
Before you restrict public access to your vault, confirm that the private connection between Confluent Cloud and your Azure Key Vault works. For setup steps, see Set up a private connection to your Azure Key Vault.
Important
If you also use keys in this vault for self-managed encryption of Kafka clusters, disabling public network access can stop those clusters from reaching their keys. For Dedicated clusters with network restrictions, set your Azure Firewall and virtual networks settings to Allow trusted Microsoft services to bypass this firewall. For Enterprise clusters, set your Azure Firewall and virtual networks settings to Allow public access from all networks.
To avoid this, use a separate vault for CSFLE keys that Schema Registry accesses over Private Link. For details, see Use Self-Managed Encryption Keys in Confluent Cloud on Azure.
Warning
Confluent Cloud doesn’t block deletion of BYOK keys that are used for CSFLE over Private Link. If you delete a BYOK key, every KEK that references it immediately and permanently loses the ability to encrypt or decrypt data.
After you confirm the private connection, disable public network access to your vault:
az keyvault update --name "$VAULT" -g "$RG" --public-network-access Disabled
To verify that Schema Registry still reaches your key over the private connection, produce and consume a record that uses the encryption rule from Create the Schema Registry encryption key and encrypt a field.