Private Linking from inside an Azure VM to multiple regional networks

We are using Private Linking - as described here - in multiple scenarios, globally distributed:

Regular deployments are usually handled by build agents in those isolated regional networks which (😀 of course only for emergency debugging) also can be used as jump boxes.

This can make some out-of-the-ordinary maintenance jobs, which are not covered with cross-regional deployment pipelines, a bit cumbersome as one practically has to connect to each of these jump boxes.

To make this easier, I created a script in PowerShell, which allows to connect to a type of resource (Key Vault in this example) in all those networks at once from a standalone Azure VM.

I considered virtual network peering from my VMs virtual network into the virtual networks of those regions but wanted to avoid to get blocked by potential IP address space overlaps. Hence I implemented linking those regional resources into my VMs virtual network and also maintaining private DNS records in a private DNS zone connected to this virtual network and with that not messing up the original environment.

In the following paragraphs I describe the most relevant parts of this script:

setting the stage

When working with Private Linking and Private DNS the private DNS zone name and the subresource or groupId has to be used according to this table:

$dnsZone = "privatelink.vaultcore.azure.net"
$groupId = "vault"

extracting VM metadata

To retrieve information about the Azure VM one is currently working on, the metadata information endpoint can be used:

$vmInfo = Invoke-RestMethod -Headers @{"Metadata" = "true" } -Method GET -NoProxy -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01"

$nicId = az vm show --subscription $vmInfo.compute.subscriptionId --resource-group $vmInfo.compute.resourceGroupName --name $vmInfo.compute.name --query networkProfile.networkInterfaces[0].id --output tsv

$subnetId = az network nic show --ids $nicId --query ipConfigurations[0].subnet.id --output tsv

$vnetInfo = $subnetId.split('/')[0..8]
$vnetId = [string]::Join("/", $vnetInfo)

This extracts the Ids of the virtual network and the subnet the current Azure VM is connected to.

vnetId is derived from subnetId by breaking down the segments as neither az network nic show nor az network vnet subnet show return a ready-to-use vnetId; not a nice approach working with those segments, but maybe someone can leave me a comment on how to improve readability here

Wait a minute - why mixing PowerShell and Azure CLI?

I decided to use Azure CLI instead of Azure PowerShell as CLI allows working with resources in subscriptions outside the current context by simply specifying the remote --subscription and --resource-group. With Azure PowerShell I would be needing to switch back and forth - from/to subscription which holds the VM and to/from subscription which holds the regional networks and resources.

the script assumes that Azure CLI session / context is set to the subscription holding the resources and that Azure VM is in another subscription

All the steps in this script can be easily achieved with Azure CLI and bash / zsh combined with jq.

private DNS zone

First we need a private DNS zone connected to the VMs virtual network which will receive the private IP address entries of the private link endpoints:

Write-Host "create/check private DNS zone" $dnsZone "for group" $groupId

if (!$(az network private-dns zone list -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId --query "[?name == '$dnsZone'].id" -o tsv)) {
    az network private-dns zone create -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId `
        -n $dnsZone
}

Write-Host "create private DNS link"

az network private-dns link vnet create -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId `
    -n $("vnet-" + $groupId + "-dns-link") `
    -z $dnsZone -v $vnetId -e false

be aware that sometimes a VM restart is required when switching a virtual network from one private DNS link to another

private link + endpoints

The script then iterates through a (here hypothetical) set of deployment instances and regions ...

• determines the regional Key Vault resourceId - again with a hypothetical / regional naming scheme
• creates names unique within the Azure VMs subscription + resource group for private link, private endpoint and DNS zone group
• creates a private endpoint and link from the Key Vault resource to VMs subnet (could also be another subnet in the VMs virtual network)
• creates a private DNS zone group which automatically maintains the private DNS zone record for the private endpoint within the private DNS zone connected to the VMs virtual network

Write-Host "private link all KeyVault instances"

foreach ($instance in "int", "qa", "stage", "prod") {
    foreach ($location in "eus", "weu", "sea") {
        $keyVaultName = "some-prefix-" + $instance + "-" + $location

        $keyVault = az keyvault show -n $keyVaultName -o json | ConvertFrom-Json

        $linkName = $keyVaultName + "-" + $vmInfo.compute.name + "-link"
        $endpointName = $keyVaultName + "-" + $vmInfo.compute.name + "-pep"
        $groupName = $keyVaultName + "-" + $vmInfo.compute.name + "-zonegroup"

        az network private-endpoint create --connection-name $linkName `
            --name $endpointName `
            -g $vmInfo.compute.resourceGroupName `
            --subscription $vmInfo.compute.subscriptionId `
            --private-connection-resource-id $keyVault.Id `
            --group-id $groupId `
            --subnet $subnetId

        az network private-endpoint dns-zone-group create `
            -g $vmInfo.compute.resourceGroupName `
            --subscription $vmInfo.compute.subscriptionId `
            --endpoint-name $endpointName `
            --name $groupName `
            --private-dns-zone $dnsZone `
            --zone-name $dnsZone            
    }
}

That's it. To connect to other types of resources "just" $dnsZone and $groupId need to be exchanged as well as the determination of the target resource's resourceId.

In our scenarios we also have globally distributed resources like

• CosmosDB
• API Management
• Container Registry

(compared to regional resources like Key Vault, Storage, SQL, Service Bus).

For those global resources it makes no sense to connect to all endpoints in all regions. Hence the script would need to be adapted to only connect to the main region.

housekeeping

At beginning of the script existing virtual network peerings (which in my case could potentially cause conflicts with private links), existing private endpoints and private DNS links are deleted from the VMs virtual network to give the script a clean slate to work on:

Write-Host "delete existing (VM's) virtual network links"

az network vnet peering list --vnet-name $vnetInfo[8] -g $vnetInfo[4] --subscription $vnetInfo[2] --output json | ConvertFrom-Json | % {
    az network vnet peering delete --ids $_.Id
}

Write-Host "delete existing (VM's) private endpoint links"

az network private-endpoint list -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId --output json | ConvertFrom-Json | % {
    az network private-endpoint delete --ids $_.Id
}

Write-Host "delete existing (VM's) private DNS links"

az network private-dns link vnet list -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId --zone-name $dnsZone -o json | ConvertFrom-Json | % {
    if ($_.virtualNetwork.id -eq $vnetId) {
        Write-Host "deleting link: $($_.id)"
        az network private-dns link vnet delete -g $vmInfo.compute.resourceGroupName --subscription $vmInfo.compute.subscriptionId --zone-name $dnsZone --name $_.name --yes
    }
}

This section of the script would only delete private endpoints and links created in the context of the VM. In case such endpoints and private DNS links exist in other subscriptions or resource groups, az network private-endpoint list + delete as well as az network private-dns link vnet list + delete need to be enhanced.