The Ops Community ⚙️: Kai Walter

Challenging n8n AI Agent with a personal productivity flow

Kai Walter — Sun, 12 Oct 2025 16:29:20 +0000

I am out, mostly in the mornings for a walk or run, and I just want to drop a thought or a task immediately. Sometimes even complete sections of an upcoming presentation. Or rushing between meetings, the same: Just drop a voice recording and have it turned into a task or just as a note into my email inbox.

That is my use case. Plain and simple.

For that I started using n8n a while back. A bit clumsy, but it worked. I had a flow running in the cloud, triggered by a new file in OneDrive, which downloaded the file, transcribed it using OpenAI Whisper API, then classified the intent using GPT-4.1-mini and based on that either created a task or sent an email to myself with the transcription.

In case that sounds familiar: Yes, I converted that flow with Dapr Agents already, decribed in this post - so skip that part.

How it works:

on my Android phone I use the paid version of Easy Voice Recorder Pro which allows to automatically upload into a predefined OneDrive folder (which is /Apps/Easy Voice Recorder Pro)
the recording is downloaded by the n8n flow on a trigger, when a file is created in that folder
before downloading, to be safe and not crash the transcription unnecessarily, the flow filters on audio/x-wav or audio/mpeg MIME types
additionally, the flow downloads a prompt text file from OneDrive which contains the instructions for classifying the intent in the transcription; I wanted to be on OneDrive, so I can modify it easily without having to touch the flow
then transcribe using OpenAI Whisper API
with the transcription and the prompt run through a model like GPT-4.1-MINI
that classification step also has access to a simple tool - referenced in the prompt: a list of relevant person and other entity names to make the transcription more precise
based on the intent resolved then either create a task (using a webhook, as I did not want to mess around in our corporate environment) or just send an email to my corporate-self with the plain transcription
as part of housekeeping, copy the file to an archive folder and delete the original

That worked pretty well. I especially liked the capability of n8n to copy runtime data of a particular execution into the editor, which makes mapping and debugging so much easier. I moved the cloud-based flow to one to my own machines, so I could run it basically for free (download it, import it from file, rewire cloud credentials).

Since the previous post I explored Dapr Agents some more and also did an exemplary implementation with Microsoft Agent Framework and .NET Aspire in this repository. Just to get a better understanding on the relevance of prompts, tools and the clean segregation between orchestration and agents or MCP servers.

During all that time I kept the n8n flow running as I (myself) was not able to muster comparable fulfillment reliability with the other implementations. Then a few days ago I watched a guy using an "AI Agent" node in n8n which exaclty looked like something to even more simplify my flow while adding more capabilities and flexibility.

Improving the flow with n8n AI Agent

In the previous flow I ran the transcription through a model to classify the intent. From that structured output I wired up a set of tasks to fulfill the intent. That worked, but was a bit clumsy and not very flexible. If I wanted to add more capabilities, I had to modify the flow and add more branches.

With the AI Agent node I can do all that in one single node. The node has access to the transcription, the prompt and a set of tools. Based on that it can decide what to do.

As most of the tools connected describe their capabilities to a certain proper degree, the prompt can be simplified. I just have to make sure that the agent understands what it can do and how to use the tools. The prompt I am using is this one:

From the voice recorded transcript between these dashed lines, determine the user's intent.

---

## {{ $json.text }}

---

Examples

- create a todo item
- synchronize tasks
- unknown

## PREPARATION

MUST: Determine current date and time to have a reference point for relative date and time expressions in the user's intent.

MUST: Determine relevant person names and only use those to spell names correctly if those are stated in the transcript. Do not add to intent or content.

## PROCESSING

RULE: For the intent to create a to-do item determine a title, try to determine a due date and time for the task and also a reminder date and time. When due dates or reminders are expressed relatively (e.g. next Monday) use available tools to convert into ISO datetime and assume a CET or CEST time zone. Omit reminder or due date in the tool call, if not determined. Do not pass empty strings.

MUST: Creating a todo item makes it mandatory to synchronize todo list. For that synchronization the id of the created todo item is required.

RULE: For the intent to synchronize to-do items or tasks retrieve all open to-do items and initiate a synchronization with the id of each to-do item.

RULE: Only if the intent bears more content than a simple command like "synchronize tasks", archive the transcript and archive the recording file.

RULE: If user's intent can be fulfilled with one of the tools and the input information for the tool is sufficient, do not ask for approval and just proceed and conclude the request.

## CONCLUSION

MUST: In any circumstance conclude by sending a summary email.

Some tweaks

for archiving the transcript I used the OneDrive upload file node and hence I had to adjust the tool description
same for archiving the recording file, where I used the OneDrive copy file node
I cannot (and shall not) connect n8n to my company resources (like to-do lists) directly, hence I use a webhook to trigger a protected synchronization flow to synchronize the tasks with my Outlook tasks
I used a simple code node to return the current date and time in ISO format, so the agent can use that to resolve relative date and time expressions
also I added a code node to provide a list of relevant person names, which the agent can use to spell names correctly in the transcription (yeah you get that with German names in an english recording)
for a start I did not make use of agent memory as each individual flow is rather ephemeral at the moment

Switching to Mistral API 🇪🇺

For the agent I switched from OpenAI to Mistral API, as I wanted to try that out. With mistral-medium-latest I was able to produce reliable results.

There is no Mistral node for speech to text in n8n yet, but I was able to use the HTTP request node to call the API. Nice thing in n8n is, that HTTP request node has a notion of the many credentials n8n supports, so I could use my existing Mistral credentials without the need to figure out authentication headers and so on.

Also here I observed, that the model is not so overly crucial to the result anymore. As with the other frameworks I tried, the prompt and the tools are much more relevant to get a good result.

Cherry on top

I particularly like the execution log of the AI Agent node. It shows the reasoning steps, the tool calls and the final response. That is super helpful to understand what is going on and why.

Conclusion

I like the simplicity of the new flow. It is much easier to understand and to modify. If I want to add more capabilities, I just have to add more tools and adjust the prompt. No need to modify the flow itself.

In terms of implementation effort and speed I easily outpace my other agent framework implementations, although there I heavily build on specification driven agentic coding. To be fair I have to say, that with those I am mainly targeting enterprise scenarios, where many more aspects have to be considered. Anyway for this personal use case n8n is a great fit.

Kudos to n8n !!!

Inject NixOS into an Azure VM with nixos-anywhere and Azure Container Intances

Kai Walter — Sat, 14 Dec 2024 13:20:15 +0000

Motivation

In a 2023 post I showed how to use Nix package manager in an Azure VM - in that case CBL Mariner / Azure Linux. As I've been intensifying using NixOS on my home systems and with that creating an extensive multi-host NixOS Flakes based configuration repository I wanted to get that native NixOS experience also over on my occasional cloud tinkering VMs.

Options

Custom Image

For me the most obvious approach would be to generate a custom image, upload it to the cloud provider and stamp up the VM with that image. I succeeded using Society for the Blind's nixos-azure-deploy repository with minor modifications. However that approach seemed too resource intensive for me and required Nix running on the initiating (source) system.

nixos-infect and nixos-anywhere

Early I was pulled towards nixos-anywhere, a set of scripts to install NixOS over SSH on an arbitrary target system having kexec support. When struggling I tried my luck with nixos-inject, another way to install NixOS from within an existing target system.

Basically flipping back and forth between the 2 I tried with an approach to deploy an Azure VM (and to cut out cloud platform side effects also AWS EC2 instance) - with Ubuntu or Azure/AWS Linux - and then initiate the infection already during cloud-init. Going down that rabbit hole for some time, trying to resolve issues around the right boot and disk configuration which made the target system not boot up again properly, I reverted back a bit and succeeded by injecting from an outside, a NixOS based source system with a command line like:

nix run github:nix-community/nixos-anywhere -- --flake .#az-nixos --generate-hardware-config nixos-facter ./facter.json root@$FQDN

nixos-anywhere with Azure Container Instances

Nice but not yet where I wanted to be. For boot-strapping I prefer to work with a very limit set of tools/dependencies, aiming for only having shell scripting and Azure CLI, cutting out NixOS or a Nix-installation on none Linux source systems. Already using ACI/Azure Container Instances for other temporary jobs - as temporary certificate authority or to handle an ACME challenge response I thought it to be a proper candidate to bring up a temporary NixOS source system. This post describes all components to achieve that kind of setup based on scripts in this repository.

As the name nixos-cloud-deploy may suggest, I want to keep this repository open for other cloud providers to be included. Out of necessity I might add AWS soon.

VM creation

Script create-azvm-nixos-anywhere.sh drives the whole VM creation process. All general parameters to control the process, can be overwritten by command line arguments

argument	command line argument(s)	purpose
VMNAME=az-nixos	-n --vm-name	sets the name of the VM
RESOURCEGROUPNAME=$VMNAME	-g --resource-group	controls the Azure resource group to create and use
VMUSERNAME=johndoe	-u --user-name	sets the user name (additional to root) to setup on the VM
LOCATION=uksouth	-l --location	controls the Azure region to be used
VMKEYNAME=azvm	--vm-key-name	controls the name of the SSH public key to be used on the VM
GITHUBSSHKEYNAME=github	--github-key-name	controls the name of the GitHub SSH keys to be used to pull the desired Nix configuration repository
SIZE=Standard_B4ms	-s --size	controls the Azure VM SKU
MODE=aci	-m --mode	controls the source system mode: `aci` using ACI, `nixos` assuming to use the local Nix(OS) configuration
IMAGE=Canonical:ubuntu-24_04-lts:server:latest	-i --image	controls the initial Azure VM image to be used on the target system to inject NixOS into; needs to support `kexec`
NIXCHANNEL=nixos-24.05	--nix-channel	controls the NixOS channel to be used for injection and installation

sensitive information / SSH keys

Keys are not passed but pulled into the script:

# obtain sensitive information
. ./common.sh
prepare_keystore
VMPUBKEY=$(get_public_key $VMKEYNAME)

To make adaptation easier, I centralized keystore access - in my case to 1Password CLI - in a shared script common.sh:

prepare_keystore () {
  op account get --account my &>/dev/null
  if [ $? -ne 0 ]; then
      eval $(op signin --account my)
  fi
}

get_private_key () {
  echo "$(op read "op://Private/$1/private key?ssh-format=openssh")"
}

get_public_key () {
  echo "$(op read "op://Private/$1/public key")"
}

So to adapt it for just using keys on the local file system those functions could (no warranties) like:

prepare_keystore () {
  # nothing to do
}

get_private_key () {
  cat ~/.ssh/$1
}

get_public_key () {
  cat ~/.ssh/$1.pub
}

From that these keys are injected either in the Nix configuration files to set directly over SSH on the target system. To keep it simple I did not trouble myself with adapting a secret handler like sops-nix.

Azure resource creation

Creation of VM is handled pretty straight forward with Azure CLI. I added an explicit Storage Account to be able to investigate boot diagnostics, in case the provisioning process failed.

Injecting NixOS

Before starting injection, the script waits for SSH endpoint to be available on the target VM and cleans up known_hosts from entries which might be left from prior attempts.

FQDN=`az vm show --show-details -n $VMNAME -g $RESOURCEGROUPNAME --query fqdns -o tsv | cut -d "," -f 1`

wait_for_ssh $FQDN
cleanup_knownhosts $FQDN

Again for re-used those 2 functions are defined in common.sh:

cleanup_knownhosts () {
  case "$OSTYPE" in
    darwin*|bsd*)
      sed_no_backup=( -i "''" )
      ;;
    *)
      sed_no_backup=( -i )
      ;;
  esac

  sed ${sed_no_backup[@]} "s/$1.*//" ~/.ssh/known_hosts
  sed ${sed_no_backup[@]} "/^$/d" ~/.ssh/known_hosts
  sed ${sed_no_backup[@]} "/# ^$/d" ~/.ssh/known_hosts
}

wait_for_ssh () {
  echo "Waiting for SSH to become available..."
  while ! nc -z $1 22; do
      sleep 5
  done
}

The $OSTYPE case handles the varying sed flavors on MacOS, BSD and Linux regaring in-place replacement.

Making root available for SSH

nixos-anywhere relies on having root SSH access to the target system. Default Azure VM provisioning generates authorized_keys which prevents root to be used for connecting. As a remedy the script copies over VM user's SSH key to root.

echo "configuring root for seamless SSH access"
ssh -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' $VMUSERNAME@$FQDN sudo cp /home/$VMUSERNAME/.ssh/authorized_keys /root/.ssh/

echo "test SSH with root"
ssh -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' root@$FQDN uname -a

Skipping this step would show an error like:

test SSH with root
Warning: Permanently added 'az-nixos.uksouth.cloudapp.azure.com' (ED25519) to the list of known hosts.
Please login as the user "johndoe" rather than the user "root".

initiating inject

For ACI based injection, script config-azvm-nixos-aci.sh is invoked, which is described below:

./config-azvm-nixos-aci.sh --vm-name $VMNAME \
    --resource-group $RESOURCEGROUPNAME \
    --user-name $VMUSERNAME \
    --location $LOCATION \
    --nix-channel $NIXCHANNEL \
    --vm-key-name $VMKEYNAME

For direct injection with Nix, nixos-anywhere is invoked directly:

TEMPNIX=$(mktemp -d)
trap 'rm -rf -- "$TEMPNIX"' EXIT
cp -r ./nix-config/* $TEMPNIX
sed -e "s|#PLACEHOLDER_PUBKEY|$VMPUBKEY|" \
    -e "s|#PLACEHOLDER_USERNAME|$VMUSERNAME|" \
    -e "s|#PLACEHOLDER_HOSTNAME|$VMNAME|" \
    ./nix-config/configuration.nix > $TEMPNIX/configuration.nix

nix run github:nix-community/nixos-anywhere -- --flake $TEMPNIX#az-nixos --generate-hardware-config nixos-facter $TEMPNIX/facter.json root@$FQDN

VM's SSH key and host/username are replaced in a copy of the configuration files which then will be used by nixos-anywhere.

concluding VM installation

When one of the 2 injection methods succeed, the Azure VM should be ready with NixOS installed and SSH-access available on the desired VM user. From that final steps to finalize the installation are executed:

set the NixOS channel to be used for the installation
transfer GitHub SSH keys to pull the repository with the desired NixOS configuration
transfer the VM's public key to a spot, where it can be picked up by my NixOS configuration definition later
configure GitHub SSH environment; dos2unix was required to bring the SSH key exported from 1Password CLI from CRLF into LF line endings
pull the configuration repository and switch into the final configuration

# finalize NixOS configuration
ssh-keyscan $FQDN >> ~/.ssh/known_hosts

echo "set Nix channel"
ssh $VMUSERNAME@$FQDN "sudo nix-channel --add https://nixos.org/channels/${NIXCHANNEL} nixos && sudo nix-channel --update"

echo "transfer VM and Git keys..."
ssh $VMUSERNAME@$FQDN "mkdir -p ~/.ssh"
get_private_key "$GITHUBSSHKEYNAME" | ssh $VMUSERNAME@$FQDN -T 'cat > ~/.ssh/github'
get_public_key "$GITHUBSSHKEYNAME" | ssh $VMUSERNAME@$FQDN -T 'cat > ~/.ssh/github.pub'
get_public_key "$VMKEYNAME" | ssh $VMUSERNAME@$FQDN -T 'cat > ~/.ssh/azvm.pub'

ssh $VMUSERNAME@$FQDN bash -c "'
chmod 700 ~/.ssh
chmod 644 ~/.ssh/*pub
chmod 600 ~/.ssh/github

dos2unix ~/.ssh/github

cat << EOF > ~/.ssh/config
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/github

EOF

chmod 644 ~/.ssh/config
ssh-keyscan -H github.com >> ~/.ssh/known_hosts
'"

echo "clone repos..."
ssh $VMUSERNAME@$FQDN -T "git clone -v git@github.com:johndoe/nix-config.git ~/nix-config"
ssh $VMUSERNAME@$FQDN -T "sudo nixos-rebuild switch --flake ~/nix-config#az-vm --impure"

Injection with ACI

Script config-azvm-nixos-anywhere.sh is called by the creation script above to bring up an Azure Container Instance with NixOS to drive the injection process. This script could be used standalone on an existing Azure VM.

argument	command line argument(s)	purpose
VMNAME=az-nixos	-n --vm-name	specifies the name of the VM
RESOURCEGROUPNAME=$VMNAME	-g --resource-group	specifies the Azure resource group to use
VMUSERNAME=johndoe	-u --user-name	specifies the user name
LOCATION=uksouth	-l --location	specifies he Azure region to be used
VMKEYNAME=azvm	--vm-key-name	specifies the name of the SSH public key to be used on the VM
SHARENAME=nixos-config	-s --share-name	specifies the Azure file share name to be used to hold configuration files
CONTAINERNAME=$VMNAME	-c --container-name	specifies the ACI container name to be used
NIXCHANNEL=nixos-24.05	--nix-channel	controls the NixOS channel to be used for injection and installation

handling sensitive information

Obtaining secrets and setting the configuration is done similar to the creation script. It might look redundant, but for certain cases I wanted this script to have its own lifecycle.

# obtain sensitive information
. ./common.sh
prepare_keystore
VMPUBKEY=$(get_public_key $VMKEYNAME)
VMPRIVKEY=$(get_private_key $VMKEYNAME | tr "[:cntrl:]" "|")

# parameters obtain sensitive information
TEMPNIX=$(mktemp -d)
trap 'rm -rf -- "$TEMPNIX"' EXIT
cp -r ./nix-config/* $TEMPNIX
sed -e "s|#PLACEHOLDER_PUBKEY|$VMPUBKEY|" \
  -e "s|#PLACEHOLDER_USERNAME|$VMUSERNAME|" \
  -e "s|#PLACEHOLDER_HOSTNAME|$VMNAME|" \
  ./nix-config/configuration.nix > $TEMPNIX/configuration.nix

Control characters in private key coming from 1Password needed to be replaced by a basic character |, so that this key is passed properly into ACI. A lot of time, sometimes hours goes into resolving such tiny issues. That might seem wasted energy for some, but for me, not being on any project or other pressure, this actually is fun and helps me recharge my batteries.

Uploading configuration files to Azure storage file share

All files, copied to the temporary configuration and then patched for the occasion, are uploaded to the file share:

STORAGENAME=$(az storage account list -g $RESOURCEGROUPNAME --query "[?kind=='StorageV2']|[0].name" -o tsv)

AZURE_STORAGE_KEY=`az storage account keys list -n $STORAGENAME -g $RESOURCEGROUPNAME --query "[0].value" -o tsv`
if [[ $(az storage share exists -n $SHARENAME --account-name $STORAGENAME --account-key $AZURE_STORAGE_KEY -o tsv) == "False" ]]; then
  az storage share create -n $SHARENAME --account-name $STORAGENAME --account-key $AZURE_STORAGE_KEY
fi

# upload Nix configuration files
for filename in $TEMPNIX/*; do
  echo "uploading ${filename}";
  az storage file upload -s $SHARENAME --account-name $STORAGENAME --account-key $AZURE_STORAGE_KEY \
    --source $filename
done

Running the container

Finally the ACI container is created with the file share mounted to /root/work and relevant parameters passed as secure-environment-variables.

Special considerations:

it turned out, that the process really needs 2GB memory - hence --memory 2
in order to keep the container active, the entrypoint process is sent into a loop with --command-line "tail -f /dev/null"
nixos-anywhere still needs some preparation in the container which is accommodated by the script aci-run.sh (descibed below)

az container create --name $CONTAINERNAME -g $RESOURCEGROUPNAME \
    --image nixpkgs/nix:$NIXCHANNEL \
    --os-type Linux --cpu 1 --memory 2 \
    --azure-file-volume-account-name $STORAGENAME \
    --azure-file-volume-account-key $AZURE_STORAGE_KEY \
    --azure-file-volume-share-name $SHARENAME \
    --azure-file-volume-mount-path "/root/work" \
    --secure-environment-variables NIX_PATH="nixpkgs=channel:$NIXCHANNEL" FQDN="$FQDN" VMKEY="$VMPRIVKEY" \
    --command-line "tail -f /dev/null"

az container exec --name $CONTAINERNAME -g $RESOURCEGROUPNAME --exec-command "sh /root/work/aci-run.sh"

az container stop --name $CONTAINERNAME -g $RESOURCEGROUPNAME
az container delete --name $CONTAINERNAME -g $RESOURCEGROUPNAME -y
az storage share delete -n $SHARENAME --account-name $STORAGENAME --account-key $AZURE_STORAGE_KEY

Process inside container

Script aci-run.sh prepares the container for nixos-anywhere:

configuring Nix to allow "new" Nix commands and flakes
copying configuration files from file share to a local folder as this folder needs to be initialized with Git to work properly
configuring Git
convert the basic character | passed in VM's private key to proper LF line endings

The last 3 steps for some may seem straightforward (the proper way to get Nix flakes working somewhere from scratch) or overdone. Again a lot of time went into getting this run smoothly.

#!/bin/sh

set -e

echo "configure Nix..."
mkdir -p /etc/nix
cat << EOF >/etc/nix/nix.conf
experimental-features = nix-command flakes
warn-dirty = false
EOF

echo "initialize Nix configuration files..."
mkdir -p /root/nix-config
cp -v /root/work/*nix /root/nix-config/

git config --global init.defaultBranch main
git config --global user.name "Your Name"
git config --global user.email "your_email@example.com"

cd /root/nix-config
git init
git add .
git commit -m "WIP"
nix flake show

echo "set SSH private key to VM..."
mkdir -p /root/.ssh
KEYFILE=/root/.ssh/vmkey
echo $VMKEY | tr "|" "\n" >$KEYFILE
chmod 0600 $KEYFILE

nix run github:nix-community/nixos-anywhere -- --flake /root/nix-config#az-nixos --generate-hardware-config nixos-facter /root/nix-config/facter.json -i $KEYFILE root@$FQDN

Nix configuration files

nix-config/configuration.nix is roughly only a copy of existing samples with only minor adjustments (e.g. adding dos2unix, git and vim).

nix-config/disk-config.nix also is a copy of a samples adjusted to fit the requirements for an Azure VM's disk layout as good as possible.

The brunt of "hardware" detection is handled by including Facter in the nixos-anywhere configuration process.

Result

So by just simple running the creation script ./create-azvm-nixos-anywhere.sh I get a VM configured with my own Nix Flake configuration, no VM image dangling somewhere.

SSHing on the VM and checking, gives me:

$ nix-info
system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.24.10, channels(root): "nixos-24.05", nixpkgs: /etc/nix/path/nixpkgs`

Automating Azure VM Ubuntu install without fancy tools

Kai Walter — Tue, 23 Jul 2024 04:43:16 +0000

Motivation

In a 2022 post I showed how to bring up a disposable CBL-Mariner VM using cloud-init and (mostly) the DNF package manager. As I explained in that post, it takes some fiddling around to find sources for various packages and also to mix installation methods. To achieve a more concise installation approach I tried mixing CBL Mariner with Nix package manager in a later 2023 post.

Since then I have been using NixOS on my tinkering computers (x86 & ARM64) at home because I liked this one-file-format-declarative-definition of machines. With some new cloud technology evaluations ahead, for which I usually bring up dedicated disposable VMs, I wanted to transfer some of my NixOS learnings and create a disposable NixOS Azure VM. As I did want to create a (lame, everbody does that) custom image I was focusing a while on some infection methods (use any installed system and then "infect" with NixOS) nixos-anywhere and nixos-infect. I did only succeed to a certain point but had to stop because time was running out. One thing I learned in the past 3 decades: pull back in time before you get stuck in a rabbit hole, contain your frustration, swallow your professional pride and move on. Maybe someone reading this already has figured out how to bring up NixOS on an Azure VM in this or another way.

Ambition

Moving on, I decided to go for the simplest solution in my eyes: Ubuntu. Why? When in the weeds of experimenting, in my experience, most on-the-spot tool installations are documented and work usually well with Ubuntu or rather Debian - basically avoiding yak shaving when trying to transfer provided installation methods to exactly your environment. After this mental simplification, to still make it interesting, I set this "bar" for me:

use basic tools like cloud-init and scripts - no Ansible, Chef, Puppet, ... - to start VM installation quickly without too many dependencies from my local machine (currently MacOS)
not to use persisted SSH keys - rather read directly with CLI from 1Password
make my regular working environment like NeoVim, TMUX, ZSH available on the VM
pre-install Node.js, Python, Rust, Docker with the scripts and methods which bring exactly the desired versions for my dev workloads

This all might not seem very exiting, however, I still had to explore and learn many things (coming from a more or less homogenous NixOS & Home Manager ecosystem) which I want to share here.

Structure

I will share 4 files I use to drive the installation and then pick out and comment on interesting sections within those files:

create.sh - driving the installation process
cloud-init.txt - basic installation of VM
check-creation.sh - connect to VM and check whether cloud-init installation step concluded
install-stages.sh - install all requirements in several stages which build up on each other

Have fun extracting whatever is interesting or useful for you!

scripts will contain names of primitives - username, SSH keys, repositories. Those are already renamed or obfuscated - so it makes no sense for anybody out there to spend energy in finding those objects out there in the wild.

create.sh

In general this script

reads SSH public key to be authorized on VM (file authorized_keys) from 1Password
creates a Resource Group and a Storage Account for Boot Diagnostics (which I used to debug NixOS infection progress and which I wanted to keep)
create a VM, 1TB OS disk, cloud-init.txt for initialization
sets Auto Shutdown to 22:00 UTC
removes probably existing SSH entries from known_hosts on my local machine and removes empty lines

#! /bin/sh

set -e

VMNAME=${1:-thevm}
USERNAME=${2:-theuser}
PUBKEYNAME=${3:-theVmSshKey}
LOCATION=${4:-uksouth}
STORAGENAME=`echo $VMNAME$RANDOM | tr -cd '[a-z0-9]'`

op account get --account my
if [ $? -ne 0 ]; then
    eval $(op signin --account my)
fi

PUBKEY=`op read "op://Private/$PUBKEYNAME/public key"`

az group create -n $VMNAME -l $LOCATION

az storage account create -n $STORAGENAME -g $VMNAME \
  --sku Standard_LRS \
  --kind StorageV2 \
  --allow-blob-public-access false

az vm create -n $VMNAME -g $VMNAME \
 --image "Canonical:ubuntu-24_04-lts:server:latest" \
 --public-ip-sku Standard \
 --public-ip-address-dns-name $VMNAME \
 --ssh-key-values "$PUBKEY" \
 --admin-username $USERNAME \
 --os-disk-size-gb 1024 \
 --boot-diagnostics-storage $STORAGENAME \
 --size Standard_DS2_v2 \
 --custom-data "$(cat ./cloud-init.txt)"

az vm auto-shutdown -n $VMNAME -g $VMNAME \
  --time "22:00"

case "$OSTYPE" in
  darwin*|bsd*)
    sed_no_backup=( -i "''" )
    ;;
  *)
    sed_no_backup=( -i )
    ;;
esac

sed ${sed_no_backup[@]} "s/$VMNAME.*//" ~/.ssh/known_hosts
sed ${sed_no_backup[@]} "/^$/d" ~/.ssh/known_hosts

Azure Storage Account Name

Reduce Storage Account name, derived from VM name to alphanumeric characters as other characters like - are not allowed. Add a random number to somewhat ensure that the Storage Account name is unique.

STORAGENAME=`echo $VMNAME$RANDOM | tr -cd '[a-z0-9]'`

Get SSH public key from 1Password

This section tests whether 1Password CLI is already signed in, if not does the signin and then reads the public key portion from the secret. myis the account (could be more than one) and Private is the vault's name.

op account get --account my
if [ $? -ne 0 ]; then
    eval $(op signin --account my)
fi

PUBKEY=`op read "op://Private/$PUBKEYNAME/public key"`

Clean up known_hosts

In case that VM name had been used before and was signed in to with SSH, these statements remove the previous entries and potential resulting empty lines.

sed swicthing is based on this StackOverflow answer

case "$OSTYPE" in
  darwin*|bsd*)
    sed_no_backup=( -i "''" )
    ;;
  *)
    sed_no_backup=( -i )
    ;;
esac

sed ${sed_no_backup[@]} "s/$VMNAME.*//" ~/.ssh/known_hosts
sed ${sed_no_backup[@]} "/^$/d" ~/.ssh/known_hosts

cloud-init.txt

This files defines

installation of a basic set of standard apt packages
installation scripts in various stages which are copied to user's home folder for later installation

#cloud-config
package_upgrade: true
apt_sources:
- source: "ppa:zhangsongcui3371/fastfetch"
packages:
- apt-transport-https
- ca-certificates
- curl
- wget
- less
- lsb-release
- gnupg
- build-essential
- python3
- zsh
- tmux
- jq
- xclip
- dos2unix
- fzf
- ripgrep
- fastfetch
write_files:
  - path: /tmp/install-stage1.sh
    content: |
      #!/usr/bin/env bash

      # Azure CLI
      curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

      # Rust
      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

      # NVM / Node part 1
      curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

      # TMUX TPM part 1
      git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm

      # Python
      sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10

      # ZSH oh-my-sh part 1
      sudo chsh -s $(which zsh) $USER
      sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

    permissions: '0755'
  - path: /tmp/install-stage2.sh
    content: |
      #!/usr/bin/env bash
      ssh -T git@github.com

      # clone script folders
      if ! [ -d ~/scripts ]; then git clone git@github.com:theuser/bash-scripts.git ~/scripts; fi
      if ! [ -d ~/.dotfiles.git ]; then git clone git@github.com:theuser/dotfiles.git ~/.dotfiles.git; fi

      # configurations
      [ -e ~/.zshrc ] && rm ~/.zshrc
      [ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.zshrc ~/.zshrc

      [ -e ~/.tmux.conf ] && rm ~/.tmux.conf
      [ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.tmux.conf ~/.tmux.conf

      [ -e ~/.configgit ] && rm ~/.configgit
      [ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.configgit ~/.configgit

      ([ ! -L ~/.config ] && [ -d ~/.dotfiles.git ]) && ln -s ~/.dotfiles.git/.config ~/.config

      # TMUX TPM part 2
      .tmux/plugins/tpm/scripts/install_plugins.sh

      # NeoVim
      [ -e ~/scripts/install-neovim.sh ] && ./scripts/install-neovim.sh

      # NVM / Node part 2
      source .nvm/nvm.sh
      nvm install --lts

    permissions: '0755'
  - path: /tmp/install-stage3.sh
    content: |
      #!/usr/bin/env bash
      ZSH=$HOME/.oh-my-zsh
      [ ! -d $ZSH/custom/plugins/zsh-autocomplete ] && git clone --depth 1 -- https://github.com/marlonrichert/zsh-autocomplete.git $ZSH/custom/plugins/zsh-autocomplete

    permissions: '0755'
runcmd:
- export USER=$(awk -v uid=1000 -F":" '{ if($3==uid){print $1} }' /etc/passwd)

- curl -fsSL https://test.docker.com -o test-docker.sh
- sh test-docker.sh
- rm test-docker.sh
- usermod -aG docker $USER

- mv /tmp/install-stage* /home/$USER/

Determine user name

This line extracts non-root user name with user id 1000 from passwd to reference later in variable $USER.

- export USER=$(awk -v uid=1000 -F":" '{ if($3==uid){print $1} }' /etc/passwd)

Install Docker Beta version

- curl -fsSL https://test.docker.com -o test-docker.sh
- sh test-docker.sh
- rm test-docker.sh
- usermod -aG docker $USER

Map configuration files to dotfiles folder

On my disposable VMs I only map selective files and folder from .dotfiles.git folder to user's home:

# configurations
[ -e ~/.zshrc ] && rm ~/.zshrc
[ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.zshrc ~/.zshrc

[ -e ~/.tmux.conf ] && rm ~/.tmux.conf
[ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.tmux.conf ~/.tmux.conf

[ -e ~/.configgit ] && rm ~/.configgit
[ -d ~/.dotfiles.git ] && ln -s ~/.dotfiles.git/.configgit ~/.configgit

([ ! -L ~/.config ] && [ -d ~/.dotfiles.git ]) && ln -s ~/.dotfiles.git/.config ~/.config

check-creation.sh

This is used to SSH into the newly created VM and wait for the cloud init process to finish (or fail):

#!/bin/sh

VMNAME=${1:-thevm}
USERNAME=${2:-theuser}
GITHUBSSHKEYNAME=${3:-theGitHubSshKey}
FQDN=`az vm show --show-details -n $VMNAME -g $VMNAME --query fqdns -o tsv | cut -d "," -f 1`
ssh $USERNAME@$FQDN sudo tail -f /var/log/cloud-init-output.log

Determine VM's FQDN

Azure CLI has an option --show-details which returns (among also the provisioning/running state) the VM's FQDNs as a comma separated list.

FQDN=`az vm show --show-details -n $VMNAME -g $VMNAME --query fqdns -o tsv | cut -d "," -f 1`

install-stages.sh

This script is called after create.sh and check-creation.sh which prepares SSH keys for GitHub and then runs the installation stages scripts:

#!/bin/sh

VMNAME=${1:-thevm}
USERNAME=${2:-theuser}
GITHUBSSHKEYNAME=${3:-theGitHubSshKey}
FQDN=`az vm show --show-details -n $VMNAME -g $VMNAME --query fqdns -o tsv | cut -d "," -f 1`

op account get --account my
if [ $? -ne 0 ]; then
    eval $(op signin --account my)
fi

op read "op://Private/$GITHUBSSHKEYNAME/private key?ssh-format=openssh" | ssh $USERNAME@$FQDN -T "cat > /home/$USERNAME/.ssh/github"
op read "op://Private/$GITHUBSSHKEYNAME/public key" | ssh $USERNAME@$FQDN -T "cat > /home/$USERNAME/.ssh/github.pub"

ssh $USERNAME@$FQDN bash -c "'
chmod 700 ~/.ssh
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/*pub
chmod 600 ~/.ssh/github

dos2unix ~/.ssh/github

cat << EOF > ~/.ssh/config
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/github

EOF

chmod 644 ~/.ssh/config
'"

echo "SSH config finished ... press key"
read -s -n 1

ssh -t $USERNAME@$FQDN ./install-stage1.sh

echo "Stage 1 finished ... press key"
read -s -n 1

ssh -t $USERNAME@$FQDN ./install-stage2.sh

echo "Stage 2 finished ... press key"
read -s -n 1

ssh -t $USERNAME@$FQDN ./install-stage3.sh

Retrieve and set GitHub SSH keys

These 2 statements extract private and public SSH keys and transfers those with SSH to the VM:

op read "op://Private/$GITHUBSSHKEYNAME/private key?ssh-format=openssh" | ssh $USERNAME@$FQDN -T "cat > /home/$USERNAME/.ssh/github"
op read "op://Private/$GITHUBSSHKEYNAME/public key" | ssh $USERNAME@$FQDN -T "cat > /home/$USERNAME/.ssh/github.pub"

On the VM line endings need to be converted from CR/LF to LF:

dos2unix ~/.ssh/github

Multiple line SSH commands

This one I had to figure out first and comes in handy when multiple lines have to be send over SSH to a remote machine:

ssh $USERNAME@$FQDN bash -c "'
...
cat << EOF > ~/.ssh/config
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/github

EOF
...
'"

Use SSH with terminal allocated

I ran into a problem ...

Host key verification failed.

... with this section when just SSHing with ssh -t $USERNAME@$FQDN ./install-stage2.sh

ssh -T git@github.com

# clone script folders
if ! [ -d ~/scripts ]; then git clone git@github.com:theuser/bash-scripts.git ~/scripts; fi
if ! [ -d ~/.dotfiles.git ]; then git clone git@github.com:theuser/dotfiles.git ~/.dotfiles.git; fi

I had to change to -t option:

ssh -t $USERNAME@$FQDN ./install-stage2.sh

install-neovim.sh

I noticed that when installing NeoVim with the various package managers (DNF, apt, AUR) different configuration postures are put on the systems. Hence I always install with this script to end up with a reproducable configuration.

#!/bin/bash

set -e

case $1 in
    nightly)  # Ok
        tag=tags/nightly
        ;;
    *)
        tag=latest
        ;;
esac

latest_nv_linux=$(curl -sL https://api.github.com/repos/neovim/neovim/releases/$tag | jq -r ".assets[].browser_download_url" | grep -E 'nvim-linux64.tar.gz$')
wget $latest_nv_linux -O ~/nvim-linux64.tar.gz
sudo tar xvf ~/nvim-linux64.tar.gz -C /usr/local/bin/
rm ~/nvim-linux64.tar.gz
mkdir -p ~/.local/bin

if [ ! -e ~/.local/bin/nvim ]; then
    sudo ln -s /usr/local/bin/nvim-linux64/bin/nvim ~/.local/bin/nvim
fi

Taking Spin for a spin on AKS

Kai Walter — Fri, 15 Mar 2024 19:41:42 +0000

TL;DR

In this post I want to illustrate, whether a WebAssembly (aka Wasm) framework like Spin can already be utilized on Kubernetes in coexistency with existing workloads using SpinKube.

Motivation

In recent posts - Comparing Azure Functions vs Dapr throughput on Azure Container Apps and How to tune Dapr bulk publish/subscribe for maximum throughput - I compared throughput of the same asynchronous message distribution scenario with various flavors of .NET deployments (Dapr, Functions) on Azure Container Apps.

Observing the space of Wasm on the back end now for a while, I was wondering whether the same scenario already could be achieved and which of the suggested benefits are in reach to be leveraged. Those benefits being faster scalability and higher density as compiling WASI-compatible code into a Wasm module potentially produces a smaller artifact then regular OCI containers (still containing fragments of an OS even in the smallest distroless variants).

We explored the capabitlies in a team - hence in this post from now on I use we and our.

Test Environment

As the Wasm ecosystem for our kind of usage is still ramping up and not all capabilities are present yet to "go all in" and migrate complete workloads, the question is, how to utilize Wasm on the back end in coexistence with existing workloads selectively.

In our environment many of the enterprise-ish workloads are hosted on Kubernetes. To get Kubernetes or rather containerd executing Wasm, a so called containerd Wasm Shim is required which itself utilizes RunWasi.

Programming Languages

To be more flexible regarding the programming languages available for such a test scenario and to have a good-enough developer-inner-loop experience, we turned to Spin. Of the languages available we used Rust to get a feeling for maximum scaling & density capabitlies and TypeScipt/Node.js to see behavior with a more runtime-heavy environment - as our prefered choice .NET in spring 2024 was not yet supporting a setup like that good enough.

Cloud Resources

Currently Spin does not feature all triggers and APIs required to get to a comparable environment as in my previous posts, so we leveraged Dapr to connect the Spin application with cloud resources using HTTP trigger and Outbound HTTP API.

Dapr Shared

When deploying Spin runtime with regular Kubernetes primitives like deployment/service/..., Dapr obviously can be leveraged in its default sidecar mode. Ultimate density with Spin is reached by getting some of this primitives out of the way - hence SpinKube with its own SpinOperator. As SpinOperator does not support injecting side-cars yet, Dapr Shared is used, which provides Dapr being hosted in a Kubernetes daemonset or deployment. That additionally enables a higher cardinality of Dapr vs (Spin) App: sidecar is a 1:1 relation while with Dapr Shared a 1:n relation can be achieved - contributing further to the higher density capabitlies that Wasm itself offers.

Test Flow

The flow corresponds to the other 2 posts with minor adjustments:

local TypeScript application generates test data set e.g. with 10k orders/messages and places it in a blob storage (using a local Dapr output binding) - this is done to always feed the same shape of data into the test flow and hence provide a similar processing stream regardless of tech stack being tested
for one test run local TypeScript application loads this test data set from blob storage, splits into single messages and schedules these messages for a specified time in to the future to get activated all at once basically simulating, that a bulk of orders where placed on your doorstep to be processed at once
distributor endpoint of sample service receives incoming messages over a Dapr input binding - not pub/sub, not bulk to cause as much of erratic traffic as possible - and then decides whether to put the messages on a queue for virtual express or standard orders again using a Dapr outbound binding
receiver endpoint for either a express or standard receives incoming message and places it in a blob object for each message
the throughput is measured from time of schedule until all generated messages are written in blob storage

Compared to arbitrarily hammering on a HTTP endpoint to evaluate performance, latency, ... this approach is intended to observe the behavior of the tech stack used within a certain environment, checking whether it is sensitive to any outside influences or side-effects or itself inflicts such for other components.

Observations

Detailed setup and code can be found in this repository and shall be explained in further depth in coming posts. For now I want to share some high level observations.

comparing Spin-Rust with Warp-Rust throughput is noticable better; even when operating Warp-Rust with classic containers on Azure DS2_v2 VM SKU nodes while running Spin-Rust on ~60% cheaper D2pds_v5 (also leveraging on the strength of Wasm that it can be moved to nodes with a different architecture without the need to explicitly building and pushing a dedicated set of platform specific OCI containers)
when building Warp-Rust into a stripped down static container comparable results can be achieved which suggests, that packing size impacts scaling capabilities
comparing Spin-TypeScript with Express-TypeScript throughput is matched or only slightly better suggesting that runtime heavy languages languages/stacks like JavaScript, .NET, Java, ... cannot yet fully exploit Wasm potentials (as of spring 2024); but again, operating Express-TypeScript on Azure DS2_v2 VM SKU nodes while running Spin-TypeScript on ~60% cheaper D2pds_v5 which is a cost benefit in itself
.NET 7 with Spin SDK is not yet a serious contender - let's check again with .NET 8 or 9

Disclaimers

as the goal has been to measure scaling behavior of a Spin App or its corresponding contender in isolation, we did not dynamically scale Dapr Shared replicas, setting them on a fixed replica count to handle as much incoming messages as possible
results could possibly change if we would enhance our relatively plain AKS setup with improved networking and load balancing e.g. by using Cillium
using KEDA HTTP scaling potentially also could yield different results due to a closer provisioned vs required replicas ratio
comparing workloads Express vs Spin or Warp vs Spin is conducted on the same cluster, with a dedicated classic or wasm node pool to show coexistency capabilities and to ensure a clean baseline

Conclusion

The observations stated here shall be considered preliminary. There is still a lot of movement in that space. Adding further capabilities into Wasm modules like observability or improving runtime compatibility may result in absorbing some of the density or performance advantages Wasm modules currently here show to have. Gains like cutting down on multi-platform builds and image retention already speak for Wasm on the back end. Tooling (although Spin and its Fermyon Cloud ecosystem is a fantastic closed loop offering for Wasm modules), developer inner and outer loop definitely need to mature before Wasm can be fully exploited for enterprise workloads and compete to the current main stream stacks.

Mentions

the Fermyon guys Radu, Justin, Danielle, Rajat, Mikkel, Matt for entertaining and supporting us with our thoughts and setup
Ralph from Microsoft for providing context and calibration
salaboy for helping to get Dapr Shared into a state that it can be utilized for our use case

How to tune Dapr bulk publish/subscribe for maximum throughput

Kai Walter — Wed, 14 Feb 2024 13:25:35 +0000

TL;DR

In this post I show

how to convert a .NET C# program processing messages from Azure Service Bus with Dapr input bindings to Dapr bulk pubsub
how to activate and remove Dapr components even with an incremental ARM/Bicep deployment
how to use Dapr multi run for testing locally
what to consider and where to measure to achieve desired throughput goals

Motivation

real motivation: In my day job as an architect I am often too far away from coding artifacts that really go into production (and would deliver value and sustain there). Folks just keep me away from production code for a good reason : my prime days for such kind of work are over, as practices evolved too far for me to keep up with just coding from time to time.
However, to unwind, I grant myself the occassional excursion, try out and combine stuff that is hard to do in product or project context's with ambitioned deadlines and value-delivery expectations.

This post builds up or is a sequel to my previous post on comparing Azure Functions vs Dapr throughput on Azure Container Apps. I detected back then, that processing a given workload of 10k messages with an ASP.NET application backed with Dapr seems to perform faster than doing that with a Functions container or the Functions on ACA footprint.

Some of the underpinnings of that offering are going to get improved, on which I will make a post when released. These changes however already show a significant improvement in processing of the Functions stacks, with now the Dapr variant falling behind:

timestamp	stack	cycle time in seconds
2024-02-05T17:09:54	FUNC	30
2024-02-05T17:19:17	ACAF	36
2024-02-05T17:28:49	DAPR	46
2024-02-05T17:38:35	FUNC	32
2024-02-06T05:50:08	ACAF	47
2024-02-06T06:11:08	DAPR	53
2024-02-06T06:32:11	FUNC	31
2024-02-06T06:52:40	ACAF	34
2024-02-06T07:15:33	DAPR	46
2024-02-06T07:37:28	FUNC	38
2024-02-08T13:38:10	ACAF	50
2024-02-08T13:45:23	DAPR	63
2024-02-08T13:52:40	FUNC	36
2024-02-08T13:59:45	ACAF	37
2024-02-08T14:06:37	DAPR	43
2024-02-08T14:13:58	FUNC	35
2024-02-08T14:21:01	ACAF	36
2024-02-08T14:27:43	DAPR	47
2024-02-08T14:35:01	FUNC	37
2024-02-08T14:42:00	ACAF	38
2024-02-08T14:48:43	DAPR	46

average / standard deviation:	ACAF = Functions on ACA	39.7 / 6.2
	DAPR = Dapr in Container on ACA	49.1 / 6.8
	FUNC = Function in Container on ACA	34.1 / 3.1

measured with Azure Container Apps Consumption workload profile, not a dedicated workload profile!

to achieve a more realistic measurement, different to the original post, not the timestamps (in telemetry, Application Insights) between the first and last request processed are measured but the time from when the 10k messages are scheduled until all messages have been stored into blobs

Back then I did not max out the capabilities of Dapr's message processing capabilities, as the point we had to prove internally - "What is faster: Dapr or Functions?" - already showed the desired results. Also the Dapr implementation with Azure Service Bus Dapr input bindings reflected the majority of our implementations.

This post is about what I did, to squeeze more performance out of the Dapr implementation. The goal would be also have a cycle time of 30-35 seconds for the Dapr implementation.

Basic Assumption

In the Dapr implementation so far, for each of the 10k messages an input binding endpoint is frantically hit by the Dapr sidecar which has the application to invoke respective output binding endpoints in the same frequency:

app.MapPost($"/q-order-ingress-{testCase}-input", async (
    [FromBody] Order order,
    [FromServices] DaprClient daprClient
    ) =>
{
    switch (order.Delivery)
    {
        case Delivery.Express:
            await daprClient.InvokeBindingAsync($"q-order-express-{testCase}-output", "create", order);
            break;
        case Delivery.Standard:
            await daprClient.InvokeBindingAsync($"q-order-standard-{testCase}-output", "create", order);
            break;
    }

    return Results.Ok(order);
});

While exploring other hosting and compute options like Dapr, WebAssembly and Spin I realized, that this kind of implementation causes a lot of traffic in the Dapr to application communication which can easily stand in the way of faster compute processing.

Dapr still has an option available (currently in alpha status) to reduce this kind of noice by reducing amount of invocations exchanged between Dapr and the App : Publish and subscribe to bulk messages.

Conversion

all code snippets shown hereinafter, can be followed along and found in dapr-pubsub branch of the repository I already used in the previous post

Convert To Single PubSub

I try to take small steps forward when converting or transforming implementations - just to to be able keep track on cause and effect.

Conversion required in .NET code to get from input bindings to pubsub are minor. I still keep Azure Service Bus queues as messaging resource and will not switch to topics. This is a preference on my end as I prefer the dedicated dead letter handling of queues on single edges on a given messaging path than over dead letter handling with topics.

When using Azure Service Bus queues with Dapr pubsub, order-pubsub in PublishEventAsync below refers to a single Dapr component in ACA (Azure Container Apps environment) - compared to indvidual components required for input and output bindings - while q-order-... determines that queue itself.

Also at his point I do not burden my self with a conversion to CloudEvents and keep message processing "raw" : { "rawPayload", "true"} when publishing and avoiding app.UseCloudEvents(); on application startup. I noticed that, when combining bulk pubsub with CloudEvents, the message payload is suddenly rendered in JSON camelCase (!) instead of the expected PascalCase and that is definetely a worry for another day.

app.MapPost($"/q-order-ingress-{testCase}-pubsub-single", async (
    [FromBody] Order order,
    [FromServices] DaprClient daprClient
    ) =>
{
    var metadata = new Dictionary<string, string>
        {
          { "rawPayload", "true"}
        };

    switch (order.Delivery)
    {
        case Delivery.Express:
            await daprClient.PublishEventAsync("order-pubsub", $"q-order-express-{testCase}", order, metadata);
            break;
        case Delivery.Standard:
            await daprClient.PublishEventAsync("order-pubsub", $"q-order-standard-{testCase}", order, metadata);
            break;
    }

    return Results.Ok(order);
});

As I want to manage pubsub subscriptions programmatically, an endpoint to configure these subscriptions is required:

app.MapGet("/dapr/subscribe", () => Results.Ok(new[]{
new {
    pubsubname = "order-pubsub",
    topic = $"q-order-ingress-{testCase}",
    route = $"/q-order-ingress-{testCase}-pubsub-single"
}}));

this is the daprdistributor implementation; daprreceiver implementation is adapted accordingly receiving messages from pubsub instead from input binding

Dapr Component Switching

To be able to switch the environment between both bindings and pubsub, as the general flow is controlled by Dapr components not by the application, I need to have a Bicep parameter like ...

@description('determines whether bindings or pubsub is deployed for the experiment')
@allowed([
  'bindings'
  'pubsub'
])
param daprComponentsModel string

... which then controls relevance of Dapr components through scoping for the individual applications:

// switch valid application ids for the respective deployment model
var scopesBindings = daprComponentsModel == 'bindings' ? {
  distributor: 'daprdistributor'
  recvexp: 'daprrecvexp'
  recvstd: 'daprrecvstd'
} : {
  distributor: 'skip'
  recvexp: 'skip'
  recvstd: 'skip'
}

var scopesPubSub = daprComponentsModel == 'pubsub' ? [
  'daprdistributor'
  'daprrecvexp'
  'daprrecvstd'
] : [
  'skip'
]

These scopes are then linked to Dapr components relevant for the desired model.

  resource pubSubComponent 'daprComponents' = {
    name: 'order-pubsub'
    properties: {
      componentType: 'pubsub.azure.servicebus.queues'
      version: 'v1'
      secrets: [
        {
          name: 'sb-root-connectionstring'
          value: '${listKeys('${sb.id}/AuthorizationRules/RootManageSharedAccessKey', sb.apiVersion).primaryConnectionString};EntityPath=orders'
        }
      ]
      metadata: [
        {
          name: 'connectionString'
          secretRef: 'sb-root-connectionstring'
        }
        {
          name: 'maxActiveMessages'
          value: '1000'
        }
        {
          name: 'maxConcurrentHandlers'
          value: '8'
        }
      ]
      scopes: scopesPubSub
    }
  }

Although this way of handling switching Dapr components in a ARM/Bicep incremental deployment context is not production grade and requires a post-deployment step like ...

az containerapp env dapr-component list -g $RESOURCE_GROUP_NAME -n $ACAENV_NAME \
  --query "[?properties.scopes[0]=='skip'].name" -o tsv | \
  xargs -n1 az containerapp env dapr-component remove -g $RESOURCE_GROUP_NAME -n $ACAENV_NAME --dapr-component-name

... to clean up non-used components, it is OK-ish for such an evaluation environment at the moment.

In the end I then just need to modify the .env file in the repositories root, to switch the desired model:

AZURE_LOCATION=eastus
AZURE_ENV_NAME=my-env
DAPR_COMPONENTS_MODEL=bindings
DAPR_PUBSUB_MODEL=single

Local Testing

With all those small changes and re-tests ahead I invested some more time in local testability of the whole environment. For that I created a Dapr Multi-App Run file which allows to run all applications in concert:

version: 1
apps:
  - appID: distributor
    appDirPath: ./src/daprdistributor/
    resourcesPath: ./components-pubsub
    configFilePath: ./components-pubsub/config.yaml
    appProtocol: http
    appPort: 3001
    daprHTTPPort: 3501
    appHealthCheckPath: "/health"
    placementHostAddress: ""
    logLevel: "debug"
    command: ["dotnet", "run"]
    env:
      ASPNETCORE_URLS: http://localhost:3001
      ASPNETCORE_ENVIRONMENT: Development
      TESTCASE: dapr
  - appID: receiver-express
    appDirPath: ./src/daprreceiver/
    resourcesPath: ./components-pubsub-express
    configFilePath: ./components-pubsub-express/config.yaml
    appProtocol: http
    appPort: 3002
    daprHTTPPort: 3502
    appHealthCheckPath: "/health"
    placementHostAddress: ""
    logLevel: "debug"
    command: ["dotnet", "run"]
    env:
      ASPNETCORE_URLS: http://localhost:3002
      ASPNETCORE_ENVIRONMENT: Development
      TESTCASE: dapr
      INSTANCE: express
  - appID: receiver-standard
    appDirPath: ./src/daprreceiver/
    resourcesPath: ./components-pubsub-standard
    configFilePath: ./components-pubsub-standard/config.yaml
    appProtocol: http
    appPort: 3003
    daprHTTPPort: 3503
    appHealthCheckPath: "/health"
    placementHostAddress: ""
    logLevel: "debug"
    command: ["dotnet", "run"]
    env:
      ASPNETCORE_URLS: http://localhost:3003
      ASPNETCORE_ENVIRONMENT: Development
      TESTCASE: dapr
      INSTANCE: standard

Accompanied by a small script to publish both types of messages into the local environment and observe the flow:

#!/bin/bash

set -e

ORDERID=0

function generate_message()
{
  UUID=`uuidgen`
  if [[ $((ORDERID % 2)) -eq 0 ]]; then
    DELIVERY=Express
  else
    DELIVERY=Standard
  fi
  jq -c -n \
        --arg uuid "$UUID" \
        --arg orderid "$ORDERID" \
        --arg delivery "$DELIVERY" \
        '{OrderId: $orderid|tonumber, OrderGuid: $uuid, Delivery: $delivery}'
}

ORDERID=$((ORDERID + 1))
dapr publish --publish-app-id distributor \
  --topic q-order-ingress-dapr \
  --pubsub order-pubsub \
  --data "$(generate_message)" \
  --metadata '{"rawPayload":"true"}'

ORDERID=$((ORDERID + 1))
dapr publish --publish-app-id distributor \
  --topic q-order-ingress-dapr \
  --pubsub order-pubsub \
  --data "$(generate_message)" \
  --metadata '{"rawPayload":"true"}'

First Pit Stop

It seems using pubsub puts a toll on the performance.

implementation	average, 5 cycles	standard deviation
Dapr input bindings, before conversion	48.6s	6.2s
Dapr single pubsub	103.4s	7.5s

The Valley Of Tears

I thought, with just implementing bulk pubsub performance will increase dramatically. I reduce single message transfers to block transfers and it seems logical, that just by that messages are bursted through the environment at light speed. Well, ...

Converting To Bulk PubSub

To apply bulk pubsub on subscription programmatically, just meta information needs to be enhanced:

app.MapGet("/dapr/subscribe", () => Results.Ok(new[]{
new {
    pubsubname = "order-pubsub",
    topic = $"q-order-ingress-{testCase}",
    route = $"/q-order-ingress-{testCase}-pubsub",
    bulkSubscribe = new {
      enabled = true,
      maxMessagesCount = 100,
      maxAwaitDurationMs = 40,
    }
}}));

On the processing endpoint changes are more substantial:

messages are passed in a bulk, hence looping is required to process single messages
response collections need to be maintained to tell Dapr, which of the incoming entries/messages processed successful, which need to be retried or dropped
messages to be published in a bulk also need to be collected for sending outbound
again, let no CloudEvents slip in at that moment

app.MapPost($"/q-order-ingress-{testCase}-pubsub", async (
    [FromBody] BulkSubscribeMessage<Order> bulkOrders,
    [FromServices] DaprClient daprClient,
    ILogger<Program> log
    ) =>
{
    log.LogInformation("{Count} Orders to distribute", bulkOrders.Entries.Count);
    List<BulkSubscribeAppResponseEntry> responseEntries = new List<BulkSubscribeAppResponseEntry>(bulkOrders.Entries.Count);
    var expressEntries = new List<BulkSubscribeMessageEntry<Order>>(bulkOrders.Entries.Count);
    var standardEntries = new List<BulkSubscribeMessageEntry<Order>>(bulkOrders.Entries.Count);

    foreach (var entry in bulkOrders.Entries)
    {
        var order = entry.Event;

        switch (order.Delivery)
        {
            case Delivery.Express:
                expressEntries.Add(entry);
                break;
            case Delivery.Standard:
                standardEntries.Add(entry);
                break;
        }
    }

    var metadata = new Dictionary<string, string>
        {
          { "rawPayload", "true"}
        };

    if (expressEntries.Count > 0)
    {
        try
        {
            var orders = from e in expressEntries select e.Event;
            await daprClient.BulkPublishEventAsync("order-pubsub", $"q-order-express-{testCase}", orders.ToArray(), metadata);
            foreach (var entry in expressEntries)
            {
                responseEntries.Add(new BulkSubscribeAppResponseEntry(entry.EntryId, BulkSubscribeAppResponseStatus.SUCCESS));
            }
        }
        catch (Exception e)
        {
            log.LogError(e.Message);
            foreach (var entry in expressEntries)
            {
                responseEntries.Add(new BulkSubscribeAppResponseEntry(entry.EntryId, BulkSubscribeAppResponseStatus.RETRY));
            }
        }
    }

    if (standardEntries.Count > 0)
    {
        try
        {
            var orders = from e in standardEntries select e.Event;
            await daprClient.BulkPublishEventAsync("order-pubsub", $"q-order-standard-{testCase}", orders.ToArray(), metadata);
            foreach (var entry in standardEntries)
            {
                responseEntries.Add(new BulkSubscribeAppResponseEntry(entry.EntryId, BulkSubscribeAppResponseStatus.SUCCESS));
            }
        }
        catch (Exception e)
        {
            log.LogError(e.Message);
            foreach (var entry in standardEntries)
            {
                responseEntries.Add(new BulkSubscribeAppResponseEntry(entry.EntryId, BulkSubscribeAppResponseStatus.RETRY));
            }
        }
    }


    return new BulkSubscribeAppResponse(responseEntries);
});

this is the daprdistributor implementation; daprreceiver implementation is adapted accordingly receiving bulk messages but storing to blob with single requests

Second Pit Stop

Even with having 2 cycles faster then 40s, still there are outliers which take 60-70s. Which in turn results in this average:

implementation	maxMessagesCount	average, 5 cycles	standard deviation
Dapr input bindings	n.a.	48.6s	6.2s
Dapr single pubsub	n.a.	103.4s	7.5s
Dapr bulk pubsub	100	57.6s	19.2s

Crank It Up

A maxMessagesCount of 100 does not seem to make a dent. Let's increase:

      maxMessagesCount = 500,

Although having half of the requests with a significant better cycle time, the other half still took twice the time (standard deviation ~21s on a 64s average):

timestamp	cycle time
2024-02-12T06:15:09	36
2024-02-12T06:22:50	43
2024-02-12T06:29:41	80
2024-02-12T06:37:04	81
2024-02-12T06:45:04	78
2024-02-12T06:52:34	38

Using a query like ...

requests
| where timestamp between( todatetime('2024-02-12T06:45:04.7285069Z') .. todatetime('2024-02-12T06:52:34.2132486Z') )
| where name startswith "POST" and cloud_RoleName matches regex "^[\\d\\w\\-]+dapr"
| where success == true
| summarize count() by cloud_RoleName, bin(timestamp, 5s)
| render columnchart

... shows a processing gap towards the end of processing ...

... while in this sample processing is in one block without a gap:

Looking at the replica count for all 3 services during the testing period shows, that those go consitently up to 10 and down again. Also no replica restarts are reported. Hence non-availibilty of replicas can be ruled out for causing these gaps.

Additionally I found many RETRY errors captured for during bulk subscribe processing, ...

time="2024-02-12T06:30:00.08158319A3Z" level=error msg="App handler returned an error for message 5888d18e-da39-44b4-81d9-a2e11bef21bb on queue q-order-ingress-dapr: RETRY required while processing bulk subscribe event for entry id: 27180f25-502b-4d13-a738-77306a6d3f01"

... spread out over all test cycles but only logged for Distributor service.

Looking for the root cause of these retries I found ...

dependencies
| where timestamp between( todatetime('2024-02-12T06:15:00') .. todatetime('2024-02-12T07:00:00') )
| where cloud_RoleName endswith "daprdistributor"
| where name == "/dapr.proto.runtime.v1.dapr/bulkpublisheventalpha1"
| order by timestamp asc

error when publish to topic q-order-express-dapr in pubsub order-pubsub: the message is too large

... which means, that a maxMessagesCount of 500 can cause Distributor to generate too large bulk publish packets.

OK, so let's increase carefully ...

      maxMessagesCount = 250,

... and then check for errors again.

dependencies
| where customDimensions.error != ""
| order by timestamp desc

Third Pit Stop

Not quite, but close.

implementation	maxMessagesCount	average, 5 cycles	standard deviation
Dapr input bindings	n.a.	48.6s	6.2s
Dapr single pubsub	n.a.	103.4s	7.5s
Dapr bulk pubsub	100	57.6s	19.2s
Dapr bulk pubsub	250	42.8s	13.1s

Still for the slow test cycles there is a gap until all records are processed:

Exactly for these kind of measurements I log the bulk messsage count with log.LogInformation("{Count} Orders to distribute", bulkOrders.Entries.Count); to have an indication on how many messages are handed over in one bulk.

At this point I even tried to switch to a dedicated workload profile type - instead of consumption - to rule out that "noisy neighbors" are causing the processing to lag. But no, same effects also with dedicated this processing gap occured.

Side node: I had to go up to E8 workload profile type to get to a processing speed similar to consumption model.

Beware Of Message Locks

Fine. Back to telemetry. Checking what else is logged on the containers ...

ContainerAppConsoleLogs_CL
| where TimeGenerated between( todatetime('2024-02-13T16:07:05.8171428Z') .. todatetime('2024-02-13T16:45:47.5552143Z') )
| where ContainerAppName_s contains "daprdist"
| where Log_s !contains "level=info"
| order by TimeGenerated asc

... reveals that almost exactly at the point before final processing continues, this type of error is logged:

...
time="2024-02-13T16:14:05.778260655Z" level=warning msg="Error renewing message locks for queue q-order-ingress-dapr (failed: 43/758):
couldn't renew active message lock for message ff3d2095-effb-428e-87bf-e131c961ac4e: lock has been lost
...

Looking at the order-pubsub component in the containerapps.bicep I realized I missed a spot. Measuring earlier how many bulks could be processed at full size (e.g. 100) I observed, that very soon during the processing cycle, the Dapr Service Bus implementation was creating smaller bulks (even with size 1) as 8 maxConcurrentHandlers, after waiting maxAwaitDurationMs = 40 (see pubsub subscribe above), were just pulling "what they get". Hence I tuned down maxConcurrentHandlers to 1 to avoid that kind of congestion.

resource pubSubComponent 'daprComponents' = {
    name: 'order-pubsub'
    properties: {
      componentType: 'pubsub.azure.servicebus.queues'
      version: 'v1'
      secrets: [
        {
          name: 'sb-root-connectionstring'
          value: '${listKeys('${sb.id}/AuthorizationRules/RootManageSharedAccessKey', sb.apiVersion).primaryConnectionString};EntityPath=orders'
        }
      ]
      metadata: [
        {
          name: 'connectionString'
          secretRef: 'sb-root-connectionstring'
        }
        {
          name: 'maxActiveMessages'
          value: '1000'
        }
        {
          name: 'maxConcurrentHandlers'
          value: '1'
        }
      ]
      scopes: scopesPubSub
    }
  }
}

However I did not pay attention to maxActiveMessages - which before were divided by 8 handlers, what in turn allowed Dapr sidecar and the app to process a block of 125 locked message in time (before the lock expires). Now 1 handler was locking all 1000 messages and even with bulk processing some message locks could expire.

Consequently I think it makes more sense to tune the amount of active messages pulled at a time to the potential bulk size:

        {
          name: 'maxActiveMessages'
          value: '250'
        }

The Last Mile

Changing this setting did not make the warning completely go away. As Alessandro Segala states in this issue : "This happens because when it's time to renew the locks, at the interval, we "snapshot" the active messages and then renew each one's lock in sequence. If you have a high number for maxActiveMessages, it takes time for the component to renew the locks for each one, and if the app has ACK'd the message in the meanwhile, then the lock renewal fails."

However as results show: the better noise and congestion is reduced in the environment, the better and more reliable the throughput.

implementation	maxMessagesCount	maxActiveMessages	maxConcurrentHandlers	average, 5 cycles	standard deviation
Dapr input bindings	n.a.	1000	8	48.6s	6.2s
Dapr single pubsub	n.a.	1000	8	103.4s	7.5s
Dapr bulk pubsub	100	1000	1	57.6s	19.2s
Dapr bulk pubsub	250	1000	1	42.8s	13.1s
Dapr bulk pubsub	250	250	1	28.0s	3.6s

Disclaimer: This combination of configuration values is just tuned for this particular scenario. With this post I just want to show, what to consider, what to look out for and where to calibrate.

Conclusion

Running all 3 stacks on the same environment now shows, that an invest in bulk pubsub definitely pays out.

stack	average, 6 cycles	standard deviation
ACAF = Functions on ACA	36.0s	3.5s
DAPR = Dapr in Container on ACA	27.8s	1.6s
FUNC = Function in Container on ACA	34.0s	2.4s

As always there is no free lunch and some points have to be considered:

Dapr HTTP and gRPC payload size
just switching to bulk pubsub is not enough; the additional aggregation and collection times (see maxAwaitDurationMs) need to be outweighed by the gains achieved with packaging data
with larger payloads moving through an environment, impact on all involved components needs to be considered and then balanced for optimum outcome

When I set out writing this post I already observed some of these lags shown above, not sure whether I would be able to identify and overcome those. But hey, that are exactly the challenges I seek.

Comparing throughput of Azure Functions vs Dapr on Azure Container Apps

Kai Walter — Mon, 09 Oct 2023 09:54:48 +0000

TL;DR

In this post I show

(mainly) how .NET Azure Functions (in the 2 currently available hosting options on Azure Container Apps) can be compared to a ASP.NET Dapr application in terms of asynchronous messaging throughput
some learnings when deploying the 3 variants on ACA (=Azure Container Apps):
- Azure Functions in a container on ACA, applying KEDA scaling
- Azure Functions on ACA, leaving scaling up to the platform
- ASP.NET in a container on ACA using Dapr sidecar, apply KEDA scaling
extending ApplicationInsights cloud_RoleName and cloud_RoleInstance for Dapr to see instance names in telemetry

jump to results

Although the sample repo additional to Bash/Azure CLI contains a deployment option with Azure Developer CLI, I never was able to sustain stable deployment with this option while Azure Functions on Container Apps was in preview.

Motivation

Azure Container Apps hosting of Azure Functions is a way to host Azure Functions directly in Container Apps - additionally to App Service with and without containers. This offering also adds some Container Apps built-in capabilities like the Dapr microservices framework which would allow for mixing microservices workloads on the same environment with Functions.

Running a sufficiently big workload already with Azure Functions inside containers on Azure Container Apps for a while, I wanted to see how both variants compare in terms of features and above all : scaling.

With another environment we heavily rely on Dapr for synchronous invocations as well as asynchronous message processing. Hence additionally I wanted to see whether one of the frameworks promoted by Microsoft - Azure Functions host with its bindings or Dapr with its generic components and the sidecar architecture - substantially stands out in terms of throughput.

Solution Overview

The test environment can be deployed from this repo - README.md describes the steps required.

Approach

To come to a viable comparison, I applied these aspects:

logic for all contenders is written in C# / .NET 7
all contenders need to process the exact same volume and structure of payloads - which is generated once and then sent to them for processing
test payload (10k messages by default) is send on a queue and scheduled to exactly the same time to force the stack, to deal with the amount at once
both Functions variants are based on .NET isolated worker, as Functions on Container Apps only support this model
all 3 variants run staggered, not at the same time, on the same Container Apps environment, hence same region, same nodes, same resources ...

Limitations

Only Service Bus queues are tested. Of course, a scenario like this can also be achieved with pub/sub Service Bus topics and subscriptions. However in our enterprise workloads, where we apply this pattern, we work with queues as these allow a dedicated dead-lettering at each stage of the process - compared to topics, where moving messages from dead-letter to active results in all subscribers (if not explicitly filtered) receivng these messages again.
Currently not all capabilities of the contesting stacks - like Dapr bulk message processing - are maxed out. Hence there is obviously still some potential for improving individual throughput.

Measuring Throughput

Throughput is measured by substracting the timestamp of the last message processed from the timestamp of the first message processed - for a given scheduling timestamp. A generic query to Application Insights with $TESTPREFIX representing one of the codes above and $SCHEDULE refering to the scheduling timestamp for a particular test run:

query="requests | where cloud_RoleName matches regex '$TESTPREFIX(dist|recv)' | where name != 'Health' and name !startswith 'GET' | where timestamp > todatetime('$SCHEDULE') | where success == true | summarize count(),sum(duration),min(timestamp),max(timestamp) | project count_, runtimeMs=datetime_diff('millisecond', max_timestamp, min_timestamp)"

Evaluating Scaling

While above query is used in the automated testing and recording process, I used this type of query ...

requests
| where cloud_RoleName startswith "func"
| where name != "Health"
| where timestamp > todatetime('2022-11-03T07:09:26.9394443Z')
| where success == true
| summarize count() by cloud_RoleInstance, bin(timestamp, 15s)
| render columnchart

... to see whether the platform / stack scales in an expected pattern, ...

... which pointed me to a strange scaling lag for Azure Functions on ACA:

Microsoft Product Group looked into this observation and provided an explanation in this GitHub issue:

"Initially, some default numbers of nodes are allocated for any ACA environment. During scaling, ACA uses these nodes to create app instances. For container apps scaling, the default number of nodes are sufficient as it uses less cpu, memory per instance. For function apps scaling, the default number of nodes is not sufficient and thus, ACA environment requests more nodes in back end. After new nodes are available to ACA environment, it uses them to create remaining instances for Function app. It takes some time to fetch new nodes and create remaining instances, therefore, we see a gap in processing between both deployments."

When conducting the final battery of tests in October'23 this behavior was partially gone (see results below) when sufficient Functions relevant nodes already had been scaled.

Solution Elements

a Generate in Function App testdata generates a test data payload (e.g. with 10k orders) and puts it in a blob storage
one of the PushIngress... functions in the very same Function App then can be triggered to schedule all orders at once on an ingress Service Bus queue - either for Functions or for Dapr
each of the contestants has a Dispatch method which picks the payload for each order from the ingress queue, inspects it and puts it either on a queue for "Standard" or "Express" orders
then for these order types there is a separate Receiver function which finally processes the dispatched message

C# project names and queues use a consistent coding for each contestant:

code used for solution elements	implementation and deployment approach
ACAF	.NET Azure Functions on ACA deployment
DAPR	ASP.NET with Dapr in a container on ACA
FUNC	.NET Azure Functions in a container on ACA

Dispatcher

As .NET isolated does not support multiple outputs for Functions, an optional message output is required to either put message into StandardMessage or ExpressMessage:

using Azure.Messaging.ServiceBus;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using Models;
using System.Text;
using System.Text.Json;

namespace funcdistributor
{
    public class Dispatch
    {
        [Function("Dispatch")]
        public DispatchedOutput Run(
            [ServiceBusTrigger("q-order-ingress-func", Connection = "SERVICEBUS_CONNECTION")] string ingressMessage,
            ILogger log)
        {
            ArgumentNullException.ThrowIfNull(ingressMessage, nameof(ingressMessage));

            var order = JsonSerializer.Deserialize<Order>(ingressMessage);

            ArgumentNullException.ThrowIfNull(order, nameof(ingressMessage));

            var outputMessage = new DispatchedOutput();

            switch (order.Delivery)
            {
                case Delivery.Express:
                    outputMessage.ExpressMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(order)))
                    {
                        ContentType = "application/json",
                        MessageId = order.OrderId.ToString(),
                    };
                    break;
                case Delivery.Standard:
                    outputMessage.StandardMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(order)))
                    {
                        ContentType = "application/json",
                        MessageId = order.OrderId.ToString(),
                    };
                    break;
                default:
                    log.LogError($"invalid Delivery type: {order.Delivery}");
                    break;
            }

            return outputMessage;
        }
    }

    public class DispatchedOutput
    {
        [ServiceBusOutput("q-order-express-func", Connection = "SERVICEBUS_CONNECTION")]
        public ServiceBusMessage? ExpressMessage { get; set; }

        [ServiceBusOutput("q-order-standard-func", Connection = "SERVICEBUS_CONNECTION")]
        public ServiceBusMessage? StandardMessage { get; set; }
    }
}

For Dapr this dispatcher is implemented with minimal API just in the top-level file Program.cs - a very concise way almost in Node.js style:

app.MapPost("/q-order-ingress-dapr", async (
    [FromBody] Order order,
    [FromServices] DaprClient daprClient
    ) =>
{
    switch (order.Delivery)
    {
        case Delivery.Express:
            await daprClient.InvokeBindingAsync("q-order-express-dapr", "create", order);
            break;
        case Delivery.Standard:
            await daprClient.InvokeBindingAsync("q-order-standard-dapr", "create", order);
            break;
    }

    return Results.Ok(order);
});

Receiver

in Functions:

using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using Models;
using System.Text.Json;
using System;

namespace acafrecvexp
{
    public class Receiver
    {
        [Function("Receiver")]
        public void Run(
            [ServiceBusTrigger("q-order-express-acaf", Connection = "SERVICEBUS_CONNECTION")] string ingressMessage,
            FunctionContext executionContext
            )
        {
            var logger = executionContext.GetLogger("Receiver");

            ArgumentNullException.ThrowIfNull(ingressMessage, nameof(ingressMessage));

            var order = JsonSerializer.Deserialize<Order>(ingressMessage);

            ArgumentNullException.ThrowIfNull(order, nameof(order));

            logger.LogInformation("{Delivery} Order received {OrderId}", order.Delivery, order.OrderId);
        }
    }
}

in Dapr with minimal API:

app.MapPost("/q-order-express-dapr", (
    ILogger<Program> log,
    [FromBody] Order order
    ) =>
{
    log.LogInformation("{Delivery} Order received {OrderId}", order.Delivery, order.OrderId);
    return Results.Ok();
});

Scaling

For the Functions and Dapr Container App, a scaling rule can be set. For Functions on ACA this is handled by the platform.

      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'queue-rule'
            custom: {
              type: 'azure-servicebus'
              metadata: {
                queueName: entityNameForScaling
                namespace: serviceBusNamespace.name
                messageCount: '100'
              }
              auth: [
                {
                  secretRef: 'servicebus-connection'
                  triggerParameter: 'connection'
                }
              ]
            }

          }
        ]
      }

This setting makes ACA scale replicas up when there are more than 100 messages in active queue.

Results

A first batch of tests in August'23 revealed no substantial disparity between the stacks:

To capture the final results in October'23, I ...

upgraded dependencies of the .NET projects (e.g. to Dapr 1.11)
switched from Azure Service Bus Standard Tier to Premium because of that throttling issue explained below, which imho gave the whole scenario a major boost

After these upgrades and probably back-end rework done by Microsoft now a much clearer spread of average duration can be seen: Dapr is obviously handling the processing faster than Functions in Container on ACA and then (currently) Functions on ACA shows the worst performance in average:

West Europe	West US

To be sure to have no regional deployment effects, I deployed and tested in 2 regions.

Looking on the time dimension one can see that Functions on ACA has a wider spread of durations - even processing faster than Dapr at some points:

I am sure, that throughput of all variants can be improved by investing more time in measuring and fine tuning. My approach was to see what I can get out of the environment with a feasible amount of effort.

Nuggets and Gotchas

Apart from the plain throughput evaluation above, I want to add the issues I stumbled over along the way - I guess this is the real "meat" of this post:

Deploying Container Apps with no App yet built

When deploying infrastructure without the apps yet being build, a Functions on ACA already needs a suitable container image to spin up. I solved this in Bicep evaluating whether a ACR container image name was provided or not. Additional challenge then is that DOCKER_REGISTRY... credentials are required for the final app image but not for the tempory dummy image.

...
var effectiveImageName = imageName != '' ? imageName : 'mcr.microsoft.com/azure-functions/dotnet7-quickstart-demo:1.0'

var appSetingsBasic = [
  {
    name: 'AzureWebJobsStorage'
    value: 'DefaultEndpointsProtocol=https;AccountName=${stg.name};AccountKey=${stg.listKeys().keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
  }
  {
    name: 'STORAGE_CONNECTION'
    value: 'DefaultEndpointsProtocol=https;AccountName=${stg.name};AccountKey=${stg.listKeys().keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
  }
  {
    name: 'SERVICEBUS_CONNECTION'
    value: '${listKeys('${serviceBusNamespace.id}/AuthorizationRules/RootManageSharedAccessKey', serviceBusNamespace.apiVersion).primaryConnectionString}'
  }
  {
    name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
    value: appInsights.properties.ConnectionString
  }
]

var appSetingsRegistry = [
  {
    name: 'DOCKER_REGISTRY_SERVER_URL'
    value: containerRegistry.properties.loginServer
  }
  {
    name: 'DOCKER_REGISTRY_SERVER_USERNAME'
    value: containerRegistry.listCredentials().username
  }
  {
    name: 'DOCKER_REGISTRY_SERVER_PASSWORD'
    value: containerRegistry.listCredentials().passwords[0].value
  }
  // https://github.com/Azure/Azure-Functions/wiki/When-and-Why-should-I-set-WEBSITE_ENABLE_APP_SERVICE_STORAGE
  // case 3a
  {
    name: 'WEBSITES_ENABLE_APP_SERVICE_STORAGE'
    value: 'false'
  }
]

var appSettings = concat(appSetingsBasic, imageName != '' ? appSetingsRegistry : [])

resource acafunction 'Microsoft.Web/sites@2022-09-01' = {
  name: '${envName}${appName}'
  location: location
  tags: union(tags, {
      'azd-service-name': appName
    })
  kind: 'functionapp'
  properties: {
    managedEnvironmentId: containerAppsEnvironment.id

    siteConfig: {
      linuxFxVersion: 'DOCKER|${effectiveImageName}'
      appSettings: appSettings
    }
  }
}

Exactly at this point I struggle with Azure Developer CLI currently: I am able to deploy infra with the dummy image but as soon as I want to deploy the service, the service deployment does not apply the above logic and set the DOCKER_REGISTRY... credentials. Triggering the very same Bicep templates with Azure CLI seems to handle this switch properly.
I had to use these credentials as managed identity was not working yet as supposed.

Extending ApplicationInsights cloud_RoleName and cloud_RoleInstance for Dapr

When hosting ASP.NET with Dapr on Container Apps, cloud_RoleName and cloud_RoleInstance are not populated - which I needed to evaluate how many instances / replicas are scaled.

AppInsightsTelemetryInitializer.cs:

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;

namespace Utils
{

    public class AppInsightsTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(ITelemetry telemetry)
        {
            if (string.IsNullOrEmpty(telemetry.Context.Cloud.RoleName))
            {
                telemetry.Context.Cloud.RoleName = System.Environment.GetEnvironmentVariable("CONTAINER_APP_NAME") ?? "CONTAINER_APP_NAME-not-set";
            }
            if (string.IsNullOrEmpty(telemetry.Context.Cloud.RoleInstance))
            {
                telemetry.Context.Cloud.RoleInstance = System.Environment.GetEnvironmentVariable("HOSTNAME") ?? "HOSTNAME-not-set";
            }
        }
    }

}

Program.cs:

...
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) =>
{
    o.TelemetryInitializers.Add(new AppInsightsTelemetryInitializer());
});
...

Channeling .env values into Bash scripts for Azure CLI

Coming from Azure Developer CLI where I channel environment values with source <(azd env get-values) into Bash, I wanted to re-use as much of the scripts for Azure CLI as possible.

For that I created a .env file in repository root like ...

AZURE_ENV_NAME="kw-md"
AZURE_LOCATION="westeurope"

... and then source its values into Bash from which I then derive resource names to operate on with Azure CLI

#!/bin/bash
source <(cat $(git rev-parse --show-toplevel)/.env)

RESOURCE_GROUP_NAME=`az group list  --query "[?starts_with(name,'$AZURE_ENV_NAME')].name" -o tsv`
AZURE_CONTAINER_REGISTRY_NAME=`az resource list --tag azd-env-name=$AZURE_ENV_NAME --query "[?type=='Microsoft.ContainerRegistry/registries'].name" -o tsv`
AZURE_CONTAINER_REGISTRY_ENDPOINT=`az acr show -n $AZURE_CONTAINER_REGISTRY_NAME --query loginServer -o tsv`
AZURE_CONTAINER_REGISTRY_ACRPULL_ID=`az identity list -g $RESOURCE_GROUP_NAME --query "[?ends_with(name,'acrpull')].id" -o tsv`
AZURE_KEY_VAULT_SERVICE_GET_ID=`az identity list -g $RESOURCE_GROUP_NAME --query "[?ends_with(name,'kv-get')].id" -o tsv`
...

Dapr batching and bulk-message handling

Dapr input binding and pub/sub Service Bus components need to be set to values much higher than the defaults to get a processing time better than Functions - keeping defaults shows Dapr E2E processing time almost factor 2 compared to Functions.

        {
          name: 'maxActiveMessages'
          value: '1000'
        }
        {
          name: 'maxConcurrentHandlers'
          value: '8'
        }

While activating bulk-message handling on the ServiceBus Dapr component did not show any significant effect.

        {
          name: 'maxBulkSubCount'
          value: '100'
        }

Functions batching

Changing from single message dispatching to batched message dispatching and thus using batching "MaxMessageBatchSize": 1000 did not have a positive effect - on the contrary: processing time was 10-20% longer.

single message dispatching

        [FunctionName("Dispatch")]
        public void Run(
            [ServiceBusTrigger("q-order-ingress-func", Connection = "SERVICEBUS_CONNECTION")] string ingressMessage,
            [ServiceBus("q-order-express  {
    name: 'WEBSITE_SITE_NAME'
    value: appName
  }
ollector<ServiceBusMessage> outputExpressMessages,
            [ServiceBus("q-order-standard-func", Connection = "SERVICEBUS_CONNECTION")] ICollector<ServiceBusMessage> outputStandardMessages,
            ILogger log)
        {
            ArgumentNullException.ThrowIfNull(ingressMessage,nameof(ingressMessage));

            var order = JsonSerializer.Deserialize<Order>(ingressMessage);

            ArgumentNullException.ThrowIfNull(order,nameof(ingressMessage));

batched

        [FunctionName("Dispatch")]
        public void Run(
            [ServiceBusTrigger("q-order-ingress-func", Connection = "SERVICEBUS_CONNECTION")] ServiceBusReceivedMessage[] ingressMessages,
            [ServiceBus("q-order-express-func", Connection = "SERVICEBUS_CONNECTION")] ICollector<ServiceBusMessage> outputExpressMessages,
            [ServiceBus("q-order-standard-func", Connection = "SERVICEBUS_CONNECTION")] ICollector<ServiceBusMessage> outputStandardMessages,
            ILogger log)-func", Connection = "SERVICEBUS_CONNECTION")] IC
        {
            foreach (var ingressMessage in ingressMessages)
            {
                var order = JsonSerializer.Deserialize<Order>(Encoding.UTF8.GetString(ingressMessage.Body));
                ArgumentNullException.ThrowIfNull(order, nameof(ingressMessage));

Functions not processing all messages

scheduleTimeStamp	variant	total message count	duration ms
2023-10-08T10:30:02.6868053Z	ACAFQ	20000	161439
2023-10-08T10:39:04.8862227Z	DAPRQ	20000	74056
2023-10-08T10:48:03.0727583Z	FUNCQ	19890 <---	81700
2023-10-08T10:57:43.6880713Z	ACAFQ	20000	146270
2023-10-08T11:06:50.3649399Z	DAPRQ	20000	95292
2023-10-08T11:15:49.0727755Z	FUNCQ	20000	85025
2023-10-08T11:25:05.3765606Z	ACAFQ	20000	137923
2023-10-08T11:34:03.8680341Z	DAPRQ	20000	67746
2023-10-08T11:43:11.6807872Z	FUNCQ	20000	84273
2023-10-08T11:52:36.0779390Z	ACAFQ	19753 <---	142073
2023-10-08T12:01:34.9800080Z	DAPRQ	20000	55857
2023-10-08T12:10:34.5789563Z	FUNCQ	20000	91777
2023-10-08T12:20:03.5812046Z	ACAFQ	20000	154537
2023-10-08T12:29:01.8791564Z	DAPRQ	20000	87938
2023-10-08T12:38:03.6663978Z	FUNCQ	19975 <---	78416

Looking at the queue items triggering distributor logic ...

requests
| where source startswith "sb-"
| where cloud_RoleName endswith "distributor"
| summarize count() by cloud_RoleName, bin(timestamp,15m)

cloud_RoleName	timestamp [UTC]	count_
acafdistributor	10/8/2023, 10:30:00.000 AM	10000
funcdistributor	10/8/2023, 10:45:00.000 AM	9890 <---
acafdistributor	10/8/2023, 10:45:00.000 AM	10000
funcdistributor	10/8/2023, 11:15:00.000 AM	10000
acafdistributor	10/8/2023, 11:15:00.000 AM	10000
funcdistributor	10/8/2023, 11:30:00.000 AM	10000
acafdistributor	10/8/2023, 11:45:00.000 AM	10000
funcdistributor	10/8/2023, 12:00:00.000 PM	10000
acafdistributor	10/8/2023, 12:15:00.000 PM	10000
funcdistributor	10/8/2023, 12:30:00.000 PM	9975 <---
acafdistributor	10/8/2023, 12:45:00.000 PM	10000

... which is strange, considering that the respective PushIngressFuncQ (at ~12:30) sent exactly 10.000 messages into the queue.

Checking how much Service Bus dependencies have been generated for a particular request:

dependencies
| where operation_Id == "cbc279bb851793e18b1c7ba69e24b9f7"
| where operation_Name == "PushIngressFuncQ"
| where type == "Queue Message | Azure Service Bus"
| summarize count()

So it seems, that between sending messages into and receiving messages from a queue, messages get lost - which is not acceptable for a scenario that assumed to be enterprise grade reliable. Checking Azure Service Bus metrics reveals, that the namespace is throttling requests:

OK, but why? Reviewing how Azure Service Bus Standard Tier is handling throttling and considering the approach of moving 10.000 messages at once from scheduled to active hints towards this easily crashing the credit limit applied in Standard Tier. After changing to Premium Tier these throttlings definetely were gone. However when packing so much load simultaneously on the Functions stacks it seems to be system immanent, that not 100% of Functions requests are logged to Application Insights. According to my information this limitation should be put to Azure Functions some time soon.

Conclusion

From results above one might immediately jump to conclude that Dapr (in an ASP.NET frame) suits best for such a message forwarding scenario, because it seems to offer best throughput and when combined with C# minimal APIs a simple enough programming model. Knowing from experience, this simple programming model will not necessarily scale to complex solutions or services with many endpoints and where a certain structure of code (see Clean Architecture etc.) and re-usability is required. Here the simplicity of Functions programming model input-processing-output really can help scale even with not so mature teams - for certain scenarios. So as always in architecture it is about weighing aspects which are important to a planned environment: here technical performance vs team performance.

Azure Functions on Container Apps combined with Dapr extension may help bringing some other aspects together: the capability to connect a huge variety of cloud resources with Dapr paired with the simple programming model of Azure Functions. I shall write about this topic soon in a future post.

Cheers,
Kai

Get NeoVim plugins with build processes working on Windows

Kai Walter — Wed, 27 Sep 2023 18:35:05 +0000

TL;DR

In this post I show or share

a PowerShell script to install LLVM / mingw / make toolchains on Windows to be used by NeoVim plugin build processes
how to install Microsoft Build Tools with winget (that in the end did not qualify for all build cases and was discarded)
a sample plugin configurations with Lazy plugin manager
an observations I made with plugin managers not working smoothly through corporate proxy configurations

Motivation

I just want to have equal editing experience on Windows and Linux - not having myself to adapt when flipping back and forth. With an ecosystem like Visual Studio Code that is practically given without the need to care. But that's not how I am wired - I want to understand what's going on. Here imho NeoVim with Lua plugins is easier to digest and understand.

Background

In a previous post I was showing how I was sharing one configuration for NeoVim based on one version of dotfiles in Linux and Windows.

That works pretty well for Lua-only plugins, as long as underlying command line tools like e.g. ripgrep or lazygit are also avaible on Windows. As soon as plugins require a tool chain to build those command line tools from source, some more work is required.

Selecting the right build tool chain

MS Build Tools and Gnu Make

With this combination I was able to get Treesitter installed and built on Windows. It requires an installation like ...

# install make
$makePath = Join-Path ${env:ProgramFiles(x86)} "GnuWin32" "bin"
if(!(Test-Path $makePath -PathType Container)) {
  winget install GnuWin32.Make
}

# install MS Build Tools
$msbtProgramFolder = Join-Path ${env:ProgramFiles(x86)} "Microsoft Visual Studio" "2022" "BuildTools"
if(!(Test-Path $msbtProgramFolder -PathType Container)) {
    winget install Microsoft.VisualStudio.2022.BuildTools
    winget install --id Microsoft.VisualStudio.2022.BuildTools --override $("--passive --config " + (Join-Path $PSScriptRoot "BuildTools.vsconfig"))
}

... with an accompaning BuildTools.vsconfig to define components to be installed ...

{
  "version": "1.0",
  "components": [
    "Microsoft.VisualStudio.Component.Roslyn.Compiler",
    "Microsoft.Component.MSBuild",
    "Microsoft.VisualStudio.Component.CoreBuildTools",
    "Microsoft.VisualStudio.Workload.MSBuildTools",
    "Microsoft.VisualStudio.Component.Windows10SDK",
    "Microsoft.VisualStudio.Component.VC.CoreBuildTools",
    "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
    "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
    "Microsoft.VisualStudio.Component.Windows11SDK.22000",
    "Microsoft.VisualStudio.Component.TextTemplating",
    "Microsoft.VisualStudio.Component.VC.CoreIde",
    "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
    "Microsoft.VisualStudio.Workload.VCTools",
    "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.Spectre",
    "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.Spectre",
    "Microsoft.VisualStudio.Component.VC.14.36.17.6.ATL.Spectre",
    "Microsoft.VisualStudio.Component.VC.14.36.17.6.MFC.Spectre"
  ]
}

... and that, on plugin-installation, NeoVim is started from "Visual Studio Developer PowerShell" command prompt while adding Gnu Make to the path:

$makePath = Join-Path ${env:ProgramFiles(x86)} "GnuWin32" "bin"
$env:Path += ";" + $makePath
nvim

Yet this setup was not sufficient for Telescope/fzf, as only a clang but obviously no gcc compiler seemed to be available with MS Build Tools. I succeeded when adding a gcc into the mix, but was not really happy with the extra "Visual Studio Developer PowerShell" command prompt required.

LLVM/Clang/LLD based mingw-w64

In this toolchain I found clang and gcc compilers. Also it allowed me to just add it to the path (see script NeoVimPluginInstall.ps1 below) of my current shell when directing NeoVim into a plugin installation - without an extra command prompt like above.

That helped me to use the same build process configuration for Telescope/fzf on Linux and Windows without any changes:

return {
    "nvim-telescope/telescope.nvim",
    branch = "0.1.x",
    dependencies = {
        "nvim-lua/plenary.nvim",
        {
            "nvim-telescope/telescope-fzf-native.nvim",
            build = "make", -- <<===== make initiates build process on Windows and Linux
        },
        "nvim-tree/nvim-web-devicons",
    },
    config = function() 
...

In the case of this plugin I tried to modify the build string from make to cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release && cmake --install build --prefix build when running on Windows, but even with this small patch I was not able to cleanly succeed with MS Build Tools.

NeoVim installation script

Since my previous post I extended and hardened the installation script a bit. It now additionally installs

ripgrep for Telescope live_grep string finder
lazygit for the equally named plugin
Gnu make to drive some of the build processes
llvm-mingw64 the LLVM/Clang/LLD based mingw-w64 toolchain from above

[CmdletBinding()]
param (
    [Parameter()]
    [switch]
    $ResetState
)

# install NeoVim with WinGet, if not already present on system
if (!$(Get-Command nvim -ErrorAction SilentlyContinue)) {
    winget install Neovim.Neovim
}

# install ripgrep
if (!$(Get-Command rg -ErrorAction SilentlyContinue)) {
  winget install BurntSushi.ripgrep.MSVC
}

# install lazygit
if (!$(Get-Command lazygit -ErrorAction SilentlyContinue)) {
  winget install JesseDuffield.lazygit
}

# install make
$makePath = Join-Path ${env:ProgramFiles(x86)} "GnuWin32" "bin"
if(!(Test-Path $makePath -PathType Container)) {
  winget install GnuWin32.Make
}

$llvmFolder = Get-ChildItem -Path $env:LOCALAPPDATA -Filter "llvm*x86_64" | Select-Object -ExpandProperty FullName
if(!$llvmFolder -or !(Test-Path $llvmFolder -PathType Container)) {
  $downloadFile = "llvm-mingw.zip"
  . .\getLatestGithubRepo.ps1 -Repository "mstorsjo/llvm-mingw" -DownloadFilePattern "llvm-mingw-.*-msvcrt-x86_64.zip" -DownloadFile $downloadFile
  Expand-Archive -Path $(Join-Path $env:TEMP $downloadFile) -DestinationPath $env:LOCALAPPDATA
}

# clone my Dotfiles repo
$dotFilesRoot = Join-Path $HOME "dotfiles"

if (!(Test-Path $dotFilesRoot -PathType Container)) {
    git clone git@github.com:KaiWalter/dotfiles.git $dotFilesRoot
}

# link NeoVim configuration
$localConfiguration = Join-Path $env:LOCALAPPDATA "nvim"
$dotfilesConfiguration = Join-Path $dotFilesRoot ".config" "nvim"

if (!(Test-Path $localConfiguration -PathType Container)) { 
    Start-Process -FilePath "pwsh" -ArgumentList "-c New-Item -Path $localConfiguration -ItemType SymbolicLink -Value $dotfilesConfiguration".Split(" ") -Verb runas
}

# reset local state if required
$localState = Join-Path $env:LOCALAPPDATA "nvim-data"

if($ResetState) {
    if(Test-Path $localState -PathType Container) {
        Remove-Item $localState -Recurse -Force
        New-Item $localState -ItemType Directory
    }
}

ATTENTION: git@github.com:KaiWalter/dotfiles.git is my private Dotfiles repo - if you want to replicate my approach you would need to work from your own version;
script .\getLatestGithubRepo.ps1 downloads the latest binary / installation file from a GitHub's repo release page and is shown below

Lua configuration

When I started with my NeoVim journey a few months back, I followed the suggestions from the NeoVim main protagonists and looked into some of the NeoVim distros like LazyVim, LunarVim, AstroVim, and NVChad to make life easier (coming fresh from Visual Studio Code even to make life even bearable). Having no knowledge in the NeoVim plugin ecosystem I struggled and stopped to try to get these distros working in parallel on Linux and Windows. Hence I decided to build a configuration with Packer from scratch to find and understand the spots, where it breaks.

While succeeding in getting NeoVim working smoothly with plugins on my own Windows machines, I struggled on my company laptop. Plugin installation with Packer was lagging at best sometimes even hanging. Digging deeper I was able to pin the problem to our companies proxy which was interfering in the package downloads.

Anyway as of August'23 it is announced on the Packer repo README, that it is not maintained anymore and suggested to move to another package manager.

lazy.nvim seemed to be the next best one package manager for me - also LazyVim distro which is based on that package manager and which I checked out earlier best related to what I was looking for. Additionally the download problems with our company proxy did not manifest here.

When converting from Packer to Lazy I wanted to clean up my configuraton file structure and follow some good practise (which is always subjective, I know) and hence I leaned on the NeoVim configuration of Josean Martinez which he explains in this video:

~/.config/nvim $ tree -n --charset UTF-16
|-- init.lua
|-- lazy-lock.json
`-- lua
    `-- kws
        |-- init.lua
        |-- lazy.lua
        |-- plugins
        |   |-- colorschema.lua
        |   |-- comment.lua
        |   |-- dap.lua
        |   |-- dressing.lua
        |   |-- harpoon.lua
        |   |-- init.lua
        |   |-- lsp
        |   |   |-- lspconfig.lua
        |   |   |-- mason.lua
        |   |   `-- null-ls.lua
        |   |-- lualine.lua
        |   |-- nvim-cmp.lua
        |   |-- nvim-tree.lua
        |   |-- nvim-treesitter.lua
        |   |-- nvim-treesitter-text-objects.lua
        |   |-- telescope.lua
        |   `-- which-key.lua
        |-- remap.lua
        `-- utils.lua

So basically lazy.lua just bootstraps the package manager itself and then pulls in the plugin specifications from plugins and plugins/lsp folders.

utility scripts

NeoVimPluginInstall.ps1

While the setup PowerShell script above installs all the tools, I created another script which I use when starting NeoVim with the intention to install or update plugins. I did not want to put these folders on my search path permanently to not pollute the search path too much.

$llvmPath = Join-Path $(Get-ChildItem -Path $env:LOCALAPPDATA -Filter "llvm*x86_64" | Select-Object -ExpandProperty FullName) "bin"
$makePath = Join-Path ${env:ProgramFiles(x86)} "GnuWin32" "bin"
$env:Path += ";" + $llvmPath + ";" + $makePath
nvim $args

getLatestGithubRepo.ps1

This script is a generalization of another post to find and download a file with a given complete file name or a file pattern from a GitHub repo's latest releases:

[CmdletBinding()]
param (
    [Parameter(Mandatory = $true, Position = 1)]
    [string] $Repository,
    [Parameter(Position = 2)]
    [string] $DownloadFile,
    [string] $DownloadFilePattern = "NOT_TO_BE_FOUND"
)

$latestRelease = Invoke-RestMethod -Method Get -Uri https://api.github.com/repos/$Repository/releases/latest -StatusCodeVariable sc

if ($sc -eq 200) {
    Write-Host $latestRelease.tag_name $latestRelease.published_at
    foreach ($asset in $latestRelease.assets) {
        Write-Host $asset.name $asset.size
        if($asset.name -eq $DownloadFile -or $asset.name -match $DownloadFilePattern) {
            $target = Join-Path $env:TEMP $DownloadFile
            Invoke-WebRequest -Uri $asset.browser_download_url -OutFile $target
            Write-Host "downloaded" $asset.browser_download_url "to" $target
        }
    }
}

The End

With this setup I am content for the moment. From here I will add LSP servers, stylers, linters and other plugins I expect to improve my productivity. Working exclusively with NeoVim for my very few coding workloads now for ~4 months gives me enough proficiency to really enjoy editing code in my spare time. If I just could have VIM motions in MS Outlook and MS Word 😏 I would not need to reconfigure my brain when switching from day job to spare time activity.

Azure VM based on CBL-Mariner with Nix package manager

Kai Walter — Sat, 29 Jul 2023 16:47:11 +0000

Motivation

In a previous post I was showing how to bring up a disposable CBL-Mariner* VM using cloud-init and (mostly) the dnf package manager. As I explained in that post, it takes some fiddling around to find sources for various packages and also to mix installation methods.

* when reviewing I found that I had a small typo in the post - "CBM-Mariner" - I guess my subconscious mind partially still lives in the 8-bit era

When I was coming across Nix package manager a few days back - while again distro hopping for my home experimental machine - I thought to combine both and maybe make package installation simpler and more versatile for CBL-Mariner - with its intended small repository to keep attack surface low.

This approach, to bring in Nix over cloud-init, should work with a vast amount of distros and should not be limited to CBL-Mariner only!

create.sh - Creation script

In this post I want to use a Bash script and Azure CLI to drive VM installation:

#!/bin/bash

set -e

user=admin
name=my-cblnix
location=westeurope
keyfile=~/.ssh/cblnix

az deployment sub create -f ./rg.bicep \
  -l $location \
  -p computerName=$name \
  resourceGroupName=$name \
  location=$location \
  adminUsername=$user \
  adminPasswordOrKey="$(cat $keyfile.pub)" \
  customData="$(cat ./cloud-init-cbl-nix.txt)" \
  vmSize=Standard_D2s_v3 \
  vmImagePublisher=MicrosoftCBLMariner \
  vmImageOffer=cbl-mariner \
  vmImageSku=cbl-mariner-2

fqdn=$(az network public-ip show -g $name -n $name-ip --query 'dnsSettings.fqdn' -o tsv)
echo "ssh -i $keyfile $user@$fqdn"

ssh-keygen -R $fqdn

Key elements and assumptions:

Azure CLI using a set of Bicep templates (shown below) to deploy a Resource Group and a Virtual Machine with the same name
it is assumed that public SSH key file has the same and is in the same location than the private SSH key file, just with a .pub extension
after VM creation FQDN is determined and printed
ssh-keygen is used to clean up potentially existing entries in SSH's known_hosts

cloud-init.txt

As in the previous post a cloud-init.txt is required to bootstrap the basic installation of the VM - but now in a much cleaner shape:

#cloud-config
write_files:
  - path: /tmp/install-nix.sh
    content: |
      #!/bin/bash
      sh <(curl -L https://nixos.org/nix/install) --daemon --yes
    permissions: '0755'
  - path: /tmp/base-setup.nix
    content: |
        with import <nixpkgs> {}; [
          less
          curl
          git
          gh
          azure-cli
          kubectl
          nodejs_18
          rustup
          go
          dotnet-sdk_7
          zsh
          oh-my-zsh
        ]
    permissions: '0644'
runcmd:
- export USER=$(awk -v uid=1000 -F":" '{ if($3==uid){print $1} }' /etc/passwd)

- sudo -H -u $USER bash -c '/tmp/install-nix.sh'

- /nix/var/nix/profiles/default/bin/nix-env -if /tmp/base-setup.nix

- - sudo -H -u $USER bash -c '/nix/var/nix/profiles/default/bin/rustup default stable'

Gotchas:

daemon installation of Nix package manager needs to be executed in the context of the VM main user
after daemon installation nix-env and all installed binaries reside in /nix/var/nix/profiles/default/bin folder but as shell has not been restarted links to those binaries are not available to the session and have to be started from that location

do not forget to sudo tail -f /var/log/cloud-init-output.log to check or observe the finalization of the installation which will take some time after the VM is deployed

rg.bicep

To achieve VM installation including its Resource Group, installation is framed with this Bicep template:

targetScope = 'subscription' // Resource group must be deployed under 'subscription' scope

param location string
param resourceGroupName string
param computerName string
param vmSize string = 'Standard_DS1_v2'
param adminUsername string = 'admin'
@secure()
param adminPasswordOrKey string
param customData string

param vmImagePublisher string
param vmImageOffer string
param vmImageSku string

resource rg 'Microsoft.Resources/resourceGroups@2021-01-01' = {
  name: resourceGroupName
  location: location
}

module vm 'vm.bicep' = {
  name: 'vm'
  scope: rg
  params: {
    location: location
    computerName: computerName
    vmSize: vmSize
    vmImagePublisher: vmImagePublisher
    vmImageOffer: vmImageOffer
    vmImageSku: vmImageSku
    adminUsername: adminUsername
    adminPasswordOrKey: adminPasswordOrKey
    customData: customData
  }
}

vm.bicep

Then VM is deployed with another Bicep in the scope of the Resource Group:

param location string = resourceGroup().location
param computerName string
param vmSize string = 'Standard_D2s_v3'

param adminUsername string = 'admin'
@secure()
param adminPasswordOrKey string
param customData string = 'echo customData'

var authenticationType = 'sshPublicKey'
param vmImagePublisher string
param vmImageOffer string
param vmImageSku string

var vnetAddressPrefix = '192.168.43.0/27'

var vmPublicIPAddressName = '${computerName}-ip'
var vmVnetName = '${computerName}-vnet'
var vmNsgName = '${computerName}-nsg'
var vmNicName = '${computerName}-nic'
var vmDiagnosticStorageAccountName = '${replace(computerName, '-', '')}${uniqueString(resourceGroup().id)}'

var shutdownTime = '2200'
var shutdownTimeZone = 'W. Europe Standard Time'

var linuxConfiguration = {
  disablePasswordAuthentication: true
  ssh: {
    publicKeys: [
      {
        path: '/home/${adminUsername}/.ssh/authorized_keys'
        keyData: adminPasswordOrKey
      }
    ]
  }
}

var resourceTags = {
  vmName: computerName
}

resource vmNsg 'Microsoft.Network/networkSecurityGroups@2022-01-01' = {
  name: vmNsgName
  location: location
  tags: resourceTags
  properties: {
    securityRules: [
      {
        name: 'in-SSH'
        properties: {
          protocol: 'Tcp'
          sourcePortRange: '*'
          destinationPortRange: '22'
          sourceAddressPrefix: '*'
          destinationAddressPrefix: '*'
          access: 'Allow'
          priority: 1000
          direction: 'Inbound'
        }
      }
    ]
  }
}

resource vmVnet 'Microsoft.Network/virtualNetworks@2022-01-01' = {
  name: vmVnetName
  location: location
  tags: resourceTags
  properties: {
    addressSpace: {
      addressPrefixes: [
        vnetAddressPrefix
      ]
    }
  }
}

resource vmSubnet 'Microsoft.Network/virtualNetworks/subnets@2022-01-01' = {
  name: 'vm'
  parent: vmVnet
  properties: {
    addressPrefix: vnetAddressPrefix
    networkSecurityGroup: {
      id: vmNsg.id
    }
  }
}

resource vmDiagnosticStorage 'Microsoft.Storage/storageAccounts@2019-06-01' = {
  name: vmDiagnosticStorageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'Storage'
  tags: resourceTags
  properties: {}
}

resource vmPublicIP 'Microsoft.Network/publicIPAddresses@2019-11-01' = {
  name: vmPublicIPAddressName
  location: location
  tags: resourceTags
  properties: {
    publicIPAllocationMethod: 'Dynamic'
    dnsSettings: {
      domainNameLabel: computerName
    }
  }
}

resource vmNic 'Microsoft.Network/networkInterfaces@2022-01-01' = {
  name: vmNicName
  location: location
  tags: resourceTags
  properties: {
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          publicIPAddress: {
            id: vmPublicIP.id
          }
          subnet: {
            id: vmSubnet.id
          }
        }
      }
    ]
  }
}

resource vm 'Microsoft.Compute/virtualMachines@2022-03-01' = {
  name: computerName
  location: location
  tags: resourceTags
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    priority: 'Spot'
    evictionPolicy: 'Deallocate'
    billingProfile: {
      maxPrice: -1
    }
    hardwareProfile: {
      vmSize: vmSize
    }
    storageProfile: {
      imageReference: {
        publisher: vmImagePublisher
        offer: vmImageOffer
        sku: vmImageSku
        version: 'latest'
      }
      osDisk: {
        createOption: 'FromImage'
        diskSizeGB: 1024
      }
    }
    osProfile: {
      computerName: computerName
      adminUsername: adminUsername
      adminPassword: adminPasswordOrKey
      customData: base64(customData)
      linuxConfiguration: ((authenticationType == 'password') ? null : linuxConfiguration)
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: vmNic.id
        }
      ]
    }
    diagnosticsProfile: {
      bootDiagnostics: {
        enabled: true
        storageUri: vmDiagnosticStorage.properties.primaryEndpoints.blob
      }
    }
  }
}

resource vmShutdown 'Microsoft.DevTestLab/schedules@2018-09-15' = {
  name: 'shutdown-computevm-${computerName}'
  location: location
  tags: resourceTags
  properties: {
    status: 'Enabled'
    taskType: 'ComputeVmShutdownTask'
    dailyRecurrence: {
      time: shutdownTime
    }
    timeZoneId: shutdownTimeZone
    notificationSettings: {
      status: 'Disabled'
    }
    targetResourceId: vm.id
  }
}

Key elements and assumptions:

VM is installed with its own virtual network - in case VM would need to be integrated in an existing VNET, that part would need adaption
a Network Security Group is created and added to the Subnet which opens SSH-port 22 - for non-experimental use it is advised to place the VM behind a Bastion service, use Just-In-Time access or protect otherwise
automatic VM shutdown is achieved with a DevTestLab/schedules resource, be aware that such a resource is not available everywhere e.g. missing in Azure China; additionally time zone and point of time are hard-wired currently, please adapt to your own needs

What else can you do with Nix?

nix-shell - temporarily running packages

nix-shell can be used to bring a package temporarily - without modifying your system persistently - to your system and shell into an environment, where you can use the package until you exit:

$ nix-shell -p python311 --run python
Python 3.11.4 (main, Jun  6 2023, 22:16:46) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

In the case of the Nix Python package, it can even be extended that particular Python libraries are made available temporarily:

$ nix-shell -p '((import <nixpkgs> {}).python311.withPackages (p: [p.numpy, p.pandas]))' --run python

nix-store - managing Nix store

A useful command to clean up local packages from the store, which are no longer linked, is nix-store --gc.

A slightly advanced configuration

This configuration adds

activating experimental feature Nix Flakes
installing Docker as a service
separates packages I want to have system wide in /nix/var/nix/profiles/default/bin (Docker, less and curl) and only for the user in ~/.nix-profile/bin (Git and Rust toolchain)

#cloud-config
write_files:
  - path: /tmp/install-nix.sh
    content: | 
      #!/bin/bash
      sh <(curl -L https://nixos.org/nix/install) --daemon --yes
      mkdir -p ~/.config/nix
      echo "experimental-features = nix-command" > ~/.config/nix/nix.conf
    permissions: '0755'
  - path: /tmp/root-setup.nix
    content: | 
        with import <nixpkgs> {}; [
          less
          curl
          docker
        ]
    permissions: '0644'
  - path: /tmp/user-setup.nix
    content: | 
        with import <nixpkgs> {}; [
          git
          rustup
        ]
    permissions: '0644'
  - path: /usr/lib/systemd/system/docker.service
    content: | 
        [Unit]
        Description=Docker Application Container Engine
        Documentation=https://docs.docker.com
        After=network.target

        [Service]
        Type=notify
        # the default is not to use systemd for cgroups because the delegate issues still
        # exists and systemd currently does not support the cgroup feature set required
        # for containers run by docker
        ExecStart=/nix/var/nix/profiles/default/bin/dockerd
        ExecReload=/bin/kill -s HUP $MAINPID
        # Having non-zero Limit*s causes performance problems due to accounting overhead
        # in the kernel. We recommend using cgroups to do container-local accounting.
        LimitNOFILE=infinity
        LimitNPROC=infinity
        LimitCORE=infinity
        # Uncomment TasksMax if your systemd version supports it.
        # Only systemd 226 and above support this version.
        #TasksMax=infinity
        TimeoutStartSec=0
        # set delegate yes so that systemd does not reset the cgroups of docker containers
        Delegate=yes
        # kill only the docker process, not all processes in the cgroup
        KillMode=process

        [Install]
        WantedBy=multi-user.target
runcmd:
- export USER=$(awk -v uid=1000 -F":" '{ if($3==uid){print $1} }' /etc/passwd)

- sudo -H -u $USER bash -c '/tmp/install-nix.sh'

# root configuration
- /nix/var/nix/profiles/default/bin/nix-env -if /tmp/root-setup.nix
- systemctl enable docker
- systemctl start docker

# VM user configuration
- sudo -H -u $USER bash -c '/nix/var/nix/profiles/default/bin/nix-env -if /tmp/user-setup.nix'
- sudo -H -u $USER bash -c '~/.nix-profile/bin/rustup default stable'

Conclusion

With this configuration I have a slim distribution combined with a powerful package management environment available to add and remove packages in a clean way - exactly what I need for experimental and development workloads.

I assume that Nix offers far more capabilities than just installing packages - which I will continue to explore to have an alternative for the relatively clunky and sensitive cloud-init installation approach.

P.S.

If you want start posting articles "at scale" and want to see how I post on https://dev.to, https://ops.io and https://hashnode.com in one go, check out my repo + script. It is not really elegant, has a bulky usability, it is PowerShell, but it does the trick - hence it is "me".

Share NeoVim configuration between Linux and Windows

Kai Walter — Mon, 29 May 2023 07:37:46 +0000

Motivation

As posted on techhub.social I am currently to stretch myself again and move out of the VS Code comfortzone into NeoVim ecosystem. That also entails that I want to use NeoVim on both : Linux and Windows.

On Linux I already create a basic NeoVim configuration which I now want to share with Windows - also to see where I hit limits of that idea. On Linux I use Dotfiles to contain and share configurations for Bash, Zsh, Sway, Tmux, NeoVim, ... So how can I leverage Neovim's Dotfiles and link it to the appropriate folder on Windows?

Background

On Windows NeoVim gets its configuration from %userprofile%\AppData\Local\nvim and keeps its data in %userprofile%\AppData\Local\nvim-data. Hence the .config/nvim folder from my Dotfiles needs to be linked to the said configuration folder and a plugin like Packer.nvim needs to be cloned in a sub-folder in the data folder.

Script

# install NeoVim with WinGet, if not already present on system
if (!$(Get-Command nvim -ErrorAction SilentlyContinue)) {
    winget install Neovim.Neovim
}

# clone my Dotfiles repo
$dotFilesRoot = Join-Path $HOME "dotfiles"

if (!(Test-Path $dotFilesRoot -PathType Container)) {
    git clone git@github.com:KaiWalter/dotfiles.git $dotFilesRoot
}

# link NeoVim configuration
$localConfiguration = Join-Path $env:LOCALAPPDATA "nvim"
$dotfilesConfiguration = Join-Path $dotFilesRoot ".config" "nvim"

if (!(Test-Path $localConfiguration -PathType Container)) { 
    Start-Process -FilePath "pwsh" -ArgumentList "-c New-Item -Path $localConfiguration -ItemType SymbolicLink -Value $dotfilesConfiguration".Split(" ") -Verb runas
}

# clone Packer.nvim, if not already present on system
$localPacker = Join-Path $env:LOCALAPPDATA "nvim-data" "site" "pack" "packer" "start" "packer.nvim"

if (!(Test-Path $localPacker -PathType Container)) { 
    git clone https://github.com/wbthomason/packer.nvim $localPacker
}

ATTENTION: git@github.com:KaiWalter/dotfiles.git is my private Dotfiles repo - if you want to replicate my approach you would need to run from your own version

After running the script and starting NeoVim a :PackerSync is required to install all the plugins.

My Dotfiles on Linux

There are plenty of posts with various flavors on how to go about setting up Dotfiles. I could not get myself to suggest a particular one, so... when I setup a new Linux system, I use these commands to clone it locally:

if [ ! -d ~/.dotfiles.git ]; then git clone --bare git@github.com:KaiWalter/dotfiles.git ~/.dotfiles.git; fi
echo ".dotfiles.git" >> ~/.gitignore
git --git-dir=$HOME/.dotfiles.git/ --work-tree=$HOME/ checkout

which brings in an alias dotfiles in .zshrc or .bashrc to be used when interacting with the particular repository later.

if [ -d ~/.dotfiles.git ]; then
    alias dotfiles="/usr/bin/git --git-dir=$HOME/.dotfiles.git/ --work-tree=$HOME/"
fi

Private linking an Azure Container App Environment (May'23 update)

Kai Walter — Sun, 21 May 2023 10:03:04 +0000

This post is an updated version of my Feb'22 post "Private linking an Azure Container App Environment", uses Bicep instead of Azure CLI to deploy Private Linking configuration but is reduced to the pure configuration without jump VM and sample applications.

+plus it applies Bicep CIDR functions to calculate sample network address prefixes

Motivation

When originally posting in spring 2022 our challenge was, that we would not be granted multiple large enough (/21 CIDR range, 2048 IP addresses) address spaces within our corporate cloud address spaces - as being one of many, many workloads in the cloud - which could hold the various Container Apps environments - while still being connected to corporate resources. Now that this limitation is more relaxed - /23, 512 IP addresses for consumption only and /27, 32 IP addresses for workload profile environments - we could rework our configuration. However over time we learned to appreciate some of the other advantages this separation delivered:

clear isolation of container workloads with a high degree of control what types of traffic go in (this post) and out (see also posts on Private Linking back from Container Apps to API Management or port forwarding) the compute environment
having enough breathing space in terms of IP addresses so to almost never hit any limitations (e.g. in burst scaling scenarios)
being able to stand up additional environments at any time as the number of IP addresses required in corporate address space is minimal

Solution Elements

To simplify terms for this post I assume the corporate network connected virtual network with limited address space would be the hub network and the virtual network containing the Container Apps Environment (with the /21 address space) would be the spoke network.****

This is the suggested configuration:

a private link service within spoke network linked to the kubernetes-internal Load balancer
a private endpoint in the hub network linked to private link service above
a private DNS zone with the Container Apps domain name and a * A record pointing to the private endpoint's IP address

DISCLAIMER: the approach in this article is based on the assumption, that the underlying AKS node resource group is visible, exposed and the name matches the environments domain name (in my sample configuration domain was redrock-70deffe0.westeurope.azurecontainerapps.io which resulted in node pool resource group MC_redrock-70deffe0-rg_redrock-70deffe0_westeurope) which in turn allows one to find the kubernetes-internal ILB to create the private endpoint; checking with the Container Apps team at Microsoft, this assumption still shall be valid after GA/General Availability

Below I will refer to shell scripts and Bicep templates I keep in this repository path: https://github.com/KaiWalter/container-apps-experimental/tree/main/ca-private-bicep.

In previous year's post I had to use a mix of CLI and Bicep as not yet all Container App properties like staticIp and defaultDomain could be processed as Bicep outputs. Now the whole deployment can be achieved purely in multi-staged Bicep modules.

Prerequisites

Bicep CLI version >=0.17.1 (for CIDR calculation functions)
Azure CLI version >=2.48.1, containerapp extension >= 0.3.29 (not required for deployment but useful for configuration checks)

Main Deployment

main.bicep deploys the target resource group in the given subscription and then invokes resources.bicep to deploy the actual resources within the resource group.
resources.bicep uses network.bicep and logging.bicep to deploy the hub and spoke network as well as basic Log Analytics workspace with Application Insights, then continues with the 3-staged deployment of the Container Apps Environment including the Private Linking and DNS resources

Separating deployment stages into Bicep modules allows Azure Resource Manager/Bicep to feed information into deployment steps of resources

Stage 1 - Container Apps Environment

stage1.bicep with environment.bicep deploys the Container Apps Environment and outputs the generated DefaultDomain for further reference in Private Linking resources.

Stage 2 - Private Link Service and Private Endpoint

stage2.bicep references the generated Load Balancer by using previous DefaultDomain information and uses privatelink.bicep to create the Private Link Service resource on the Load Balancer + the Private Endpoint linking to the Private Link Resource. It outputs the Private Endpoint's Network Interface Card/NIC name to be referenced in the Private DNS configuration.

Stage 3 - Private DNS Zone

stage3.bicep references Private Endpoint's NIC by using its NIC name to extract Private IP address information required for private DNS configuration in privatedns.bicep.

Finally in privatedns.bicep

a Private DNS Zone with the domain name of the Container App Environment
an A record pointing to the Private Endpoint
Virtual Network Link from the Private DNS Zone to the hub network

is created.

Credits

Thank you @Lebrosk for reaching out and asking for the all-Bicep solution which we created in the past months but I did not care to share here.

Private linking and port forwarding to non-Azure resources

Kai Walter — Wed, 05 Apr 2023 16:47:52 +0000

TL;DR

What can be seen in this post:

use a Load Balancer combined with a small sized VM scaleset (VMSS) configured with iptables to forward and masquerade incoming connections to 2 IP addresses which represent 2 on-premise servers; this installation is placed in a hub network that can be shared amount several spoke networks
link this Load Balancer to another virtual network - without virtual network peering - by utilizing Private Link Service and a Private Endpoint which is placed in a spoke network
use Azure Container Instances to connect into hub or spoke networks and test connections
how to feed cloud-init.txt for VM customData into azd parameters
how to stop Azure Container Instances immediately after azd / Bicep deployment with the post-provisioning hook
how to persistent iptables between reboots on the VMSS instance without an additional package
how to use a NAT gateway to allow outbound traffic for an ILB/Internal Load Balancer

Context

For a scenario within a corporate managed virtual network, I private linked the Azure Container Apps environment with its own virtual network and non-restricted IP address space (here Spoke virtual network) to the corporate Hub virtual network.

back then, there was no official icon for Container Apps Environments available, hence I used an AKS icon

One challenge that had to be solved back then is how to let the workloads running in Azure Container Apps environment call back into an API Management instance in the Hub virtual network. To achieve that I private linked the Application Gateway, that forwards to the API Management instance, into the Spoke virtual network:

A New Challenge

Just recently a new challenge came up: We needed to forward TCP traffic on a specific port to 2 specific - usually load balanced - servers in a downstream / connected on-premise network.

The first reflex was to try to put both IP addresses into a backend pool of a Load Balancer in the Hub virtual network. Then trying to establish a Private Endpoint in the Spoke virtual network to allow traffic from Azure Container Apps environment over private linking into the Load Balancer and then to the downstream servers. However some limitations got in the way of this endeavor:

Limitations

IP based backends can only be used for Standard Load Balancers

The backend resources must be in the same virtual network as the load balancer for IP based LBs

A load balancer with IP based Backend Pool can't function as a Private Link service

...

Going Down the Rabbit Hole

As I usually "Don't Accept the Defaults" (Abel Wang) or just am plain and simple stubborn, I tried it anyway - which in its own neat way also provided some more learnings, I otherwise would have missed.

To let you follow along I created a sample repo which allows me to spin up an exemplary environment using Azure Developer CLI and Bicep.

I like using azd together with Bicep for simple Proof-of-Concept like scenarios as I can easily azd up and azd down the environment without having to deal with state - as with other IaC stacks.

Learning 1: I was not able to bring up the Load Balancer directly linked with the server IP addresses with Bicep in one go. Deployment succeeded without error but backend pool just was not configured.

Learning 2: Deploying with CLI, configured the Load Balancer backend pool correctly ... but forwarding did not work, because ...

source <(azd env get-values)

az network lb delete -g $RESOURCE_GROUP_NAME --name ilb-$RESOURCE_TOKEN

az network lb create -g $RESOURCE_GROUP_NAME --name ilb-$RESOURCE_TOKEN --sku Standard \
--backend-pool-name direct \
--subnet $(az network vnet subnet show -g $RESOURCE_GROUP_NAME -n shared --vnet-name vnet-hub-$RESOURCE_TOKEN --query id -o tsv)

az network lb probe create -g $RESOURCE_GROUP_NAME --lb-name ilb-$RESOURCE_TOKEN -n direct --protocol tcp --port 8000

az network lb address-pool create -g $RESOURCE_GROUP_NAME --lb-name ilb-$RESOURCE_TOKEN -n direct \
--backend-address name=server65 ip-address=192.168.42.65 \
--backend-address name=server66 ip-address=192.168.42.66 \
--vnet $(az network vnet show -g $RESOURCE_GROUP_NAME  -n vnet-hub-$RESOURCE_TOKEN --query id -o tsv)

az network lb rule create -g $RESOURCE_GROUP_NAME --lb-name ilb-$RESOURCE_TOKEN -n direct --protocol tcp \
--frontend-ip LoadBalancerFrontEnd --backend-pool-name direct \
--frontend-port 8000 --backend-port 8000 \
--probe direct

az network lb show -g $RESOURCE_GROUP_NAME --name ilb-$RESOURCE_TOKEN

source <(azd env get-values) sources all main.bicep output values generated by azd up or azd infra create as variables into the running script

Learning 3: ... specifying IP addresses together with a virtual network in the backend pool is intended for the Load Balancer to hook up the NICs/Network Interface Cards of Azure resources later automatically when these NICs get available. It is not intended for some generic IP addresses.

Learning 4: Anyway Azure Portal did not allow to create a Private Link Service on a Load Balancer with IP address configured backend pool. So it would not have worked for my desired scenario anyway.

Other Options

Virtual Network Peering Hub and Spoke is not an option as we
- do not want to mix up corporate IP address ranges with the arbitrary IP addresses ranges of the various Container Apps virtual networks
- want to avoid BGP/Border Gateway Protocol mishaps at any cost
with a recently reduced required subnet size for Workload profiles moving the whole Azure Container Apps environment or just the particular single Container App in question back to corporate IP address space would have been possible, but I did not want to give up this extra level of isolation this separation based on Private Link in and out gave us; additionally it would have required a new / separate subnet to keep it within network boundaries
deploy this one containerized workload into the corporate VNET with Azure App Service or Azure Functions, but that would have messed up the homogeneity of our environment; additionally it would have required a new / separate subnet allowing delegation for these resources

Bring In some IaaS and iptables Magic

Being a passionate PaaS-first guy, I usually do not want (or let other people need) to deal with infrastructure / IaaS. So for this rare occasion and isolated use case I decided to go for it and keep and eye out for a future Azure resource or feature that might cover this scenario - as with our DNS forwarded scaleset which we now can replace with Azure DNS Private Resolver. For our team such a decision in the end is a matter of managing technical debt.

Making such a decision easier for me is, that this is a stateless workload. VMSS nodes as implemented here can recreated at anytime without the risk of data loss.

Solution Overview

All solution elements can be found in this repo:

infra/modules/forwarder/* : a VMSS / VM Scaleset based on a Ubuntu 22.04 image, configure and persist netfilter / iptables
infra/modules/forwarder/forwarder.bicep : a Load Balancer on top of the VMSS
infra/modules/forwarder/forwarder-spoke-privatelink.bicep : a Private Link Service linked to the Load Balancer
infra/modules/forwarder/forwarder-spoke-privatelink.bicep : a Private Endpoint in the Spoke network connecting to the Private Link Server paired with a Private DNS zone to allow for name resolution
infra/modules/network.bicep : a NAT gateway on the Hub virtual network for the Internal Load Balancer
infra/modules/containergroup.bicep : Azure Container Instances (ACI) in Hub and Spoke virtual networks to hop onto these network for basic testing

general note: the sample repo is forwarding to web servers on port 8000 - for that I could have used (a layer 7) Application Gateway; however in our real world scenario we forward to another TCP/non-HTTP port, so the solution you see here should work for any TCP port (on layer 4)

VM Scaleset

I chose an image with a small footprint and which is supported for enableAutomaticOSUpgrade: true to reduce some of the maintenance effort:

  imageReference: {
    publisher: 'MicrosoftCBLMariner'
    offer: 'cbl-mariner'
    sku: 'cbl-mariner-2-gen2'
    version: 'latest'
  }

I chose a small but feasible VM SKU:

  sku: {
    name: 'Standard_B1s'
    tier: 'Standard'
    capacity: capacity
  }

HealthExtension

I wanted the scaleset to know about "application's" health, hence whether the forwarded port is available:

  extensionProfile: {
    extensions: [
      {
        name: 'HealthExtension'
        properties: {
          autoUpgradeMinorVersion: false
          publisher: 'Microsoft.ManagedServices'
          type: 'ApplicationHealthLinux'
          typeHandlerVersion: '1.0'
          settings: {
            protocol: 'tcp'
            port: port
          }
        }
      }
    ]
  }

I had to learn, that this check is done by the extension from within the VM, hence I had to open up an additional OUTPUT:

iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport $PORT -j DNAT --to-destination $ONPREMSERVER

cloud-init.txt

IP forwarding needs to be enabled, also on the outbound:

...
write_files:
- path: /etc/sysctl.conf
  content: | 
    # added by cloud init
    net.ipv4.ip_forward=1
    net.ipv4.conf.all.route_localnet=1
  append: true
...

I added a basic distribution of load to the 2 on-premise servers based on the last digit of the VMSS node's hostname:

if [[ $HOSTNAME =~ [02468]$ ]]; then export ONPREMSERVER=192.168.42.65; else export ONPREMSERVER=192.168.42.66; fi

cloud-init.txt / iptables configuration

To dig into some of the basics of iptables I worked through these 2 posts:

and got the idea of a simplistic persistence of iptables accross reboots

https://dev.to/oryaacov/3-ways-to-make-iptables-persistent-4pp

# clear filter table
iptables --flush
# general policy : deny all incoming traffic
iptables ${IPTABLES_WAIT} -P INPUT DROP
# general policy : allow all outgoing traffic
iptables ${IPTABLES_WAIT} -P OUTPUT ACCEPT
# general policy : allow FORWARD traffic
iptables ${IPTABLES_WAIT} -A FORWARD -j ACCEPT
# allow input on loopback - is required e.g. for upstream Azure DNS resolution
iptables ${IPTABLES_WAIT} -I INPUT -i lo -j ACCEPT
# further allow established connection
iptables ${IPTABLES_WAIT} -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# drop invalid connections
iptables ${IPTABLES_WAIT} -A INPUT -m conntrack --ctstate INVALID -j DROP
# allow incoming SSH for testing - could be removed
iptables ${IPTABLES_WAIT} -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# clear nat table
iptables --flush -t nat
# allow outgoing connection to target servers from inside VMSS node - is required for ApplicationHealthLinux
iptables ${IPTABLES_WAIT} -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport $PORT -j DNAT --to-destination $ONPREMSERVER
# allow incoming traffic from Load Balancer! - important!
iptables ${IPTABLES_WAIT} -t nat -A PREROUTING -s 168.63.129.16/32 -p tcp -m tcp --dport $PORT -j DNAT --to-destination $ONPREMSERVER:$PORT
# allow incoming traffic from Hub virtual network - mostly for testing/debugging, could be removed
iptables ${IPTABLES_WAIT} -t nat -A PREROUTING -s $INBOUNDNET -p tcp -m tcp --dport $PORT -j DNAT --to-destination $ONPREMSERVER:$PORT
# masquerade outgoing traffic so that target servers assume traffic originates from "allowed" server in Hub network
iptables ${IPTABLES_WAIT} -t nat -A POSTROUTING -d $ONPREMSERVER -j MASQUERADE

I learned that I also could have used nftables or ufw, I just found the most suitable samples with iptables

Load Balancer

Nothing special here.

In our pretty locked-down environment I had to pair the internal Load Balancer with a Public IP address fronted Load Balancer to arrange outbound traffic for the VMSS instances. That configuration still needs to be replaced with a NAT gateway which in turn needs reconfiguration of our corporate virtual network setup. If there is something relevant to share, I will update here or create a separate post.

Private Link Service

I got confused when I started using private linking a few months back: This service needs to be created in the virtual network of the service to be linked as a kind of pick-up point. So in my sample in the Hub virtual network.

Private Endpoint & Private DNS Zone

Also pretty straight forward: I chose a zone internal.net which is unique in the environment and potentially not appearing somewhere else.

forwarder-private-nic-to-ip.bicep : Within the same Bicep file, after the deployment of the private endpoint, its private IP address is not available. This script allows to resolve the private endpoints's private address with an extra deployment step from the NIC:

param pepNicId string

resource nic 'Microsoft.Network/networkInterfaces@2021-05-01' existing = {
  name: substring(pepNicId, lastIndexOf(pepNicId, '/') + 1)
}

output nicPrivateIp string = nic.properties.ipConfigurations[0].properties.privateIPAddress

It is injected after deploying the private endpoint to receive the private IP address for the DNS zone, as there is no linked address update possible as with regular Azure resource private endpoints:

module nic2pip 'forwarder-private-nic-to-ip.bicep' = {
  name: 'nic2pip'
  params: {
    pepNicId: pep.properties.networkInterfaces[0].id
  }
}

resource privateDnsZoneEntry 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: 'onprem-server'
  parent: dns
  properties: {
    aRecords: [
      {
        ipv4Address: nic2pip.outputs.nicPrivateIp
      }
    ]
    ttl: 3600
  }
}

NAT gateway

NAT gateway can be used to scale out outbound IP traffic with a range of public IP addresses to have a defined to avoid SNAT port exhaustion. In this scenario it is required to allow outbound IP traffic for the ILB/Internal Load Balancer - so for package manager updates during VMSS instance provisioning and for applying updates while running. It is defined in infra/modules/network.bicep ...

resource publicip 'Microsoft.Network/publicIPAddresses@2021-05-01' = {
  name: 'nat-pip-${resourceToken}'
  location: location
  tags: tags
  sku: {
    name: 'Standard'
  }
  properties: {
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
  }
}

resource natgw 'Microsoft.Network/natGateways@2022-09-01' = {
  name: 'nat-gw-${resourceToken}'
  location: location
  tags: tags
  sku: {
    name: 'Standard'
  }
  properties: {
    idleTimeoutInMinutes: 4
    publicIpAddresses: [
      {
        id: publicip.id
      }
    ]
  }
}

... and then linked to Hub virtual network, shared subnet where the Internal Load Balancer is placed into:

// hub virtual network where the shared resources are deployed
resource vnetHub 'Microsoft.Network/virtualNetworks@2022-09-01' = {
  name: 'vnet-hub-${resourceToken}'
  location: location
  tags: tags
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.1.0/24'
      ]
    }
    subnets: [
      {
        name: 'shared'
        properties: {
          addressPrefix: '10.0.1.0/26'
          privateEndpointNetworkPolicies: 'Disabled'
          privateLinkServiceNetworkPolicies: 'Disabled'
          natGateway: {
            id: natgw.id
          }
        }
      }
...

Azure Container Instances

I use a publicly available image hacklab/docker-nettools which contains some essential tools like curl for testing and debugging.

In the containerGroup resource I overwrite the startup command to send this shell-like container into a loop so that it can be re-used for a shell like az container exec -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "/bin/bash" later.

****  command: [
    'tail'
    '-f'
    '/dev/null'
  ]

Other Gadgets

I use azd env set to funnel e.g. cloud-init.txt or SSH public key id_rsa content into deployment variables - see scripts/set-environment.sh

#!/bin/bash

set -e

azd env set SSH_PUBLIC_KEY "$(cat ~/.ssh/id_rsa.pub)"
azd env set CLOUD_INIT_ONPREM "$(cat ./infra/modules/onprem-server/cloud-init.txt | base64 -w 0)"
azd env set CLOUD_INIT_FWD "$(cat ./infra/modules/forwarder/cloud-init.txt | base64 -w 0)"

a. converting to base 64 avoids having to deal with line breaks or other control characters in the variables

b. I started having scripts/set-environment.sh as a preifnracreate or preup hook but experienced that changes made in such a hook script to e.g. cloud-init.txt were not picked up consistently. Values seemed to be cached somewhere. This lagging update of cloud-init.txt on the scaleset let me to figure out how to check, which content of cloud-init.txt was used during deployment or reimaging.

After the deployment I immediately stop Azure Container Instances to not induce cost permanently - using the post provisioning hook hooks/post-provision.sh ...

#!/bin/bash

set -e

source <(azd env get-values | grep NAME)

az container stop -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME --verbose
az container stop -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME --verbose

... which and then spin up for testing on demand

source <(azd env get-values | grep NAME)
az container start -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME
az container exec -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "curl http://onprem-server.internal.net:8000"
az container stop -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME

source <(azd env get-values | grep NAME) is a simple way to source some azd / Bicep deployment outputs into shell variables.

Deployment

Preparation

This setup assumes it works with ~/.ssh/id_rsa key pair - to use other key pairs adjust ./hooks/preprovision.sh. If you don't already have a suitable key pair, generate one or modify the preprovision script to point to another public key file:

ssh-keygen -m PEM -t rsa -b 4096

az login
azd login
azd init
scripts/set-environment.sh

be sure to set Azure CLI subscription to same subscription with az account set -s ... as specified for azd init

Deploy

azd up

Check, that connection to sample server is working from within Hub network directly:

source <(azd env get-values | grep NAME)
az container start -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME
az container exec -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "wget http://192.168.42.65:8000 -O -"
az container exec -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "wget http://192.168.42.66:8000 -O -"

Check, that connection to sample servers is working from over Load Balancer within Hub network directly:

source <(azd env get-values | grep -E 'NAME|TOKEN')
az container start -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME
ILBIP=`az network lb list -g $RESOURCE_GROUP_NAME --query "[?contains(name, '$RESOURCE_TOKEN')].frontendIPConfigurations[].privateIPAddress" -o tsv`
az container exec -n $HUB_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "wget http://$ILBIP:8000 -O -"

Check, that connection to sample servers is working from Spoke network

source <(azd env get-values | grep NAME)
az container start -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME
az container exec -n $SPOKE_JUMP_NAME -g $RESOURCE_GROUP_NAME --exec-command "curl http://onprem-server.internal.net:8000"

Final Words

As stated I am not a huge fan of bringing in IaaS components into our cloud infrastructures. Just one more spinning piece one has to keep an eye on. Particularly for smaller team sizes have too many IaaS elements does not scale and shifts a substantial portion of the focus on operations instead of delivering business value - features.

However, after careful consideration of all options, it sometimes makes sense to bring in such a small piece to avoid to reworking your whole infrastructure or deviate from fundamental design goals.

Using Azure private links and private DNS zones with globally distributed resources

Kai Walter — Thu, 23 Mar 2023 17:40:05 +0000

Azure private link / endpoints allow you to connect resources to your private virtual network and with that - when removing public access - shield resources from being accessed or even attacked from the internet. For most of enterprise mission critical systems I help designing and implementing in the cloud, this kind of locked down environment is a hard requirement.

Private link as a way of restricting access to resources only for a defined range of virtual networks is an additional offering for service endpoints which I used so far in projects till spring 2020. This post by Sameera Perera shows the basic differences between these 2 offerings.

When bringing up a new environment I learned that even some resources like Azure Container Registry have a better support for private linking then for service endpoints. Hence I started looking into this other offering to check whether I can achieve a similar or even better behavior.

TL;DR

linking multiple private DNS zones to a virtual network is possible if none or not more than 1 DNS zone has auto registration enabled
(as observed) most resource types do not autoregister into the private DNS zone linked to a virtual network anyway -> manual creation of private DNS recordsets is required
the private DNS zone name is also the resource name and so (if required) the same private DNS zone name can only be created once in a resource group

target setup

The application is deployed in multiple regions (more than 2) across the globe to allow for a certain degree of autonomous operation or even take over operation if a region is down.

global resources

Resources that are globally deployed or replicated hold state or configuration data that is relevant throughout all regions:

Front Door
API Management
Cosmos DB (with multi master write)
Container Registry

regional resources

Resources that are deployed in individual regions, hold region specific data, process regional data and should be only accessible from within the region:

Application Gateway to handle ingress from Front Door
API Management (Gateway)
AKS cluster
Storage
SQL Server
ServiceBus
KeyVault

considerations

Front Door / Application Gateway

... are just used to control global ingress into regions and have no attachment to this private link / DNS scenario.

API Management

... can be deployed globally and is linked into frontend virtual networks in each region with a dedicated IP address. API Management currently has no affiliation with private link and hence also no way to sensibly bring regional gateway name resolution into private DNS zones. As API gateways are only addressed internally from containers I feed IP adress / FQDN pairs into K8S hostaliases.

AKS

... setup is based on this post courtesy of Dennis Zielke.

challenges

multiple private DNS zones linked to a virtual network

Creating a single resource and private linking it to a virtual network is pretty straight forward and has docs dedicated to each of these resources e.g. for Cosmos DB. To achieve DNS name resolution - without standing up an own custom DNS server or fiddling around with hosts. files (which btw would not work e.g. for API Management) - private DNS zones can be used. But: each resource type needs a dedicated private DNS zone to be created and maintained.

In my scenario I created required private DNS zones with in the ARM template used to create the network configuration, immediately linking these zones to the frontend and backend virtual networks.

...
        "privateZoneNames": {
            "type": "array",
            "defaultValue": [
                "privatelink.database.windows.net",
                "privatelink.vaultcore.azure.net",
                "privatelink.blob.core.windows.net",
                "privatelink.servicebus.windows.net"
            ]
        },
...
        // private DNS zones
        {
            "type": "Microsoft.Network/privateDnsZones",
            "apiVersion": "2018-09-01",
            "name": "[parameters('privateZoneNames')[copyIndex()]]",
            "location": "global",
            "copy": {
                "name": "zonecopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameBackend'),'-link')]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]",
                "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
                }
            },
            "copy": {
                "name": "zonebackendlinkcopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameFrontend'),'-link')]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]",
                "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
                }
            },
            "copy": {
                "name": "zonefrontendlinkcopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        },
...

Only the DNS zones for the 4 regional resource types are created here for the overall network. Creation of DNS zones for the global resources is covered later.

When creating private endpoints for multiple resources and with that linking multiple private DNS zones to the same virtual network, auto registration cannot be enabled on more than one private DNS zone.

Auto registration for me seems to make sense for automatically registering VMs in other type of scenarios - not so much when handling Azure PaaS resources.

However when auto registration is disabled, you have to create DNS recordsets manually or to script the creation - not saying when it is enabled that resources - at least based on my observations - would necessarily register automatically.

Creating these private DNS recordsets can be achieved by looping through the FQDN entries registered for the created endpoint. These entries look like this e.g. for Azure SQL:

CustomDnsConfigs           : [
                               {
                                 "Fqdn": "my-sqlsvr.database.windows.net",
                                 "IpAddresses": [
                                    "10.2.6.42"
                                 ]
                               }
                             ]

This entry needs to be created in a private DNS zone privatelink.database.windows.net but without the public domain name database.windows.net - hence this domain name suffix needs to be removed before:

   $resourceGroup = "resourcegroupfordnszone"
   $globalDnsSuffx = ".database.windows.net"
   $dnsZoneName = "privatelink.database.windows.net"
...
   # get custom DNS entries for endpoint just created
   (Get-AzPrivateEndpoint -Name $privateEndpoint.Name).CustomDnsConfigs | % {
      $recordSetName = $_.Fqdn -replace $globalDnsSuffx, ""
      # remove existing record
      if (Get-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
            -Name $recordSetName -RecordType A -ErrorAction SilentlyContinue) {
         Remove-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
            -Name $recordSetName -RecordType A
      }
      # create new record
      New-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
         -Name $recordSetName `
         -PrivateDnsRecord (New-AzPrivateDnsRecordConfig -IPv4Address $_.IpAddresses[0]) `
         -RecordType A -Ttl 3600
   }

handling DNS entries for global resources with mutliple IP addresses

Creating the DNS entries for the regional resources was possible because the resource names (SQL, KeyVault, Storage, ...) had the region somewhere in the resource name anyway (e.g. fancy-sql-westus, fancy-sql-eastus) and with that providing unique name to IP address mappings.

For global resources there is one name (e.g. fancy-cosmos-global, fancy-cr-global) with an IP address / a set of IP addresses for each private link endpoint created in the regional virtual networks.

For CosmosDB it would result in a list like this:

fancy-cosmos-global                   {10.1.6.17,10.2.6.15}
fancy-cosmos-global.eastus.data       {10.1.6.16,10.2.6.14}
fancy-cosmos-global.westus.data       {10.1.6.15,10.2.6.13}

No way for a consuming resources in backend or frontend network trying to resolve fancy-cosmos-global.documents.azure.com or in fact fancy-cosmos-global.privatelink.documents.azure.com.

In my first attempts I failed with the assumption or expectation that somehow the source network would be considered here in the private DNS resolution.

To work around this, one would only need to create a dedicated private DNS zone for each region and feed the entries/IP addresses relevant for this region, but ... as private DNS zone name is also the resource name within a resource group, you can only have one private DNS zone (for a given resource type) in one resource group (until that stage I only had one resource group holding all the network resources including private DNS zones).

Hence I created additional resource groups for private links, endpoints and DNS zones specifically for a region but keep the common resource group for all the private DNS zones where this is not required.

For that I use a similar ARM template to create private DNS zones and link those to the virtual networks:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vnetNameFrontend": {
            "type": "string",
            "defaultValue": "frontend",
            "metadata": {
                "description": "VNet frontend name"
            }
        },
        "vnetNameBackend": {
            "type": "string",
            "defaultValue": "backend",
            "metadata": {
                "description": "VNet backend name"
            }
        },
        "networkResourceGroup": {
            "type": "string",
            "defaultValue": "network",
            "metadata": {
                "description": "network resource group name"
            }
        },
        "privateZoneNames": {
            "type": "array",
            "defaultValue": [
                "privatelink.documents.azure.com",
                "privatelink.azurecr.io"
            ]
        }
    },
    "variables": {
    },
    "resources": [
        // private DNS zones
        {
            "type": "Microsoft.Network/privateDnsZones",
            "apiVersion": "2018-09-01",
            "name": "[parameters('privateZoneNames')[copyIndex()]]",
            "location": "global",
            "copy": {
                "name": "zonecopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameBackend'),'-link')]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[resourceId(parameters('networkResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
                }
            },
            "copy": {
                "name": "zoneclusterlinkcopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameFrontend'),'-link')]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[resourceId(parameters('networkResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
                }
            },
            "copy": {
                "name": "zonebackendlinkcopy",
                "count": "[length(parameters('privateZoneNames'))]"
            }
        }
    ]
}

putting it all together

I placed the create endpoint & maintain DNS entries in a common function (PowerShell module) as it was called for several resources.

function Update-PrivateLink {
   param (
      [string]$locationCode,
      [string]$resourceName,
      [string]$resourceId,
      [string]$groupId
   )

   # mapping from https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns
   # list of groupIds / sub resources https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-overview#private-link-resource

   # place DNS entries for global resources in location specific resource group private DNS zone to allow dedicated assignment to local VNETs
   $dnsZoneResourceGroupName = "network-common"
   $linkPrefix = $resourceName
   switch ($groupId) {
      # global resources have regional resource groups for private DNS zones -----------
      "registry" {
         $globalDnsSuffx = ".azurecr.io"
         $dnsZoneName = "privatelink.azurecr.io"
         $dnsZoneResourceGroupName = "network-" + $locationCode
         $linkPrefix = $resourceName + "-" + $locationCode
      }
      "Sql" {
         $globalDnsSuffx = ".documents.azure.com"
         $dnsZoneName = "privatelink.documents.azure.com"
         $dnsZoneResourceGroupName = "network-" + $locationCode
         $linkPrefix = $resourceName + "-" + $locationCode
      }
      # regional resources are handled with the common network resource group -----------
      "blob" {
         $globalDnsSuffx = ".blob.core.windows.net"
         $dnsZoneName = "privatelink.blob.core.windows.net"
      }
      "namespace" {
         $globalDnsSuffx = ".servicebus.windows.net"
         $dnsZoneName = "privatelink.servicebus.windows.net"
      }
      "sqlServer" {
         $globalDnsSuffx = ".database.windows.net"
         $dnsZoneName = "privatelink.database.windows.net"
      }
      "vault" {
         $globalDnsSuffx = "(.vaultcore.azure.net|.vault.azure.net)"
         $dnsZoneName = "privatelink.vaultcore.azure.net"
      }
      Default {
         throw $("no DNS name mapping defined for groupId:" + $groupId)
      }
   }

   subnetId = "{subnet-id-from-another-magic-function}"

   $subnet = Get-AzVirtualNetworkSubnetConfig `
      -ResourceId $subnetId `
      -ErrorAction Stop

   Write-Host "update private endpoint+DNS for" $resourceName $groupId "to" $subnet.Id

   $privateLink = New-AzPrivateLinkServiceConnection -Name $($linkPrefix + "-link") `
      -PrivateLinkServiceId $resourceId `
      -GroupId $groupId

   $privateEndpoint = New-AzPrivateEndpoint -ResourceGroupName $dnsZoneResourceGroupName `
      -Name $($linkPrefix + "-endpoint") `
      -Location $location.name `
      -Subnet $subnet `
      -PrivateLinkServiceConnection $privateLink `
      -Force

   (Get-AzPrivateEndpoint -Name $privateEndpoint.Name).CustomDnsConfigs | % {
      $recordSetName = $_.Fqdn -replace $globalDnsSuffx, ""
      if (Get-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
            -Name $recordSetName -RecordType A -ErrorAction SilentlyContinue) {
         Remove-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
            -Name $recordSetName -RecordType A
      }
      New-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
         -Name $recordSetName `
         -PrivateDnsRecord (New-AzPrivateDnsRecordConfig -IPv4Address $_.IpAddresses[0]) `
         -RecordType A -Ttl 3600
   }
}

This function is called after creation of each of the resource types.

The source of resourceId to be passed to the function varies by resource type. It can be .Id or .ResourceId.

groupId passed in is based on this list and refers to the private link service -GroupId.

Storage

   $storage = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageName -ErrorAction SilentlyContinue
   if ($storage) {
      Update-PrivateLink -locationCode $locationCode `
         -resourceName $storageName -resourceId $storage.Id `
         -groupId "blob"
   }

SQL

   $sqlServer = Get-AzSqlServer -ResourceGroupName $resourceGroupName -ServerName $sqlServerName -ErrorAction SilentlyContinue
   if ($sqlServer) {
      Update-PrivateLink -locationCode $locationCode `
         -resourceName $sqlServerName -resourceId $sqlServer.ResourceId `
         -groupId "sqlServer"

       # can only be set after private endpoint is created
      Set-AzSqlServer -ResourceGroupName $resourceGroupName -ServerName $sqlServerName -PublicNetworkAccess "Disabled"
   }

SQL server resource public network access needs to be enabled when creating the resource and no private endpoints yet defined. It can be switched to disabled after the endpoint creation.

ServiceBus

   $sbNamespace = Get-AzServiceBusNamespace -Name $serviceBusNamespaceName -ResourceGroupName $resourceGroupName
   if ($sbNamespace) {
      Update-PrivateLink -locationCode $locationCode `
      -resourceName $($sbNamespace.Name+"-sb") -resourceId $sbNamespace.Id `
      -groupId "namespace"
   }

KeyVault

   $kv = Get-AzKeyVault -VaultName $keyVaultName -ResourceGroupName $resourceGroupName -ErrorAction SilentlyContinue
   if ($kv) {
      Update-PrivateLink -locationCode $locationCode `
         -resourceName $keyVaultName -resourceId $kv.ResourceId `
         -groupId "vault"
   }

CosmosDB

For CosmosDB I iterate through an array with all the relevant global locations (which I used before to setup multi master write regions) so that endpoint and DNS entries are created for and in each region.

   $cosmosDb = Get-AzCosmosDBAccount -ResourceGroupName $resourceGroupName -Name $accountName
   if ($cosmosDb) {
      foreach ($instanceLocationCode in $instanceLocationCodes) {
         Update-PrivateLink -locationCode $instanceLocationCode `
            -resourceName $accountName -resourceId $cosmosDb.Id `
            -groupId "Sql"
      }
   }

Container Registry

Same goes for the 2nd global resource.

   $acr = Get-AzContainerRegistry -ResourceGroupName $resourceGroupName -Name $acrName
   if ($acr) {
      foreach ($instanceLocationCode in $instanceLocationCodes) {
         Update-PrivateLink -locationCode $instanceLocationCode `
            -resourceName $acrName -resourceId $acr.Id `
            -groupId "registry"
      }
   }

limitation: ACR build does not work out-of-the-box when placed behind a firewall or as in this case in a closed down network - except when using ACR private build agents; as I did not want to have 2 types of build agents to maintain I choose regular Azure DevOps build agents (several Docker containers running on the jump VMs) to build Docker images

conclusion

Service endpoints are easier to setup and handle. Private link requires more planning and a higher sophistication in infrastructure automation but with that allows really fine grain control on network access paths.

Currently I have a mix of imperative and declarative infrastructure code which I basically do not prefer. This is tributed to the flexible way we want to spin up regions and instances of the infrastructure. Maybe in some other iteration we refactor and reintegrate in the one way or the other.

Let me know whether this makes sense to you and whether it helped you out in your work.

credits

Thanks to my good buddy Matthias for reviewing.

March 2023 addendum - Blob Triggers in container hosted Azure Functions

Having this environment in place for a while we experienced a strange issue:Azure Functions hosted inside containers in above environment, which access storage account over private link, did not get triggered by blob triggers. Another esteemed team mate of mine - Fabian - figured out that in such a case a private link to only the blob endpoint is not sufficient but also a private link+DNS to the blob queue endpoint is required see here.