The Ops Community ⚙️: Massimiliano Donini

Azure WebJobs, Service Bus and Managed Identity: Lesson learned

Massimiliano Donini — Tue, 09 Aug 2022 13:24:58 +0000

Today I was converting some Azure webjobs to connect to Azure Service Bus using managed service identity (MSI).

The application is a simple C# Azure WebJob built using the Azure WebJob SDK that subscribe to a topic and process incoming message writing to a database.

These are the nuget packages used:

Microsoft.Azure.WebJobs v 3.0.33

Microsoft.Azure.WebJobs.Extensions.ServiceBus v 5.6.0 Please note that since Azure Functions are built on top of the WebJobs SDK, you may encounter the same issue there, I haven't verified though.

In order to grant the required permission, I created a security group and added the managed identity of the app service to the group, then I proceeded to grant this service group the Azure Service Bus Data Owner

According to the description, this role should have full access to the whole Service Bus namespace, so imagine my surprise when I tried to run the application and got an error that looks like the following:

Unauthorized access. 'Listen' claim(s) are required to perform this operation. 
Resource: 'sb://{namespace-name}.servicebus.windows.net/{topic-name}/subscriptions/{service-name}'.
TrackingId:4e956a067b044b1089b5e327c0d08fd0_G9, SystemTracker:gateway7, Timestamp:2022-08-05T15:32:58

After several trial and error, this is what I found:

If you assign the permission Azure Service Bus Data Owner to a group and the managed identity of the application is part of the group, it won't work, no matter which permission you grant ❌
If you assign the managed service identity the permission Azure Service Bus Data Owner it won't work ❌
If you assign the permission Azure Service Bus Data Receiver to a group and the managed identity of the application is part of the group, it will work ✅
If you assign the managed service identity the permission Azure Service Bus Data Receiver, it will work ✅

Please note it may take up to 5 minutes for the permission changes to be applied, so you may still experience failures after applying them.

The only way to get it working was to assign the managed identity the Azure Service Bus Data Receiver role to either the service identity or the security group.

This seems either a bug in the Azure SDK or in the Service Bus itself, I'm not the only one that ran into this issue and here you can find additional information.

Till the next time.

Dynamically scale down AppService outside business hours to save 💰💰

Massimiliano Donini — Tue, 26 Jul 2022 16:26:56 +0000

The other day I was on a quest to lower a bit our Azure spending.

Im my current company we have several environment that we use for different purposes, Development, Test, Acceptance and so on.

All these environments have slightly different tiers for various services and I was wondering how to lower App Service Plan tier outside business hours.

App Services have some built-in, albeit limited, capabilities to scale but this only involves scaling out.

Scaling out is the process of adding additional instances of our application to adapt to an increasing load.
Scaling up is the process of running the application on a more performant hardware.

Since there's no built-in support to scale up & down in App Services, I had to come up with a custom solution.

After a bit of research, I ended up creating an Azure automation account, two runbooks that execute on a schedule the scale down and scale up of our App Service Plan.

It turned out to be extremely simple to implement yet effective.

Disclaimer: This is just one of the possible way to scale services up & down outside business hours, you can achieve the same with a scheduled github action or Azure DevOps pipeline than runs your IaC code with different Sku parameter values for example.

Here's what I've created:

Azure automation account
Azure Runbooks
Azure automation schedule
Azure automation account variables

Azure automation account - Docs

This is the go to resource to automate processes in Azure, where you define the runbooks, the schedule and the variables.

Azure Runbooks - Docs

This is where I defined what needs to happen when the schedule triggers the runbook and starts a job.
There are several different types of runbooks, here I chose the powershell one.

Azure automation schedule - Docs

This is where I defined when to execute our runbooks, I went with a weekly schedule to scale down resource in the evening and scale them back up early in the morning.

Azure automation account variables - Docs

This is where I defined few variables used by the runbook.
This step is optional since you can potentially hardcode everything in the runbook itself, but if you want to use the same runbook across different environment, you can define variables and read them in the runbook.
I defined few variables, one for the resource group name, one for the app service plan name and the desired scale down sku and the one the needs to be used during business hours.

In order for the runbook to successfully change the App Service Plan, we also need to grant the identity - either managed identity or user assigned one - of the automation account enough grant on the App Service Plan. I went with managed identity.

Putting it all together

After creating the variables, the schedules and the runbooks I linked the schedule to the runbook.
I created two schedules called scale-down and scale-up, two runbooks named the same way and linked the schedule to the runbook. You link a runbook to a schedule in the overview page of the runbook itself.

Last missing part is the code of the runbook itself, so here's the shortest possible version of it (of course you can make it smarter based on your needs) used to scale down, the scale up version is exactly the same but read a different variable for the sku.

Till the next one!

Zero downtime deployment with Azure Container Apps and Github Actions - Part 1

Massimiliano Donini — Fri, 24 Jun 2022 16:29:35 +0000

Introduction

As you may know, Azure Container Apps went out of preview during Microsoft Build in late May this year.
Azure Container Apps is a very interesting service that runs on top of Kubernetes adding some additional powerful capabilities in a simple and covinient way.
Some of these capabilities are:

Built in support for Keda autoscalers
Built in support for Dapr components
Ability to scale to zero

There's a lot more to it, you can dig deeper on the official Microsoft documentation here

TL;DR; All the code described in this article is available on Github here
I'm still working on improvements so main branch may be updated by the time you read this.

Requirements

I would like to migrate several Azure App Services to Azure Container Apps, I have two different type of services, http api that write to a Service Bus queue/topic and web jobs that consume and process the messages, a pretty common setup these days.

Azure Container Apps allows me to easily scale the web jobs based on the amount of messages present in the queue/topic and also makes it easy to scale to zero outside business hours.

In order to migrate from Azure App Service to Azure Container Apps I want to implement a zero downtime deployment for the http api services.

Problem

Azure Container Apps has built in support for health probes, there are 3 types of health probes:

Liveness
Readiness
Startup

Given the built-in health probes support, I went under the assumption that, while using single revision mode, we could just deploy another revision and the control plane could take care of warming up and then swapping traffic to the new revision without downtime.
To my surprise I figured out that's not the case and if you're reading this blog post, you might have noticed that too.

I also double checked it on Discord with the Azure Container Apps team if I was missing something on my end, but they confirmed my findings.

After doing a bit of research, I figured it out that I can implement my own workflow to implement zero downtime deployment. This is far from ideal but still better than a deployment process that causes downtime.
Hopefully Azure Container Apps will implement built support for zero downtime deployment, but in the meantime the following approach is an acceptable workaround.

Solution

In order to implement zero downtime deployment in (pseudo) single revisions mode (meaning using multiple revision mode with a single active revision at a time, serving all the traffic), we need to do the following steps:

Redirect all the traffic to the latest revision by name (more on that later)
Deploy a new revision
Warm up the new revision
Redirect the traffic to the newly deployed revision

In this blog post I am using multiple revision mode because it happened to me that moving from single revision to multiple revision mode caused downtime - I'm currently investigating it and will update this post as soon as I found.
The idea is to have the container app configured in multiple revision mode but only have one active revision at a time.

Regarding step 1, Redirect all the traffic to the latest revision by name, has to do with the latest revision alias.
I found that if the traffic is set to the latest revision, when deploying a new revision, the traffic get redirected to the new revision even while it's still in the provisioning state, leading to possible timeouts and failures on the caller side.

The workflow above is a bit long but not particularly difficult, and we can easily implement it with the help of the Azure Cli containerapp extension.

Out of all the steps above, point 3 (Warm up the new revision) is the most tricky since different services may have different health probe configurations. I really didn't want to duplicate health probe configurations in the infrastructure and in the deployment pipeline but rather reuse what has been configured in the Container App Container instead.

Dynamically discovering and calling the health probe in bash is doable but I am a bit more proficient writing that code in a high level language so I decided to write a small C# application to do that.

In order to manage Azure resources, we can use the Azure Management SDK, this is a set of packages that allows you to manage Azure resources in your language of choice.
You can find all the supported resources here.

Luckily, Azure Container Apps already have an SDK available, although in beta at the moment of writing.

Since I should be able to invoke this application from a Github Action, I decided to implementing a web application that exposes an api to make my life easy.

Essentially all the steps described above but steps 4 are executed in a github action, while step 4 is executed by a web application invoked by cURL in the Github Actions.

The next point to solve is where should be this web application deployed, we have few options:

Have it always available
Make it available on demand

If you deploy an internal Azure Container App Environment and the application is not exposed to the internet, point two may be a bit more complicated since you need to make sure the github action runner can reach the Azure Container App you want to warm up.

I initially went with the first approach, having the application always available, but since I didn't want to also add authentication to the mix (to make sure only verified clients can call the warmup endpoint), I later decided against it.

In order to make this application available on demand, I decided to use Github Actions service container.
This post is getting already a bit too long so I won't go into detail of what service container is, let's just say that it allows you to run a container and made it available on the runner. If you wanna dig deeper, you can check the documentation here

After sorting the Github Actions service container, last problem that I had to tackle was how to authenticate the application against Azure.
Thanks to Azure.Identity package and the DefaultAzureCredential class, we can just set some environment variables and authenticate with a previously defined service principal.

Azure management SDK authentication

In order to get the required credentials to authenticate, we need to create a service principal with the Reader role on the resource group that contains the Azure Container Apps.
We can quickly create this with the az cli and the following command:

az ad sp create-for-rbac -n "HealthProbeSp" --role Contributor --scopes /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupId}

Remember to replace the subscription and resource group names

After setting all this up, I was able to implement zero downtime deployment.

Here's an extract of the Github Action:

jobs:
  build:
    name: build ${{ matrix.services.appName }}
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read
    services:
      health-invoker:
        image: ghcr.io/${{ github.repository }}/health-invoker:main
        ports:
          - 5000:80
        env:
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
          AZURE_CLIENT_ID:  ${{ secrets.AZURE_CLIENT_ID }}
          AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
          Azure__SubscriptionId: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
          Azure__ResourceGroupName: ${{ secrets.RESOURCE_GROUP_NAME }}

# Clone repo, Build and push omitted for brevity 

      - name: Deploy azure container app without downtime
        if: github.event_name != 'pull_request' && matrix.services.zeroDowntime == true
        run: |
          echo "Installing containerapp extension"
          az extension add --name containerapp --upgrade &> /dev/null
          echo "Get latest active revision name"
          latest_revision=$(az containerapp show -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --query properties.latestRevisionName -o tsv)
          echo "Redirect traffic to active revision $latest_revision"
          az containerapp ingress traffic set -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --revision-weight $latest_revision=100 &> /dev/null
          echo "Create new revision"
          az containerapp update -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} -i ${{ steps.image-tag.outputs.tag }} &> /dev/null
          new_revision=$(az containerapp show -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --query properties.latestRevisionName -o tsv)
          echo "Warmup new revision at ${{ env.WARMUP_APP }}/warmup/${{ matrix.services.appName }}"
          health_response_status=$(curl -m 180 --write-out "%{http_code}\n" -s ${{ env.WARMUP_APP }}/warmup/${{ matrix.services.appName }} --output backend.txt)
          if [ $health_response_status = "200" ]; then
            echo "Redirect traffic to new revision $new_revision"
            az containerapp ingress traffic set -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --revision-weight $new_revision=100 $latest_revision=0 &> /dev/null
            echo "Deactivate revision $latest_revision"
            az containerapp revision deactivate -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --revision $latest_revision &> /dev/null
          else
            echo "Warmup failed with status code $health_response_status"
            cat ./backend.txt
            echo "Redirect traffic to active revision $latest_revision"
            az containerapp ingress traffic set -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --revision-weight $latest_revision=100 &> /dev/null
            if [ ! -z "$new_revision" ]; then
              echo "Deactivate revision $new_revision"
              az containerapp revision deactivate -n ${{ matrix.services.appName }} -g ${{ secrets.RESOURCE_GROUP_NAME }} --revision $new_revision &> /dev/null
            fi
            exit 1
          fi

Here's the output of wrk while deploying a new revision:

wrk -t12 -c400 -d30s https://xxxxxxxx.azurecontainerapps.io/api/echo/ping
Running 30s test @ https://xxxxxxxx.azurecontainerapps.io/api/echo/ping
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   268.53ms  123.73ms   1.02s    68.90%
    Req/Sec   137.15    103.17     1.15k    66.59%
  41604 requests in 30.10s, 8.89MB read
Requests/sec:   1382.32
Transfer/sec:    302.38KB

After deployment has been completed:

wrk -t12 -c400 -d30s https://xxxxxxxx.azurecontainerapps.io/api/echo/ping
Running 30s test @ https://xxxxxxxx.azurecontainerapps.io/api/echo/ping
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   251.00ms  135.47ms   1.24s    77.22%
    Req/Sec   148.38    104.52   434.00     63.10%
  44970 requests in 30.09s, 9.61MB read
Requests/sec:   1494.36
Transfer/sec:    326.89KB

As you can see there's almost no difference and, most importantly, wrk doesn't indicate any non 2XX or 3XX response meaning that we were able to serve all requests while deploying a new revision.

The numbers are quite low because I'm using a very small configuration for testing purposes (0.25 Cores and 0.5 Gi of memory)

I hope you find this helpful and if you have suggestions , don’t hesitate to comment or reach me out via twitter at twitter.com/maxx_don.

Stay tuned for part 2 that will be out very soon!