The Ops Community ⚙️: Javier Marasco

Kubernetes probes easily explained

Javier Marasco — Sun, 24 Dec 2023 11:30:39 +0000

Recently I needed to work with kubernetes probes (to implement them actually) and came across some questions about them, I think it is a good idea to write a quick explanation without being too deep in the technical aspects of it but more on the why they exist and what are they used for, let's take a look.

In Kubernetes you deploy your applications in forms of images which are a single file that contains your application code and all needed dependencies for it to run.

Idealy your application will start right away and put in place to receive requests (from other containers in your cluster or from the outside world) but this some times is not so quick, your application might need some extra time to start and during that time it can't respond properly to requests, let's see an example.

Imagine your application is exposing and enpoint to the world and other applications will connect to it and send a certain payload in a POST request to that endpoint. Your application before receiving or even be able to respond to a request, it needs to do some preparations, it needs to connect to certain service to download a configuration file, then it needs to connect to another system to retrieve some secrets to connect to other services (SQL server for example) and open some connections to a database. While doing all this your application can't do much, any request you send to it will simply fail as your application is not running yet, it is initializing, so if Kubernetes tries to route traffic to it, your consummers will simply see an error. Now what?

How does traffic reach your application?

When you create a pod (wrong, you should run deployments and not pods) it has it's own IP inside the cluster, you don't send traffic directly to your pods as they can be destroyed at any time and the new ones can have a different IP, so it is not reliable to expect a pod to get traffic, instead we have services which are another Kubernetes resource which has a name (fqdn in the form ) which your other pods or ingresses should use to reach your pods, this approaches ensures you always reach your application using a reliable approach (an fqdn). Each service contains a kind of "backend pool" where your pods are assigned to receive traffic, they are assigned to this pool as soon as they are ready, and this is the tricky part, how does Kubernetes knows your applications is ready?

Startup, liveness and readiness probes

Kubernetes defines three types of probes to check the status of your pods, let's view what each one of them does and when they are executed.

Startup probes
This is the first probe to be executed (if defined) and it only runs once when the pod is started, the intention of this probe is to wait until the pod is ready to start working, but watch out, this doesn't means the application is ready to receive traffic, it means the code inside the image is ready to start working, the main intention of this probe is to wait before start running other probes (liveness and/or readiness) ensuring kubelet doesn't kill the pod before it even start operating (this is specially true in slow starting applications).

liveness probes
This is a probe that (again, if defined) will run constantly during the whole life of the pod, this is intended to allow kubelet to know when to destroy a pod and create a new one, when defining a liveness probe keep in mind it should return a success code constantly while your application is running, do not configure it for something that is only "true" while starting or a short period of time as that will cause kubelet to kill your pod when that test if failed but your application might be totally healty.

Readiness probes
This probe is probably the most important one for the topic we are covering (knowing when your application can accept traffic). The readiness probe will be executed only once just after your startup probe (if defined) and once it completes it will let Kubernetes determine what to do with your pod. If the probe is a success the pod IP will be added in the backed of the apropriate service to start receiving traffic, if the probe is failed Kubrenetes will kill your pod and start a new one (which will do all the probes defined again), this will prevent an unhealty pod to receive requests.

Wrapping up
So, basically our deployment should specify the three probes, the "startup" will fire as soon as the pod is scheduled into a node, upon completion a "readiness" probe will be fires which can return a success code which will add the IP of the pod into the service backend but if it fails kubelet will kill the pod and restart the process again. If the readiness probe completes then your application is healthy and will start receiving traffic but then during it's whole life kubelet will keep executing periodically a "liveness" probe on the pod to determine if it is needed to kill the pod or not, on the first failure from the liveness probe, your pod will be restarted.

Reference material
This is a short article to summarize the concept of probes in Kubernetes, there are a lot of technical details on how to properly implement and configure them, if you need more details on it, you can find it in the official documentation here.

Managed cluster vs unmanaged clusters in Kubernetes

Javier Marasco — Tue, 27 Jun 2023 08:30:00 +0000

As we see in the latest article here a Kubernetes cluster can contain a lot of components just to function properly and as you can imagine, install, maintain, update, upgrade and operate such an infrastructure might be haunting.

What does it look like to build a Kubernetes cluster?

As discussed there are two kinds of nodes for our cluster, the master node also known as control nodes, and the worker nodes. Each of those nodes contains different components that will be supporting our cluster, a minimum configuration will have one control node and one worker node, which can be physical machines or virtual machines connected by a network.

The control nodes will run the API server, an etcd database, the scheduler, and some controllers while the worker nodes will run only tree components being them the kubelet, kube-proxy, and the container runtime.

This might sound simple, right? but then you need to configure all those components, you also need to do all the networking needed for those nodes to communicate, and in the networking part there is one additional thing, you will (eventually) need to expose your applications to the internet using an ingress controller which will require to have an external IP exposed to the internet from where all the incoming traffic to your applications will ingress your cluster, this means you will need to administer the creation (and maintenance) of those public IPs.

This becomes a complex task pretty quick but if this is so complicated, why would anyone want to build a cluster for themselves? well, the answer is pretty easy, security when you build your cluster, you keep control of everything, patching, kernel versions, os versions, packages in the nodes, everything is possible to customize to your needs. It might sound like overkill but think in heavily regulated industries like the financial sector, there you would like to have control over every piece of your infrastructure, you don't want to have a bug in a library running in the OS of your worker node that allows malicious actors to exploit it and gain root in one of your pods.

Managed clusters and the cloud providers offerings

So, we are not working in a highly regulated industry but we do need one or more clusters and we don't want to spend months learning how to build a cluster and then need to maintain them manually, what can we do?

Luckily each cloud provider (at least the most popular ones) does offer automated ways to build a cluster for you and give you access to it to simply deploy resources into them. This is very convenient for most of the scenarios where a Kubernetes cluster is needed, even for production workloads, those cloud providers have (normally) multiple levels of SLAs for those clusters meaning that if you are running a development cluster you can opt for a lower SLA but also a cheaper billing while for production workloads you can opt for a higher SLA which includes highly available resources but that will also cost you more, but you can choose what is best for your case and environment.

Those managed clusters have a clear downside, and that is the fact that the cloud providers will retain the control of the control plane and their control nodes meaning you can't decide on how to configure it and even when it is possible to adjust and tune your worker nodes (even apps, libraries and kernel configurations) as soon as you modify them, you loose support from the cloud provider. This might sound like a bad move from the providers but think what would it mean for them to support every possible change made in any cluster in the world? it would be impossible for them to provide proper support so they stick to a proven configuration and they support it, if you move away from that you are on your own.

Patching and upgrading your cluster

We discussed a lot about building and configuring your cluster, but what about version upgrades? what happens when Kubernetes releases a new version? well, here is again a big difference, cloud providers tend to be a bit delayed with the latest version being released because they need to test it first, pass certain internal validations, and then expose them for you to consume in a trustable manner, this takes time meaning if you are wanting to implement a super shiny new feature of the latest release of Kubernetes and you are running a managed cluster you will need to wait a bit before this version is available for you to pick it up, while in a managed cluster you can upgrade whenever you want.

But what happens if something goes wrong with my upgrade? I moved to a newer version and now everything is broken (see the note after this paragraph for a good example). In a managed cluster it is very complicated to revert to an older Kubernetes version I would say it is impossible but I actually know cases where the cloud provider managed to revert the version in VERY specific cases but I would take this as "something you can't do" but in your unmanaged cluster it is as simple as just reinstall the older version or revert to a backup snapshot from before your upgrade (because of course you do backups before upgrading your cluster version) and you are again in a case like nothing happened, very easy.

A note about this, recently with the introduction of version 1.25 there was a change in a configuration called cgroups in the kernel of the OS being used by Kubernetes, this caused some applications using an old (but still functional) version of their frameworks to start consuming a lot of memory and making the kubelet constantly kill those pods, a simple solution is to upgrade your framework in your app, but if that is not possible you can also revert the change in the worker node kernel, but this last option will make you unsupported for your cloud provider.

So, what is the takeaway of this article?

Essentially you now know what a managed and unmanaged cluster, you know what are the advantages and disadvantages of each one, and also you know what it means to have a managed cluster and what can you do with it.

The main point here is to know your situation, know your use case and know what you expect from Kubernetes and your cluster/s. Will you need to meet very strict standards and provide proof that you have certain configurations? then you are tied to an unmanaged cluster, you will need to learn and understand the details of Kubernetes and manage all by yourself, which might sound haunting but with time and training, you will be perfectly fine.
If on the other hand, you will be deploying applications that you know are secure, you don't need to provide configurations to audits and you can rely on a cloud provider having control over your control plane, you are more than fine with a managed cluster and you can skip the deep part of Kubernetes for now (it is always advisable to learn the concepts anyway, but you will not be rushing to learn and understand everything because you need to build you cluster quickly).

So what are your thoughts? would you prefer a managed cluster? an unmanaged cluster? would you feel comfortable knowing your control plane is not under your control? Let me know in the comments box 👍

Thank you for reading 📚 !

What is Kubernetes, how does it works and why do we need it

Javier Marasco — Mon, 19 Jun 2023 15:15:42 +0000

A lot of people are getting into Kubernetes and the first thing they normally do is google "How to deploy an application to Kubernetes" which could make sense but then you will find thousands of articles (including the official documentation) explaining how to deploy an application, but then you see "deployment", "replica set", "pods", "resource quotas", "secrets" and everything starts getting confusing, complex and not making sense and then you get overwhelmed by the amount of information.

I believe the first thing to start with a technology that is new to you is to understand why such technology exists, what problems it solves, and how it works (conceptually), this will let you have a clear picture of the technology and will ultimately let you decide if it is the best fit for your particular case, so let's start with that.

A little of the history and background

Previous to Kubernetes applications used to be deployed in virtual machines (and previous to that into physical machines) but those machines needed to have libraries, dependencies, networking configurations, etc. You can imagine managing that was very complex and changes needed a long time and coordination between multiple teams. Then in 2013, there was a presentation that changed everything, a person called "Solomon Hykes" presented a new project his company (with two other persons) was working on, it was Docker.

The key difference with Docker was nothing new actually, was a set of capabilities that already existed in the Linux kernel for a long time but now it was being exposed to the application level in a more comprehensive way, with this "Docker" tool, it was possible to pack your code with all dependencies in a "container" which you could take to any other system where Docker was running and it will behave the same, that was exactly what was needed!

So now a few months/years had passed and you have containers everywhere, everyone is happy with the approach but the more containers you have it becomes a bit more complicated to manage, so we started using "docker-compose" which was a way to group containers into "logical units" to deploy them together and have some sense of integration between them, quickly it was obvious that there was a need for some way to manage large amounts of containers, then "Docker Swarm" appeared which was a tool to orchestrate the deployment of complex applications with multiple containers. This worked fine as an orchestrator but then in 2014 Google releases Kubernetes as an alternative to Docker Swarm but with more functionalities, more features were added with time and the community adopted Kubernetes as the de facto orchestrator, Kubernetes used Docker under the hood for a quite long time to execute the containers the same as Docker Swarm but later the community of Kubernetes decided to make it possible to use any container engine you want so they adjusted Kubernetes to support virtually any container engine removing the need to have Docker as the only container engine.

But... how does it work?

Kubernetes at the very top level is very simple in concept, you have your image (your application code and dependencies packed into a single file made of multiple "layers") and you will deploy it into a set of machines running "something" that will take your image and make it run this "something" will take care of checking your image is running in a healthy machine, will restart the image when something bad happens to it, will kill it and restart it if it starts consuming too many resources, it will handle communication in/out from the world to your image, etc.

To do this, Kubernetes has its own internal components, being them:

API Server
Scheduler
Kubelet
Kube proxy
etcd

The API Server is the interface between Kubernetes and the rest of the universe, every time you run a command using kubectl (the CLI to manage any Kubernetes cluster) it will make a REST call to the API server of your cluster and provide the parameters and files you pass to kubectl as the payload of the command.

The Scheduler is a process that will take the container (note I am not talking about an image anymore, more on this later) and will validate which node in the cluster meets the needs to have that container running (resources, exclusions, affinity, etc.)

Kubelet is a process running in each "worker" node that will do the actual work of making your container run, it will do all the needed tasks to ensure your code runs in the node, resource allocation, resource monitoring, process handling, etc. this is a key part as it also is responsible to support different container engines.

Kube proxy is a component also running on each worker node that will take care of all the networking work for our containers, one important piece of information about kube proxy that many people get confused about is that it will not route or be in the middle of the traffic at all, kube proxy does the needed networking configuration in the worker node for the traffic to reach the correct container (adding and removing rules from iptables for example) which means that it can crash and the applications running in your node will continue to work.

And of course, we have a lot of components in the control nodes and the worker nodes, we have container running, network routes defined and a lot of information about our infrastructure but how does the API server "remembers" all this? well, there is where etcd enters into the scene, etcd is a highly available and scalable key/value data store (what a lot of words, no worries, is not that complex) which takes the role to store all the cluster state constantly, it contains information about your cluster and the API server is the one updating it constantly.

Now you have a basic understanding of how is it possible for Kubernetes to run a container, by knowing this you can decide if Kubernetes is the right tool for your application.
I often tend to not recommend Kubernetes for small applications that are just in the initial stages of being deployed or if your application is already running in another infrastructure and the migration to Kubernetes will only bring you more work to do without much gain.

Just remember Kubernetes is a tool and like any other tool it serves a purpose and that should drive your intentions to move into it or not, adopting Kubernetes is a task that will require a lot of investigation, learning, and effort (translated into time), be mindful to move to Kubernetes only if it makes sense for your use case.

I hope this first article of the series helped you to understand the basics of Kubernetes and decide if Kubernetes is the right choice for your next steps.

If you enjoyed this article please consider following me to get alerted on every new article and consider leaving a message with your ideas and/or feedback.

Thank you for reading!

The ever-growing kubernetes manual

Javier Marasco — Mon, 19 Jun 2023 07:17:14 +0000

After several months not posting and trying to think a way to articulate this idea I had in mind, I think I got how to present this to the community.

This post will serve as an index for a series of other articles about kubernetes from scratch, I am planing on writing a more elavorated version of it (possibly in a ebook format) but I also wanted to give a free version to the community with the basics of Kubernetes in an easy to follow format clarifying the basic concepts you will need to go from 0 to a degree where you will feel confortable enough to deploy apps to Kubernetes and do some basic troubleshooting by using a easy to follow narrative to not only read technical information but also to nuderstand the reasoning behind it.

I will try to add more articles into the list so if you feel there are items not being presented please let me know and I will add them into the index.

So, let's jump into it:

1 - What is Kubernetes, how does it works and why do we need it
2 - Managed cluster vs unmanaged clusters
3 - Basic elements of an application running in Kubernetes
4 - Deployments, replicasets and pods
5 - Services, networking and ingress
6 - Storage classes, volumes and how to store your app data
7 - Secrets, config maps and how they are used

I will be adding those articles soon and update this index with the links to the corresponding articles, follow me to get notified every time I upload one of them and don't miss a single post :)

As always, comments and feedback is always welcome and appreciated.

See you later!

Terraform providerless (or how to work with non terraform resources)

Javier Marasco — Sat, 03 Dec 2022 18:21:28 +0000

Intro

A few days ago I needed to integrate a service with Terraform in a way that allows my team to create infrastructure (with Terraform) while storing some information of that infrastructure in a remote service which doesn't have a Terraform provider, I also wanted to include the possibility to extract some information from this remote service to use in the resources managed by Terraform.

It is not important which service I was working with but the fact that you can interact with any external service that exposes an API or that you have access to query (and write) to it.

By the end of this tutorial you will be able to write a Terraform module that will allow you to read and write to an external service that doesn't have a terraform provider.

Initial investigation

While thinking on how to integrate this provider-less service I found that Terraform does have a way to interact with external services, it is by no means the same as a fully Terraform provider, the main difference is that with a provider you can manage the complete lifecycle of the resources managed with Terraform very easily while with this approach I will explain it is a bit limited, functional but limited.

We will be using two scenarios in this tutorial, how to read information from an external resource and how to write to that external resource. Both approaches use a different implementation that we will be covering here.

Let's read!

For this tutorial I will use the pokemon api as an external resource to retrieve the name of the first ability of a pokemon and use it as name of my terraform resource.

The first thing we need to know is that Terraform has something called External Data Source which allows you to execute an script and send back to terraform the output of this script, we will need to write an script that connects to the API and returns something that out data source will expose later, here is an example:

/module/poke.ps1

$stdio = [Console]::In.ReadLine()
$stdio = ConvertFrom-Json $stdio

$url = "https://pokeapi.co/api/v2/pokemon/$($stdio.name)"
$response = Invoke-RestMethod "$url" -Method Get;
$ability_name = $response.abilities[0].ability.name

Write-Output "{""value"" : ""$ability_name""}"

This simple script will read from the stdio and parse it as a Powershell object to extract the name of the pokemon and then access our external API, then will retrieve the name of the first ability and write a json object to stdio containing this value.

This is basically what the terraform documentation details on how this works. Let's write now a terraform module to use this:
/module/poke.tf

variable "name" {
  type        = string
  default     = ""
  description = "Name of the pokemon"
}

data "external" "read_ability" {
  program = ["powershell.exe", "./main.ps1"]
  query = {
    name    = var.name
  }
}
output "content" {
  value = data.external.read_ability.result.value
}

Whit this terraform code we can pack this .tf and the .ps1 files into a single folder and use them as a module from other .tf files as follow:
/main.tf

module "read_ability" {
  source = "./module"
  name = "ditto"
}

resource "azurerm_resource_group" "example" {
  name     = module.read_ability.content
  location = "West Europe"
}

keep in mind the folder structure and where each file needs to be to make this work.

When you run a Terraform plan on the main.tf file you will see the name of the resource group will be limber, in this way you can pass other names to the poke module and obtain different names, of course this is a very limited example just to demonstrate how this works but you are encouraged to do your own implementation with more functionality.

Let's write

Now that we can read from our external service, we can create resources in Terraform using that information as input, but what if we want to store something we created with terraform in an external service. Let's suppose we have an external store and we want to keep track of the ID of some resources created in Azure in that store. For this example we will be using kind of the same approach as for read , this means we will create a terraform module that will take an input and write it to some external service.

/module/write.ps1

param (
  $Value
)

# Implement here your code to write to the external service
# here we only do a write-host to demonstrate the functionality
Write-host $Value

/module/main.tf

variable "name" {
  type        = string
  default     = ""
}

resource "null_resource" "write_to_external_service" {
  triggers = {
    "command" = "${timestamp()}"
  }
  provisioner "local-exec" {
    command     = "powershell -file write.ps1 -Value ${var.name} "
    working_dir = "${path.module}"
  }
}

main.tf



resource "azurerm_resource_group" "example" {
  name     = module.read_ability.content
  location = "West Europe"
}

module "write_value" {
  source = "./module"
  name = resource.azurerm_resource_group.example.name
}

The /module/main.tf is using a null_resource to execute a command and passing a single argument to the script, in this case "name" and the trigger will force to run every time this null resource is called, be aware you need to implement in your script a mechanism to not write multiple times the same outputs to your external service, but that is up to you and the way your external service works.

Wrapping up

So in this short tutorial you ended up with two modules, one to read from the PokeAPI and another to write to an external service, in both cases using a Powershell script to interact with the external services, this is an easy way to integrate a service without a terraform provider into a terraform deployment without the extra effort that means build a fully functional terraform provider (which of course is the prefered way to deal with external sources).

As a workaround to quickly get it to work, this is a very good alternative that Hashicorp provides us, I hope this helps you and in case you find this useful, please consider to share it with others so we all benefit from others contributions!.

Follow me in my other networks and get in touch if you have any question or need help 👍

Prometheus en AKS (en español)

Javier Marasco — Fri, 16 Sep 2022 07:24:29 +0000

Introducción

La intención principal de esta publicación (así como de las demás) no es tener un tutorial muy profundo en todos los detalles de cada tecnología/aplicación que se describe, sino una guía breve y concisa de algo en particular que puede ayudar a las personas que están comenzando a conocer el tema descrito en este artículo, por supuesto, si tenés alguna recomendación para mejorar esto, no dudes en enviarme un mensaje/comentario.

¿Qué es Prometheus?

Arranquemos, vayamos directamente al tema que estaremos hablando hoy acá, Prometheus es una herramienta de monitoreo que es muy popular en el mundo de Kubernetes al igual que una de los proyectos de la CNCF (Cloud Native Computing Foundation) y esto significa que es un producto muy maduro con un gran apoyo de la comunidad.

Hay varias características que hacen de Prometheus una de las herramientas favoritas para monitorear entornos, algunas se enumeran aquí:

Metodología "pull" (significa que el servidor de Prometheus extrae métricas en lugar de esperar a que la aplicación envíe las métricas al servidor de Prometheus)
Muy rápido al recopilar métricas y realizar agregaciones
Una arquitectura que permite monitorear no solo los entornos de Kubernetes sino otras aplicaciones como bases de datos o servidores web (usando "exporters")
Soporta cientos de aplicaciones, hardware, plataformas, etc. para monitorear aquí está la lista

Con esto en mente, continuemos instalándolo y configurándolo para un primer intento.

Cómo instalarlo

Para nuestro ejemplo, instalaremos Prometheus en un clúster de AKS (clúster de Kubernetes que se ejecuta como PaaS en Azure).

Hay varias formas de implementar Prometheus en un clúster de Kubernetes, pero la más simple y la que tiene más sentido es utilizar algo llamado Helm charts, imagina a un Helm chart como un conjunto de archivos YAML vinculados como un único recurso para implementar, esto significa que lo instala como un elemento "único" (un chart) pero, en su lugar, implementará los recursos necesarios en tu clúster para que la solución funcione correctamente (como pods, replica sets, deployments, services, secrets, etc.)

Qué es Helm

Helm es una tecnología que nos permite empaquetar un montón de archivos YAML que se usarán como un todo para hacer que una solución funcione (en este caso, la solución es Prometheus), al usar Helm estás eliminando la complejidad de tener que administrar de forma independiente todos los recursos para que la solución funcione, en su lugar, proporcionas un archivo de configuración al chart y lo implementás, que creará todos los recursos y los configurará correctamente.

Una de las ventajas de usar Helm es que se puede confiar en los repositorios donde se mantienen esos charts y usarlos, pero también si querés podés mantener tu propia versión del chart localmente o en un repositorio privado, para que puedas ajustarlo a tus necesidades (como usar una imagen de container personalizada para los deployments del chart en lugar de usar la predeterminada, esto podría ser un requerimiento de seguridad, por ejemplo)

Otra característica es que se puede almacenar esos charts en los mismos repositorios donde se almacenan sus imágenes de contenedor y versionarlos de la misma manera que lo haces con una imagen de contenedor.

Cada chart tiene su propio conjunto de archivos porque representan un grupo de recursos que funcionan para hacer que una aplicación funcione, por lo que los recursos necesarios para un chart para Prometheus no son los mismos recursos necesarios para cert-manager, por ejemplo, pero la idea es la misma, un conjunto de archivos YAML que una vez implementados trabajarán juntos para hacer que la aplicación se ejecute.

Para utilizar los charts de helm, tenés que tener helm instalado en tu sistema y agregar los repositorios que utilizará, cada chart de helm vive en un repositorio que se debe agregar para descargar el chart y sus archivos.

agregando un repositorio *

Para este artículo usaremos los charts de Prometheus estándar.

Requisitos previos

Bueno, para este artículo necesitaremos:

Un clúster de AKS en funcionamiento, nada especial, solo la implementación base está bien, podés seguir este tutorial (Haré un tutorial en el futuro)
Una terminal con kubectl y helm instalados
Instala el repositorio de helm de prometheus-community

Instalación

Lo primero que haremos es agregar el repositorio de helm de prometheus-community y actualizar nuestra lista de repositorios locales ejecutando el siguiente comando:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Ahora podemos revisar todos los gráficos que podemos instalar ahora que agregamos el repositorio, veamos:

helm search repo prometheus-community

¡Perfecto! Vemos muchos charts allí, esos son elementos diferentes que podemos instalar, pero hoy nos centraremos en el llamado "prometheus".

Hay un nombre de columna ** CHART VERSION **, esta es la versión del gráfico en sí y no de Prometheus, esto se debe a que se pueden realizar modificaciones en la forma en que está compuesto el gráfico y lo que hay dentro, pero aun así usar la misma versión de Prometheus que la versión de gráfico anterior. Se pueden ver todas las versiones de charts ejecutando:

helm search repo prometheus-community/prometheus -l | grep -v "community-prometheus/prometheus-"

El comando grep es para eliminar todos los demás charts de la lista *

Si instalas Prometheus sin decir la versión del gráfico que deseas, se instalará la última (14.11.1 en el momento en que escribo este artículo).

Vamos a instalarlo ahora:

helm install prometheus prometheus-community/prometheus

el "prometheus" antes del nombre del repositorio/gráfico es el nombre que queremos darle a esta implementación, podés elegir otro nombre.

Ahora que tenemos nuestro servidor Prometheus instalado y listo, verifiquemos en Kubernetes lo que implementamos (tenía el espacio de nombres "default" seleccionado al instalar el gráfico, por lo que mi gráfico se implementó en el espacio de nombres "default")

** Una cosa importante es que cuando instala un gráfico de helm, se instala en un namespace en Kubernetes, si cambia a otro namespace e intentas ver los gráficos instalados, no verás el que instalaste en el otro namespace. Dejame poner esto en una imagen para explicarlo mejor: **

Esto es un helm list en mi namespace" default "
Esto es un helm list en otro namespace

En Kubernetes podemos ver todos los recursos creados automáticamente:

kubectl get all

Siguiendo los pasos descritos después de la instalación del gráfico de helm, debemos reenviar un puerto desde nuestra máquina al pod donde se ejecuta el servidor Prometheus con:

export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus, component=server" -o jsonpath="{. items [0].metadata.name}")

kubectl --namespace default port-forward $POD_NAME 9090

** Si estás utilizando WSL para ejecutar tus comandos de Kubernetes, debes realizar algunos pasos adicionales para que esto funcione **

Primero verifica cuál es la IP de tu WSL ejecutando wsl hostname -I, ya que no se puede acceder a los puertos en tu máquina host (Windows) ejecutando localhost: port si está exponiendo los puertos dentro de WSL.
En segundo lugar, el comando port-forward debe incluir --address 0.0.0.0 como kubectl --namespace default port-forward --address 0.0.0.0 $POD_NAME 9090
En tercer lugar, debe usar la IP de WSL (la del paso uno) en lugar de localhost para acceder a Prometheus

Con esto ahora podemos ir a nuestro navegador y acceder a localhost: 9090 para ver este dashboard:

Prometheus tiene muchas métricas predeterminadas que se recopilan de forma predeterminada, para verlas, podes comenzar a escribir algo en el cuadro de búsqueda y se completará automáticamente con las métricas disponibles.

Como ejemplo:

Y ahora, lo único que queda es profundizar en las métricas que Prometheus está recopilando, tal vez agregando algunos exportadores, ¿configurar el administrador de alertas tal vez? (este es un tema para otra publicación), ¿consumir esas métricas desde Grafana? (El artículo sobre esto ya está en camino : guiño :)

Últimas palabras

Espero que esto te ayude a comenzar con Prometheus, ya que es muy simple de implementar y al mismo tiempo muy poderoso, si tenés algún problema siguiendo esta guía o alguna recomendación, hacémelo saber en la sección de comentarios.
Aprovecho y les dejo un link a mis redes sociales donde tambien publico algunos tips.

Saludos!

Let's test our configurations with Powershell and Pester

Javier Marasco — Tue, 13 Sep 2022 13:25:55 +0000

I tend to automate everything, it makes sense that if there is something you are requested to do more than once and the time you need to invest to automate it is not huge, you will spend some time automating it. But I often found myself having a configuration file to not need to deal with modifying my scripts, I simply create a script that does the job and provide the script with configuration files as input.
This is a very nice approach when you don't want to have parameters in your scripts as well.

But then, you communicate this in your company and more people start using your automation, you of course know how the config file should look like, but what if others are not aware? what if that automation ends up in a pipeline? you have the usage documented (of course you do!) but probably others are not aware of it.

So.... how do you ensure your script is used in a safe way and your configuration file is honored? well, keep reading and I will show you how I do it :)

General idea

Let's think that we have a terraform file that will build some resources in the cloud, and you are providing this terraform plan a JSON file, inside the terraform plan you decode the JSON and use the contents to provide of values to you resources.

Since this JSON is something you create, the format is what ever it makes sense for your needs, it can have any format and any amount of fields, but you need to be aware that depending on the structure you give to it the tests might change a bit.

Example configuration file

For this example I will go with a general JSON file that I made for this example, if contains arrays, nested objects, booleans and arrays inside nested objects.

[
    {
        "vnet_name": "demo-vnet-name-1",
        "resource_group_name": "resource_groupname",
        "address_space": [
            "10.0.0.0/23"
        ],
        "dns_servers": [
            "10.0.1.0",
            "10.0.0.128"
        ],
        "vnet_location": "eastus2",
        "Subnets": {
            "name": "misubnet",
            "address_prefixes": [
                "10.0.0.0/24",
                "10.0.0.128/24"
            ]
        },
        "Enabled" : true
    },
    {
        "vnet_name": "demo-vneT-name",
        "resource_group_name": "resource_groupname",
        "address_space": [
            "10.0.2.0/24"
        ],
        "dns_servers": [
            "10.0.1.12"
        ],
        "vnet_location": "eastus3",
        "Subnets": {
            "name": "misubnet2",
            "address_prefixes": [
                "10.0.2.0/24",
                "10.0.2.128/24"
            ]
        },
        "Enabled" : false
    }
]

This configuration file is an array that contains 2 definitions to build two hypothetical network resources.

Thinking process

The first thing we want to do is to sit, relax and watch our configuration file, think on what make sense to validate and what doesn't, we don't want to write kilometers of tests when we don't need to validate all, maybe there are fields that are ok to have any value on them (like tags, we probably don't care if the tag is correct), while others are very important to validate, like a naming convention.

Once we saw what we want to write a test for, our next step is to write down a list of test we want to address, let's do that.

I want to test:

1) My vnet_name is not empty
2) I want to check the vnet_name is following my naming convention (very simple needs to have 4 sections)
3) My vnet_name is composed of only lower case letters
4) The location for the resources needs to be one that I "approve" to build on

With this list, we are ready to start writing our tests.

Pester

What it is

Pester is a test framework for powershell, is very easy to use, contains a lot of methods/keywords to run our assertions and it's installation and configuration can't be simpler.
Using Pester we will be writing blocks called "Descriptions" defined by the word "Describe" which are logical ways to split our assertions, inside each "Describe" block we will be creating one or more "It" asserts, each one of those will run a validation and will be reported as a "Passed" or "Failed" assertion.

In the output, when you run the "Invoke-Pester" command with the "-Output Detailed" parameter, the "Describe" block will be grouping the "It" asserts, so it will be easier to read as an output.

How to install (Windows)

To install Pester is as simple as install it from the PSGallery following this guide.

The key steps are:

1) Open a powershell terminal as administrator
2) Run Install-Module -Name Pester -Force -SkipPublisherCheck

No big mystery here, it will install pester as a module in your host and let it ready to use.

How it works

As mentioned before, writing a test is simply create Describe blocks to group similar assertions and inside those blocks, write one It block for each assertion.

For the Describe blocks, there is not much to say, you place a name on them and nothing more.

For the It asserts is different, in them you can pass a TestCase which is an array of elements that the assert will evaluate one by one or you can simply skip that and inside the It block write the code to make the validation.

When you write an assertion you want to do an operation and then pipe it to the should operator (which is part of what you install with Pester). This should operator has some parameters that you can use to describe what are you expecting the evaluation will return.

Some parameters for should are:

Be: Compares the evaluation result to a desired value
Not: Inverts the boolean of the evaluation
BeNullOrEmpty: Checks if the evaluation is an empty string or not defined at all
BeGreaterThan: Checks if the evaluation is greater than a defined value
BeLessThan: Checks if the evaluation is less than a defined value

Here you can find the complete list

Once you have your test written, you can call it by running Invoke-Pester -Path <file.ps1> and if you like the verbose output (like me) add -Output Detailed

Example Pester test

We are going to be using Pester 5.3.1 in this guide, next I will split my test in sections to explain them next:

Pre tests data

We will need some elements before we can start testing stuff, this is to define certain values for the tests.

# We retrieve the configuration to test
$configfile = Get-Content -path "./configuration.json" | ConvertFrom-Json -Depth 4

# Define the list of approved regions to deploy resources
$Regions = @('eastus2','eastus')

# We create an empty array to pass to our tests
$TestCases = @()

# We populate our test cases creating elements named "Instance" for each entry in our config file
foreach ($item in $configfile) {
    $TestCases += @{Instance = $item }
}

"My vnet_name is not empty"

Let's define the test for this

# Example to verify if the value was defined, this prevents missing important fields.
Describe "Check vnet_name is defined." -Verbose {
    It "Verify the name is set in <Instance.vnet_name>." -TestCases $TestCases -Verbose {
        Param($Instance)
        $Instance.vnet_name | should -not -Benullorempty
    }
}

In here we are using the "should", "not" and "benullorempty" to compare with the value we got from the configuration, Pester already have all this functions ready for use to use.

Check naming convention

In this one we are going to assume our naming convention is something that needs to have 4 segments separated by a "-". This is a very simple check, think that you can even validate each of those sections and see if their values are correct.
Another useful check is to validate if there are no other resources already created with this name.
Keep in mind you can have multiple "It" commands inside a "Describe" section.

# Example to verify namingconvention, this helps to enforce we don't create resources wrongly named.
Describe "Check naming convention for vnet_name." -Verbose {
    It "Verify the vnet_name for <Instance.vnet_name> matches naming convention length." -TestCases $TestCases -Verbose {
        Param($Instance)
        $Instance.vnet_name.split("-").count | should -be 4
    }
}

Only lower letters are allowed on the vnet_name

This one is very interesting, we are going to use regular expressions to check if the name we are providing is composed of lower case letters, regular expressions are very powerfull and we can write really good tests by using them to define what we are expecting.

In this example cmath is for case sensitive matches and imatch is used fo case insensitive matches.

# Example to validate our names are all lowercase, useful for resources that doesn't support uppercase
# here the cmatch uses a regular expression, this can be adjusted to match any patter we need.
Describe "Check name for vnet_name should be all lowercase." -Verbose {
    It "Verify <Instance.vnet_name> is all lowercase." -TestCases $TestCases -Verbose {
        Param($Instance)
        $Instance.vnet_name -cmatch "^[^A-Z]*$" | should -be $true
    }
}

Validate we are only deploying to approved locations

At the beginning of this section we defined a list of approved locations, we will use that to validate the location in our configuration is in that list.

# Example on how to validate a value in an array of values, like in this case where an 
# approved list of regions is given to the test to validate we build in the approved locations/regions.
Describe "Check location/region to deploy." -Verbose {
    It "Verify if region for <Instance.vnet_name> is approved." -TestCases $TestCases -Verbose {
        Param($Instance)
        $Regions -contains $Instance.vnet_location | should -be $true
    }
}

Final notes

As usual, you can find my other networks in here

If you find this useful or have any recommendation, please let me know in the comments, follow me for future posts so I know which content is more desired by the community and I can focus on produce more of it and I hope you enjoyed it.

Thanks for reading!!

Counting message in Azure ServiceBus at subscription level

Javier Marasco — Wed, 07 Sep 2022 08:20:49 +0000

A bit of background

In some applications the usage of messaging systems is a must, you need to communicate different components in your application in an asynchronous way, this means putting a message in some place hoping another piece of your application pick up those messages and process them. This is fairly simple and doesn't present a problem by itself but sometimes a piece of your application (the one that should be picking up those messages) fails and you only notice when you see the implication of those messages not being processed (which could be too late and your customers are noticing the bad effect).

Presenting the scenario

Let's suppose we have a single service bus (a messaging/queue PaaS component you can consume in Azure) with a single topic and 2 subscriptions in that topic. This is a very simple scenario but the idea is to present the way to monitor the amount of messages in each subscription to detect if our application is not processing those messages. We will have an application that will be publishing messages into one of the subscription at a regular basis and will be running always, another application will process those messages but will be doing it a bit slower than the one pushing messages, this will show the messages piling up a little bit (just to have a nice graph :) ) and then, for some wild reason, this application stops working, we should notice in the graph that the messages in that subscription starts piling up noticing that they are not being processed.
This example doesn't include an alert, but in Azure Monitor it is possible to set an alert to trigger when we detect a certain amount of messages in the subscription

The limitations currently present in Azure

At this moment, we do have a metric in Service Bus that presents the count of messages, but it is at a topic level and not at subscription level, even when the information is accessible thru the Service Bus API it is not possible to set alerts or create graphs based on that information (yet), if you are only interested in the messages at a topic level, those metrics (still in preview) are enough but if you are interested in knowing which subscription is not being processed, this article is definitely for you.

What we will need

We will need:

A Service Principal Name, which is called "App Registration" in the Azure portal. This SPN will be what we will use to interact with our Service Bus to read the message counters and to push the counter to the Log Analytics Workspace.
A Log Analytics Workspace which is a really nice resource where we can store metrics we extract from azure or even push our own metrics there (like we are about to do with the counters of the amount of messages per subscription).
A Service Bus with one Topic and 2 subscriptions in the topic, this is an over simplified scenario but will help present the case and then you can adjust to your needs.

Configuring the SPN

The first item we are going to need is a Service Principal, this is created in the Azure Active Directory under the "App Registration" section.

Configuration of new app registration

The configuration is fair simple, you need to provide a Name and later when it is created setup a password of it and you are done.

Setting a name for our SPN

Setting a secret for our SPN

At this point take note of the secret created (it is automatically created by Azure) because you will not be able to retrieve it again in the future, if for some reason you lost it or it gets compromised, you can always create a new secret and remove this one.

Also very important to note the Application ID from this SPN as you will need it in the script to identify to Azure as this SPN.

After this is completed there is an extra step that is very important, we need to provide permissions to this Service Principal to operate with our resources otherwise it will not be able to even read anything, for this example I will simplify this by giving "Contributor" in the subscription where the resources are, this is not ideal at all but for this example will be ok, please setup yours with less permissions to limit the actions this SPN can do.

For this we simply go to the Azure Subscription pane and click on "Access control (IAM)" and then add a role assignment as follows.

Define permissions for our SPN

We setup our SPN as Contributor for simplicity of this example

Setup the Log Analytics Workspace

We will need a Log Analytics Workspace to store the results of our script and later build alerts on top of it.

Once you have the Log Analytics Workspace deployed you will need to obtain it's:

WorkspaceID
Primary Key (or Secondary, it is the same)

This two items will be used to tell the script which one is the Log Analytics to store the information and to give permissions to write in it.

You will find this information in the "Advanced settings" blade of the Log Analytics.

Details of our Log Analytics workspace

Then we can go to the Logs in our Log Analytics and check what are the stored tables in there, this ones are the defaults, you will see later that we will add a new one to store our informations.

Default tables of our Log Analytics

Lets start the coding

Obtain a valid header for our calls to Azure API

Ok, now that we have the SPN setup and the Log Analytics configured we can start with the needed code to work with them.
Our first piece of code will create the needed header to interact with the Azure API to perform calls.

function GetAzureAuthHeader {
    $token = Invoke-RestMethod `
             -Uri "https://login.microsoftonline.com/$TenantId/oauth2/token?api-version=1.0" `
             -Method Post `
             -Body @{
                        "grant_type" = "client_credentials";
                        "resource" = "https://management.core.windows.net/";
                        "client_id" = $client_id;
                        "client_secret" = $client_secret;
                    };
    $header = @{
        'Authorization' = $("Bearer " + $token.access_token);
    };
    write-host "Logging to Azure finished.";
    return $header;
}

Posting information to Log Analytics

In order to post information to your Log Analytics you will need a signature, this signature is build based on the information of your workspace and also with the information you are posting, here is a simple function that will calculate the signature for you.

Function BuildSignature ($WorkspaceId, $sharedKey, $date, $contentLength, $method, $contentType, $resource){
    $xHeaders = "x-ms-date:" + $date
    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
    $keyBytes = [Convert]::FromBase64String($sharedKey)

    $sha256 = New-Object System.Security.Cryptography.HMACSHA256
    $sha256.Key = $keyBytes
    $calculatedHash = $sha256.ComputeHash($bytesToHash)
    $encodedHash = [Convert]::ToBase64String($calculatedHash)
    $authorization = 'SharedKey {0}:{1}' -f $WorkspaceId,$encodedHash
    return $authorization
}

Now you have your signature and you will need to post the information to Log Analytics, we will encapsulate that in a function to make it easier.

Function PostLogAnalyticsData($WorkspaceId, $sharedKey, $body, $logType){
    $method = "POST"
    $contentType = "application/json"
    $resource = "/api/logs"
    $rfc1123date = [DateTime]::UtcNow.ToString("r")
    $contentLength = $body.Length
    $signature = BuildSignature `
        -WorkspaceId $WorkspaceId `
        -sharedKey $sharedKey `
        -date $rfc1123date `
        -contentLength $contentLength `
        -method $method `
        -contentType $contentType `
        -resource $resource
    $uri = "https://" + $WorkspaceId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"
    $TimeStampField = "TimeGenerated"
    $headers = @{
        "Authorization" = $signature;
        "Log-Type" = $logType;
        "x-ms-date" = $rfc1123date;
        "time-generated-field" = $TimeStampField;
    }
    $response = Invoke-RestMethod -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body
    return $response.StatusCode
}

With this two functions you have everything that is neede to post your information to Log Analytics, now we need to obtain the information and simply push it.

Populate the subscriptions with some messages for our example

In this example we will simply add a few messages to a few subscriptions to show the functionality of the script, this example is not intended to be a real life scenario.

In our Service Bus we will have a single topic called "simpletopic" and two subscriptions in it called "Subscription1" and "Subscription2" (how creative), we will have 2 messages in "Subscription1" and 3 messages in "Subscription2"

Our demo topic with it's subscriptions

Allright, all the setup is done and we have the tree basic functions we need to retrieve information and post it to Log Analytics, let's put it all together!

Querying the Log Analytics Workspace to see the messages

Let's first get the topics of our Service Bus

$Header = GetAzureAuthHeader
$ServiceBusTopicQueryURL = "https://management.azure.com//subscriptions/$Subscription/resourceGroups/$ResourceGroupName/providers/Microsoft.ServiceBus/namespaces/$ServiceBusName/topics?api-version=2017-04-01"
$ServiceBusTopics = $(Invoke-RestMethod -Uri $ServiceBusTopicQueryURL -Headers $Header -Method Get  -ErrorAction Stop)

In $ServiceBusTopics.value we will have all the founded topic names, in our case is only "simpletopic"

Then we will do our next call to retrieve the subscriptions for that topic. In a real world example you will want to do this in a foreach loop to look in all the topics returned.

$ServiceBusSubscriptionQueryURL = "https://management.azure.com//subscriptions/$Subscription/resourceGroups/$ResourceGroupName/providers/Microsoft.ServiceBus/namespaces/$ServiceBusName/topics/simpletopic/subscriptions?api-version=2017-04-01"
$ServiceBusSubscriptions = $(Invoke-RestMethod -Uri $ServiceBusSubscriptionQueryURL -Headers $Header -Method Get  -ErrorAction Stop)

This will return the two subscriptions, here is a picture of the output.

Subscriptions defined

you can see in the "properties" the counts of messages already, they will look like this

Properties of our subscriptions

And there you have the count of messages as "messageCount" which is the sum of all messages in the subscription no matter if they are active, deadletter, scheduled, transfer or transferdeadletter, but in "countDetails" we DO have them split by message status and this is what we are looking for!

Now we will want to do something for every message that we found, yes you are right, store them in Log Analytics!. Let's do that using the function "PostLogAnalyticsData" that we created earlier.

For ActiveMessages

$jsonActive = @{
            "ServiceBusTopicName" = $Topic
            "ServiceBusSubscriptionName" = $ServiceBusSubscription.name
            "ServiceBusSubscriptionActiveMessageCount" = $ServiceBusSubscription.properties.countDetails.activeMessageCount
        }
        $json = $jsonActive | ConvertTo-Json
        # Submit the data to the API endpoint for active messages
        PostLogAnalyticsData -WorkspaceId $WorkspaceId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType

The tricky part in here is the "logType" this is the name we will give to our table in Log Analytics, it is any name we want it to have and then we need to provide a json as the body with the information that we want to store in Log Analytics, keep in mind that this is whatever you want to put in there, I am choosing to store the name of the topic, the name of the subscription and the amount of active messages but you can put whatever you want.

Executing all together

When you put all this together you will want to run it in a fixed schedule to constantly count the messages, keep in mind that the information you push to Log Analytics can take up to 5 minutes to show there, so is not instant when you push it.

After your first push to Log Analytics you will see a "Custom Log" appear and below a table with the name you gave to this ("ServiceBusMessageCount" in our example)

Our custom table appears

After a minutes of our first execution we see the numbers there.

We start getting the numbers in our Log Analytics

OK! the information is in there from here on you can build a graph and place it in a dashboard, an Azure Monitor alert or consume this information with another third party application (like grafana).

Final notes

I am putting this in a github repository in a single file so is easy for you to clone and tune on top of it.

Github: https://github.com/javiermarasco/devopsjourney/tree/master/MessageCounter

If you liked this article, please consider letting me know by liking this article, leaving a comment or sharing this article with others, you can also follow me in my networks.

PowerGrafana, what is it and how to use it

Javier Marasco — Tue, 06 Sep 2022 10:19:36 +0000

Lets begin with a bit of context and how it all started

Sometimes it happens that we have to monitor some application, service or component (among other things) and the tool that we use or our company uses is Grafana, while we have few things to monitor it is not much of a problem but as we add more elements to the panels it becomes more complex to maintain and / or configure.

Imagine that you have to deploy 20 applications, each one running in an app service so we would surely want to monitor your CPU and Memory usage (to begin with) and surely a few more things.
In this simplistic scenario we have to configure some panels in a dashboard and in each panel put the metrics that we want to show (CPU and Memory) in our case.

This sounds simple but in a short time we will surely need to add another metric or deploy a new version of our application, even worse, we could deploy a new version of some components and not others while we have to keep all the versions (the initial one plus the new one) of each component, as you can see this becomes more and more complex to show in Grafana, it is a lot of time clicking on the interface or (the alternative) editing json files to paste them in the Grafana web interface and generate the dashboards or panels ( believe me that it is very easy to make mistakes when editing those files).

The alternative

PowerGrafana was created to solve this problem (or try to make it easier to handle) by extracting all the complexity of dealing with the web interface, json files, or even entering the names of the resources to monitor by hand.
Using a simple PowerShell module we can quickly iterate through our resources and for each of them create a panel that shows CPU and Memory usage.

Each command has it's own help, which can be accessed by executing:

PS> Get-Help New-GrafanaDashboard 

NAME
    New-GrafanaDashboard

SYNOPSIS
    Creates a dashboard in Grafana


SYNTAX
    New-GrafanaDashboard [-DashboardName] <Object> [[-Tags] <String[]>] [<CommonParameters>]


DESCRIPTION
    This cmdlet will create an empty dashboard in Grafana that can be used as starting point to create your grafana monitoring.

EXAMPLE
    New-GrafanaDashboard -DashboardName "My new dashboard" -Tags @('Web','Azure','Production')

RELATED LINKS

REMARKS
    To see the examples, type: "Get-Help New-GrafanaDashboard -Examples"
    For more information, type: "Get-Help New-GrafanaDashboard -Detailed"
    For technical information, type: "Get-Help New-GrafanaDashboard -Full"
    For online help, type: "Get-Help New-GrafanaDashboard -Online"

One day I woke up to a crashed AKS cluster and this is what I did to get it back to life

Javier Marasco — Mon, 05 Sep 2022 18:57:35 +0000

[Article updated]

One day I just found a crashed AKS cluster where I used to have a healthy cluster, yes, you read it right, a cluster with 3 nodes simply stopped working in the tree nodes at the same time.

At first for some reason I thought it was a problem with the portal and that the pods were still working fine but I only lost proper response from the control plane (the side of Kubernetes you don't manage in a managed cluster), to my surprise, no, the complete cluster crashed with all the pods in "Terminating" as the control plane was desperately trying to reschedule them in a healthy node (sadly there were no healthy nodes).

And now what?

Well, my first attempt was check what the portal was reporting, so I went to the portal, selected my cluster and then went directly to "node pools", to my (not) surprise, the nodepool was in "not ready" status, indicating 3/3 nodes were in "Failed" status... nice :)

So my second attempt was "simply" spin up a new node_pool, super simple.... well, I was in an unsupported version, so you can't do that

Lesson learned #1

If your control plane is in a version that is not
available in azure to create new clusters, it
means that you can't create node_pools with that
version, you need to update your control plane
first in order to create node_pools

Ok, no new node_pool for me (bad Javi, you should keep your clusters updated!!). Next idea? yes, add another node to the same/existing node_pool, that will create another node with the same image of the others, this is something that DOES work even when you are in outdated version of kubernetes.

Well... when you create your cluster you can choose two kinds of network plugins in AKS, the regular "kubenet" and the "Azure CNI", there are slights differences between them, the most notorious one is that Azure CNI pre-allocates an IP in the subnet of your cluster when you create a new node in the node_pool, this is because it allocates an IP of the subnet to each pod that could be created in the node, in this case my subnet was too small to fit another node, so no, I couldn't create another node because of the ip starvation in my subnet...

Lesson learned #2

When you plan your kubernetes environment, think
on the future growth, maybe make extra room for
a few more nodes, if your don't create them
you don't pay for them, but you still have plenty
of space in your subnet.

Not all is darkness and desolation

Al this point nothing was working, so I was not loosing anything by cordoning my nodes and trying to delete one of them. From the portal was not possible, any scaling operation was taking around 20 minutes to simply fail.
I ran kubectl cordon <node_name> on my tree nodes and then tried a kubectl drain <node_name on one of them, it complained about the daemonsets and the stateful pods, force them to be removed anyway and then..... it failed to drain the node, if you think a bit about it it actually makes sense, who is gonna drain the nods (a simple delete pod in the background) if the kubelet was dead... absolutely no one, so I simple deleted the node kubectl delete node what can I lose?

IT WORKED! the node was gone, I checked the Azure portal and it was actually deleted (I have the suspicion that the nodes were not even there, they crashed and the Portal never updated the status).

The next thing I noticed is that my node count in the Portal was still 3... despite the fact I had only 2 visible in there.That was weird, but I decided to wait 20 minutes and see it a new node spawned by it's own.... no, it never happened, so I simple scaled up to 4 nodes, my idea was that AKS will try to match the new max nodes by creating a new node, and that's exactly what happened!!!

Lesson learned #3

The Azure Portal some times is a bit buggy,
take what it shows with careful, trust you
tool kit and try to obtain as much information
as you can in different ways.

Lesson learned #4

Think outside of the box, if I was in 3 nodes
already and I was seeing only 2, to create
a new one I decided to scale up to 4, this
makes no sense, but it actually worked.

The (slow) resurrection of the cluster

So now I had a new node, WITH the same version my previous one had AND without needing to update my control plane.
As soon as this node was ready all my deployments tried desperately to schedule their pods in it (good luck fitting all of them there), I repeated the step for the second node, this time scaling to 5, and it worked again!

So after a few minutes I had 3 nodes (1 crashed, 2 healthy and all my pods running fine)

Recommendations

Here is what I learned at the end and the knowledge I can transfer to you:

ALWAYS send our kubernetes logs to some other place, in my case log analytics, once the node crashed, you can't check the logs anymore.
Setup a nice set of alerts to know when something doesn't goes nice, even for those cases you might think you are covered when running a managed cluster
Remember, if everything is broken, you can't broke it even more, don't be afraid of deleting or breaking a little bit more something in the pursue of fixing the problem, after all the cluster was already not working
If you can, document every step you do, it is helpful for when you have a similar problem or just to share with others (or your team)
Have your kubernetes cheat sheet handy, you never know when you will need some command that you rarely use (I will be posting something about this soon)

Update (30-Sept-2022)

Some things to add in this case that I found useful to share are:

1) The whole problem was caused initially because the service principal used by AKS to interact with Azure resources had it secret/password expired, so part of the solution was to update the secret in AAD, update AKS to use the new one and then follow the steps described here.

2) If this service principal expires, AKS gets into a "failed" status, which block any kubernetes version update in the future (but the workloads will keep working), to resolve this you need to use the azcli to upgrade the version of the AKS control plane to the same version you are currently running as described here, this seems to do kind of a restart of the control plane, but remember to do this after you do the service principal password reset, update AKS and do the other steps in this article.

Up to here the update on this case, if I get more information in the future, I will update this article.

Wrapping up

I am still investigating what was the root cause of this, it is not common at all that suddenly all the nodes in your cluster crash, but I don't think I can get much information, the node is completely crashed, I can't even debug it and my only hope are the node logs (which I store in another place)

I apologize for not including pictures in the post, I was trying to fix the problem while trying to extract something educative about the incident and totally forget about capturing some screenshots.

If you enjoyed this post/history please let me know in the comments or with a hearth, and if you are interested in knowing more about my adventures on the wild, remember to follow me in my networks

See you in another post, thank you for reading!

Install and run Grafana in AKS cluster

Javier Marasco — Mon, 05 Sep 2022 07:19:16 +0000

Introduction

In this post I will go thru the steps needed to install Grafana in AKS as well as explaining what is Grafana used for and how it looks after installed.

What is Grafana

To start this post let's first focus on what is Grafana and what is it used for. Grafana is a monitoring tool very widely used because of it's simplicity and power, you can monitor basically anything because of it's many datasources and it comes with an alert engine to send out notifications when a certain threshold is met.

It's configuration is not complicated and you can even version the dashboards you make in Grafana, but we will also cover in this post a powershell module that can help automate the creation of dashboards and panels in Grafana (At this moment it only supports Azure resources but I will work in add other datasources like Prometheus) this module is called PowerGrafana.

How to install it

To install Grafana you have multiple options:

Run a VM with Grafana installed inside
Run Grafana as a container in a VM
Deploy a Grafana container inside a Kubernetes cluster deploying the resources by yourself
Deploy a Grafana container inside a Kubernetes cluster using Helm Charts (we are going to use this one)

This post will cover the deployment of Grafana as a Helm Chart as we did in the post about Prometheus, I will link an article about a deeper explanation on Helm Charts in the near future to go thru the benefits of Helm Charts and specially how to adjust them to cover your particular needs.

To quickly start with this let's go and add the repository and update our repository list.

And now we can check the Charts we have in the repository we just added, let's take a look.

Perfect! we are ready to start working!

Prerequisites

Well for this article we will need:

An AKS cluster up and running, nothing special, just the base deployment is fine, you can follow this tutorial
A terminal with kubectl and helm installed
Install the grafana helm repository

Installation

Since we want to keep our grafana resources in a separate namespace to have a better view of what was created, we will start by creating a namespace called monitoring

kubectl create namespace monitoring

You can see here I use kubens to switch to the monitoring namespace, this is a very cool tool that makes your work with kubernetes easy by changing your namespace to the one you define so your next commands doesn't need to include the namespace on them

helm search repo grafana

Nice!, lot's of charts there, you can see there are charts for multiple solutions, including Grafana Enterprise, Loki (focused on logs more than metrics) and the base grafana one, that's the one we will be using, let's install it.

helm install grafana grafana/grafana

Here we can see the helm command completed and we can notice several things, let's go over them one by one:

Warnings about deprecations:

W1115 09:12:11.499698    9800 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1115 09:12:11.529960    9800 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1115 09:12:12.040639    9800 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1115 09:12:12.040639    9800 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+

This is because my kubernetes cluster is running version v1.21.2 which means the PodSecurityPolicy kind of resource for the apiVersion v1beta1 is deprecated in my version and will be completely removed in the version v1.25 of kubernetes, this is something to cover in the post about helm, I will link it at the end of this article as soon as it is finished, but for now we don't need to worry about this.

Grafana admin password:

Of course, the initial setup needs to configure a default password for the admin account, after your first login you must change this password but to get what this initial password is you need to retrieve it from a secret, and then decode it (secrets are stored encoded as base64)

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

This command will first retrieve the secret called grafana and from it will take the admin-password data, decode it and output to the terminal, let's see what it looks like:

If you don't have base64 installed in your system you can use brew (linux, macos) or chocolatey (windows)

I am running the commands in Windows so I should run:

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 -d

and my output is xzUvc2esXl2aSivqhQQ5X3ZZ01TZ2HMCEPdpWVSJ so that's the password of my admin account in the Grafana deployment I just installed.

Accessing our Grafana!

The last step is how to reach our Grafana instance, and it states that we can access it reaching the service called grafana.monitoring.svc.cluster.local but that will work if you are running kubernetes locally, in my case I am running an AKS cluster in the cloud, which meands I need to port forward the port 3000 of the Grafana server pod to my localmachine port 3000 (or any other port that is not in use), to do this port-forward I need the pod name which we can get by running this:

kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}"

Why do I use port 3000 and not 80 as stated in the output of the helm install command?, that is because the service will listen in port 80 as the snippet states, but the pods listes in port 3000 which is the default port for Grafana and since I am accessing the pod directly I need to use the pod port (3000) instead of the service port (80)

This will get the pods in namespace monitoring that matches the labels name=grafana and instance=grafana then it will apply a jsonpath query to extract the name of the pod, in my case that is grafana-5c999c4fd5-czxdw.

Now we can simply do a:

kubectl port-forward grafana-5c999c4fd5-czxdw 3000:3000

Use the username admin and the password you got in the step 1 of the instructions of the helm chart.

To this point Grafana is fully configured but keep in mind that his installation is very basic, it will not persist (keep) your dashboards or any other configuration if the pod is killed, and this only has one single pod.

All this can changed by downloading the Chart locally with helm pull grafana/grafana --untar and then adjusting our needs in there. If you do this you will notice the fact that there is no persistence once the pod is killed is because the storage for the /var/lib/ directory (where all grafana config is stored) is an EmptyDir which is a very quick way to give the pod some place where to store data but it will be removed when the pod dies.

I will cover more on how to customize Helm Charts locally in the post about that in the near future.

Final words

I hope the explanation was clear and you have now the basics to start playing around with Grafana, if you liked this post, please feel free to leave a comment and/or share it!. Thank you for reading 😊

If you liked this article, please consider letting me know by liking this article, leaving a comment or sharing this article with others, you can also follow me in my networks.

How to auto-document your deployments

Javier Marasco — Wed, 31 Aug 2022 17:00:19 +0000

A bit of background

For most of my infrastructure deployments (if not all) I use a combination of terraform and terragrunt (I could write an article on it if you like, let me know in the comments) which relies on a configuration file to know what the automation should build.
You can think of it like there is a pipeline that reads a config file (json), extracts the information from it and passes it to terraform/terragrunt to run the build.

After each automation I create, there is a documentation on how to use it, what does what and where every piece of code exists and how to extend functionality if needed, this is very important for my team (so everyone can use it without me even present) but also as a guideline for another automations I will be doing in the future so I can re-use some patterns.

Functional documentation vs build documentation

After I create the documentation on how the solution works (functional documentation) there is another kind of documentation that is not so common to have, I am talking about a place where you can store details of what my automation built, and you can imagine this is kind of hard to track as (remember anyone in my team can run the pipeline now reading my documentation) more runs are done on the pipeline.

Approach (what if.....)

A few days ago I woke up having an idea and I started coding it right away (yeah... about 6am in the morning on a Monday...), I was excited to test if the idea was actually possible and how would it work out in the reality.
The idea was simple apparently: "Every time the pipeline runs, it will read the config files (json files) and extract information from them and format them nicely in html and publish that into some place". That sounds easy, so let's see what else can we add.

Folders, files and configs

Ok, I wrote a very simple folder structure to simulate an imaginary repository, this is like this.

/folder/app1/config.json
/folder/app2/config.json
/folder/app3/config.json

Each one of the config.json contains this structure with different values:

{
    "Name": "My resource in one",
    "UUID": "232131232-13eqwesd2-12321w-dsdasda",
    "ConnectionString": "My-connection-one.com:2020",
    "IPs": [
        "10.1.1.2",
        "23.2.2.12",
        "192.233.123.3"
    ]
}

This is a very simple configuration file, imagine this will be sent as input to a terraform module that would build something for you.

Code approach

I will be writing this in Python. First we will define a basic html structure and a variable that will contain our dynamic content.

import os
import json

base_template = '''
<body>
    <table style="border: 1px solid black;">
        <th style="border: 1px solid black;">Name</th>
        <th style="border: 1px solid black;">Connection string</th>
        <th style="border: 1px solid black;">IPs</th>
        {tr}
    </table>
</body>
'''

temporal_tr = ''

base_template contains an html table with a template place holder {tr} which will be replaced later by the content of temporal_tr

Then, we need to iterate over any .json file in our folder structure where we have our configuration files.

for root, subdirectories, files in os.walk(os.path.dirname(__file__)+'/configs'):    
    for file in files: 
        print(os.path.join(root, file))
        if file.__contains__('.json'):
            temporal_tr += '<tr>'
            config_file = os.path.join(os.path.dirname(__file__), os.path.join(root, file))
            with open(config_file, 'r') as read_json:
                content_json = json.load(read_json)
                temporal_tr += '<td style="border: 1px solid black;">'
                temporal_tr += content_json['Name']
                temporal_tr += '</td>'
                temporal_tr += '<td style="border: 1px solid black;">'
                temporal_tr += content_json['ConnectionString']
                temporal_tr += '</td>'
                temporal_tr += '<td style="border: 1px solid black;">'
                temporal_tr += ' '.join(content_json['IPs'])
                temporal_tr += '</td>'
            temporal_tr += '</tr>'

Keep in mind this is VERY hardcoded to match the config files I am using in this test, this definitely needs adjusts to match your case or even be more performant.

Lastly we will need to grab an index.html file, write our table inside, replace the {tr} place holder and save the index.html file.

path_html = os.path.join(os.path.dirname(__file__), 'index.html')
with open(path_html, 'w') as write_base:
    write_base.write(base_template)
tmpl = open(path_html, 'rt').read()
text = tmpl.format(tr=temporal_tr)
with open(path_html, 'w') as writer:
    writer.write(text)

At this point, if we execute this file, if will update an index.html file with the table format and the dynamic content extracted from our files.

You can try adding more config files (following the folder structure) and run the script again to see the changes in the file.

Just one more thing

Why stop here and have an index.html file with our content? why not do something more? ... since confluence is so used and they have a free tier, I opened an space there and I added a functionality to update a page in a particular space in confluence with the content of the index.html file.

Before we can start this section you need to do some steps.

Go to Confluence and open a free cloud account
Log in into here and create a token for your access
Create an empty page and retrieve the id of the page

How does that works? I wrote a simple function that does this for you:

Get's the current version of your page
Increases the version by 1
Creates the content of the page based on our previous work
Publish the content to the confluence page

def publish_to_confluence(user,token,url,page_id,title,content):
    import requests
    confluence_page_version_url = '%s?expand=version' % (url)
    page_current_version = requests.get(url=confluence_page_version_url, auth=(user,token)).json()['version']['number']
    content = content.replace('\n','')
    content = content.replace('"', "'")
    body = '''
    {"id":"%s","type":"page", "title":"%s","body":{"storage":{"value": "%s","representation":"storage"}}, "version":{"number":%i}}
    ''' % (page_id, title, content,page_current_version + 1)
    update_page_content = requests.put(url=url, auth=(user,token), data=body.replace('\n',''), headers={'content-type': 'application/json'})

At this point, your confluence page is updated and every time your pipeline runs you get this page updated.

We have a small problem here, and that is you will eventually end with a lot of versions there, so we can do some plean up and remove any older version and retain only the last X amount of versions. For that I have this other function:

def clean_old_versions(user,token,url,threshold):
    import requests
    versions = requests.get(url='%s/version' % (url), headers={'content-type': 'application/json'},auth=(user,token))

    if len(versions.json()['results']) >= threshold:
        max = len(versions.json()['results'])
        amount_to_delete = max - threshold
        while amount_to_delete >= 0:
            deleted = requests.delete(url='%s/version/%s' % (url,max-amount_to_delete),auth=(user,token),headers={'content-type': 'application/json'})
            amount_to_delete -= 1

What if I don't want to use confluence but I still want to show this to others?

Well.... there is another possibility, Azure storage accounts have a functionality called static pages, basically there is a repository in them that you can upload an index.html file and expose it to the world using a well known azure dns FQDN (you can later configure your DNSs to use a more friendly name if you want).

I just created an azure storage account in Azure and enabled "Static websites" this gives us an url https://javilabsstaticpage.z6.web.core.windows.net/ (in my case)
and also creates a $web container so we store there our static page (you need to specify the page you will be serving and also optionally an error page, I called mine index.html) so you should upload your page into this container.

I will use the python SDK for this so we keep the same language and integrate all this in a single big script.

You will need to install the Azure SDK (mainly the authorization module and the storage one, nothing else).

pip install azure-storage-blob

After you installed this, you can execute this code

def publish_to_storageaccount(account_conn_str,content_file_location):
    from azure.storage.blob import BlobServiceClient
    container = '$web'
    blob_client = BlobServiceClient.from_connection_string(account_conn_str)
    my_blob = blob_client.get_blob_client(container=container, blob='index.html')
    my_blob.delete_blob(delete_snapshots='include')
    with open(content_file_location, "rb") as blob:
        my_blob.upload_blob(blob)

With this you are uploading into the $web container, the html file you created previously (the one you are using to update your confluence page).

In this way, you also stored this information in a cheap storage that can be accessed from internet.

And how does all this works with a pipeline?

Well, at this point you have all the code needed to use any CI/CD tool you know/use and execute this script pointing to the place where your configuration files are.
Every time the pipeline runs, you will be updating your confluence page and/or your storage account static page, no more manual updates and a real reflection of your deployed resources in a handy place.

Wrap up

So in this article you saw how to make your pipeline update a document on each run, so basically on every run, you will be updating the documentation with the list of what was created by the pipeline while you don't need to do anything for it to keep current.

I hope you find this articule useful and let me know if you have any question, if you find this interesting I will appreciate you share if with others, leave a heath, a comment or follow me in any of my networks.

Thank you for reading!

The Ops Community ⚙️: Javier Marasco

Kubernetes probes easily explained

How does traffic reach your application?

Startup, liveness and readiness probes

Managed cluster vs unmanaged clusters in Kubernetes

What does it look like to build a Kubernetes cluster?

Managed clusters and the cloud providers offerings

Patching and upgrading your cluster

So, what is the takeaway of this article?

What is Kubernetes, how does it works and why do we need it

The ever-growing kubernetes manual

Terraform providerless (or how to work with non terraform resources)

Intro

Initial investigation

Let's read!

Let's write

Wrapping up

Prometheus en AKS (en español)

Introducción

¿Qué es Prometheus?

Cómo instalarlo

Qué es Helm

Requisitos previos

Instalación

Últimas palabras

Let's test our configurations with Powershell and Pester

General idea

Example configuration file

Thinking process

Pester

What it is

How to install (Windows)

How it works

Example Pester test

Pre tests data

"My vnet_name is not empty"

Check naming convention

Only lower letters are allowed on the vnet_name

Validate we are only deploying to approved locations

Final notes

Counting message in Azure ServiceBus at subscription level

A bit of background

Presenting the scenario

The limitations currently present in Azure

What we will need

Configuring the SPN

Setup the Log Analytics Workspace

Lets start the coding

Obtain a valid header for our calls to Azure API

Posting information to Log Analytics

Populate the subscriptions with some messages for our example

Querying the Log Analytics Workspace to see the messages

Executing all together

Final notes

PowerGrafana, what is it and how to use it

Lets begin with a bit of context and how it all started

The alternative

Links

One day I woke up to a crashed AKS cluster and this is what I did to get it back to life

And now what?

Not all is darkness and desolation

The (slow) resurrection of the cluster

Recommendations

Update (30-Sept-2022)

Wrapping up

Install and run Grafana in AKS cluster

Introduction

What is Grafana

How to install it

Prerequisites

Installation

Warnings about deprecations:

Grafana admin password:

Accessing our Grafana!

Final words

How to auto-document your deployments

A bit of background

Functional documentation vs build documentation

Approach (what if.....)

Folders, files and configs