The Ops Community ⚙️: Alexandru Dejanu

EKS: Guide to create Kubernetes clusters in AWS

Alexandru Dejanu — Thu, 13 Jul 2023 15:47:22 +0000

Amazon Elastic Kubernetes Service (EKS) is a managed service within AWS, that allows you to run a Kubernetes cluster. Managed service translates into the fact that there's no need to install, or maintain the cluster's control or data plane.

Prerequisites

You're going you need a user with the right policies for services like EKS, CloudFormation, EC2, IAM, and an access key for that user (guide here).

Tooling

kubectl - CLI for communicating with Kubernetes (installation guide here)
aws - CLI for interacting with AWS services (installation guide here)
eksctl - CLI for creating and managing clusters on EKS (installation guide here)

Configuration

aws iam authenticator that allows you to use AWS IAM credentials to authenticate (installation guide here)

Configure AWS CLI: run the following command without arguments in order to get prompted for configuration values (e.g. AWS Access Key Id and your AWS Secret Access Key, AWS region).

aws configure

IAM AWS CLI: for eksctl you will need to have AWS API credentials configured. Amazon EKS uses the IAM service to provide authentication to your Kubernetes cluster through the AWS IAM authenticator for Kubernetes.

Next you can verify if you're authenticated by running the following command:

aws iam get-user

Creating Kubernetes cluster

First you can list if there are any existing clusters (normally you should not have them if this is a fresh setup).

eksctl get clusters

To allow SSH access to nodes, eksctl imports by default the ssh public key from ~/.ssh/id_rsa.pub , but if you want you can use another SSH public key by passing the absolute path to the key to --ssh-public-key flag.

EKS clusters run in a VPC, therefore you need an Amazon VPC with public and private subnets. The VPC must have a sufficient number of IP addresses available for the cluster, any nodes, and other Kubernetes resources that you want to create, and also it must have DNS hostname and DNS resolution support (otherwise nodes can't register to the cluster). You can't change which subnets you want to use after cluster creation.

The beauty of it, is that eksctl will do all the heavy lifting for you, and even more it allows to customize your Kubernetes cluster as needed (number of nodes, region, size of the nodes).

Example for 2 cluster node in the eu-west-1 region:

eksctl create cluster --name=demo_cluster --nodes=2 --region=eu-west-1 --ssh-public-key=eks_key.pub

Behind the scenes eksctl uses CloudFormation, you can see that in this case, it creates 2 CloudFormation stacks, one for cluster itself (control plane) and one for the initial managed nodegroup (woker nodes).

Furthermore you can use CloudFormation console to check the status of it.

After the cluster was created everything is set, you can verify that kubectl point to the correct cluster by running:

kubectl config current-context

Leveraging eksctl you can deploy a Kubernetes cluster in AWS in a matter of minutes.

Linux: Execute commands in parallel without parallel-ssh or Ansible

Alexandru Dejanu — Thu, 08 Jun 2023 12:07:06 +0000

I’ve decided to write this bite-sized article after reading Linux: Execute commands in parallel with parallel-ssh, but I was confronted with a particular use case in which I didn’t have access to the internet and I faced constraints that prevented me from installing new packages.

Without much at hand, I’ve defined a server list, and scripted the following approach:

#!/bin/bash
## #####################################################################
## run concurrently command passed as argv on multiple remote servers ##
## UPDATE: servers array and user variable                            ##
########################################################################
# define an array of remote servers
servers=("server1.fqdn" "server2.fqdn" "server3.fqdn")
# Function to execute command on a remote server
execute_command() {
    server=$1
    command=$2
    # define USER for the ssh connection 
    ssh user@"$server" "$command"
}

for server in "${servers[@]}"
do
    # exec arg command as background process
    execute_command "$server" "$1" &
done
# wait for all background processes to complete
wait

The script initiates multiple SSH connections and uses wait and & operator approach to achieve a rudimentary concurrency, and the usage is pretty straightforward:

./pssh.sh "hostname -A"

Also if you fancy a sequential one-liner, define hosts_file that contains the servers, replace user and run:

while read -u10 line;do ssh user@$line 'hostname -A';done 10<hosts_file

Or if you prefer to pass a script to the one-liner, just replace script.sh with the desired script.

while read -u10 line;do echo $line;ssh user@$line 'bash -s' < script.sh;done 10< servers.txt

Iron Bank: Secure Registries, Secure Containers

Alexandru Dejanu — Thu, 09 Feb 2023 07:46:08 +0000

Iron Bank – DoD Centralized Artifacts Repository (DCAR)

Long story short...Iron Bank Centralized Artifacts Repository is a trusted repository management system to store U.S. Department of Defense artifacts. All artifacts stored in Iron Bank are hardened according to the Container Hardening Guide. DoD containers are OCI compliant and also have DoD-wide reciprocity across classifications.

Some of the main points that reduce the container's attack surface:

Image building must start with a trusted image with known content from a trusted source.
Use distroless images (which contain only application and its runtime dependencies, and don't include package managers/shells or any other programs you would expect to find in a standard Linux distribution). All distroless images are signed by cosign.
The container image must be built with a non-privileged user in the build file.

Without further ado, let's give it a try. Platform one provides two front-ends when it comes to user experience:

1) Iron Bank DCAR

2) Harbor instance registry

One of the most used base images is Alpine Linux (a security-oriented, lightweight Linux distribution),which is a "security-oriented, lightweight Linux distribution based on musl libc and busybox". Next I'll search for the official image:
docker search --format "{{.Name}}: {{.StarCount}}: {{.IsOfficial}}" alpine

The Docker Official Images are a curated set of Docker repositories hosted on Docker Hub. I'm going to use Snyk to scan the alpine image.

The scan found 18 issues most of which are Low severity vulnerabilities.

Now I'll scan ironbank's alpine image, and the result of the scan showed that accordingly to Snky's vulnerability database:

you are currently using the most secure version of the selected base image

In conclusion I highly recommend Iron Bank container registry since is one of the central registries used by the U.S. Department of Defense, and is maintained by the Platform One team.

Azure Kubernetes Service — behind the scenes

Alexandru Dejanu — Mon, 22 Aug 2022 08:50:00 +0000

If you’re using the managed Kubernetes offering from Microsoft also known as AKS you’re probably interested in the following.

AKS control plane

Right off the bat if you do kubectl get nodes you're not going to see any control-plane,master nodes, because the Azure platform manages the AKS control plane, and you only pay for the AKS worker nodes that run your applications.

AKS provides a single-tenant control plane, and its resources reside only in the region where you created the cluster. To limit access to cluster resources, you can configure the role-based access control (RBAC) (which is enabled by default), and optionally integrate your cluster with Azure AD.

AKS worker nodes, virtual machines and node pools

When you create an AKS cluster, Azure sets up the Kubernetes control plane for you and will make one or more _VMSS _(virtual machine scale sets) as worker nodes, in your subscription. Alternatively, you can pay for a control plane that’s backed by an SLA.

Azure virtual machine scale sets let you create and manage a group of load-balanced virtual machines. The VMSS can only manage virtual machines that are implicitly created based on the same virtual machine configuration model.

According to your workloads, you can choose different virtual machines flavors, some of them are:

A-series (for entry-level workload best suited for dev/test environments)
B-series (economical burstable VMs)
D-series (general purpose compute)

Going further in Azure Kubernetes Service, nodes of the same configuration are grouped together into node pools.

Existing node pools can be scaled, upgraded, or if needed they can be deleted, but don’t support resizing in place. Meaning vertical scaling (changing the VM “flavor”/size) is not supported and the alternative way to do this is to create a new node pool with a new target and either delete the previous one or just use multiple node pools.

Another interesting piece of information is that in Azure Kubernetes Service containerd has become the default container runtime starting with Kubernetes 1.19.

AKS networking

By default, AKS clusters use kubenet networking option, and a virtual network and subnet are created for you.

The cluster nodes get an IP address from a subnet in a Virtual Network. The pods running on those nodes get an IP address from an overlay network, which uses a different address space from the nodes. Pod-to-pod networking is enabled by NAT (Network Address Translation). The benefit of kubenet is that only nodes consume an IP address from the cluster subnet.

AKS deployment

There’re multiple ways to create an Azure Kubernetes Service, using Azure Portal, Azure CLI, Azure Resource Manager templates, or even Terraform.

To deploy an AKS cluster you must have a resource group to manage the resources consumed by the cluster.

An important aspect is that an AKS deployment spans two resource groups, 1st group contains only the Kubernetes service resources, and the 2nd resource group (node resource group, name starting with MC_) contains all of the infrastructure resources associated with the cluster (k8s nodes VMSSS, virtual networking, and storage).

Closing notes

In conclusion, AKS simplifies deploying a managed Kubernetes cluster by offloading the operational tasks to Azure, but you always should be aware of the tradeoffs that come with it. If you liked this article you can also read about 🔎 probing Kubernetes architecture or even 📚 a tale of a Kubernetes upgrade.

Kubernetes disaster recovery

Alexandru Dejanu — Thu, 11 Aug 2022 12:17:00 +0000

Disaster recovery is not an easy job, because applications are not deployed on a "fixed" server/machine and also Kubernetes workloads cannot be backed up in a traditional manner.

We're going to talk about how can we maintain/regain the use of stateful components, after a disaster occurred. By disaster we mean a non-graceful shutdown of Kubernetes nodes. If you're not familiar with Kubernetes architecture, check Probing K8s Architecture.

Non-graceful shutdown?
The situation in which the node shutdown action takes place without the kubelet daemon (on each worker node) knowing about it, and as a result the pods on that node also shut down ungracefully.

For stateless workloads, that's often not a problem but for stateful ones(e.g. controlled by StatefulSet) it might imply that the pods on that specific node will also shut down ungracefully.

One example of stateful workloads is applications that need persistence across Pod (re)scheduling a.k.a persistent storage, e.g: Opensearch, Rabbit, Redis, Habor.

So we can be in the situation in which a Pod might be stuck indefinitely in Terminatingand the StatefulSet cannot create a replacement for the pod because the pod still exists in the cluster (maybe on a failed node), resulting in the fact that the application running on StatefulSet may be degraded or even offline.

Lucky for us Kubernetes v1.24 introduces alpha support for Non-Graceful Node Shutdown. This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as hardware failure or broken OS. Even more Kubernetes team plans to push the Non-Graceful Node Shutdown implementation to Beta in either 1.25 or 1.26.

In this particular case, we have NodeOutOfServiceVolumeDetach feature which currently is in Alpha(meaning disabled by default).In order to use it we need to enable it using --feature-gates command line flag the kube-controller-manager component, which is in kube-system namespace: kubectl get pods -n kube-system | grep controller-manager.

Note: After enabling the aforementioned feature, we can apply the out-of-service taint kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute, only after the node has been powered off.

As a result, the PersistentVolumes attached to the shutdown node will be detached, and for StatefulSets, replacement pods will be created successfully on a different running node. Last but not least we need to manually remove the out-of-service taint after the pods are moved to a new node and the user has checked that the shutdown node has been recovered since the user was the one who originally added the taint.

Ingress in one minute

Alexandru Dejanu — Thu, 30 Jun 2022 09:56:32 +0000

Once your services have been established in Kubernetes, most probably you want to get external traffic into your cluster.

This can be achieved using the Service Kubernetes object OR Ingress.

"Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource."

What the latter means, is you need an Ingress controller and some Ingress rules(routing rules to manage external users' access to the services in the cluster) which will be passed to the controller.

⚠️ You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.

Ingress controllers come in various flavors with different capabilities: Istio, Google Cloud Load Balancer, Nginx, Contour...more ingress controllers here.

In conclusion, Ingress represents a single entry point to your cluster that routes the request to different services and acts as a reverse proxy...usually what happens in the real world it's that the request "hits" a service-mesh via Ingress, a.k.a Ingress through Istio gateways 🤯.

kubectl context like a pro

Alexandru Dejanu — Fri, 03 Jun 2022 10:32:07 +0000

If your work involves working with multiple Kubernetes clusters, then this might be for you.

☸️ A file that is used to configure access to a cluster is called kubeconfig(usually placed at ~/.kube/config), but you can easily override the location by using --kubeconfig=<path_to_config> flag or using KUBECONFIG environment variable export KUBECONFIG=<path/to/your_kubeconfig>. A Kubernetes config file describes clusters, users, and contexts. You can use multiple contexts to target different Kubernetes clusters.

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
  name: <cluster_name>
...

users:
- name: <user_name>
...

contexts:
- context:
  name: <context_name>

You can render your current config: kubectl config view.

Without further ado, let's take the example of merging two config files(one for each cluster):

# create backup for current config
cp ~/.kube/config ~/.kube/config.bak

# merge the 2 configs
KUBECONFIG=~/.kube/config:/path/to/new/config kubectl config view --flatten > ~/intermediate_config

# replace the current config with the intermediate_config
mv ~/intermediate_config ~/.kube/config

A context is a combination of a cluster and user credentials.You can:

List contexts: kubectl config get-contexts
Check the current context: kubectl config current-context
Switch to desired context: kubectl config use-context <context_name>
Check defined clusters: kubectl config get-clusters

⚠️ Running aws eks update-kubeconfig --region <us-east-1> --name <cluster_name> pulls your cluster's API endpoint and CA data to create/update a context in your local kubeconfig. This automatically sets that cluster as your active target, so any immediate kubectl commands will execute against it.

$ aws eks update-kubeconfig --region us-east-1 --name demo-eks
...
$ kubectl config current-context
arn:aws:eks:us-east-1:255656399702:cluster/demo-eks

kubectl output like a pro

Alexandru Dejanu — Mon, 30 May 2022 11:09:09 +0000

If your work involves kubectl, even more wrapping kubectl commands in bash scripts, then this might be for you.

So let's say you want to take only the pods names for all namespaces. You can use things la AWK, grep but also you can customize your output:

kubectl get po -o name -A

Pretty simple, and it looks like this:

pod/harbor-jobservice-5f56f77c85-75fzj
pod/harbor-nginx-858bb8887f-8grwb
pod/harbor-portal-59ff88c985-2k878
pod/harbor-redis-0

If you don't like that fact that pod's names are prefixed with pod/...say no more:

kubectl get po -o=custom-columns=NAME:.metadata.name -A

harbor-jobservice-5f56f77c85-75fzj
harbor-nginx-858bb8887f-8grwb
harbor-portal-59ff88c985-2k878
harbor-redis-0

Would be nice though to know the namespace in which each pod is running:

kubectl get po -A -o go-template='{{range .items}} --> {{.metadata.name}} in namespace: {{.metadata.namespace}}{{"\n"}}{{end}}'

--> harbor-jobservice-5f56f77c85-75fzj in namespace: harbor
--> harbor-nginx-858bb8887f-8grwb in namespace: harbor
--> harbor-portal-59ff88c985-2k878 in namespace: harbor
--> harbor-redis-0 in namespace: harbor

That's about it, also this applies for Docker check this article.