The Ops Community ⚙️: Divine Odazie

How to Set Environment Variables on a Linux Machine

Divine Odazie — Wed, 14 Sep 2022 15:43:11 +0000

This article was originally posted on Everything DevOps.

When building software, you start in a development environment (your local computer). You then move to another environment(s) (Staging, QA, etc.), and finally, the production environment where users can use the application.

While moving through each of these environments, there may be some configuration options that will be different. For example, in development, you may want to test CRUD operations with a dummy database with varying configuration values to the live database with real user data.

To ensure a seamless workflow and not have to regularly change the database configurations in code when moving to different environments, you can set environment variables for each.

In this tutorial, you will learn:

What environment variables are, and
How to set environment variables on a Linux machine.

Prerequisite

To follow along in this tutorial, you must have the following:

Basic knowledge of the terminal.
Access to a Linux machine — This article uses Ubuntu 22.04 (LTS) x64 distribution.

What are environment variables?

Environment variables are variables whose values are set outside the code of an application. They are typically set through the built-in functionality of an operating system. Environment variables are made up of name and value pairs, and you can create as many as you wish to be available for reference at a point in time.

Setting environment variables on a Linux machine

To set environment variables on a Linux machine, normally, in the shell session of your terminal, you would run the export command on each environment variable’s name and value like the following:

export ENVIRONMENT_VARIABLE_NAME = <value>

But doing so, if and when the particular shell session ends, all the environment variables will be lost. All the environment variables will be lost because the export command exports variables to the shell session’s environment, not the Linux machine environment.

To persist environment variables on a Linux machine, in any directory aside from your application’s directory, create an environment file with the vi editor using the following command:

$ vi .env

The above command will create and open a .env file, then to edit the file using the vi editor, press i and add your environment variables like in the image below.

After adding your environment variables, to save the file, press esc, then type :wq and press enter.

After saving the file, in the root directory of your Linux machine, run $ ls -la to view all files, including hidden ones, which should show you a .profile as in the image below.

Open up the profile file with $ vi .profile, press i to edit the file, and at the end of the file, add the following configuration:

set -o allexport; source /<path_to_the_directory_of_.env_file>/.env; set +o allexport

The above configuration will loop through all the environment variables you added to the .env file and set them on the Linux machine.

To save the configuration, press esc, then type :wq and press enter as you did previously.

To confirm that the configuration took effect and your environment variables have been set, log out of your current shell session, log back in and then run:

$ printenv

After running the above command, you should see your environment variables, as shown in the image below.

Conclusion

This tutorial explained environment variables and taught how to set them on a Linux machine. There is more to learn about environment variables in Linux. To learn more, check out the following resources:

How to avoid merge commits when syncing a fork

Divine Odazie — Mon, 22 Aug 2022 01:21:26 +0000

This article was originally posted on Everything DevOps.

Whenever you work on open source projects, you usually maintain your copy (a fork) of the original codebase. To propose changes, you open up a Pull Request (PR). After you create a PR, there are chances that during its review process, commits will be made to the original codebase, which will require you to sync your fork.

To sync your fork with the original codebase, ideally, you would use the web UI provided by your Git hosting service or run a git fetch and git merge in your terminal, as indicated in this Github tutorial. But with a PR open, syncing your fork that way will introduce an unwanted merge commit to your PR.

In this article, you will learn what merge commits are and how to avoid them with git rebase when syncing a fork with an original codebase.

What is a merge commit?

A merge commit is just like any other commit, it is the state of a repository at a point in time plus the history it evolved from. But there is one thing unique about a merge commit: it has at least two parent commits.

When you create a merge commit, Git automatically merges the histories of two separate commits. This merge commit can cause conflicts and mess up a project’s Git history if present in a merged PR.

The annotated section in the image above shows a merge commit of two parent commits. Here is the link to the merge commit to the image.

Now you know what a merge commit is. Next, you will learn how to avoid it when syncing your fork with an original codebase.

How to avoid merge commits when syncing a fork in Git.

**
To avoid merge commits, you need to rebase the changes from the original remote codebase in your local fork before pushing them to your remote fork by following the steps below.

Step 1:
Create a link with the original remote repository to track and get the changes from the codebase with the command below:

$ git remote add upstream https://github.com/com/original/original.git

After running the above command, you will now have two remotes. One for your fork and one for the original codebase. If you run $ git remote -v, you will see the following:

origin https://github.com/your_username/your_fork.git (fetch)
origin https://github.com/your_username/your_fork.git (push)
upstream https://github.com/original/original.git (fetch)
upstream https://github.com/original/original.git (push)

In the above, upstream refers to the original repository from which you created the fork.

Step 2:
In this step, you fetch all the branches of the remote upstream with:

$ git fetch upstream

Step 3:
Next, you rewrite your fork’s master with the upstream’s master using git rebase.

$ git rebase upstream/master

Step 4:
Then finally, push the updates to your remote fork. You may need to force the push with “--force.”

$ git push origin master --force

Note:
You can skip the git fetch step by using git pull which is git fetch + git merge but with the --rebase flag to override the git merge. The pull command will be:

$ git pull --rebase upstream master

Conclusion

In this article, you learned about merge commits and how you can avoid them when syncing your fork in Git using git rebase. There is a lot more to learn about git rebase. To learn more, check out the following resources:

Building x86 Images on an Apple M1 Chip

Divine Odazie — Tue, 16 Aug 2022 14:59:07 +0000

This article was originally posted on Everything DevOps.

A few months ago, while deploying an application in Amazon Elastic Kubernetes Service (EKS), my pods crashed with a standard_init_linux.go:228: exec user process caused: exec format error error.

After a bit of research, I found out that the error tends to happen when the architecture an image is built on differs from the architecture it is running on. I then remembered that I was building the image on a MacBook with Apple M1 Chip which is based on ARM64 architecture, and the worker nodes in the EKS cluster I deployed on are based on x86 architecture.

I had two options to fix the error: create new ARM-based worker nodes or build the image on x86 architecture. I couldn’t create new worker nodes for obvious reasons, so I had to figure out how to build x86 images on my Apple M1 chip.

In this article, I will walk you through how I built my application’s Docker image with x86 architecture on an Apple M1 chip using Docker Buildx.

What is Docker Buildx?

Docker Buildx is a CLI plugin that extends the docker command. Docker Buildx provides the same user experience as docker build with many new features like the ability to specify the target architecture for which Docker should build the image. These new features are made possible with the help of the Moby BuildKit builder toolkit.

Before you can build x86-64 images on an Apple M1 chip with Docker Buildx, you first need to install Docker Buildx.

Installing Docker Buildx

If you use Docker Desktop or have Docker version 20.x, Docker Buildx is already included in it, and you don’t need a separate installation. Verify that you have Docker Buildx with:

docker buildx version

But if you are like me that use another tool to get the Docker runtime, install Docker Buildx through the binary with the following commands:

$ ARCH=arm64
$ VERSION=v0.8.2

The above commands set temporary environment variables for the architecture and version of the Docker Buildx binary you will download. See the Docker Buildx releases page on GitHub for the latest version.

After setting the temporary environment variables, download the binary with:

$ curl -LO https://github.com/docker/buildx/releases/download/${VERSION}/buildx-${VERSION}.darwin-${ARCH}

After downloading the binary, create a folder in your home directory to hold Docker CLI plugins with:

$ mkdir -p ~/.docker/cli-plugins

Then move the binary to the Docker CLI plugins folder with:

$ mv buildx-${VERSION}.darwin-${ARCH} ~/.docker/cli-plugins/docker-buildx

After that, make the binary executable with:

$ chmod +x ~/.docker/cli-plugins/docker-buildx

To verify the installation, run:

$ docker buildx version

Building x86-64 images on an Apple M1 chip with Docker Buildx

After installing Docker Buildx, you can now easily build your application image to x86-64 on an Apple M1 chip with this command:

$ docker buildx build --platform=linux/amd64 -t <image-name> .

In the above command:

buildx builds the image using the BuildKit engine and does not require the DOCKER_BUILDKIT=1 environment variable to start the builds.
The --platform flag specifies the target architecture (platform) to build the image for. In this case, linux/amd64, which is of x86 architecture.
And the <image-name> is a placeholder for putting an image tag.

To verify that Docker built the image to linux/amd64, use the docker image inspect <image_name> command as you can see annotated screenshot above.

The inspect command will display detailed information about the image in JSON format. Scroll down, and you should see the Architecture and Os information as in the image below.

Conclusion

This article explored building images based on x86 architecture on an Apple M1 chip using Docker Buildx. There is so much more to learn about Docker Buildx. To learn more, check out the following resources:

Kubernetes Architecture Explained: Worker Nodes in a Cluster

Divine Odazie — Mon, 08 Aug 2022 19:14:09 +0000

This article was originally posted on Everything DevOps.

When you deploy Kubernetes, you get a cluster. And the cluster you get upon deployment would consist of one or more worker machines (virtual or physical) called nodes for you to run your containerized applications in pods.

For each worker node to run containerized applications, it must contain a container runtime for running the containers, a kubelet to ensure everything runs, and the kube-proxy for handling networking.

In this article, you will learn more about what each of the above components does to enable the running of containerized applications in a Kubernetes cluster.

Prerequisites

To properly understand this article, you should have an understanding of the Kubernetes control plane.

Container runtime

The container runtime in a worker node is responsible for running containers. The container runtime is also responsible for pulling container images from a repository, monitoring local system resources, isolating system resources for the use of a container, and managing the container lifecycle.

In Kubernetes, there is support for container runtimes such as:

containerd: An industry-standard container runtime with an emphasis on simplicity, robustness, and portability.
CRI-O: A lightweight container runtime specifically built for Kubernetes.
And any other implementation of the Kubernetes Container Runtime Interface (CRI) — a plugin enabling the kubelet to use other container runtimes without recompiling.

You may wonder why you didn’t see Docker (a major container runtime) in the list above. Docker isn’t there because of the removal of dockershim — the component that allows use of Docker as a container runtime — in Kubernetes release v1.24.

But not to worry, you can still use Docker as a container runtime in Kubernetes using the cri-dockerd adapter. cri-dockerd provides a shim for Docker Engine that lets you control Docker via the Kubernetes CRI. ****

kubelet

The kubelet is the primary Kubernetes node agent. The kubelet is responsible for running the containers for the pods scheduled to its node. The kubelet runs on every machine (control plane, too) in the cluster.

The kubelet for each node keeps a watch on pod resources in the control plane’s kube-apiserver. Whenever the kube-scheduler assigns a pod to a node, the kubelet for that node reads the PodSpec (a YAML or JSON object that describes a pod) and instructs the container runtime using the CRI to spin up containers to satisfy that spec. The container runtime will then pull the container images if they aren’t present in the node and start them.

It is important to note that the kubelet is a Kubernetes component that does not run in a container. The kubelet, along with the container runtime, are installed and run directly on the machine that forms the node in the cluster, and that’s why they are responsible for managing the containerized workloads.

kube-proxy

Like the kubelet, kube-proxy runs on every node in the cluster, but unlike the kubelet, kube-proxy typically runs in a Kubernetes pod as a part of a Kubernetes DaemonSet.

kube-proxy implements essential functionalities for Kubernetes services. kube-proxy maintains network rules that allow communication to your Pods from network sessions inside or outside your cluster.

To dig deep into kube-proxy’s role, you must understand how Service resources work in Kubernetes.

Service in Kubernetes
A Service provides a stable IP address for connecting to pods. There are Service resources in Kubernetes because without them:

When a client application needs to connect to server pods in a cluster, it would need to retrieve and maintain a list of each pod’s IP address, which will burden the client.
And also, it will be hard to maintain those connections, seeing that in Kubernetes, pods will come and go due to scaling, update or hardware failure.

With the above setup using a Service resource, the client will just need the address of the Service, and the rest is taken care of for you automatically.

And how that’s taken care of automatically is that when you create a Service with a selector that references a label applied to a set of pods, the Endpoints controller in the kube-controller-manager will create an Endpoints resource with the pods IP addresses. In the Endpoints resource, the addresses are maintained and if they change, they will be updated automatically and the client’s requests routed accordingly.

In the above image, you might get the impression that a Service is a literal proxy that load balances requests to backends like Nginx. But that’s almost not the case in modern Kubernetes clusters.

Now with your understanding of Service resources, let’s dig into how kube-proxy implements essential functionalities for Kubernetes services.

How kube-proxy implements functionalities for Kubernetes Service resources.
There is an Endpoints controller in the kube-controller-manager that manages the endpoint resources. The Endpoints controller also manages the associations between services and pods.

With the kube-proxy running on each node in the cluster, it watches Service and Endpoint resources. So when a change that needs updating occurs, the kube-proxy will update the rules in iptables. iptables is a network packet filtering utility that allows rules to be set in the network stack of the Linux kernel.

Now, when a client pod sends a request to a Service’s IP, the Linux kernel routes the request to one of the pod’s IP according to the rule kube-proxy sets.

Note: kube-proxy supports alternatives to using iptables to manage network packet routing. But iptables is the most common and suitable for the majority of installations.

Also, note that with iptables, the pod selection will be random. It would not be round robin or other common load balancing strategies. For load balancing, you will need to use IPVS (IP Virtual Server), an alternative to iptables that implements a layer for load balancing in the Linux kernel.

Also, It is important to point out that a Service’s IP is a “virtual IP”, so if you try to ping that IP, you won’t get a response the way you usually would if you ping a pod’s IP, which is an actual endpoint on the network.

A Service’s IP is essentially a key in the rules set by iptables that gives network packet routing instructions to the host’s kernel. But the important part is that a client pod can use the Service IP like it usually would and get the expected behavior as if it were calling an actual pod IP.

Conclusion

This article explained the roles of the container runtime, kubelet, and kube-proxy components in a Kubernetes worker node. There is so much more to learn about these three node components.

To learn more, check out the following resources:

How to run Minikube on an Apple M1 chip without Docker Desktop

Divine Odazie — Fri, 29 Jul 2022 19:33:36 +0000

This article was originally posted on Everything DevOps.

Docker Desktop failed me, and at the time (13th July 2022) of writing this article, Hyperkit, VirtualBox, and other hypervisors specified here don’t work on the Apple M1 chip.

Weeks ago, while using Docker Desktop, it suddenly got stuck in a start-stop loop. I spent hours trying to resolve it, combed through GitHub issues and stackoverflow, and even downloaded older versions of Docker Desktop but to no avail.

I then started searching for an alternative to Docker Desktop. And oh, did I see so many! But though they were alternatives, most didn’t work seamlessly on the M1 chip until I found Colima.

This article will teach you how to run minikube on an Apple M1 chip without Docker Desktop using Colima.

Prerequisite

To follow along with this article, you should have:

minikube installed on your machine. Learn how to here.
Docker Desktop uninstalled. It’s important to note that Colima from v0.3.0 upwards can run alongside Docker Desktop. # How to run Minikube without Docker Desktop

With your current M1 minikube installation, if you run just $ minikube start, it will fail (see image below). And this is because minikube needs a container or virtual machine manager to run, which ideally, Docker desktop (VM + container), hyperKit, etc., would provide.

But since you are having issues or don’t want to use Docker Desktop and can’t use other hypervisors on the M1 chip. You can replace Docker Desktop with Colima reducing the process to run minikube to $ colima start and then $ minikube start.

What is Colima?

Colima is an open-source project that provides container runtimes on macOS with minimal setup. The name Colima means containers in Lima — Linux virtual machines on macOS.

Aside from the minimal setup, Colima offers the following:

Simple CLI interface
Docker and Containerd support
Port Forwarding
Volume mounts
Kubernetes

As this article is to run minikube, you will use Colima with Docker runtime without enabling its Kubernetes feature.

Installing Colima to run Docker containers on macOS

As minikube (with Docker driver) will create a docker container, you must install Docker’s CLI alongside Colima.

Installing Colima
The fastest way to install Colima is through Homebrew. There are other installation options; check here to see them.

$ brew install colima

To check your installation, run:

$ colima

And ensure you get a help message similar to the image below:

Installing Docker’s CLI
After Colima installs, install Docker and Docker Compose.

$ brew install docker docker-compose

After the above installation, configure docker-compose as a Docker plugin so you can use the docker compose command instead of the legacy docker-compose script. To do that, first create a folder in your home directory to hold Docker CLI plugins:

$ mkdir -p ~/.docker/cli-plugins

Then symlink docker-compose into that folder with:

$ ln -sfn $(brew --prefix)/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

And to test, run:

$ docker compose

Then ensure you get an output similar to the image below.

Now that you have Colima installed on your machine, you will create a virtual machine instance with Colima. minikube will use the VM instance, which provides the Docker runtime to start.

Creating a virtual machine instance with Colima

To create a virtual machine with Colima, run:

$ colima start

The above command will create a VM with the default configuration of 2 CPUs, 2GiB memory, and 60GiB storage which should be enough for moderate use. But if you’d like, you can customize your VM by passing additional flags such as --cpu, --memory, --disk, etc., to colima start. Or by editing the config file with colima start --edit. To learn more, check here.

After you run the start command, you will see the following output as Colima downloads the image required the create the VM and then boots the VM:

Once the download and boot process finishes, you will see an output showing that Colima is ready for use.

To confirm, you can run $ colima list or $ colima status to see the Colima running and with the default Docker runtime which minikube will use to start.

Now with Colima running, you can efficiently run minikube without specifying any driver.

Starting minikube

With Colima running, run the $ minikube start command to create your minikube cluster.

Now, run $ docker ps. You will see the minikube container running on your machine with the image created on the first $ minikube start.

Conclusion

This article taught you how to run minikube on an Apple silicon without Docker Desktop and other hypervisors using Colima. Colima is a great tool and has made my life easier since I started using the Apple M1 chip. To learn more about Colima, see answers to some frequently asked questions.

How to create and apply a Git patch file with git diff and git apply commands

Divine Odazie — Sat, 09 Jul 2022 11:27:10 +0000

This article was originally posted on Everything DevOps.

An engineer just joined their first company. While debugging an issue, their co-worker shared a patch (also known as diff) file to apply changes listed in the file to their local branch of the same Git repository.

The engineer could go line by line to retype and apply the changes, but doing so, will subject the code to human error. How can that engineer apply the changes easily?

In this article, you will learn:

what a patch file is,
how the co-worker created the patch file with the git diff command, and
how the engineer can apply the changes using git apply command.

GitHub repository

This article will show screenshots of commands run on a cloned Git repository. Here is the GitHub repository.

What is a patch file?

A patch file is a file that contains changes to be applied to another file or files. A patch file records all the way files are different from each other. Patch files are sometimes called diff files and use .patch or .diff extension.

The `git diff` command and its use cases:

The git diff command works by outputting the changes between two sources inside a Git repository. The sources can be two different files, commits, branches, etc.

To understand the above explanations better, below are common use cases of the git diff command.

$ git diff
When you run the $ git diff command, it outputs all the modified changes which are you yet to add to Git.

See the image of $ git diff in use below:

$ git diff
When you run the $ git diff <filename> command, it will output the changes of the file (<filename>) to its previous committed state.

See image of $ git diff <filename> in use below:

$ git diff
When you run $ git diff <branch_1_name> <branch_2_name> command, it will output the modifications between the two branches.

See image of $ git diff <branch_1_name> <branch_2_name> in use below:

Also, you can output the changes of your current branch to the mentioned branch to its previous committed state using just $ git diff branch_name. See the example image below:

$ git diff --staged path/to/file
When you add changes to Git or move the change to staging, you will not be able to see the difference between the files. You can use the git diff command with --staged or --cached option to see the staged changes.

$ git diff commit_id1 commit_id2
When you want to see the difference between any two commits, you can use the $ git diff commit_id1 commit_id2 command.

To get the id's of the commits you want to see the difference in, use the $ git log command to see the list of commits (starting from the latest commit) made in the Git repo and their respective commit ids.

Creating a Git patch file with `git diff`

For the co-worker to create the Git patch file they shared with the engineer, they used any git diff commands you saw above to get the changes. And then save the changes to a patch file using >, as you can see below.

$ git diff > patch_file.diff

$ git diff commit_id1 commit_id2 > patch_file.diff

$ git diff <filename> > patch_file.diff

, etc.

The above command will create the patch file in the current working directory, as seen in the image below.

Also, remember that the file can also have a .patch extension.

Applying a Git patch file using `git apply`

After the created patch file has been shared with the engineer, they can apply it in the directory of the Git repository that requires the change using the command below.

$ git apply patch_file.diff

For more information about the git apply, check out its man page.

Conclusion

This article showed you how Git patch files are created using git diff and how the patch changes can be applied using git apply.

There are other ways to apply changes from a patch file. To learn more about them, check out this conversation on stackoverflow.

Persisting Data in Kubernetes with Volumes

Divine Odazie — Sat, 09 Jul 2022 11:25:11 +0000

Persisting Data in Kubernetes is complex. And this is because though Pods have readable and writable disk space, the disk space still depends on the Pod’s lifecycle.

When building applications on Kubernetes, there are use cases where developers would want storage for their Pods that doesn’t depend on a Pod’s lifecycle, is available for all nodes, and can survive cluster crashes.

To meet those use cases, Kubernetes provides developers with PersistentVolumes, which are volume plugins like volumes that have a lifecycle independent of any individual Pod that uses it.

In this article, with the help of this Kubernetes docs demo example, you will learn what PersistentVolumes are and how to configure a Pod to use a PersistentVolume for storage.

Prerequisite

To follow this article, you would need a single Node Kubernetes cluster with the kubectl command-line tool configured to communicate with the cluster.

If you don’t have a single Node cluster, you can create one by using Minikube — which this article uses.

What are Persistent Volumes?

PersistentVolumes (PVs) are units of storage in a Kubernetes cluster that an administrator has provisioned. As a Node is a cluster resource, a PersistentVolume is a resource in the cluster.

A PersistentVolume provides storage in a Kubernetes cluster through an API object that captures the details of the actual storage implementation, be that NFS, iSCSI, or a cloud-provider-specific storage system.

How to use PersistentVolume for storage

To use a PersistentVolume, you must request it through PersistentVolumeClaims (PVC). A PersistentVolumeClaim is a request for storage used to mount a PersistentVolume into a Pod.

One can summarize the entire process in the following three steps:

The cluster administrator creates a PersistentVolume backed by a physical storage. And they do not associate the volume with any Pod.
The developer (cluster user) creates a PersistentVolumeClaim that is automatically bound to a suitable PersistentVolume.
The developer then creates a Pod that uses the created PersistentVolumeClaim for storage.

In this article, you will take the role of a cluster administrator and user to walk through the steps.

Configuring a Pod to use a PersistentVolume for storage

In this article, you will create a hostPath Persistent Volume. Kubernetes supports hostPath volumes for development and testing on a single Node cluster. A hostPath PersistentVolume uses a file or directory on the single Node to emulate network-attached storage.

In a production cluster, it is not recommended you use hostPath as it presents many security risks. Instead, the cluster administrator would provision a network resource like an NFS share, a Google Compute Engine persistent disk, or an Amazon Elastic Block Store volume.

Creating a file for the hostPath

To create a file for the hostPath, open a shell to the single Node in your cluster. As this article uses Minikube, the terminal command will be:

$ minikube ssh

In the shell, create a directory for the file:

    # This article assumes that your Node uses 'sudo' to run commands as the superuser
    $ sudo mkdir /mnt/tmp

In the /mnt/tmp directory, create an index.html file:

$ sudo sh -c "echo 'Hello from a Kubernetes storage' > /mnt/tmp/index.html"

Test that the index.html file exists:

$ cat /mnt/tmp/index.html

The output of the above command should be:

Hello from a Kubernetes storage

Now, you can close the shell to your single Node. Next, you will create a PersistentVolume for the hostPath.

Creating a hostPath PersistentVolume

You use a YAML configuration file to create a PersistentVolume, like any Kubernetes component.

Below is the YAML configuration file for a hostPath PersistentVolume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: learn-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/tmp"

The YAML configuration file specifies that the volume is at /mnt/tmp on the cluster's Node. The configuration also defines a storage of 10 gibibytes and an access mode of ReadWriteOnce. The access mode of ReadWriteOnce means the volume can be mounted as read-write by a single Node. The configuration file also defines the PersistentVolume StorageClassName as manual, which Kubernetes will use to bind the PersistentVolumeClaim requests to this PersistentVolume.

Create the hostPath PersistentVolume with the above configuration using the command below:

$ kubectl apply -f https://raw.githubusercontent.com/Kikiodazie/persisting-data-in-kubernetes-with-persistent-volumes/master/hostPathPersistentVolume.yaml

Verify the creation of the PersistentVolume with the command below:

    // learn-pv-volume is the pv name defined in the configuration file above
    $ kubectl get pv learn-pv-volume

The above command will show the PersistentVolume STATUS as Available, which means the PersistentVolume is yet to be bound to a PersistentVolumeClaim.

Next, you will create a PersistentVolumeClaim to request the PersistentVolume you’ve created.

Creating a PersistentVolumeClaim

Same with PersistentVolume, you use a YAML file. Below is a PersistentVolumeClaim configuration file that requests a volume of at least three gibibytes and can allow read-write access for at least one Node.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: learn-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

To create a PersistentVolumeClaim with the above configurations, run the command below:

$ kubectl apply -f https://raw.githubusercontent.com/Kikiodazie/persisting-data-in-kubernetes-with-persistent-volumes/master/persistentVolumeClaim.yaml

After running the above command, the Kubernetes control plane will look for a PersistentVolume that satisfies the claim's requirements. Supposed the control plane finds a suitable PersistentVolume with the same StorageClass, it binds the claim to that PersistentVolume.

To confirm that the PersistentVolumeClaim binds to the PersistentVolume, get the PersistentVolume’s STATUS with the command below:

$ kubectl get pv learn-pv-volume

The above command will show a STATUS of Bound, as shown in the screenshot below.

You can also see that the PersistentVolumeClaim is bound to the PersistentVolume by looking at the PersistentVolumeClaim:

$ kubectl get pvc learn-pv-claim

Next, you will create a Pod to use the PersistentVolumeClaim as a volume.

Creating a Pod

Below is the YAML configuration file for the Pod you will create:

apiVersion: v1
kind: Pod
metadata:
  name: learn-pv-pod
spec:
  volumes:
    - name: learn-pv-storage
      persistentVolumeClaim:
        claimName: learn-pv-claim
  containers:
    - name: learn-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: learn-pv-storage

Looking at the above configuration file, you would notice that it specifies the PersistentVolumeClaim and not the PersistentVolume. This is because the claim is a volume from the Pod’s perspective.

To create the Pod with the above configurations, run the command below:

$ kubectl apply -f 
https://raw.githubusercontent.com/Kikiodazie/persisting-data-in-kubernetes-with-persistent-volumes/master/persistentVolumePod.yaml

To verify that the container (Ngnix) in the Pod is running, run the command below:

$ kubectl get pod learn-pv-pod

Next, you will confirm that the Pod has been configured to use the hostPath PersistentVolume as storage from the PersistentVolumeClaim.

Testing the configuration

To test and confirm the configuration, you verify that the Nginx server is serving the index.html file from the hostPath volume.

To do so, enter the shell of the Nginx container running in your Pod with the command below:

$ kubectl exec -it learn-pv-pod -- /bin/bash

In the shell, run the commands below to install curl, which you will use to serve the index.html file.

$ apt update
$ apt install curl

After installing curl, use it to verify the configuration works with the command below:

$ curl http://localhost/

The above command will output the text you wrote to the index.html file on the hostPath volume:

Now, even if the Pod dies or restarts, upon recreation, it will have access to the same storage.

To test it out, delete the Pod with:

$ kubectl delete pod learn-pv-pod

And create it again with:

$ kubectl apply -f https://raw.githubusercontent.com/Kikiodazie/persisting-data-in-kubernetes-with-persistent-volumes/master/persistentVolumePod.yaml

Then get into the shell of the Pod and rerun the curl command:

$ kubectl exec -it learn-pv-pod -- /bin/bash
$ curl http://localhost/

You will still get the output of theindex.html` file.

Cleaning up

Clean up the entire setup by deleting the Pod, PersistentVolumeClaim, and the PersistentVolume with the commands below:

$ kubectl delete pod learn-pv-pod
$ kubectl delete pvc learn-pv-claim
$ kubectl delete pv learn-pv-volume

Also, delete the index.html file and the directory by running the following commands in the Node shell:

$ sudo rm /mnt/tmp/index.html

$ sudo rmdir /mnt/tmp

Conclusion

This article introduced you to PersistentVolumes in Kubernetes. It also showed you how to configure a Pod to use a PersistentVolume for storage.

There is more to learn about persisting data in Kubernetes with volumes. To learn more, check out the following resources:

Linux background and foreground process management

Divine Odazie — Thu, 26 May 2022 17:51:52 +0000

This article was originally posted on Everything DevOps.

There is support for background and foreground job processing in Linux-based operating systems. A job in this context is just a command launched from a terminal window. Any running command is a process.

This tutorial will show you how to manage jobs in the foreground and background of your Linux terminal window. In the end, you would've learned:

Why you would manage jobs
How to take a foreground job to the background with &
How to bring a background job to the foreground with fg
How to suspend a foreground job with CTRL + Z
How to resume a suspended foreground job with bg
The meaning of the + and - symbol on background jobs
How to run job processes even after closing the terminal window

Why manage jobs?

By default, all jobs in Linux execute in the foreground. Foreground jobs run directly from the shell. When you run one foreground job, you will have to wait for shell access until the job completes before you can run other jobs.

Waiting for jobs to complete is fine when the jobs complete quickly. But in cases where the current job is going to take a long time (even several hours) to complete, it becomes a challenge, especially when you have a single terminal window (SSH session, wconsole) to work with.

In such cases, you can run the job in the background and free the shell for other jobs and tasks. When you run a Job in the background, it will execute at low priority, which will, in turn, allow typing of other commands in the terminal window while the job runs.

Next, you will see how to manage jobs in the foreground and background.

Prerequisite

To follow along on managing jobs in foreground and background in Linux, you need access to a Linux distribution terminal window.

This tutorial uses an Ubuntu Linux distribution.

Managing jobs in the foreground and background

This tutorial will use the sleep command to create dummy jobs for illustration purposes. The sleep command, when run, delays the shell for a specified timeframe. To learn more about the sleep command, check out this article.

Taking a foreground job to the background with `&`

To start, open up your terminal window and run:

$ sleep 10000

After running the above command, it delays the shell for 10000 seconds, making it impossible to run any other command until after 10000 seconds, as shown in the image below.

To run the above sleep command in the background, after terminating the currently running job with CTRL + C, run:

$ sleep 10000 &

After running the above command, you will get an output similar to the image below and your shell back to run other commands.

In the above image, the sleep 10000 & is process 3647 job number 1, and it is currently running in the background.

To see the Job, run:

$ jobs

The above command will show all the jobs in the background of your terminal session and their current state.

Bringing a background job to the foreground with `fg`

In some cases, after taking a job to the background, you might want to do something with it; that’s where the fg command comes in. You use the fg to move a background job on your current Linux shell to the foreground.

To bring back your current running job to the foreground, run:

$ fg %1

Next to % in the above command is the job number 1. After running the command, you will see it back in the foreground.

Suspending a foreground job with `CTRL + Z`

Let’s say you want to suspend the job for a while to run other commands; you can do that with CTRL + Z. CTRL + Z puts the job in a “stopped” mode and doesn’t terminate it.

Resuming a suspended foreground job with `bg`

To resume the sleep 10000 process, you can use the bg command. bg will resume the job process and take it to the background.

In the shell, run:

$ bg %1

Meaning of the `+` and `-` symbol on background job processes

If you’ve noticed throughout the screenshots, the + symbol beside the job number, well, there’s also - and they both have meanings, as you can see in the quote below:

In output pertaining to jobs command), the current job is always flagged with a +, and the previous job with a -. — "JOB CONTROL" section on the bash manpage.

To show the above quote, create two more dummy jobs in the background.

$ sleep 10001 &
$ sleep 10002 &

After running the above commands, run $ jobs to see all the job processes:

As you can see in the image above, the most recent just you created is flagged with the + symbol and the previous with -.

So far, you’ve learned how to take foreground jobs to the background, bring them back to the foreground, suspend them and resume them. Doing all this helps you be more efficient when you have a single terminal window to work.

Next, you will learn how to make your job run even after closing the terminal window.

Running jobs even after closing the terminal window

When you have jobs running in the background or foreground, all the jobs will be terminated if you close the terminal window. The job termination is because they are associated with that particular terminal session. And upon exit, a hang-up (HUP) signal is sent to all the processes it started.

To practicalize this, if the previous jobs you created are still running, close the terminal window, open it back up and then run $ jobs. You will see that no job process is running like in the image below.

To make jobs continue running even after closing the terminal window, you can use nohup, disown, and other commands. This tutorial uses only nohup and disown. To learn about the other commands, check out these recommendations by other Linux users on askubuntu.

When to use nohup? When to use disown?
If you’re able to think ahead of time that you will close your terminal window and not want to hang up your running jobs, use nohup. If not, use disown, which allows you to disconnect a process from a terminal session after its already been created.

Using nohup
To use nohup and continue running jobs even after closing the terminal window, when creating the job, add nohup.

To practicalize, in your terminal window, create a dummy job with nohup:

$ nohup sleep 10000 &

After running the above command, you will see a terminal output similar to the image below:

The annotated terminal output above means that nohup will append (add) any output from the job process sleep 10000 to a file called nohup.out. You can find the nohup.out file in your present working directory, as you can see in the image below.

You can also redirect a particular job output to a different file. To learn how to do that, check out this question and answer on Unix & Linux StackExchange.

Now, when you close your terminal and open it back up, you can still find your sleep 10000 job running. But this time, not with the $ jobs command but with $ ps -aux | grep sleep commands, as you can see in the image below.

The second sleep process you see in the above terminal output was not started by you but by the grep command. To learn why it is there, check out these answers on stackoverflow.

To terminate the job, run:

$ pkill sleep

After running the command, you can still see the second process there like in the image below:

Using disown
Since all the jobs you’ve created are “stopped,” to learn about how to use disown, create a new dummy job process:

$ sleep 10000 &

Then disconnect it from that particular terminal session with:

$ disown %1

Now, close the terminal window, open it up again, then run $ ps -aux | grep sleep to see the job process running like in the image below

Conclusion

This tutorial taught you how to manage foreground and background job processes in a Linux terminal window. To learn more about managing processes in Linux, check out the following resources:

The Ops Community ⚙️: Divine Odazie

How to Set Environment Variables on a Linux Machine

Prerequisite

What are environment variables?

Setting environment variables on a Linux machine

Conclusion

How to avoid merge commits when syncing a fork

What is a merge commit?

How to avoid merge commits when syncing a fork in Git.

Conclusion

Building x86 Images on an Apple M1 Chip

What is Docker Buildx?

Installing Docker Buildx

Building x86-64 images on an Apple M1 chip with Docker Buildx

Conclusion

Kubernetes Architecture Explained: Worker Nodes in a Cluster

Prerequisites

Container runtime

kubelet

kube-proxy

Conclusion

How to run Minikube on an Apple M1 chip without Docker Desktop

Prerequisite

What is Colima?

Installing Colima to run Docker containers on macOS

Creating a virtual machine instance with Colima

Starting minikube

Conclusion

How to create and apply a Git patch file with git diff and git apply commands

GitHub repository

What is a patch file?

The git diff command and its use cases:

Creating a Git patch file with git diff

Applying a Git patch file using git apply

Conclusion

Persisting Data in Kubernetes with Volumes

Prerequisite

What are Persistent Volumes?

How to use PersistentVolume for storage

Configuring a Pod to use a PersistentVolume for storage

Creating a file for the hostPath

Creating a hostPath PersistentVolume

Creating a PersistentVolumeClaim

Creating a Pod

Testing the configuration

Cleaning up

Conclusion

Linux background and foreground process management

Why manage jobs?

Prerequisite

Managing jobs in the foreground and background

Taking a foreground job to the background with &

Bringing a background job to the foreground with fg

Suspending a foreground job with CTRL + Z

Resuming a suspended foreground job with bg

Meaning of the + and - symbol on background job processes

Running jobs even after closing the terminal window

Conclusion

The `git diff` command and its use cases:

Creating a Git patch file with `git diff`

Applying a Git patch file using `git apply`

Taking a foreground job to the background with `&`

Bringing a background job to the foreground with `fg`

Suspending a foreground job with `CTRL + Z`

Resuming a suspended foreground job with `bg`

Meaning of the `+` and `-` symbol on background job processes