The Ops Community ⚙️: gjorgivarelov

FinOps: the definition

gjorgivarelov — Mon, 06 Jun 2022 01:05:42 +0000

I am intrigued by the abbreviation: is FinOps dealing only with cloud processing cost? Is it also including APIs that enable e-commerce and payment processing online? From the topics posted in this part of the Ops community, it seems like cutting cost on AWS is the main focus of FinOps.
I could use a search engine to dig on the FinOps meaning, but I thought I may get a more relevant answer from the professionals of that field posting here...

SELinux: need I say more to scare you?

gjorgivarelov — Mon, 06 Jun 2022 00:56:57 +0000

SELinux: it may be the scarecrow for many and a best friend for the few. In these few following posts, I'll try to explain how to handle SELinux if one encounters it as an obstacle, with minimal explanation on historical background. Like for example the fact that it is a creation of the NSA. Yes, that government agency contributed to open source software. And that's it for the historical context.
By now, you are aware that SELinux' role is to deny access to a resource you sent a process to use. And indeed, SELinux provides an extra level of security beyond user-based and access control list prescribed access to system resources.

At the basic level...

SELinux operates on the concept of security contexts. Each resource on the system- a file, directory, device, port- has a context label on it. Each process running on the system has such a context label as well. It is at the moment that a process is trying to access a system resource that SELinux jumps into action and decides whether that process is authorized to access the resource or not. If labels of both the process and the resource that is about to be accessed match, then the process will successfully access that resource. If labels don't match, SELinux will deny access but also will produce a denial message and assign it a unique UUID that you can later use to read the denial message and take action- which usually consists of re-labeling the resource with the correct label that will allow the process to access that resource. You probably guessed it, re-labeling a resource requires elevated user privileges.

Is SELinux running on my machine?

Let's first see if SELinux itself is turned on while your system is running, and after that we'll see how to check labels of a process, a port and a file (representing every type of resource on a Unix system).
There are three available states in which SELinux may be running. To check on the current state, you don't need elevated privileges. Just issue:
getenforce
as a regular user at the command line and one of the following three may be returned: Permissive, Enforcing or Disabled.
One of the novelties/oddities of RHEL 9 is that SELinux by default is now running in Permissive mode: SELinux will not deny access to a resource when its label and the label of the process trying to access aren't matching, but will produce an error message and log it into syslog. That default setting was Enforcing on previous versions of RHEL.
Knowing now what Permissive mode does, you can safely guess what Enforcing and Disabled modes do: enforce SELinux policies that prevent access when labels are mismatched and Disabled will completely turn off SELinux- both its ability to restrict access and to log policy violations.
You may be enticed to think that setting SELinux mode to Disabled on the system is an unsafe way to operate that system. It is however best to consult higher-ups in your hierarchy at work or organization-level documentation on that system before rushing to switch to Enforcing mode: certain pieces of hardware, not always legacy, have no concept of access to them regulated beyond user-level. In the past, sysadmins were left with no choice but to disable SELinux to make the non-standard hardware work with the system.
If you are the only one in charge of the system and you felt the need to switch around SELinux modes on a running system (educational purposes or testing if SELInux is denying access to a resource- as a part of the troubleshooting process), as a user with elevated privileges, to switch up the level of enforcement, issue:
setenforce Enforcing
or if you need to lower the level of enforcement:
setenforce Permissive
Both modes have their numerical equivalents of 1 for Enforcing or 0 for Permissive. Now, disabling SELinux is done at a different level, at the kernel boot level: either as a kernel boot parameter selinux=0 at boot time (which change will not survive reboot) or adding that option as a permanent kernel boot option.
As is with all subjects Linux, there's a config file that sets the switches for SELinux operation, that file is /etc/selinux/config

Checking labels

To check the label of a specific file, add the -Z option to the ls command:
ls -Z /home/gjorgivarelov/file1
will produce the context label of that file.
Let's say there was a process p1 that tried to access the file file1 and SELinux prevented that but now that you are equipped with some awareness of SELinux being present on your system, you can quickly check the label of the process p1 as well:
ps -Z p1
and check that label against the output of the ls -Z command on file1.
That was easy, but how about checking the SELinux context label of a port? Let's say you need to check the context label of port 80, very common when setting up a web server. Elevated privileges to the rescue and:
semanage port -l | grep /\<80\>/
Labels of ports, processes and resources aren't unique to each resource or process. Several ports can have the same label and the same goes for processes and files.

Labels aren't the only game in SELinux town

SELinux is able to present additional obstacles to processes trying to access resources on the system. Just because labels match doesn't mean that process has unchecked access to the resource. Enter SEBooleans. Once a process is granted access to a resource, SEBooleans present constraints within which the process can use the resource. Many of those Booleans have to do with network access, meaning they (dis)allow network access to a resource. As you may have guessed already, these values can only be on or off. You may have already encountered the need to switch an SELinux Boolean on or off while configuring a port to be used by a web server on a system that has SELinux at Enforcing.
To check if a Boolean is switched on or off:
getsebool -a | grep 'that_boolean'
and see if a required Boolean is on the list and is it turned on or off.
Toggle the Boolean on or off for the current session, for example turning on Boolean se_bool_1:
setsebool se_bool_1 on
of course with elevated privileges. Or to toggle to a new state permanently (survives reboot):
setsebool -P se_bool_1 on
Unlike labels, SE Booleans carry names unique to the action they (dis)allow. For example, to enable access to a non-standard port for the httpd process, part of the procedure might involve switching on SELinux Booleans whose name starts with httpd_. Names of these Booleans are pretty descriptive most of the time.
This concludes the introduction to SELinux, next to present will be the semanage command and how to change labels on a resource, along with the mechanism of inheritance of SELinux labels by resources at different levels of system's hierarchy.

UPGRADING FROM RHEL 8 TO RHEL 9

gjorgivarelov — Fri, 27 May 2022 19:27:46 +0000

I was so excited to see RHEL 9.0 released out of beta stages and into GA (general availability) that I felt compelled to upgrade my RHEL 8 installation. Upgrading an existing installation to a newer version without going through a clean install is what Red Hat calls "in-place upgrade". Remembering how I was forced to do a clean install when upgrading from RHEL 7 to 8, I was intrigued by the process described in the RH docs on in-place upgrading.
All commands are executed as a user with elevated privileges, even the reboot.

Prerequisites:

Get your existing RHEL installation to version 8.6
Install the Leapp utility
Uninstall apps with non-Red-Hat-signed GPG keys. These apps were reported by the pre-installation check utility to be able to crash the entire upgrade process.
Run the pre-installation check to see what would impede or halt the in-place upgrade process.

Only after you satisfy all four can you assume safe upgrade to RHEL 9.

1. Get your existing RHEL installation to 8.6

I had my own installation at RHEL version 8.5 so I needed to update to 8.6. There was actually a lot to update. Make sure your BaseOS and AppStream repositories are enabled. They are on most RHEL 8 systems unless you heavily customized your own system since the original RHEL 8 installation.

dnf update

It will take A BIT of time. And once done with the update to 8.6, if you want to verify you are actually at 8.6:

cat /etc/redhat-release

subscription-manager list --installed

which should report RHEL 8.6 as the version you are running on. Next to do:

reboot

2. Installing the Leapp utility:

Yes, that's Leapp with "pp" at the end. Installing this utility is a prerequisite for carrying on the in-place upgrade:

dnf install leapp-upgrade

From here, you could stay with the command-line and run the pre-upgrade leapp to see if anything impedes the upgrade process- sort of a dry run, very helpful, I don't remember seeing this approach with the leap from RHEL 7 to 8. Or you could install Leapp's Cockpit add-on and carry on with running the pre-upgrade within Cockpit's terminal. The advantage of using Cockpit is that the report of pre-upgrade run will be presented graphically and you will have the option to take the required corrective steps with just a click of a button...

3. ...of course I went for the easy option and installed the Cockpit add-on:

dnf install cockpit-leapp

Log in to the web console by pointing your browser to https://localhost:9090, provided you have Cockpit up and running and listening to port 9090. Open the Terminal tab...

4. ... and then, in that terminal, run the pre-upgrade test:

leapp preupgrade --target 9.0

Test will take SOME time to complete, after which you'll be presented with a report both on the terminal and more importantly, as an option at the bottom of the left bar of Cockpit's page. Click on the Upgrade Report to view and take action on possible future complications on your upgrade path.
Color coding is done right. In both CLI version of the report and the graphical version, red means trouble. Potential problems are graded with a Risk Factor on different levels, in which High can rarely be ignored. Problems are also graded as Inhibitor yes/no. An issue can have a risk factor of High and an Inhibitor- which issue can definitely not be ignored but just follow the corrective steps suggested.
In my case, Google Chrome and VS Code were highlighted as Risk Factor: High and Inhibitors because their RPMs' GPG keys aren't signed by Red Hat. I had to uninstall both to have the report come back with issues with low level risk factor.

Prerequisites satisfied? Upgrade!

Once you get that Upgrade Report to report issues in all blue (or green in CLI), you are ready for the upgrade!

leapp upgrade --target 9.0

and for the next hour (more or less, depending on the size of the filesystem(s) you're upgrading and the speed of your internet connection) the process is pretty much automated.

Conclusion

The whole process, from updating to RHEL 8.6 to upgrading to RHEL 9 and getting back the fully upgraded system back for my usage took 3 hours. It isn't a trivial undertaking compared with upgrades on more consumer-oriented OSes, but it's much, much better compared with the upgrade from RHEL 7 to 8.