The Ops Community ⚙️: Alexey Melezhik

SparrowCI - DSL is dead, long live DSL!

Alexey Melezhik — Tue, 25 Oct 2022 22:11:43 +0000

Originally published on dev.to - https://dev.to/melezhik/sparrowci-dsl-is-dead-long-live-dsl-4l1b

TL;DR - How to write YAML DSL pipelines without frustration.

To be honest, I am not in favor of modern DSL based tools.

At least, I am not in favor of them when people try to use those tools for the things they are not very well designed for ( which starts to happen pretty quickly ).

Starting from Ansible the course of declarative style DSL languages seems has taken the market, while it does not always mean those tools are easy to use and maintain.

Well, for simple, typical cases ala make && sudo make install this approach works really well, while for more complex scenarios where a pipeline logic implies some data flow and sharing states between steps the idea of confining everything into a declarative (none imperative) approach results in ugly and hard to maintain code.

Consider this simple example of GH actions pipeline of setting 2 variables in 2 steps and using them in downstream code:

steps:
  - id: task1
    run: echo "::set-output name=test::hello"

  - id: task2
    run: echo "::set-output name=test::world"

  - name: main 
    run: |
      echo "Task1 returns: ${{ steps.task1.outputs.test }}"
      echo "Task2 returns: ${{ steps.task2_outputs.test }}"

IMHO here we have a code that is hard to read (we use none relevant print statements to set a step output), and thus to maintain.

These typical question would pop up:

If I use high level general purpose programming languages why should I use print statements to declare return output ?
How could a step return structured, complex object data?
And the last but not the least - how to pass arbitrary configuration as a task input?

This is why ( well partly because of that ) I've created SparrowCI - super fun and flexible CI system with many programming languages support

SparrowCI DSL still allows one to write pipeline code as YAML based steps, however, every step essentially is a function written in a language of choice and thus accepting some input parameters and returning some output values.

That is it.

In SparrowCI have a good balance of YAML based pipelines for people not willing to go imperative path and do any serious coding and flexibility and freedom to express elementary units of pipeline logic as functions written on regular programming languages:

  tasks:
    -
      language: Python
      name: task1
      code: |
        print("task1 start")
        print(f"you passed {config()['message']}")
        update_state({"test": "hello"})
    -
      name: task2
      language: Python
      code: |
        print("task2 start")
        print(f"you passed {config()['message']}")
        update_state({"test": "world"})
    -
      name: main
      language: Python
      default: true
      depends: 
        - 
          name: task1
          config:
            message: hello task1
        - 
          name: task2
          config:
            message: hello task2
      code: |
        print(f"Task1 returns: {config()["tasks"]["task1"]["state"]["test"]}")
        print(f"Task2 returns: {config()["tasks"]["task2"]["state"]["test"]}")

So, with that being said - DSL(YAML) is dead, long live DSL(YAML)!

One Tomtit to make it!

Alexey Melezhik — Wed, 25 May 2022 19:47:17 +0000

Photo via Good Free Photos

Tomtit is a Raku task runner with dozens of plugins, you can use it as alternative for many task runners.

Why?

it shapes best for Raku developers as it written on Raku
it has programmatic API for Raku language
it has shortcuts for many common tasks - running Bash scripts, creation of files and starting/stoping services - https://github.com/melezhik/sparrowdo/blob/master/core-dsl.md
it has dozens of plugins - https://sparrowhub.io - to solve more specific tasks
it's extendible - you can take your favourite language and write a new plugin to address your specific needs.

As a bottom line it helps you to get things done with a minimal of fuss, yet not limited you to a static DSL but allow you to orchestrate your scenarios in modern and powerful language Raku.

So, are you ready to get closer?

Installation

Tomtit is installed as Raku module.

zef install Tomtit

Once successfully installed Tomcat provides tom - command line client to execute scenarios.

Tomtit workflow

Your usual workflow with tomtit is when you define scenarios and then run them. It works especially nice with project-centric approach where you checkout the source code of application and run some related tasks.

Say, we have an application source code, where we perform 3 standard operations:

build
test
and install

We end up with scenarios, we name then build, test and install.

If use make utility to build project, it could be just a 3 invocations of make utility with related arguments.

Let's create our first Tomtit scenarios:

git checkout $git-repo

mkdir .tom

nano .tom/build.pl6
nano .tom/test.pl6
nano .tom/install.pl6

The code of every scenarios is as simple as running make though bash shortcut:

.tom/build.pl6:

bash "make"

.tom/test.pl6:

bash "make test"

.tom/install.pl6:

bash "sudo make install"

Programmatic API

The tomtit scenarios are written in Raku and basically are just calls of different tasks, those scenarios however not necessarily limited tasks calls, they just are Raku programs where you can do anything you could with Raku.

There are two syntactically different types of tasks in Tomtit. The first one you call by the means of task-run function and second one is shortcuts - pretty much the same as task-run but with mnemonic names and sometimes easier signatures.

In the scenarios for make utility we used bash shortcut to call a piece of Bash code. The full list of shortcuts as we as calling details is here - https://github.com/melezhik/sparrowdo/blob/master/core-dsl.md

Tasks and plugins

In other words both task-run and shortcuts is just a way to call small piece if code that is downloaded from SparrowHub - script repository and then executed by sparrow internal script runner for tomtit, I have not told you but we have one more birds in our zoo-repository!

These small scripts or tasks are also called Sparrow plugins.

Tomtits scenarios basically are just a list of executed Sparrow plugins or tasks.

Command line API

By running tom --list you'll see all the scenarios available now:

[scenarios list]
build
test
install

Once scenarios are defined you run them through tom --run=$scenario command.

For example:

tom run=build

tom run=test

And so on.

To recall what was the last scenario you run use tom --last command.

You'll find thorough documentation on tom command line client usage at Tomtit's gihtub pages.

Custom tasks

Like I said, you are not limited running Bash only in Tomtit scenarios. You can run any task provided you find related plugin for it.

For example, let's create scenario that configure local git repository with user's parameters like username and email.

The task is often useful when you've just clone new project, make some changes and want to push some changes back to remote. Git requires you to set your identities as a commiter.

.tom/git-setup.pl6:

task-run "set git", "git-base", %(
  email                => 'melezhik@gmail.com',
  name                 => 'Alexey Melezhik',
  config_scope         => 'local',
  set_credential_cache => 'on'
);

By setting config_scope we ask task to make settings for local git repository and by setting set_credential_cache we also ask git to cache our password, so we don't enter it every time we do git push

The list of plugins you can use in Tomtit scenarios is available at SparrowHub - sparrow plugin repository.

In the next scenario example we create task to run VSTS build remotely through the vsts-build plugin:

.tom/build-vsts.pl6:

task_run "run my build", "vsts-build", %(
    definition => "BackEndBuild"
)

There are many other plugins you can use in Tomtit scenarios as tasks. Check out SparrowHub repository.

Summary

Tomtit provides you with the plenty of small script or plugins you might use in daily tasks, especially when dealing with source code management and build automation.

Tomtit scenarios are plain Raku scripts to generate dynamic lists of executed tasks - scripts with parameters.

There is syntax sugar - predefined set of built-in functions you can use instead of referring to plugins, that make code more concise and easier to read.

If for some reasons you don't find plugin to solve your task, you can always create your own one and start using it strait away.

Tomtit is easy to use, powerful and fun - give it a try, and let me know how it's going on.

Write devopsish tests using Tomty and Raku

Alexey Melezhik — Wed, 25 May 2022 19:44:01 +0000

Tomty is a Raku test framework with focus on devops tasks. Tomty allows one to write test scenarios on Bash (or whatever) and orchestrate them using Raku.

In this post I am going to give a quick introduction into Tomty.

Installation

Tomty is installed as Raku module, using zef - Raku package manager:

zef install --/test https://github.com/melezhik/Sparrow6.git
zef install --/test https://github.com/melezhik/Tomty.git

We need bleeding edge version of Tomty and Sparrow (dependency for Tomty) to get an access to all the features mentioned in this post.

Setup Tomty

Working with Tomty requires some initialization. First of all we need to create a working directory for tests and initialize it. Optionally we can install Bash completion for tomty - Tomty cli application:

export SP6_REPO=https://sparrowhub.io/repo
mkdir work
cd work
tomty --init
tomty --completion

Create first scenario

Scenarios could be written on many languages, in this post I'll focus on Bash as it's a simple and quite popular scripting language for devops tasks and automation. Tomty allows effectively right low level tasks in Bash while gluing them together with Raku.

A Bash scenario should be just a file named task.bash and located at some directory:

mkdir -p tasks/task1/
nano tasks/task1/task.bash

echo "hello world"

Once bash task is created, we can run it through a Raku wrapper:

tomty --edit my-task

#!raku
task-run "tasks/task1"

In context of Raku a Bash task gets called as a Raku function task-run, with a parameter setting a task directory.

The approach gives us a benefit of using high level language to call low level Bash scenarios.

Now, let's run our first task:

tomty my-task

[repository] :: index updated from file:///root/repo/api/v1/index
[tasks/task1] :: hello world

The task executed successfully, now let's convert this trivial hello world example it into something more practical.

Check if a web site is available

The task of checking an http resource health check is quite common in daily devops @job. Let's see how we could do this in Tomty.

First, let's create a Bash task.

nano tasks/task1/task.bash

curl -s -f $(config url) -o /dev/null -D - | head  -n 4

The task is just using curl command to GET some http URL, pay attention that -f flag causes curl to exit with none zero exit code in case of receiving none successful http response.

A Tomty scenario, would change a bit to pass tested URL as a paramater into a Bash task:

tomty --edit my-task

#!raku
task-run "tasks/task1", %(
  url => "https://raku.org"
);

Now let's see how it works:

tomty my-task

[tasks/task1] :: HTTP/2 200
[tasks/task1] :: date: Fri, 02 Jul 2021 18:20:58 GMT
[tasks/task1] :: content-type: text/html
[tasks/task1] :: last-modified: Tue, 25 May 2021 16:30:03 GMT

As we could see Bash config name is used as function to access named parameters passed to Bash from Raku as a Raku Hash. This semantic works because Tomty is build on top of Sparrow automation tool, that provides all type of such functionality.

Many times, relying on a script exit code to check if a test passes or not works just fine, however sometime one needs to specify expected script output to define a test state.

Let's see how Tomty allows that.

Task checks

Task checks allow define an output a test expects to see in a running script:

nano tasks/task1/task.check

HTTP/2 200

So we check that https://raku.org web server servers HTTP v2 protocol:

tomty my-task

[repository] :: index updated from file:///root/repo/api/v1/index
[tasks/task1] :: HTTP/2 200
[tasks/task1] :: date: Fri, 02 Jul 2021 19:08:34 GMT
[tasks/task1] :: content-type: text/html
[tasks/task1] :: last-modified: Tue, 25 May 2021 16:30:03 GMT
[task check] stdout match <HTTP/2 200> True

And if we want to check against v1 or v2, we can use Raku regular expressions:

nano tasks/task1/task.check

regexp: "HTTP/" 1 || 2 " 200"

Refer to Sparrow task check documentation to get full explanation of the DSL and see more complex examples of validating scripts output.

Conclusion

This was a brief introduction into Tomty - test framework written on Raku and supporting many scripting languages to write low level tasks. The main philosophy of Sparrow ( the engine Tomty is built upon ) - use those languages that fits your domain best ( in our case - Bash is simple and efficient way to send http requests) and glue scripts together using high language - Raku, where scripts get called as functions.

When I develop tests for my daily devops tasks on Tomty, I switch between Bash and Raku often and this approach allows me do my work effectively and with minimal efforts.

Thanks for reading. I'd like to hear feedback as usual.

Alexey

Validating k8s deployments using Sparrow

Alexey Melezhik — Wed, 25 May 2022 19:41:16 +0000

Sparrow is a wonderful tool to automate @daily devops tasks. Recently I've dropped a new plugin called k8s-deployment-check to verify k8s deployments. It lets you with a little bit of Raku code effectively test entire k8s infrastructure, including k8s deployments.

Let's create a sample nginx deployment to show how it works:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  # replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /var/www
          name: www-data
        env:
          - name: DEMO_GREETING
            value: "Hello from the environment"
          - name: DEMO_FAREWELL
            value: "Such a sweet sorrow"
      volumes:
      - name: www-data
        persistentVolumeClaim:
          claimName: nginx-example

In this example we run simple nginx application with local folder mounted as /var/www and consuming it's data throughout a volume named www-data:

k8s apply -f nginx.yaml
deployment.apps/nginx-deployment created

Now when our deployment is up and running, let's verify it, using Raku:

tomty --edit verify

#!raku

task-run "chk-dpl", "k8s-deployment-check", %(
  name => "nginx-deployment",
  namespace => "melezhik-sandbox",
  volume-mounts => %(
    www-data => "/var/www"
  ),
);

tomty verify

[repository] :: index updated from file:///root/repo/api/v1/index
[chk-dpl] :: >>> verify deployment. name=nginx-deployment,namespace=melezhik-sandbox,container=nginx
[chk-dpl] :: [env_start]
[chk-dpl] :: [DEMO_GREETING=Hello from the environment]
[chk-dpl] :: [DEMO_FAREWELL=Such a sweet sorrow]
[chk-dpl] :: [env_end]
[chk-dpl] :: [volume_mounts_start]
[chk-dpl] :: [www-data /var/www]
[chk-dpl] :: [volume_mounts_end]
[chk-dpl] :: ==================================================================
[task check] >>> check volume mounts
[task check] stdout match (r) <[www-data /var/www]> True

As we could see Sparrow has successfully verified that

k8s deployment resource it exists
it has volume www-data mounted as /var/www folder

We can alter the scenario a bit, adding environment variable check as well:

tomty --edit verify

task-run "chk-dpl", "k8s-deployment-check", %(
  name => "nginx-deployment",
  namespace => "melezhik-sandbox",
  volume-mounts => %(
    www-data => "/var/www"
  ),
  env => {
    DEMO_GREETING => "Hello from the environment",
    DEMO_FAREWELL => "Such a sweet sorrow"
  },
);

And now run:

tomty verify

[repository] :: index updated from file:///root/repo/api/v1/index
[chk-dpl] :: >>> verify deployment. name=nginx-deployment,namespace=melezhik-sandbox,container=nginx
[chk-dpl] :: [env_start]
[chk-dpl] :: [DEMO_GREETING=Hello from the environment]
[chk-dpl] :: [DEMO_FAREWELL=Such a sweet sorrow]
[chk-dpl] :: [env_end]
[chk-dpl] :: [volume_mounts_start]
[chk-dpl] :: [www-data /var/www]
[chk-dpl] :: [volume_mounts_end]
[chk-dpl] :: ==================================================================
[task check] >>> check env
[task check] stdout match (r) <[DEMO_FAREWELL=Such a sweet sorrow]> True
[task check] stdout match (r) <[DEMO_GREETING=Hello from the environment]> True
[task check] >>> check volume mounts
[task check] stdout match (r) <[www-data /var/www]> True

Conclusion

Sparrow k8s-deployment-check plugin allows one to test k8s infrastructure by just writing a simple piece of Raku code. The full documentation is available at SparrowHub site.

I am going to add more features eventually.

Under the hood Sparrow uses Raku regular expressions to verify resources structure, this allows to write even more sophisticated checks.

For example, to check that a container run command has python 2nd or 3rd version one can write:

  command => "regexp: '/usr/bin/python' 2|3"

Stay tuned and as usual I'd like to hear your feedback

Alexey

The Ops Community ⚙️: Alexey Melezhik

SparrowCI - DSL is dead, long live DSL!

One Tomtit to make it!

Installation

Tomtit workflow

Programmatic API

Tasks and plugins

Command line API

Custom tasks

Summary

Write devopsish tests using Tomty and Raku

Installation

Setup Tomty

Create first scenario

Check if a web site is available

Task checks

Tags

Conclusion

Validating k8s deployments using Sparrow

Conclusion