How To Scale Jenkins Build Executor and Set Up New Node

By default, Jenkins provides 2 Build Executor and 2 Build Queue. So for example if we have 4 jobs all running at the same time, 2 jobs will be executed by 2 Build Executor and 2 left in Build Queue to wait the first 2 jobs are done.

Don't worry, we can add the number of active Build Executor. There are two options we can use. Those are:

1. Scale Up
The first option, we can increase the default number of executor. But please note it doesn't mean you can increase as many as you want, it should fit with your Jenkins host specification. For example, we'll add one more executor. So, now we should have 3 in total.

2. Scale Out
The second option is by adding new node to our Jenkins Master.
Here I'll tell you the steps:

• Download and Install Plugin

We need to install SSH Build Agents to be used later for connecting to the new node/agent.

• Create New VM

This VM will act as the node. What's the difference between master and node? We don't need to install Jenkins at node. Only Git and Java will be needed.

$ aws ec2 describe-instances --query 'Reservations[].Instances[].{PublicIP:PublicIpAddress, ID:InstanceId}'
- ID: i-0bd3d9b34302fdbe0
  PublicIP: 108.137.62.173
- ID: i-002903a0767e41602
  PublicIP: 108.137.94.186

$ ssh ubuntu@108.137.94.186
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1004-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jun  9 15:54:57 UTC 2022

  System load:  0.359375          Processes:             115
  Usage of /:   18.9% of 7.58GB   Users logged in:       0
  Memory usage: 22%               IPv4 address for ens5: 172.31.17.4
  Swap usage:   0%

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-172-31-17-4:~$

Alright, because I only enable SSH key as remote authentication. Now I'll enable SSH username password to make the master is easier to connect with the node.

ubuntu@ip-172-31-17-4:~$ sudo passwd ubuntu
New password: 
Retype new password: 
passwd: password updated successfully
ubuntu@ip-172-31-17-4:~$ sudo vi /etc/ssh/sshd_config
ubuntu@ip-172-31-17-4:~$ cat /etc/ssh/sshd_config | grep PasswordAuthentication
PasswordAuthentication yes
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication, then enable this but set PasswordAuthentication

Then, I'll try to SSH the node from the master and we should be asked to enter the password.

ubuntu@ip-172-31-17-9:~$ ssh ubuntu@108.137.94.186
ubuntu@108.137.94.186's password:

• Create New Agent

Go to Manage Jenkins => Manage Nodes and Clouds

For remote root directory:

ubuntu@ip-172-31-17-4:~$ pwd
/home/ubuntu

Then, add the directory name you want to put the Jenkins file configuration.

As we configured above, we'll use SSH username password as authentication method.

As we can see, the number of Build Executor also increases as we add new node to the master. Now we have three from master and two from node.

Alright, that's it for now! Any feedback are very welcome and thank you for coming. Follow me to get notified when new post is published by me! Thank you.

Let's continue to the next part!