Cloud based workers for openQA
20. May 2020 | Guillaume Gardet | CC-BY-SA-3.0
Cloud based workers for openQA
For those who do not know openQA, this is an automated test tool for operating systems and the engine at the heart of openSUSE’s automated testing initiative. It’s implemented mainly in Perl and uses QEMU, by default, for starting and controlling the virtual machines and OpenCV for fuzzy image matching. The general architecture is split in 2:
- One openQA server, aka openQA web UI
- Multiple openQA workers, which run the tests
If you want to know more about openQA, please check the documentation.
openQA workers, which run the tests, are generally on the same network as the openQA web UI server which is fine most of the time, but if some additionnal hardware must be added, they must be sent physically and only few people can take care of it, which can be problematic. One solution to this problem is to use cloud based machines, which are by definition on a separate network and accessible through Internet.
The good news is openQA supports such setups by using a local cache service on the worker. This service downloads the assets (ISO, HDD images, etc.) on demand through the openQA API via HTTPS, instead of using the legacy NFS mount method. Tests and needles are already in git repositories so they can be fetched from the remote git repositories directly instead of using them from the NFS share.
First tests have been done on local workers connected to openqa.opensuse.org (o3 for short), where NFS share has been replaced by the cache service. But this was still on the same network.
Then, more tests have been performed with additionnal aarch64 workers on:
- Public cloud: An AWS EC2 A1 bare metal instance which runs qemu/kvm based tests
- Remote machines:
- Marvell ThunderX2 and SolidRun HoneyComb LX2K which also runs qemu/kvm based tests
- A Softiron Overdrive 1000 which runs a generalhw backend to perform tests on real hardware, currently a Raspberry Pi 2 and two Raspberry Pi 3 (B and B+)
The only caveat is the developer mode which requires the worker to be reachable from the openQA web UI server at specific ports, so here, reachable on the Internet. Unfortunately, this is not the case with current aarch64 remote workers.
For qemu based tests KVM enabled systems are highly recommended otherwise you will get very poor performances with runtimes about 10x slower compared to KVM enabled systems. So, you need to use bare metal instances or nested virtualization when available.
Now, we will detail specific configurations to setup a remote cloud worker which has access only to the openQA API.
Setup
Install required software on the worker
As any other openQA worker, you need to install some packages.
You likely want to use the latest version of openQA and thus use the binaries from devel:openQA
and devel:openQA:Leap:15.1
projects (adjust the URL, if you do not use Leap 15.1):
sudo zypper ar -f https://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.1/devel:openQA.repo
sudo zypper ar -f https://download.opensuse.org/repositories/devel:/openQA:/Leap:/15.1/openSUSE_Leap_15.1/devel:openQA:Leap:15.1.repo
If you use SLE15-SP1, you need to enable the matching repositories and also PackageHub:
sudo zypper ar -f https://download.opensuse.org/repositories/devel:/openQA/SLE_15_SP1/devel:openQA.repo
sudo zypper ar -f https://download.opensuse.org/repositories/devel:/openQA:/SLE-15/SLE_15_SP1/devel:openQA:SLE-15.repo
sudo SUSEConnect -p PackageHub/15.1/aarch64
Now, you can install required packages:
sudo zypper install openQA-worker os-autoinst-distri-opensuse-deps
Get API keys from openQA web UI host
Create a new set of API keys from openQA web UI or ask someone with admin permissions to create a set for you.
Setup worker to use API keys and cache
With a remote worker, you cannot NFS mount /var/lib/openqa/cache
from the openQA server as only the openQA API is reachable. Instead, you must use the cache service as described below.
Update /etc/openqa/workers.ini
with:
[global]
HOST = http://openqa.opensuse.org http://myotheropenqa.org
CACHEDIRECTORY = /var/lib/openqa/cache
CACHELIMIT = 50 # GB, default is 50.
CACHEWORKERS = 5 # Number of parallel cache minion workers, defaults to 5
Update /etc/openqa/client.conf
with the key generated from web UI:
[openqa.opensuse.org]
key = 0123456789ABCDEF
secret = FEDCBA9876543210
Start and enable the Cache Service:
sudo systemctl enable --now openqa-worker-cacheservice
Enable and start the Cache Worker:
sudo systemctl enable --now openqa-worker-cacheservice-minion
Synchronize tests and needles
Tests and needles are not part of the cache services, but are hold in GIT repositories, so you need to setup an auto-update of those repos. Currently, the easiest way is to use the fetchneedles
script from openQA package to fetch Git repos and create a cron job to update it often enough (say, every minute).
Install required package and run the fetch script a first time.
sudo zypper in --no-recommends openQA system-user-wwwrun
sudo /usr/share/openqa/script/fetchneedles
Also, if you plan to run MicroOS tests:
cd /var/lib/openqa/share/tests
sudo ln -s opensuse microos
cd /var/lib/openqa/share/tests/opensuse/products/microos
sudo ln -s ../opensuse/needles/
Now, add a cron job to fetch tests and needles every minute /etc/cron.d/fetchneedles
:
-*/1 * * * * geekotest env updateall=1 /usr/share/openqa/script/fetchneedles
And restart the service:
sudo systemctl restart cron
Enjoy your remote worker
Now you can (re)start your worker(s):
sudo systemctl enable --now openqa-worker@1
And, your remote worker should be registered on the openQA server. Check the /admin/workers
page from the openQA web UI. Enjoy!
Have a lot of fun!