If you have provided a fleet as your workers configuration, then Leviathan will search that fleet for an available autokit connected to the device type you want to run tests on. For example, check out the config below shortenned for brevity:
{
deviceType: "raspberrypi3",
suite: `${__dirname}/../suites/os`,
config: {
balenaApiKey: '',
balenaApiUrl: 'balena-cloud.com',
...
},
image: false,
workers: {
balenaApplication: 'balena/testbot-rig',
apiKey: "", // BM
}
}
Using this config.js, we can determine that Leviathan will be searching for an available autokit with a DUT: raspberrypi3 tag in the balena/testbot-rig fleet in the balena-cloud.com environment to run the os test suite. Refer to the fleet list to find balena's official autokit fleets used for testing.
Once you find the right balenaCloud fleet where autokits are provisioned, you can check the DUT tags on the devices page to see which autokits are connected to the device type you want to test on. Once identified, run your tests targetting that autokit directly by providing the Public Device URL of the autokit in the workers configuration. For example:
workers: ['https://PUBLIC-DEVICE-URL.balena-devices.com/']
This will target the test run directly to the specific autokit, bypassing the fleet search that Leviathan conducts. If the autokit is available, then Leviathan will run the tests.
Testing with autokit requires several components to work together. All steps in the Getting Started guide must be completed to ensure the autokit works correctly. You can do the following steps to verify if the setup is correct.
$ curl -X POST PUBLIC-DEVICE-URL-OF-AUTOKIT/state
workers section.If the test completes without any errors, then the autokit should be ready to run the tests.
Invalid device type: <device-type-without>
If you see this, ensure the contract for your device type is available in the contracts submodule in leviathan/core/contracts/contracts/hw.device-type.
OS image not found: ENOENT: no such file or directory, access '/usr/src/app/workspace/image.img.gz'
Ensure your OS image is stored in the leviathan/workspace directory.
No workers found for deviceType: DEVICETYPE-SLUG
No device could be found for this device type slug in the Autokit fleet you specified. There could be several reasons for this error in order to resolve it:
DUT tag of the autokit devices in the fleet containing the device type slug. If you don't find a DUT tag matching the slug, then the worker doesn't exist. $ curl -X POST PUBLIC-DEVICE-URL-OF-AUTOKIT/state
When the device fails during flashing - first check: has the correct TESTBOT_DUT_TYPE been selected (see above)?
If it has, we need a progression of checks.
First, confirm that the power control is working for the DUT. This can be done by:
curl -X POST <autokit_ip>/dut/on
And checking that the DUT powers on. If it does, you can turn it back off with
curl -X POST <autokit_ip>/dut/off
If that works, then check if the SD card is flashed with the image you wanted to flash to it. You can take the card out, plug it into your laptop and check.
If it has, then check that the DUT was booting from the SD card - if you have a serial cable attached to the autokit, you can do this:
balena ssh <autokit_uuid> workerapk add screenscreen <serial_dev> 115200usbsdmux /dev/sg0 dut - this will toggle the MUX to the DUT curl -X POST <autokit_ip>/dut/on to power on the DUTscreen session. Is there anything?screen session - you can see if it's failing to boot from the SD card or if it's booting into balena OS. Powering on and off can be done with:
curl -X POST <autokit_ip>/dut/on
curl -X POST <autokit_ip>/dut/off
You can manually flash the DUT outside of the test suite with an image of your choice with:
curl --data-binary @<your image (must be .gz gzipped)> <autokit_ip or public url>/dut/flash
You can create a wired network for your DUT to connect to with:
curl -X POST localhost/dut/network -H 'Content-Type: application/json' -d '{"wired": {"nat":true}}'
You can create a wireless network with:
curl -X POST localhost/dut/network -H 'Content-Type: application/json' -d '{"wireless": {"ssid":"<EXAMPLE SSID>", "psk":"<EXAMPLE PSK", "nat": "true"}}'
After setting up one of these networks, you can access the DUT via the autokit.
You can find the DUT IP address by ssh'ing into the autokit and checking the dnsmasq lease files, for example:
root@3d57642:~# cat /var/lib/NetworkManager/dnsmasq-enp1s0u1u4u1u1.leases
1708607547 d8:3a:dd:4b:6e:0f 10.42.0.248 ef0a7ac 01:d8:3a:dd:4b:6e:0f
then from the autokit command line
ssh 10.42.0.248 -p 22222
You can view the live HDMI output in your browser with <autokit_ip or public url>/dut/liveStream
Upon starting a test, you might see the following errors, or there could be other symptoms when the autokit isn't responsive.
Already running a suite. Please stop it or try again later.
This means Leviathan couldn't find an available worker to run the tests on. It does that by checking each eligible worker's availability and then selecting an IDLE one if found. If the test is stuck due to an error or the testbot can't recover from a previous test run, then run the following command to restore the autokit to IDLE state safely.
$ curl -X POST PUBLIC-DEVICE-URL-OF-AUTOKIT/teardown
OK
If it still doesn't work out, try performing a reboot of the autokit.
Connection issues have been creating numerous problems in our testing lately. This can be due to VPN/API outages, DUT disconnecting, or the worker getting stuck. Such cases would lead to retires, connection resets, and delays that interrupt or fail the tests entirely. We are still debugging these issues on a case-by-case basis. Our goal is to make Leviathan more stable and reliable and reduce our dependence on VPN/API connections.
Please do log the following issues as you see them on https://github.com/balena-os/leviathan/issues/
Leviathan is an open-source project, and we welcome contributions. If you want to add a new test suite or contribute to an existing one, start by writing your first test. For reference, check the existing balenaOS operating system suite in the meta-balena repository
To improve workflow and write tests faster on Leviathan, the following debug options can be triggered to alter the behavior of test runs. A debug object can be added to the config.js file right with the existing suite's config. The debug object can also have custom options as needed by the test suite. These properties will become available during the test run and can be used to customize the test run further as required. Example of a debug object:
debug: {
failFast: false,
globalFailFast: false,
preserveDownloads: false,
unstable: ["TITLE OF THE TEST 1", "TITLE OF THE TEST 2"]
dev: false,
// Custom value
CUSTOM_OPTION: 'Verycustomindeed',
},
The supported debug options available are as follows:
failFast: Exit the ongoing test suite if a test fails. Type: Boolean. Value: true or false. Default: true.preserveDownloads: Persist downloadeded artifacts. Type: Boolean. Value: true or false. Default: false.globalFailFast: Exit the entire ongoing test run if a test fails. Type: Boolean. Value: true or false. Default: false.unstable: Add titles of the test suite that need to be marked unstable in order to have them skipped from a test suite. Skipped tests are marked as todo in the test logs and skipped in the test results. Type: Array.dev: Run tests in development mode with validation and other checks toggled off. Type: Boolean. Value: true or false. Default: false.You can use this.suite.options to access your test suite's CUSTOM_OPTION property.
Check out the config.example.js file for a complete example.
Generated using TypeDoc