2021-12-10 09:48:34 -05:00
(integration-testing-vms)=
2023-10-06 16:37:55 -04:00
# Integration testing with NixOS virtual machines
2021-12-10 09:48:34 -05:00
2022-08-16 10:27:42 -04:00
## What will you learn?
2021-12-10 09:48:34 -05:00
2023-10-06 16:56:26 -04:00
This tutorial introduces Nixpkgs functionality for testing NixOS configurations.
It also shows how to set up distributed test scenarios that involve multiple machines.
2021-12-10 09:48:34 -05:00
2022-08-16 10:27:42 -04:00
## What do you need?
2021-12-10 09:48:34 -05:00
2023-10-06 16:56:26 -04:00
- A working [Nix installation ](<install-nix> ) on Linux, or [NixOS ](https://nixos.org/manual/nixos/stable/index.html#sec-installation )
- Basic knowledge of the [Nix language ](<reading-nix-language> )
- Basic knowledge of [NixOS configuration ](<nixos-vms> )
2021-12-10 09:48:34 -05:00
2022-08-16 10:27:42 -04:00
## Introduction
2021-12-10 09:48:34 -05:00
2023-09-11 09:14:15 -04:00
Nixpkgs provides a [test environment ](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests ) to automate integration testing for distributed systems.
2023-09-11 09:13:00 -04:00
It allows defining tests based on a set of declarative NixOS configurations and using a Python shell to interact with them through [QEMU ](https://www.qemu.org/ ) as the backend.
2022-08-16 10:27:42 -04:00
Those tests are widely used to ensure that NixOS works as intended, so in general they are called [NixOS Tests ](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests ).
2023-09-11 09:15:14 -04:00
They can be written and launched outside of NixOS, on any Linux machine[^darwin].
2023-10-06 16:38:34 -04:00
[^darwin]: Support for [running NixOS VM tests on macOS ](https://github.com/NixOS/nixpkgs/issues/108984 ) is also implemented but [currently undocumented ](https://github.com/NixOS/nixpkgs/issues/254552 ).
2023-09-11 12:10:38 -04:00
2023-10-06 17:04:46 -04:00
Integration tests are reproducible due to the design properties of Nix, making them a valuable part of a continuous integration (CI) pipeline.
2021-12-10 09:48:34 -05:00
2023-06-19 14:10:26 -04:00
## The `nixosTest` function
2021-12-10 09:48:34 -05:00
2023-09-11 09:13:00 -04:00
NixOS VM tests are defined using the `nixosTest` function.
The pattern for NixOS VM tests looks like this:
2022-08-16 10:27:42 -04:00
```nix
let
2023-06-19 14:10:26 -04:00
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
2023-09-11 09:08:42 -04:00
pkgs = import nixpkgs { config = {}; overlays = []; };
2022-08-16 10:27:42 -04:00
in
2023-10-06 16:39:28 -04:00
pkgs.nixosTest {
name = "test-name";
nodes = {
machine1 = { config, pkgs, ... }: {
# ...
};
machine2 = { config, pkgs, ... }: {
2022-08-16 10:27:42 -04:00
# ...
2023-10-06 16:39:28 -04:00
};
2022-08-16 10:27:42 -04:00
}
2023-10-06 16:39:28 -04:00
testScript = { nodes, ... }: ''
# ...
'';
}
2022-08-16 10:27:42 -04:00
```
2023-10-06 16:40:23 -04:00
The function `nixosTest` takes a [module ](https://nixos.org/manual/nixos/stable/#sec-writing-modules ) to specify the [test options ](https://nixos.org/manual/nixos/stable/index.html#sec-test-options-reference ).
2023-10-06 17:14:23 -04:00
Because this module only sets configuration values, one can use the abbreviated module notation.
2023-10-06 16:40:23 -04:00
The following configuration values must be set:
2022-08-16 10:27:42 -04:00
2023-06-19 16:19:29 -04:00
- [`name` ](https://nixos.org/manual/nixos/stable/index.html#test-opt-name ) defines the name of the test.
2022-08-16 10:27:42 -04:00
2023-06-19 16:19:29 -04:00
- [`nodes` ](https://nixos.org/manual/nixos/stable/index.html#test-opt-nodes ) contains a set of named configurations, because a test script can involve more than one virtual machine.
2023-10-06 16:40:35 -04:00
Each virtual machine is created from a NixOS configuration.
2022-08-16 10:27:42 -04:00
2023-09-11 09:17:16 -04:00
- [`testScript` ](https://nixos.org/manual/nixos/stable/index.html#test-opt-testScript ) defines the Python test script, either as literal string or as a function that takes a `nodes` attribute.
2022-08-16 10:27:42 -04:00
This Python test script can access the virtual machines via the names used for the `nodes` .
It has super user rights in the virtual machines.
2023-10-06 16:40:35 -04:00
In the Python script each virtual machine is accessible via the `machine` object.
2022-08-16 10:27:42 -04:00
NixOS provides [various methods ](https://nixos.org/manual/nixos/stable/index.html#ssec-machine-objects ) to run tests on these configurations.
The test framework automatically starts the virtual machines and runs the Python script.
## Minimal example
2022-10-12 07:08:53 -04:00
As a minimal test on the default configuration, we will check if the user `root` and `alice` can run Firefox.
2023-09-11 09:13:00 -04:00
We will build the example up from scratch.
2023-11-01 21:30:40 -04:00
1. Use a [pinned version of Nixpkgs ](ref-pinning-nixpkgs ), and [explicitly set configuration options and overlays ](nixpkgs-config ) to avoid them being inadvertently overridden by global configuration:
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
```nix
let
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
pkgs = import nixpkgs { config = {}; overlays = []; };
in
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
pkgs.nixosTest {
# ...
}
```
2022-08-16 10:27:42 -04:00
2023-10-06 17:15:14 -04:00
1. Label the test with a descriptive name:
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
```nix
name = "minimal-test";
```
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
1. Because this example only uses one virtual machine, the node we specify is simply called `machine` .
This name is arbitrary and can be chosen freely.
As configuration you use the relevant parts of the default configuration, [that we used in a previous tutorial ](<nixos-vms> ):
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
```nix
nodes.machine = { config, pkgs, ... }: {
users.users.alice = {
isNormalUser = true;
extraGroups = [ "wheel" ];
packages = with pkgs; [
firefox
tree
];
};
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
system.stateVersion = "22.11";
};
```
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
1. This is the test script:
2023-09-11 05:49:20 -04:00
2023-10-06 16:41:15 -04:00
```python
machine.wait_for_unit("default.target")
machine.succeed("su -- alice -c 'which firefox'")
machine.fail("su -- root -c 'which firefox'")
```
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
This Python script refers to `machine` which is the name chosen for the virtual machine configuration used in the `nodes` attribute set.
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
The script waits until systemd reaches `default.target` .
It uses the `su` command to switch between users and the `which` command to check if the user has access to `firefox` .
It expects that the command `which firefox` to succeed for user `alice` and to fail for `root` .
2022-08-16 10:27:42 -04:00
2023-10-06 16:41:15 -04:00
This script will be the value of the `testScript` attribute.
2022-08-16 10:27:42 -04:00
The complete `minimal-test.nix` file content looks like the following:
2021-12-10 09:48:34 -05:00
2023-10-06 16:41:15 -04:00
```nix
2022-08-16 10:27:42 -04:00
let
2023-09-11 12:19:29 -04:00
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
2023-09-11 09:08:42 -04:00
pkgs = import nixpkgs { config = {}; overlays = []; };
2022-08-16 10:27:42 -04:00
in
2023-10-06 16:41:15 -04:00
pkgs.nixosTest {
name = "minimal-test";
nodes.machine = { config, pkgs, ... }: {
users.users.alice = {
isNormalUser = true;
extraGroups = [ "wheel" ];
packages = with pkgs; [
firefox
tree
];
2022-08-16 10:27:42 -04:00
};
2023-10-06 16:41:15 -04:00
system.stateVersion = "22.11";
};
testScript = ''
machine.wait_for_unit("default.target")
machine.succeed("su -- alice -c 'which firefox'")
machine.fail("su -- root -c 'which firefox'")
'';
}
2022-08-16 10:27:42 -04:00
```
## Running tests
2023-09-11 09:13:00 -04:00
To set up all machines and run the test script:
2022-08-16 10:27:42 -04:00
```shell-session
$ nix-build minimal-test.nix
```
...
test script finished in 10.96s
cleaning up
killing machine (pid 10)
(0.00 seconds)
/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-minimal-test
2023-09-11 09:13:00 -04:00
## Interactive Python shell in the virtual machine
2022-08-16 10:27:42 -04:00
When developing tests or when something breaks, it’ s useful to interactively tinker with the test or access a terminal for a machine.
2023-09-11 09:13:00 -04:00
To start an interactive Python session with the testing framework:
2022-08-16 10:27:42 -04:00
```shell-session
$ $(nix-build -A driverInteractive minimal-test.nix)/bin/nixos-test-driver
```
2023-09-11 09:13:00 -04:00
Here you can run any of the testing operations.
Execute the `testScript` attribute from `minimal-test.nix` with the `test_script()` function.
2022-08-16 10:27:42 -04:00
2023-09-11 09:13:00 -04:00
If a virtual machine is not yet started, the test environment takes care of it on the first call of a method on a `machine` object.
2022-08-16 10:27:42 -04:00
2023-09-11 09:13:00 -04:00
But you can also manually trigger the start of the virtual machine with:
2022-08-16 10:27:42 -04:00
```shell-session
>>> machine.start()
```
for a specific node,
or
```shell-session
>>> start_all()
```
2023-09-11 09:13:00 -04:00
for all nodes.
2022-08-16 10:27:42 -04:00
You can enter a interactive shell on the virtual machine using:
```shell-session
>>> machine.shell_interact()
```
2023-09-11 09:13:00 -04:00
and run shell commands like:
2022-08-16 10:27:42 -04:00
```shell-session
uname -a
```
Linux server 5.10.37 #1 -NixOS SMP Fri May 14 07:50:46 UTC 2021 x86_64 GNU/Linux
2023-09-11 09:13:00 -04:00
< details > < summary > Re-running successful tests< / summary >
<!-- FIXME: this should be a separate recipe that can be linked to, as it's a bit of knowledge one will need now and again. -->
2022-08-16 10:27:42 -04:00
Because test results are kept in the Nix store, a successful test is cached.
This means that Nix will not run the test a second time as long as the test setup (node configuration and test script) stays semantically the same.
Therefore, to run a test again, one needs to remove the result.
If you would try to delete the result using the symbolic link, you will get the following error:
```shell-session
nix-store --delete ./result
```
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed' since it is still alive. To find out why, use: nix-store --query --roots
Instead, remove the symbolic link and only then remove the cached result:
```shell-session
rm ./result
nix-store --delete /nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed
```
This can be also done with one command:
```shell-session
result=$(readlink -f ./result) rm ./result & & nix-store --delete $result
```
2023-09-11 12:10:38 -04:00
< / details >
2022-08-16 10:27:42 -04:00
2023-10-06 16:45:41 -04:00
## Tests with multiple virtual machines
2022-08-16 10:27:42 -04:00
2023-10-06 16:46:41 -04:00
Tests can involve multiple virtual machines, for example to test client-server-communication.
2022-08-16 10:27:42 -04:00
2023-10-06 16:46:41 -04:00
The following example setup includes:
- A virtual machine named `server` running [nginx ](https://nginx.org/en/ ) with default configuration.
- A virtual machine named `client` that has `curl` available to make an HTTP request.
2022-08-16 10:27:42 -04:00
- A `testScript` orchestrating testing logic between `client` and `server` .
2023-10-06 16:47:06 -04:00
The complete `client-server-test.nix` file content looks like the following:
2022-08-16 10:27:42 -04:00
```{code-block}
2021-12-10 09:48:34 -05:00
let
2023-10-06 16:47:06 -04:00
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
2023-09-11 12:19:29 -04:00
pkgs = import nixpkgs { config = {}; overlays = []; };
2023-10-06 04:59:27 -04:00
in
2023-10-06 16:47:06 -04:00
pkgs.nixosTest {
name = "client-server-test";
nodes.server = { pkgs, ... }: {
networking = {
firewall = {
allowedTCPPorts = [ 80 ];
2021-12-10 09:48:34 -05:00
};
};
2023-10-06 16:47:06 -04:00
services.nginx = {
enable = true;
virtualHosts."server" = {};
2023-09-11 11:47:24 -04:00
};
2023-10-06 16:47:06 -04:00
};
nodes.client = { pkgs, ... }: {
environment.systemPackages = with pkgs; [
curl
];
};
testScript = ''
server.wait_for_unit("default.target")
client.wait_for_unit("default.target")
client.succeed("curl http://server/ | grep -o \"Welcome to nginx!\"")
'';
}
2021-12-10 09:48:34 -05:00
```
2023-10-06 16:47:06 -04:00
The test script performs the following steps:
1) Start the server and wait for it to be ready.
1) Start the client and wait for it to be ready.
1) Run `curl` on the client and use `grep` to check the expected return string.
The test passes or fails based on the return value.
Run the test:
```shell-session
$ nix-build server-client-test.nix
```
2021-12-10 09:48:34 -05:00
2023-10-06 16:49:30 -04:00
## Additional information regarding NixOS tests
2023-09-11 09:13:00 -04:00
2023-10-06 16:49:30 -04:00
- Running integration tests on CI requires hardware acceleration, which many CIs do not support.
2023-09-11 09:17:16 -04:00
2023-10-06 16:49:30 -04:00
To run integration tests in [GitHub Actions ](<github-actions> ) see [how to disable hardware acceleration ](https://github.com/cachix/install-nix-action#how-do-i-run-nixos-tests ).
2023-09-11 09:13:00 -04:00
2023-10-06 16:49:30 -04:00
- NixOS comes with a large set of tests that can serve as educational examples.
2023-09-11 09:17:16 -04:00
2023-10-06 16:49:30 -04:00
A good inspiration is [Matrix bridging with an IRC ](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/matrix/appservice-irc.nix ).
2023-09-11 09:13:00 -04:00
<!-- TODO: move examples from https://nixos.wiki/wiki/NixOS_Testing_library to the NixOS manual and troubleshooting tips to nix.dev -->