This guide introduces the functionality of Nix Package Manager to write automated tests to debug NixOS configurations independent of a working NixOS installation.
- A working installation of [Nix Package Manager](https://nixos.org/manual/nix/stable/installation/installation.html) or [NixOS](https://nixos.org/manual/nixos/stable/index.html#sec-installation).
- Basic knowledge of the [Nix language](https://nixos.org/manual/nix/stable/language/index.html).
NixOS provides a [test environment](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests) to automate integration testing.
You can define tests that make use of a set of declarative NixOS configurations and use a Python shell to interact with them through [QEMU](https://www.qemu.org/) as the backend.
Those tests are widely used to ensure that NixOS works as intended, so in general they are called [NixOS Tests](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests).
They can be written and launched outside of NixOS, on any Linux machine (with [MacOS support coming soon](https://github.com/NixOS/nixpkgs/issues/108984)).
Integration tests are reproducible due to the design properties of Nix, making them a valuable part of a Continuous Integration (CI) pipeline.
You define the name of the test using a descriptive name like "minimal-test":
```nix
name = "minimal-test";
```
#### Nodes
Because this example only uses one virtual machine the node you specify is simply called `machine`. As configuration you use the default configuration as {ref}`discussed before <nixos-vms>`:
```nix
nodes.machine = { config, pkgs, ... }: {
# ...
};
```
#### Test script
This is the test script:
```python
machine.wait_for_unit("default.target")
machine.succeed("su -- alice -c 'which firefox'")
machine.fail("su -- root -c 'which firefox'")
```
This Python script is referring to `machine` which is the name chosen for the virtual machine configuration used in the nodes attribute set.
## Interactive Python shell to interact with virtual machine
When developing tests or when something breaks, it’s useful to interactively tinker with the test or access a terminal for a machine.
To interactively start a Python session with the testing framework:
```shell-session
$ $(nix-build -A driverInteractive minimal-test.nix)/bin/nixos-test-driver
```
You can run any of the testing operations.
The `testScript` attribute from `minimal-test.nix` definition can be executed with `test_script()` function.
Within this Python shell you can enter a interactive shell and run Python commands like those in the test script.
If a virtual machine is not yet started, the test environment takes care of it on the first call of a method of a `machine` object.
But you can also manually trigger the start of the virtual machine by using
```shell-session
>>> machine.start()
```
for a specific node,
or
```shell-session
>>> start_all()
```
for all specified nodes.
You can enter a interactive shell on the virtual machine using:
```shell-session
>>> machine.shell_interact()
```
and run commandline commands like:
```shell-session
uname -a
```
Linux server 5.10.37 #1-NixOS SMP Fri May 14 07:50:46 UTC 2021 x86_64 GNU/Linux
## Re-run successful tests
Because test results are kept in the Nix store, a successful test is cached.
This means that Nix will not run the test a second time as long as the test setup (node configuration and test script) stays semantically the same.
Therefore, to run a test again, one needs to remove the result.
If you would try to delete the result using the symbolic link, you will get the following error:
```shell-session
nix-store --delete ./result
```
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed' since it is still alive. To find out why, use: nix-store --query --roots
Instead, remove the symbolic link and only then remove the cached result:
This example uses the use-case of a REST interface to a Postgres database.
The following example Nix expression is adapted from [How to use NixOS for lightweight integration tests](https://www.haskellforall.com/2020/11/how-to-use-nixos-for-lightweight.html).
This tutorial follows [PostgREST tutorial](https://postgrest.org/en/stable/tutorials/tut0.html), a generic [RESTful API](https://restfulapi.net/) for PostgreSQL.
If you skim over the official tutorial, you'll notice there's quite a bit of setup in order to test if all the steps work.
The setup includes:
- A virtual machine named `server` running postgreSQL and postgREST.
- A virtual machine named `client` running HTTP client queries using `curl`.
- A `testScript` orchestrating testing logic between `client` and `server`.
:::{note}
Because some of the needed packages of this example are broken in 22.11 release this example uses a specific revision of nixpkgs.
Additionally this example shows the value of {ref}`pinning <ref-pinning-nixpkgs>` a test to a specific revision of `nixpkgs`.
Tests that make use of nixpkgs versions before 22.11 need to choose names that do not contain whitespaces.
:::
The complete `postgrest.nix` file looks like the following:
- NixOS Tests section in [NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests)
- Running integration tests on CI requires hardware acceleration, which many CIs do not support.
To run integration tests on {ref}`GitHub Actions <github-actions>` see [how to disable hardware acceleration](https://github.com/cachix/install-nix-action#user-content-how-can-i-run-nixos-tests).
- NixOS comes with a large set of tests that serve also as educational examples.
A good inspiration is [Matrix bridging with an IRC](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/matrix/appservice-irc.nix).
- [NixOS.wiki on NixOS Testing library](https://nixos.wiki/wiki/NixOS_Testing_library) seems to be mostly outdated (last edit 05.11.2021)