NixOS/nix.dev

Fork 0

mirror of https://github.com/NixOS/nix.dev.git synced 2024-10-18 14:32:43 -04:00

Valentin Gagarin bfb1858a51

formatting and consistency

2023-09-11 15:08:42 +02:00

14 KiB

Raw Blame History

(integration-testing-vms)=

Integration testing using virtual machines

This tutorial aims to be compatible with NixOS release 22.11.

What will you learn?

This guide introduces the functionality of Nixpkgs to write automated tests to debug NixOS configurations independent of a working NixOS installation.

What do you need?

A working installation of Nix Package Manager or NixOS.
Basic knowledge of the Nix language.
Basic knowledge of NixOS configuration.

Introduction

Nixpkgs provides a test environment to automate integration testing. You can define tests that make use of a set of declarative NixOS configurations and use a Python shell to interact with them through QEMU as the backend. Those tests are widely used to ensure that NixOS works as intended, so in general they are called NixOS Tests. They can be written and launched outside of NixOS, on any Linux machine (with MacOS support coming soon). Integration tests are reproducible due to the design properties of Nix, making them a valuable part of a Continuous Integration (CI) pipeline.

The `nixosTest` function

To setup a test you make use of the nixosTest function. The scaffolding of a test nix file looks like the following:

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
  pkgs = import nixpkgs { config = {}; overlays = []; };
in
  pkgs.nixosTest {
    name = "test-name";
    nodes = {
      machine1 = { config, pkgs, ... }: {
        # ...
      };
      machine2 = { config, pkgs, ... }: {
        # ...
      };
    }
    testScript = { nodes, ... }: ''
      # ...
    '';
  }

The function nixosTest takes an attribute set that follows the module convention to specify the test. Because the attribute set only defines options, one can use the abbreviated form of the module convention. The attribute set needs to define the following options:

name defines the name of the test.
nodes contains a set of named configurations, because a test script can involve more than one virtual machine. Each virtual machine is setup using a NixOS configuration.
testScript defines the Python test script, either as literal string. This Python test script can access the virtual machines via the names used for the nodes. It has super user rights in the virtual machines. In the Python script is each virtual machine is accessible via the machine object. NixOS provides various methods to run tests on these configurations.

The test framework automatically starts the virtual machines and runs the Python script.

Minimal example

As a minimal test on the default configuration, we will check if the user root and alice can run Firefox. As recommended you use an explicitly pinned version of Nixpkgs:

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
  pkgs = import nixpkgs { config = {}; overlays = []; };
in
  pkgs.nixosTest {
    # ...
  }

Options

Name

You define the name of the test using a descriptive name like "minimal-test":

name = "minimal-test";

Nodes

Because this example only uses one virtual machine the node you specify is simply called machine. As configuration you use the relevant parts of the default configuration, that we used before:

nodes.machine = { config, pkgs, ... }: {
  users.users.alice = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    packages = with pkgs; [
      firefox
      tree
    ];
  };

  system.stateVersion = "22.11";
};

Test script

This is the test script:

machine.wait_for_unit("default.target")
machine.succeed("su -- alice -c 'which firefox'")
machine.fail("su -- root -c 'which firefox'")

This Python script is referring to machine which is the name chosen for the virtual machine configuration used in the nodes attribute set.

The script waits until start up is reaching systemd default.target. It utilizes the su command to switch between users and the which command to see if the user has access to firefox. It expects that the command which firefox to succeed for user alice and to fail for root.

This script needs to be located in a function inside the testScript attribute.

Test file

The complete minimal-test.nix file content looks like the following:

let
    nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
  pkgs = import nixpkgs { config = {}; overlays = []; };
in
  pkgs.nixosTest {
    name = "minimal-test";
    nodes.machine = { config, pkgs, ... }: {

      users.users.alice = {
        isNormalUser = true;
        extraGroups = [ "wheel" ];
        packages = with pkgs; [
          firefox
          tree
        ];
      };

      system.stateVersion = "22.11";
    };
    testScript = ''
      machine.wait_for_unit("default.target")
      machine.succeed("su -- alice -c 'which firefox'")
      machine.fail("su -- root -c 'which firefox'")
    '';
  }

Running tests

To set up all machines and execute the test script:

$ nix-build minimal-test.nix

...
test script finished in 10.96s
cleaning up
killing machine (pid 10)
(0.00 seconds)
/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-minimal-test

Interactive Python shell to interact with virtual machine

When developing tests or when something breaks, it’s useful to interactively tinker with the test or access a terminal for a machine.

To interactively start a Python session with the testing framework:

$ $(nix-build -A driverInteractive minimal-test.nix)/bin/nixos-test-driver

You can run any of the testing operations. The testScript attribute from minimal-test.nix definition can be executed with test_script() function.

Within this Python shell you can enter a interactive shell and run Python commands like those in the test script.

If a virtual machine is not yet started, the test environment takes care of it on the first call of a method of a machine object.

But you can also manually trigger the start of the virtual machine by using

>>> machine.start()

for a specific node,

>>> start_all()

for all specified nodes.

You can enter a interactive shell on the virtual machine using:

>>> machine.shell_interact()

and run commandline commands like:

uname -a

Linux server 5.10.37 #1-NixOS SMP Fri May 14 07:50:46 UTC 2021 x86_64 GNU/Linux

Re-run successful tests

Because test results are kept in the Nix store, a successful test is cached. This means that Nix will not run the test a second time as long as the test setup (node configuration and test script) stays semantically the same. Therefore, to run a test again, one needs to remove the result.

If you would try to delete the result using the symbolic link, you will get the following error:

nix-store --delete ./result

finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed' since it is still alive. To find out why, use: nix-store --query --roots

Instead, remove the symbolic link and only then remove the cached result:

rm ./result
nix-store --delete /nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed

This can be also done with one command:

result=$(readlink -f ./result) rm ./result && nix-store --delete $result

Tests that need multiple virtual machines

Tests can utilize multiple virtual machines.

This example uses the use-case of a REST interface to a PostgreSQL database. The following example Nix expression is adapted from How to use NixOS for lightweight integration tests.

This tutorial follows PostgREST tutorial, a generic RESTful API for PostgreSQL.

If you skim over the official tutorial, you'll notice there's quite a bit of setup in order to test if all the steps work.

The setup includes:

A virtual machine named server running PostgreSQL and PostgREST.
A virtual machine named client running HTTP client queries using curl.
A testScript orchestrating testing logic between client and server.

The complete postgrest.nix file looks like the following:

let
  # Pin nixpkgs, see pinning tutorial for more details
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz";
  pkgs = import nixpkgs { config = {}; overlays = []; };

  # Single source of truth for all tutorial constants
  database = "postgres";
  schema        = "api";
  table         = "todos";
  username      = "authenticator";
  password      = "mysecretpassword";
  webRole       = "web_anon";
  postgrestPort = 3000;

  # NixOS module shared between server and client
  sharedModule = {
    # Since it's common for CI not to have $DISPLAY available, you have to explicitly tell the tests "please don't expect any screen available"
    virtualisation.graphics = false;
  };

in 
  pkgs.nixosTest {
    # NixOS tests are run inside a virtual machine, and here you specify system of the machine.
    system = "x86_64-linux";
    name = "postgres-test";
    nodes = {
      server = { config, pkgs, ... }: {
        imports = [ sharedModule ];

        networking.firewall.allowedTCPPorts = [ postgrestPort ];

        services.postgresql = {
          enable = true;

          initialScript = pkgs.writeText "initialScript.sql" ''
            create schema ${schema};

            create table ${schema}.${table} (
                id serial primary key,
                done boolean not null default false,
                task text not null,
                due timestamptz
            );

            insert into ${schema}.${table} (task) values ('finish tutorial 0'), ('pat self on back');

            create role ${webRole} nologin;
            grant usage on schema ${schema} to ${webRole};
            grant select on ${schema}.${table} to ${webRole};

            create role ${username} inherit login password '${password}';
            grant ${webRole} to ${username};
          '';
        };

        users = {
          mutableUsers = false;
          users = {
            # For ease of debugging the VM as the `root` user
            root.password = "";

            # Create a system user that matches the database user so that you
            # can use peer authentication. The tutorial defines a password,
            # but it's not necessary.
            "${username}".isSystemUser = true;
          };
        };

        systemd.services.postgrest = {
          wantedBy = [ "multi-user.target" ];
          after = [ "postgresql.service" ];
          script =
            let
              configuration = pkgs.writeText "tutorial.conf" ''
                  db-uri = "postgres://${username}:${password}@localhost:${toString config.services.postgresql.port}/${database}"
                  db-schema = "${schema}"
                  db-anon-role = "${username}"
              '';
            in "${pkgs.haskellPackages.postgrest}/bin/postgrest ${configuration}";
          serviceConfig.User = username;
        };
      };

      client = {
        imports = [ sharedModule ];
      };
    };

    # Disable linting for simpler debugging of the testScript
    skipLint = true;

    testScript = ''
      import json
      import sys

      start_all()

      server.wait_for_open_port(${toString postgrestPort})

      expected = [
          {"id": 1, "done": False, "task": "finish tutorial 0", "due": None},
          {"id": 2, "done": False, "task": "pat self on back", "due": None},
      ]

      actual = json.loads(
          client.succeed(
              "${pkgs.curl}/bin/curl http://server:${toString postgrestPort}/${table}"
          )
      )

      assert expected == actual, "table query returns expected content"
    '';
}

Unlike the previous example, the virtual machines need an expressive name to distinguish them. For this example we choose client and server.