From f4464873f514a1aa05ca43178dfca6e1c978f0c9 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Sat, 27 Jul 2024 13:01:56 +0200 Subject: [PATCH 1/2] tests/nixos/remote-builds: Print hello world to stderr Trying to learn more about enigmatic spurious hang at https://hydra.nixos.org/build/267517233/nixlog/8 - builder1 seems to have started properly - ssh connection and session are established - ssh client doesn't exit or client.succeed does not return for some reason. Seeing the stdout on the console might give a tiny bit more info. --- tests/nixos/remote-builds.nix | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/nixos/remote-builds.nix b/tests/nixos/remote-builds.nix index 1661203ec..8813832f5 100644 --- a/tests/nixos/remote-builds.nix +++ b/tests/nixos/remote-builds.nix @@ -104,7 +104,10 @@ in builder.succeed("mkdir -p -m 700 /root/.ssh") builder.copy_from_host("key.pub", "/root/.ssh/authorized_keys") builder.wait_for_unit("sshd") - client.succeed(f"ssh -o StrictHostKeyChecking=no {builder.name} 'echo hello world'") + client.succeed(f""" + ssh -o StrictHostKeyChecking=no {builder.name} \ + 'echo hello world on $(hostname)' >&2 + """) # Perform a build and check that it was performed on the builder. out = client.succeed( From 7c5a0b06a4a7908560bca273049e561c0f8193b2 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Sat, 27 Jul 2024 13:08:30 +0200 Subject: [PATCH 2/2] tests/nixos/remote-builds: Wait for multi-user This should make the test more robust, considering the strange hang in https://hydra.nixos.org/build/267517233/nixlog/8 `builder` seems to have reached `multi-user.target` before the SSH connection was established, but this seems to be coincidental. This does tell us that enforcing this has a minimal cost in terms of runtime. Waiting for `multi-user.target` on the client is honestly paranoid, but flaky tests are very bad for productivity. --- tests/nixos/remote-builds.nix | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/nixos/remote-builds.nix b/tests/nixos/remote-builds.nix index 8813832f5..8ddf6ad02 100644 --- a/tests/nixos/remote-builds.nix +++ b/tests/nixos/remote-builds.nix @@ -104,6 +104,11 @@ in builder.succeed("mkdir -p -m 700 /root/.ssh") builder.copy_from_host("key.pub", "/root/.ssh/authorized_keys") builder.wait_for_unit("sshd") + # Make sure the builder can handle our login correctly + builder.wait_for_unit("multi-user.target") + # Make sure there's no funny business on the client either + # (should not be necessary, but we have reason to be careful) + client.wait_for_unit("multi-user.target") client.succeed(f""" ssh -o StrictHostKeyChecking=no {builder.name} \ 'echo hello world on $(hostname)' >&2