mirror of
https://github.com/NixOS/hydra.git
synced 2024-10-17 16:37:26 -04:00
578a3d2292
We've seen many fails on ofborg, at lot of them ultimately appear to come down to
a timeout being hit, resulting in something like this:
Failure executing slapadd -F /<path>/slap.d -b dc=example -l /<path>/load.ldif.
Hopefully this resolves it for most cases.
I've done some endurance testing and this helps a lot.
some other commands also regularly time-out with high load:
- hydra-init
- hydra-create-user
- nix-store --delete
This should address most issues with tests randomly failing.
Used the following script for endurance testing:
```
import os
import subprocess
run_counter = 0
fail_counter = 0
while True:
try:
run_counter += 1
print(f"Starting run {run_counter}")
env = os.environ
env["YATH_JOB_COUNT"] = "20"
result = subprocess.run(["perl", "t/test.pl"], env=env)
if (result.returncode != 0):
fail_counter += 1
print(f"Finish run {run_counter}, total fail count: {fail_counter}")
except KeyboardInterrupt:
print(f"Finished {run_counter} runs with {fail_counter} fails")
break
```
In case someone else wants to do it on their system :).
Note that YATH_JOB_COUNT may need to be changed loosely based on your
cores.
I only have 4 cores (8 threads), so for others higher numbers might
yield better results in hashing out unstable tests.
|
||
|---|---|---|
| .. | ||
| content-addressed | ||
| evaluator | ||
| Helper/AddBuilds | ||
| Hydra | ||
| input-types | ||
| jobs | ||
| lib | ||
| queue-runner | ||
| scripts | ||
| api-test.t | ||
| build-products.t | ||
| Makefile.am | ||
| perlcritic.pl | ||
| s3-backup-test.config | ||
| s3-backup-test.pl | ||
| setup-notifications-jobset.pl | ||
| test.pl | ||