Automatic Runtime Dependencies
Welcome to the 9th Nix pill. In the previous
8th pill we wrote a generic builder
for autotools projects. We fed in build dependencies and a source tarball, and
we received a Nix derivation as a result.
Today we stop by the GNU hello program to analyze build and runtime
dependencies, and we enhance our builder to eliminate unnecessary runtime
dependencies.
Build dependencies
Let's start analyzing build dependencies for our GNU hello package:
It has precisely the derivations referenced in the derivation function;
nothing more, nothing less. Of course, we may not use some of them at all.
However, given that our generic mkDerivation function always pulls
such dependencies (think of it like
build-essential
from Debian), we will already have these packages in the nix store for any future packages that
need them.
Why are we looking at .drv files? Because the hello.drv
file is the representation of the build action that builds the hello
out path. As such, it contains the input derivations needed before building
hello.
Digression about NAR files
The NAR format is the "Nix ARchive". This format was designed due to
existing archive formats, such as tar, being insufficient.
Nix benefits from deterministic build tools, but commonly used archivers
lack this property: they add padding, they do not sort files, they add timestamps,
and so on. This can result in directories containing bit-identical files turning into
non-bit-identical archives, which leads to different hashes.
Thus the NAR format was developed as a simple, deterministic
archive format. NARs are used extensively within Nix, as we will
see below.
For more rationale and implementation details behind NAR see
Dolstra's PhD Thesis.
To create NAR archives from store paths, we can use
nix-store --dump and
nix-store --restore.
Runtime dependencies
We now note that Nix automatically recognized build dependencies once our
derivation call referred to them, but we never specified the
runtime dependencies.
Nix handles runtime dependencies for us automatically. The technique it uses
to do so may seem fragile at first glance, but it works so well that the NixOS
operating system is built off of it. The underlying mechanism relies on the
hash of the store paths. It proceeds in three steps:
Dump the derivation as a NAR. Recall that this is a serialization of
the derivation output -- meaning this works fine whether the output
is a single file or a directory.
For each build dependency .drv and its relative out path,
search the contents of the NAR for this out path.
If the path is found, then it's a runtime dependency.
The snippet below shows the dependencies for hello.
We see that glibc and gcc are runtime dependencies.
Intuitively, gcc shouldn't be in this list! Displaying the
printable strings in the hello binary shows that the out path
of gcc does indeed appear:
This is why Nix added gcc. But why is that path present in the
first place? The answer is that it is the ld rpath: the list of
directories where libraries can be found at runtime. In other distributions,
this is usually not abused. But in Nix, we have to refer to particular versions
of libraries, and thus the rpath has an important role.
The build process adds the gcc lib path thinking it may be useful
at runtime, but this isn't necessary. To address issues like these, Nix provides
a tool called patchelf,
which reduces the rpath to the paths that are actually used by the binary.
Even after reducing the rpath, the hello binary would still
depend upon gcc because of some debugging information. This
unnecessarily increases the size of our runtime
dependencies. We'll explore how strip
can help us with that in the next section.
Another phase in the builder
We will add a new phase to our autotools builder. The builder has six
phases already:
The "environment setup" phase
The "unpack phase": we unpack the sources in the current directory
(remember, Nix changes to a temporary directory first)
The "change directory" phase, where we change source root to the
directory that has been unpacked
The "configure" phase: ./configure
The "build" phase: make
The "install" phase: make install
Now we will add a new phase after the installation phase, which we call
the "fixup" phase. At the end of the
builder.sh, we append:
That is, for each file we run patchelf --shrink-rpath
and strip. Note that we used two new commands here,
find and patchelf. These must be
added to our derivation.
Exercise: Add findutils
and patchelf to the baseInputs of
autotools.nix.
Now, we rebuild hello.nix...
and we see that glibc is a runtime dependency. This is
exactly what we wanted.
The package is self-contained. This means that we can copy its closure onto
another machine and we will be able to run it. Remember, only a very few
components under the /nix/store are required to
run nix.
The hello binary will use the exact version of glibc
library and interpreter referred to in the binary, rather than the system one:
Of course, the executable will run fine as long as everything is under the
/nix/store path.
Conclusion
We saw some of the tools Nix provides, along with their features.
In particular, we saw how Nix is able to compute runtime dependencies
automatically. This is not limited to only shared libraries,
but can also reference executables, scripts, Python libraries, and so
forth.
Approaching builds in this way makes packages self-contained, ensuring
(apart from data and configuration) that copying the runtime closure onto
another machine is sufficient to run the program. This enables us to run programs
without installation using nix-shell, and forms the basis for
reliable deployment in the cloud.
Next pill
The next pill will introduce nix-shell. With
nix-build, we've always built derivations from
scratch: the source gets unpacked, configured, built, and installed.
But this can take a long time for large packages. What if we want to
apply some small changes and compile incrementally instead, yet still
want to keep a self-contained environment similar to nix-build?
nix-shell enables this.