automatic runtime
+ automatic runtime dependencies
+
+
+ Welcome to the 9th Nix pill. In the previous
+ 8th pill we wrote a generic builder
+ for autotools projects. We feed build dependencies, a source tarball, and
+ we get a Nix derivation as a result.
+
+
+
+ Today we stop by the GNU hello world program to analyze build and runtime
+ dependencies, and enhance the builder in order to avoid unnecessary runtime
+ dependencies.
+
+
+
+ Build dependencies
+
+
+ Let's start analyzing build dependencies for our GNU hello world package:
+
+
+
+
+
+ It has exactly the derivations referenced in the derivation
+ function, nothing more, nothing less. Some of them might not be used at
+ all, however given that our generic mkDerivation function always pulls
+ such dependencies (think of it like
+ build-essential
+ of Debian), for every package you build from now on, you will have these
+ packages in the nix store.
+
+
+
+ Why are we looking at .drv files? Because the hello.drv file is the
+ representation of the build action to perform in order to build the hello
+ out path, and as such it also contains the input derivations needed to be
+ built before building hello.
+
+
+
+
+ Digression about NAR files
+
+
+ NAR is the Nix ARchive. First question: why not tar? Why another archiver?
+ Because commonly used archivers are not deterministic. They add padding,
+ they do not sort files, they add timestamps, etc.. Hence NAR, a very
+ simple deterministic archive format being used by Nix for deployment.
+ NARs are also used extensively within Nix itself as we'll see below.
+
+
+
+ For the rationale and implementation details you can find more in the
+ Dolstra's PhD Thesis.
+
+
+
+ To create NAR archives, it's possible to use
+ nix-store --dump and
+ nix-store --restore. Those two commands work
+ regardless of /nix/store.
+
+
+
+
+ Runtime dependencies
+
+
+ Something is different for runtime dependencies however. Build
+ dependencies are automatically recognized by Nix once they are used in
+ any derivation call, but we never specify what are the
+ runtime dependencies for a derivation.
+
+
+
+ There's really black magic involved. It's something that at first glance
+ makes you think "no, this can't work in the long term", but at the same
+ it works so well that a whole operating system is built on top of this
+ magic.
+
+
+
+ In other words, Nix automatically computes all the runtime dependencies
+ of a derivation, and it's possible thanks to the hash of the store paths.
+
+
+
+ Steps:
+
+
+
+
+
+ Dump the derivation as NAR, a serialization of the derivation output.
+ Works fine whether it's a single file or a directory.
+
+
+
+
+ For each build dependency .drv and its relative out path, search the
+ contents of the NAR for this out path.
+
+
+
+
+ If found, then it's a runtime dependency.
+
+
+
+
+
+ You get really all the runtime dependencies, and that's why Nix
+ deployments are so easy.
+
+
+
+
+
+ Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
+
+
+
+
+
+ Oh Nix added gcc because its out path is mentioned in the "hello" binary.
+ Why is that? That's the
+ ld rpath.
+ It's the list of directories where libraries can be found at runtime. In
+ other distributions, this is usually not abused. But in Nix, we have to
+ refer to particular versions of libraries, thus the rpath has an
+ important role.
+
+
+
+ The build process adds that gcc lib path thinking it may be useful at
+ runtime, but really it's not. How do we get rid of it? Nix authors have
+ written another magical tool called
+ patchelf, which
+ is able to reduce the rpath to the paths that are really used by the
+ binary.
+
+
+
+ Not only, even after reducing the rpath the hello binary would still
+ depend upon gcc. Because of debugging information. For that, the well
+ known
+ strip
+ can be used.
+
+
+
+
+ Another phase in the builder
+
+
+ We will add a new phase to our autotools builder. The builder has these
+ phases already:
+
+
+
+
+
+ First the environment is set up
+
+
+
+
+ Unpack phase: we unpack the sources in the current directory
+ (remember, Nix changes dir to a temporary directory first)
+
+
+
+
+ Change source root to the directory that has been unpacked
+
+
+
+
+ Configure phase: ./configure
+
+
+
+
+ Build phase: make
+
+
+
+
+ Install phase: make install
+
+
+
+
+
+ We add a new phase after the installation phase, which we call
+ fixup phase. At the end of the
+ builder.sh follows:
+
+
+
+
+
+ That is, for each file we run patchelf --shrink-rpath
+ and strip. Note that we used two new commands here,
+ find and patchelf. These two
+ deserve a place in baseInputs of
+ autotools.nix as findutils and
+ patchelf.
+
+
+
+ Rebuild hello.nix and...:
+
+
+
+
+
+ ...only glibc is the runtime dependency. Exactly what we wanted.
+
+
+
+ The package is self-contained, copy its closure on another machine and
+ you will be able to run it. I remind you the very few components under
+ the /nix/store necessary to run nix
+ when we installed it.
+ The hello binary will use that exact version of glibc library and
+ interpreter, not the system one:
+
+
+
+
+
+ Of course, the executable runs fine as long as everything is under the
+ /nix/store path.
+
+
+
+
+ Conclusion
+
+
+ Short post compared to previous ones as I'm still on vacation, but I hope
+ you enjoyed it. Nix provides tools with cool features. In particular, Nix
+ is able to compute all runtime dependencies automatically for us. Not
+ only shared libraries, but also referenced executables, scripts, Python
+ libraries etc..
+
+
+
+ This makes packages self-contained, because we're sure (apart data and
+ configuration) that copying the runtime closure on another machine is
+ sufficient to run the program. That's why Nix has
+ one-click install,
+ or
+ reliable deployment in the cloud.
+ All with one tool.
+
+
+
+
+ Next pill
+
+
+ ...we will introduce nix-shell. With nix-build we build derivations
+ always from scratch: the source gets unpacked, configured, built and
+ installed. But this may take a long time, think of WebKit. What if we
+ want to apply some small changes and compile incrementally instead, yet
+ keeping a self-contained environment similar to nix-build?
+
+
diff --git a/pills/09/build-hello-nix.txt b/pills/09/build-hello-nix.txt
new file mode 100644
index 0000000..b364bd8
--- /dev/null
+++ b/pills/09/build-hello-nix.txt
@@ -0,0 +1,5 @@
+$ nix-build hello.nix
+[...]
+$ nix-store -q --references result
+/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
+/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello
diff --git a/pills/09/find.txt b/pills/09/find.txt
new file mode 100644
index 0000000..361ce92
--- /dev/null
+++ b/pills/09/find.txt
@@ -0,0 +1 @@
+find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null
diff --git a/pills/09/instantiate-hello.txt b/pills/09/instantiate-hello.txt
new file mode 100644
index 0000000..53d15b7
--- /dev/null
+++ b/pills/09/instantiate-hello.txt
@@ -0,0 +1,8 @@
+$ nix-instantiate hello.nix
+/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
+$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
+/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
+$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
+/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
+/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
+/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
diff --git a/pills/09/instantiate.txt b/pills/09/instantiate.txt
new file mode 100644
index 0000000..0d5e5f7
--- /dev/null
+++ b/pills/09/instantiate.txt
@@ -0,0 +1,15 @@
+$ nix-instantiate hello.nix
+/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
+$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
+/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz
+/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv
+/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv
+/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv
+/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh
+/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv
+/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv
+/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv
+/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv
+/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv
+/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv
+/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv
diff --git a/pills/09/ldd-hello.txt b/pills/09/ldd-hello.txt
new file mode 100644
index 0000000..20c61e8
--- /dev/null
+++ b/pills/09/ldd-hello.txt
@@ -0,0 +1,4 @@
+$ ldd result/bin/hello
+ linux-vdso.so.1 (0x00007fff11294000)
+ libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000)
+ /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)
diff --git a/pills/09/strings.txt b/pills/09/strings.txt
new file mode 100644
index 0000000..f766ab9
--- /dev/null
+++ b/pills/09/strings.txt
@@ -0,0 +1,2 @@
+$ strings result/bin/hello|grep gcc
+/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64