diff --git a/pills/09-automatic-runtime.xml b/pills/09-automatic-runtime.xml index 8692de9..df7e56f 100644 --- a/pills/09-automatic-runtime.xml +++ b/pills/09-automatic-runtime.xml @@ -2,7 +2,278 @@ xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0" - xml:id="automatic-runtime"> + xml:id="automatic-runtime-dependencies"> -automatic runtime + automatic runtime dependencies + + + Welcome to the 9th Nix pill. In the previous + 8th pill we wrote a generic builder + for autotools projects. We feed build dependencies, a source tarball, and + we get a Nix derivation as a result. + + + + Today we stop by the GNU hello world program to analyze build and runtime + dependencies, and enhance the builder in order to avoid unnecessary runtime + dependencies. + + +
+ Build dependencies + + + Let's start analyzing build dependencies for our GNU hello world package: + + + + + + It has exactly the derivations referenced in the derivation + function, nothing more, nothing less. Some of them might not be used at + all, however given that our generic mkDerivation function always pulls + such dependencies (think of it like + build-essential + of Debian), for every package you build from now on, you will have these + packages in the nix store. + + + + Why are we looking at .drv files? Because the hello.drv file is the + representation of the build action to perform in order to build the hello + out path, and as such it also contains the input derivations needed to be + built before building hello. + +
+ +
+ Digression about NAR files + + + NAR is the Nix ARchive. First question: why not tar? Why another archiver? + Because commonly used archivers are not deterministic. They add padding, + they do not sort files, they add timestamps, etc.. Hence NAR, a very + simple deterministic archive format being used by Nix for deployment. + NARs are also used extensively within Nix itself as we'll see below. + + + + For the rationale and implementation details you can find more in the + Dolstra's PhD Thesis. + + + + To create NAR archives, it's possible to use + nix-store --dump and + nix-store --restore. Those two commands work + regardless of /nix/store. + +
+ +
+ Runtime dependencies + + + Something is different for runtime dependencies however. Build + dependencies are automatically recognized by Nix once they are used in + any derivation call, but we never specify what are the + runtime dependencies for a derivation. + + + + There's really black magic involved. It's something that at first glance + makes you think "no, this can't work in the long term", but at the same + it works so well that a whole operating system is built on top of this + magic. + + + + In other words, Nix automatically computes all the runtime dependencies + of a derivation, and it's possible thanks to the hash of the store paths. + + + + Steps: + + + + + + Dump the derivation as NAR, a serialization of the derivation output. + Works fine whether it's a single file or a directory. + + + + + For each build dependency .drv and its relative out path, search the + contents of the NAR for this out path. + + + + + If found, then it's a runtime dependency. + + + + + + You get really all the runtime dependencies, and that's why Nix + deployments are so easy. + + + + + + Ok glibc and gcc. Well, gcc really should not be a runtime dependency! + + + + + + Oh Nix added gcc because its out path is mentioned in the "hello" binary. + Why is that? That's the + ld rpath. + It's the list of directories where libraries can be found at runtime. In + other distributions, this is usually not abused. But in Nix, we have to + refer to particular versions of libraries, thus the rpath has an + important role. + + + + The build process adds that gcc lib path thinking it may be useful at + runtime, but really it's not. How do we get rid of it? Nix authors have + written another magical tool called + patchelf, which + is able to reduce the rpath to the paths that are really used by the + binary. + + + + Not only, even after reducing the rpath the hello binary would still + depend upon gcc. Because of debugging information. For that, the well + known + strip + can be used. + +
+ +
+ Another phase in the builder + + + We will add a new phase to our autotools builder. The builder has these + phases already: + + + + + + First the environment is set up + + + + + Unpack phase: we unpack the sources in the current directory + (remember, Nix changes dir to a temporary directory first) + + + + + Change source root to the directory that has been unpacked + + + + + Configure phase: ./configure + + + + + Build phase: make + + + + + Install phase: make install + + + + + + We add a new phase after the installation phase, which we call + fixup phase. At the end of the + builder.sh follows: + + + + + + That is, for each file we run patchelf --shrink-rpath + and strip. Note that we used two new commands here, + find and patchelf. These two + deserve a place in baseInputs of + autotools.nix as findutils and + patchelf. + + + + Rebuild hello.nix and...: + + + + + + ...only glibc is the runtime dependency. Exactly what we wanted. + + + + The package is self-contained, copy its closure on another machine and + you will be able to run it. I remind you the very few components under + the /nix/store necessary to run nix + when we installed it. + The hello binary will use that exact version of glibc library and + interpreter, not the system one: + + + + + + Of course, the executable runs fine as long as everything is under the + /nix/store path. + +
+ +
+ Conclusion + + + Short post compared to previous ones as I'm still on vacation, but I hope + you enjoyed it. Nix provides tools with cool features. In particular, Nix + is able to compute all runtime dependencies automatically for us. Not + only shared libraries, but also referenced executables, scripts, Python + libraries etc.. + + + + This makes packages self-contained, because we're sure (apart data and + configuration) that copying the runtime closure on another machine is + sufficient to run the program. That's why Nix has + one-click install, + or + reliable deployment in the cloud. + All with one tool. + +
+ +
+ Next pill + + + ...we will introduce nix-shell. With nix-build we build derivations + always from scratch: the source gets unpacked, configured, built and + installed. But this may take a long time, think of WebKit. What if we + want to apply some small changes and compile incrementally instead, yet + keeping a self-contained environment similar to nix-build? + +
diff --git a/pills/09/build-hello-nix.txt b/pills/09/build-hello-nix.txt new file mode 100644 index 0000000..b364bd8 --- /dev/null +++ b/pills/09/build-hello-nix.txt @@ -0,0 +1,5 @@ +$ nix-build hello.nix +[...] +$ nix-store -q --references result +/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 +/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello diff --git a/pills/09/find.txt b/pills/09/find.txt new file mode 100644 index 0000000..361ce92 --- /dev/null +++ b/pills/09/find.txt @@ -0,0 +1 @@ +find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null diff --git a/pills/09/instantiate-hello.txt b/pills/09/instantiate-hello.txt new file mode 100644 index 0000000..53d15b7 --- /dev/null +++ b/pills/09/instantiate-hello.txt @@ -0,0 +1,8 @@ +$ nix-instantiate hello.nix +/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv +$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv +/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello +$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello +/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 +/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3 +/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello diff --git a/pills/09/instantiate.txt b/pills/09/instantiate.txt new file mode 100644 index 0000000..0d5e5f7 --- /dev/null +++ b/pills/09/instantiate.txt @@ -0,0 +1,15 @@ +$ nix-instantiate hello.nix +/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv +$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv +/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz +/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv +/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv +/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv +/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh +/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv +/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv +/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv +/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv +/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv +/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv +/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv diff --git a/pills/09/ldd-hello.txt b/pills/09/ldd-hello.txt new file mode 100644 index 0000000..20c61e8 --- /dev/null +++ b/pills/09/ldd-hello.txt @@ -0,0 +1,4 @@ +$ ldd result/bin/hello + linux-vdso.so.1 (0x00007fff11294000) + libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000) + /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000) diff --git a/pills/09/strings.txt b/pills/09/strings.txt new file mode 100644 index 0000000..f766ab9 --- /dev/null +++ b/pills/09/strings.txt @@ -0,0 +1,2 @@ +$ strings result/bin/hello|grep gcc +/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64