NixOS/nix-pills
NixOS/nix-pills
1
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-09-19 04:00:13 -04:00
Code Activity

Port pill #9

Browse source
This commit is contained in:
Justin Wood 2017-08-13 16:38:22 -04:00
parent 06530261a2
commit d55e32f7d6
No known key found for this signature in database
GPG key ID: AD57C47E5F931181
7 changed files with 308 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

275
pills/09-automatic-runtime.xml
View file

@ -2,7 +2,278 @@
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="automatic-runtime">
xml:id="automatic-runtime-dependencies">
<title>automatic runtime</title>
<title>automatic runtime dependencies</title>
<para>
Welcome to the 9th Nix pill. In the previous
<link linkend="generic-builders">8th pill</link> we wrote a generic builder
for autotools projects. We feed build dependencies, a source tarball, and
we get a Nix derivation as a result.
</para>
<para>
Today we stop by the GNU hello world program to analyze build and runtime
dependencies, and enhance the builder in order to avoid unnecessary runtime
dependencies.
</para>
<section>
<title>Build dependencies</title>
<para>
Let's start analyzing build dependencies for our GNU hello world package:
</para>
<screen><xi:include href="./09/instantiate.txt" parse="text" /></screen>
<para>
It has exactly the derivations referenced in the <code>derivation</code>
function, nothing more, nothing less. Some of them might not be used at
all, however given that our generic mkDerivation function always pulls
such dependencies (think of it like
<link xlink:href="https://packages.debian.org/unstable/build-essential">build-essential</link>
of Debian), for every package you build from now on, you will have these
packages in the nix store.
</para>
<para>
Why are we looking at .drv files? Because the hello.drv file is the
representation of the build action to perform in order to build the hello
out path, and as such it also contains the input derivations needed to be
built before building hello.
</para>
</section>
<section>
<title>Digression about NAR files</title>
<para>
NAR is the Nix ARchive. First question: why not tar? Why another archiver?
Because commonly used archivers are not deterministic. They add padding,
they do not sort files, they add timestamps, etc.. Hence NAR, a very
simple deterministic archive format being used by Nix for deployment.
NARs are also used extensively within Nix itself as we'll see below.
</para>
<para>
For the rationale and implementation details you can find more in the
<link xlink:href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Dolstra's PhD Thesis</link>.
</para>
<para>
To create NAR archives, it's possible to use
<command>nix-store --dump</command> and
<command>nix-store --restore</command>. Those two commands work
regardless of <filename>/nix/store</filename>.
</para>
</section>
<section>
<title>Runtime dependencies</title>
<para>
Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in
any <code>derivation</code> call, but we never specify what are the
runtime dependencies for a derivation.
</para>
<para>
There's really black magic involved. It's something that at first glance
makes you think "no, this can't work in the long term", but at the same
it works so well that a whole operating system is built on top of this
magic.
</para>
<para>
In other words, Nix automatically computes all the runtime dependencies
of a derivation, and it's possible thanks to the hash of the store paths.
</para>
<para>
Steps:
</para>
<orderedlist>
<listitem>
<para>
Dump the derivation as NAR, a serialization of the derivation output.
Works fine whether it's a single file or a directory.
</para>
</listitem>
<listitem>
<para>
For each build dependency .drv and its relative out path, search the
contents of the NAR for this out path.
</para>
</listitem>
<listitem>
<para>
If found, then it's a runtime dependency.
</para>
</listitem>
</orderedlist>
<para>
You get really all the runtime dependencies, and that's why Nix
deployments are so easy.
</para>
<screen><xi:include href="./09/instantiate-hello.txt" parse="text" /></screen>
<para>
Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
</para>
<screen><xi:include href="./09/strings.txt" parse="text" /></screen>
<para>
Oh Nix added gcc because its out path is mentioned in the "hello" binary.
Why is that? That's the
<link xlink:href="http://en.wikipedia.org/wiki/Rpath">ld rpath</link>.
It's the list of directories where libraries can be found at runtime. In
other distributions, this is usually not abused. But in Nix, we have to
refer to particular versions of libraries, thus the rpath has an
important role.
</para>
<para>
The build process adds that gcc lib path thinking it may be useful at
runtime, but really it's not. How do we get rid of it? Nix authors have
written another magical tool called
<link xlink:href="https://nixos.org/patchelf.html">patchelf</link>, which
is able to reduce the rpath to the paths that are really used by the
binary.
</para>
<para>
Not only, even after reducing the rpath the hello binary would still
depend upon gcc. Because of debugging information. For that, the well
known
<link xlink:href="http://unixhelp.ed.ac.uk/CGI/man-cgi?strip">strip</link>
can be used.
</para>
</section>
<section>
<title>Another phase in the builder</title>
<para>
We will add a new phase to our autotools builder. The builder has these
phases already:
</para>
<orderedlist>
<listitem>
<para>
First the environment is set up
</para>
</listitem>
<listitem>
<para>
Unpack phase: we unpack the sources in the current directory
(remember, Nix changes dir to a temporary directory first)
</para>
</listitem>
<listitem>
<para>
Change source root to the directory that has been unpacked
</para>
</listitem>
<listitem>
<para>
Configure phase: <command>./configure</command>
</para>
</listitem>
<listitem>
<para>
Build phase: <command>make</command>
</para>
</listitem>
<listitem>
<para>
Install phase: <command>make install</command>
</para>
</listitem>
</orderedlist>
<para>
We add a new phase after the installation phase, which we call
<emphasis role="bold">fixup</emphasis> phase. At the end of the
<filename>builder.sh</filename> follows:
</para>
<screen><xi:include href="./09/find.txt" parse="text" /></screen>
<para>
That is, for each file we run <command>patchelf --shrink-rpath</command>
and <command>strip</command>. Note that we used two new commands here,
<command>find</command> and <command>patchelf</command>. These two
deserve a place in <code>baseInputs</code> of
<filename>autotools.nix</filename> as <command>findutils</command> and
<command>patchelf</command>.
</para>
<para>
Rebuild <filename>hello.nix</filename> and...:
</para>
<screen><xi:include href="./09/build-hello-nix.txt" parse="text" /></screen>
<para>
...only glibc is the runtime dependency. Exactly what we wanted.
</para>
<para>
The package is self-contained, copy its closure on another machine and
you will be able to run it. I remind you the very few components under
the <filename>/nix/store</filename> necessary to run nix
<link linkend="install-on-your-running-system">when we installed it</link>.
The hello binary will use that exact version of glibc library and
interpreter, not the system one:
</para>
<screen><xi:include href="./09/ldd-hello.txt" parse="text" /></screen>
<para>
Of course, the executable runs fine as long as everything is under the
<filename>/nix/store</filename> path.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
Short post compared to previous ones as I'm still on vacation, but I hope
you enjoyed it. Nix provides tools with cool features. In particular, Nix
is able to compute all runtime dependencies automatically for us. Not
only shared libraries, but also referenced executables, scripts, Python
libraries etc..
</para>
<para>
This makes packages self-contained, because we're sure (apart data and
configuration) that copying the runtime closure on another machine is
sufficient to run the program. That's why Nix has
<link xlink:href="http://nixos.org/nix/manual/#sec-one-click">one-click install</link>,
or
<link xlink:href="http://nixos.org/nixops/manual/#chap-introduction">reliable deployment in the cloud</link>.
All with one tool.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will introduce nix-shell. With nix-build we build derivations
always from scratch: the source gets unpacked, configured, built and
installed. But this may take a long time, think of WebKit. What if we
want to apply some small changes and compile incrementally instead, yet
keeping a self-contained environment similar to nix-build?
</para>
</section>
</chapter>

5
pills/09/build-hello-nix.txt Normal file
View file

@ -0,0 +1,5 @@
$ nix-build hello.nix
[...]
$ nix-store -q --references result
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello

1
pills/09/find.txt Normal file
View file

@ -0,0 +1 @@
find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null

8
pills/09/instantiate-hello.txt Normal file
View file

@ -0,0 +1,8 @@
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello

15
pills/09/instantiate.txt Normal file
View file

@ -0,0 +1,15 @@
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz
/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv
/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv
/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv
/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh
/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv
/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv
/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv
/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv
/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv
/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv
/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv

4
pills/09/ldd-hello.txt Normal file
View file

@ -0,0 +1,4 @@
$ ldd result/bin/hello
linux-vdso.so.1 (0x00007fff11294000)
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000)
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)

2
pills/09/strings.txt Normal file
View file

@ -0,0 +1,2 @@
$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64