nix-pills/pills/09-automatic-runtime.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         version="5.0"
         xml:id="automatic-runtime-dependencies">

  <title>Automatic Runtime Dependencies</title>

  <para>
    Welcome to the 9th Nix pill. In the previous
    <link linkend="generic-builders">8th pill</link> we wrote a generic builder
    for autotools projects. We fed in build dependencies and a source tarball, and
    we received a Nix derivation as a result.
  </para>

  <para>
    Today we stop by the GNU <code>hello</code> program to analyze build and runtime
    dependencies, and we enhance our builder to eliminate unnecessary runtime
    dependencies.
  </para>

  <section>
    <title>Build dependencies</title>

    <para>
      Let's start analyzing build dependencies for our GNU <code>hello</code> package:
    </para>

    <screen><xi:include href="./09/instantiate.txt" parse="text" /></screen>

    <para>
      It has precisely the derivations referenced in the <code>derivation</code> function;
      nothing more, nothing less. Of course, we may not use some of them at all.
      However, given that our generic <code>mkDerivation</code> function always pulls
      such dependencies (think of it like
      <link xlink:href="https://packages.debian.org/unstable/build-essential">build-essential</link>
      from Debian), we will already have these packages in the nix store for any future packages that
      need them.
    </para>

    <para>
     Why are we looking at <code>.drv</code> files? Because the <code>hello.drv</code>
      file is the representation of the build action that builds the <code>hello</code>
      out path. As such, it contains the input derivations needed before building
      <code>hello</code>.
    </para>
  </section>

  <section>
    <title>Digression about NAR files</title>

    <para>
      The <code>NAR</code> format is the "Nix ARchive". This format was designed due to
      existing archive formats, such as <code>tar</code>, being insufficient.
      Nix benefits from deterministic build tools, but commonly used archivers
      lack this property: they add padding, they do not sort files, they add timestamps,
      and so on. This can result in directories containing bit-identical files turning into
      non-bit-identical archives, which leads to different hashes.
    </para>

    <para>
      Thus the <code>NAR</code> format was developed as a simple, deterministic
      archive format. <code>NAR</code>s are used extensively within Nix, as we will
      see below.
    </para>

    <para>
      For more rationale and implementation details behind <code>NAR</code> see
      <link xlink:href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Dolstra's PhD Thesis</link>.
    </para>

    <para>
      To create NAR archives from store paths, we can  use
      <command>nix-store --dump</command> and
      <command>nix-store --restore</command>.
    </para>
  </section>

  <section>
    <title>Runtime dependencies</title>

    <para>
      We now note that Nix automatically recognized build dependencies once our
      <code>derivation</code> call referred to them, but we never specified the
      runtime dependencies.
    </para>

    <para>
      Nix handles runtime dependencies for us automatically. The technique it uses
      to do so may seem fragile at first glance, but it works so well that the NixOS
      operating system is built off of it. The underlying mechanism relies on the
      hash of the store paths. It proceeds in three steps:
    </para>

    <orderedlist>
      <listitem>
        <para>
          Dump the derivation as a NAR. Recall that this is a serialization of
          the derivation output -- meaning this works fine whether the output
          is a single file or a directory.
        </para>
      </listitem>
      <listitem>
        <para>
          For each build dependency <code>.drv</code> and its relative out path,
          search the contents of the NAR for this out path.
        </para>
      </listitem>
      <listitem>
        <para>
          If the path is found, then it's a runtime dependency.
        </para>
      </listitem>
    </orderedlist>

    <para>
      The snippet below shows the dependencies for <code>hello</code>.
    </para>

    <screen><xi:include href="./09/instantiate-hello.txt" parse="text" /></screen>

    <para>
      We see that <code>glibc</code> and <code>gcc</code> are runtime dependencies.
      Intuitively, <code>gcc</code> shouldn't be in this list! Displaying the
      printable strings in the <code>hello</code> binary shows that the out path
      of <code>gcc</code> does indeed appear:
    </para>

    <screen><xi:include href="./09/strings.txt" parse="text" /></screen>

    <para>
      This is why Nix added <code>gcc</code>. But why is that path present in the
      first place? The answer is that it is the <link xlink:href="http://en.wikipedia.org/wiki/Rpath">ld rpath</link>: the list of
      directories where libraries can be found at runtime. In other distributions,
      this is usually not abused. But in Nix, we have to refer to particular versions
      of libraries, and thus the rpath has an important role.
    </para>

    <para>
      The build process adds the <code>gcc</code> lib path thinking it may be useful
      at runtime, but this isn't necessary. To address issues like these, Nix provides
      a tool called <link xlink:href="https://nixos.org/patchelf.html">patchelf</link>,
      which reduces the rpath to the paths that are actually used by the binary.
    </para>

    <para>
      Even after reducing the rpath, the <code>hello</code> binary would still
      depend upon <code>gcc</code> because of some debugging information. This
      unnecessarily increases the size of our runtime
      dependencies. We'll explore how <command><link
      xlink:href="https://linux.die.net/man/1/strip">strip</link>
      </command> can help us with that in the next section.
    </para>
  </section>

  <section>
    <title>Another phase in the builder</title>

    <para>
      We will add a new phase to our autotools builder. The builder has six
      phases already:
    </para>

    <orderedlist>
      <listitem>
        <para>
          The "environment setup" phase
        </para>
      </listitem>
      <listitem>
        <para>
          The "unpack phase": we unpack the sources in the current directory
          (remember, Nix changes to a temporary directory first)
        </para>
      </listitem>
      <listitem>
        <para>
          The "change directory" phase, where we change source root to the
          directory that has been unpacked
        </para>
      </listitem>
      <listitem>
        <para>
          The "configure" phase: <command>./configure</command>
        </para>
      </listitem>
      <listitem>
        <para>
          The "build" phase: <command>make</command>
        </para>
      </listitem>
      <listitem>
        <para>
          The "install" phase: <command>make install</command>
        </para>
      </listitem>
    </orderedlist>

    <para>
      Now we will add a new phase after the installation phase, which we call
      the "fixup" phase. At the end of the
      <filename>builder.sh</filename>, we append:
    </para>

    <screen><xi:include href="./09/find.txt" parse="text" /></screen>

    <para>
      That is, for each file we run <command>patchelf --shrink-rpath</command>
      and <command>strip</command>. Note that we used two new commands here,
      <command>find</command> and <command>patchelf</command>. These must be
      added to our derivation.
    </para>

    <para>
      <emphasis role="bold">Exercise:</emphasis> Add <code>findutils</code>
      and <code>patchelf</code> to the <code>baseInputs</code> of
      <filename>autotools.nix</filename>.
    </para>

    <para>
      Now, we rebuild <filename>hello.nix</filename>...
    </para>

    <screen><xi:include href="./09/build-hello-nix.txt" parse="text" /></screen>

    <para>
      and we see that <code>glibc</code> is a runtime dependency. This is
      exactly what we wanted.
    </para>

    <para>
      The package is self-contained. This means that we can copy its closure onto
      another machine and we will be able to run it. Remember, only a very few
      components under the <filename>/nix/store</filename> are required to
      <link linkend="install-on-your-running-system">run nix</link>.
      The <code>hello</code> binary will use the exact version of <code>glibc</code>
      library and interpreter referred to in the binary, rather than the system one:
    </para>

    <screen><xi:include href="./09/ldd-hello.txt" parse="text" /></screen>

    <para>
      Of course, the executable will run fine as long as everything is under the
      <filename>/nix/store</filename> path.
    </para>
  </section>

  <section>
    <title>Conclusion</title>

    <para>
      We saw some of the tools Nix provides, along with their features.
      In particular, we saw how Nix is able to compute runtime dependencies
      automatically. This is not limited to only shared libraries,
      but can also reference executables, scripts, Python libraries, and so
      forth.
    </para>

    <para>
      Approaching builds in this way makes packages self-contained, ensuring
      (apart from data and configuration) that copying the runtime closure onto
      another machine is sufficient to run the program. This enables us to run programs
      without installation using <command>nix-shell</command>, and forms the basis for
      <link xlink:href="https://nixos.org/manual/nix/stable/introduction.html">reliable deployment in the cloud</link>.
    </para>
  </section>

  <section>
    <title>Next pill</title>

    <para>
      The next pill will introduce <command>nix-shell</command>. With
      <command>nix-build</command>, we've always built derivations from
      scratch: the source gets unpacked, configured, built, and installed.
      But this can take a long time for large packages. What if we want to
      apply some small changes and compile incrementally instead, yet still
      want to keep a self-contained environment similar to <command>nix-build</command>?
      <command>nix-shell</command> enables this.
    </para>
  </section>
</chapter>