nix-pills/pills/06-our-first-derivation.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0"
         xml:id="our-first-derivation">

<title>Our First Derivation</title>

  <para>
    Welcome to the sixth Nix pill. In the previous <link
    linkend="functions-and-imports">fifth pill</link> we introduced functions
    and imports. Functions and imports are very simple concepts that allow for
    building complex abstractions and composition of modules to build a flexible
    Nix system.
  </para>

  <para>
    In this post we finally arrived to writing a derivation. Derivations are the
    building blocks of a Nix system, from a file system view point. The Nix
    language is used to describe such derivations.
  </para>

  <para>
    I remind you how to enter the Nix environment: <code>source
    ~/.nix-profile/etc/profile.d/nix.sh</code>
  </para>


<section>
  <title>The derivation function</title>

  <para>
    The <link
    xlink:href="https://nixos.org/manual/nix/stable/expressions/derivations.html">derivation
    built-in function</link> is used to create derivations. I invite you to read
    the link in the Nix manual about the derivation built-in. A derivation from
    a Nix language view point is simply a set, with some attributes. Therefore
    you can pass the derivation around with variables like anything else.
  </para>

  <para>
    That's where the real power comes in.
  </para>

  <para>
    The <code>derivation</code> function receives a set as its first argument. This
    set requires at least the following three attributes:
  </para>

  <itemizedlist mark='bullet'>
    <listitem>
      <para>
        name: the name of the derivation. In the nix store the format is
        hash-name, that's the name.
      </para>
    </listitem>
    <listitem>
      <para>
        system: is the name of the system in which the derivation can be built.
        For example, x86_64-linux.
      </para>
    </listitem>
    <listitem>
      <para>
        builder: is the binary program that builds the derivation.
      </para>
    </listitem>
  </itemizedlist>

  <para>
    First of all, what's the name of our system as seen by nix?
  </para>

  <screen><xi:include href="./06/current-system.txt" parse="text" /></screen>

  <para>
    Let's try to fake the name of the system:
  </para>

  <screen><xi:include href="./06/fake-system.txt" parse="text" /></screen>

  <para>
    Oh oh, what's that? Did it build the derivation? No it didn't, but it
    <emphasis role="strong">did create the .drv file</emphasis>. <literal>nix repl</literal> does
    not build derivations unless you tell to do so.
  </para>

</section>

<section>
  <title>Digression about .drv files</title>

  <para>
    What's that <filename>.drv</filename> file? It is the specification of how
    to build the derivation, without all the Nix language fuzz.
  </para>

  <para>
    Before continuing, some analogies with the C language:
  </para>

  <itemizedlist mark='bullet'>
    <listitem>
      <para>
        <filename>.nix</filename> files are like <filename>.c</filename> files.
      </para>
    </listitem>
    <listitem>
      <para>
        <filename>.drv</filename> files are intermediate files like
        <filename>.o</filename> files. The <filename>.drv</filename> describes
        how to build a derivation; it's the bare minimum information.
      </para>
    </listitem>
    <listitem>
      <para>
        out paths are then the product of the build.
      </para>
    </listitem>
  </itemizedlist>

  <para>
    Both drv paths and out paths are stored in the nix store as you can see.
  </para>

  <para>
    What's in that <filename>.drv</filename> file? You can read it, but it's
    better to pretty print it:
  </para>

  <xi:include href="./06/show-derivation.xml" />


  <para>
    Ok, we can see there's an out path, but it does not exist yet. We never told
    Nix to build it, but we know beforehand where the build output will be. Why?
  </para>

  <para>
    Think, if Nix ever built the derivation just because we accessed it in Nix,
    we would have to wait a long time if it was, say, Firefox. That's why Nix
    let us know the path beforehand and kept evaluating the Nix expressions, but
    it's still empty because no build was ever made.
  </para>

  <para>
    <emphasis role="underline">Important</emphasis>: the hash of the out path is
    based solely on the input derivations in the current version of Nix, not on
    the contents of the build product. It's possible however to have <link
    xlink:href="https://en.wikipedia.org/wiki/Content-addressable_storage">content-addressable</link>
    derivations for e.g. tarballs as we'll see later on.
  </para>

  <para>
    Many things are empty in that <filename>.drv</filename>, however I'll write a
    summary of the <link
    xlink:href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">.drv format</link>
    for you:
  </para>

  <orderedlist>
    <listitem>
      <para>
        The output paths (there can be multiple ones). By default nix creates one
        out path called "out".
      </para>
    </listitem>
    <listitem>
      <para>
        The list of input derivations. It's empty because we are not referring
        to any other derivation. Otherwise, there would be a list of other .drv
        files.
      </para>
    </listitem>
    <listitem>
      <para>
        The system and the builder executable (yes, it's a fake one).
      </para>
    </listitem>
    <listitem>
      <para>
        Then a list of environment variables passed to the builder.
      </para>
    </listitem>
  </orderedlist>

  <para>
    That's it, the minimum necessary information to build our derivation.
  </para>

  <para>
    <emphasis role="underline">Important note</emphasis>: the environment
    variables passed to the builder are just those you see in the .drv plus some
    other Nix related configuration (number of cores, temp dir, ...). The
    builder will not inherit any variable from your running shell, otherwise
    builds would suffer from <link
    xlink:href="https://wiki.debian.org/ReproducibleBuilds">non-determinism</link>.
  </para>

  <para>
    Back to our fake derivation.
  </para>

  <para>
    Let's build our really fake derivation:
  </para>

  <screen><xi:include href="./06/build-derivation.txt" parse="text" /></screen>

  <para>
    The <code>:b</code> is a <literal>nix repl</literal> specific command to build a derivation.
    You can see more commands with <code>:?</code> . So in the output you can
    see that it takes the <filename>.drv</filename> as information on how to
    build the derivation. Then it says it's trying to produce our out path.
    Finally the error we were waiting for: that derivation can't be built on our
    system.
  </para>

  <para>
    We're doing the build inside <literal>nix repl</literal>, but what if we don't want to use
    <literal>nix repl</literal>? You can <emphasis role="strong">realise</emphasis> a
    <filename>.drv</filename> with:
  </para>

  <screen><xi:include href="./06/realise-derivation.txt" parse="text"
  /></screen>

  <para>
    You will get the same output as before.
  </para>

  <para>
    Let's fix the system attribute:
  </para>

  <screen><xi:include href="./06/fix-attribute.txt" parse="text" /></screen>

  <para>
    A step forward: of course, that <code>mybuilder</code> executable does not
    really exist. Stop for a moment.
  </para>


</section>
<section>
  <title>What's in a derivation set</title>

  <para>
    It is useful to start by inspecting the return value from the derivation function.
    In this case, the returned value is a plain set:
  </para>

  <screen><xi:include href="./06/inspect-values.txt" parse="text" /></screen>

  <para>
    You can guess what <code>builtins.isAttrs</code> does; it returns true if
    the argument is a set. While <code>builtins.attrNames</code> returns a list
    of keys of the given set. Some kind of reflection, you might say.
  </para>

  <para>
    Start from drvAttrs:
  </para>

  <screen><xi:include href="./06/drvattrs.txt" parse="text" /></screen>

  <para>
    That's basically the input we gave to the derivation function. Also the
    <code>d.name</code>, <code>d.system</code> and <code>d.builder</code>
    attributes are exactly the ones we gave as input.
  </para>

  <screen><xi:include href="./06/check-drvattrs.txt" parse="text" /></screen>

  <para>
    So out is just the derivation itself, it seems weird but the reason is that
    we only have one output from the derivation. That's also the reason why
    <code>d.all</code> is a singleton. We'll see multiple outputs later.
  </para>

  <para>
    The <code>d.drvPath</code> is the path of the <filename>.drv</filename>
    file: <filename>/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<emphasis
    role="strong">myname.drv</emphasis></filename>.
  </para>

  <para>
    Something interesting is the <code>type</code> attribute. It's
    <code>"derivation"</code>. Nix does add a little of magic to sets with type
    derivation, but not that much. To help you understand, you can create
    yourself a set with that type, it's a simple set:
  </para>

  <screen><xi:include href="./06/type-derivation.txt" parse="text" /></screen>

  <para>
    Of course it has no other information, so Nix doesn't know what to say :-)
    But you get it, the <code>type = "derivation"</code> is just a convention
    for Nix and for us to understand the set is a derivation.
  </para>

  <para>
    When writing packages, we are interested in the outputs. The other metadata
    is needed for Nix to know how to create the drv path and the out path.
  </para>

  <para>
    The <code>outPath</code> attribute is the build path in the nix store:
    <filename>/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<emphasis
    role="strong">myname</emphasis></filename>.
  </para>


</section>
<section>
  <title>Referring to other derivations</title>

  <para>
    Just like dependencies in other package managers, how do we refer to other
    packages? How do we refer to other derivations in terms of files on the
    disk? We use the <code>outPath</code>. The <code>outPath</code> describes 
    the location of the files of that derivation. To make it more convenient, 
    Nix is able to do a conversion from a derivation set to a string.
  </para>

  <screen><xi:include href="./06/outpath.txt" parse="text" /></screen>

  <para>
    Nix does the "set to string conversion" as long as there is the
    <code>outPath</code> attribute (much like a toString method in other
    languages):
  </para>

  <screen><xi:include href="./06/tostring.txt" parse="text" /></screen>

  <para>
    Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
  </para>

  <screen><xi:include href="./06/coreutils.txt" parse="text" /></screen>

  <para>
    Apart the nixpkgs stuff, just think we added to the scope a series of
    variables. One of them is coreutils. It is the derivation of the coreutils
    package you all know of from other Linux distributions. It contains basic
    binaries for GNU/Linux systems (you may have multiple derivations of
    coreutils in the nix store, no worries):
  </para>


  <screen><xi:include href="./06/list-coreutils.txt" parse="text" /></screen>

  <para>
    I remind you, inside strings it's possible to interpolate Nix expressions
    with <code>${...}</code>:
  </para>


  <screen><xi:include href="./06/interpolate.txt" parse="text" /></screen>

  <para>
    That's very convenient, because then we could refer to e.g. the bin/true
    binary like this:
  </para>


  <screen><xi:include href="./06/reference.txt" parse="text" /></screen>

</section>
<section>
  <title>An almost working derivation</title>

  <para>
    In the previous attempt we used a fake builder, <code>mybuilder</code> which
    obviously does not exist. But we can use for example bin/true, which always
    exits with 0 (success).
  </para>

  <screen><xi:include href="./06/test-build.txt" parse="text" /></screen>

  <para>
    Another step forward, it executed the builder (bin/true), but the builder
    did not create the out path of course, it just exited with 0.
  </para>


  <para>
    <emphasis role="underline">Obvious note</emphasis>: everytime we change the
    derivation, a new hash is created.
  </para>

  <para>
    Let's examine the new <filename>.drv</filename> now that we referred to
    another derivation:
  </para>

  <xi:include href="./06/examine-build.xml" />

  <para>
    Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv.
    Before doing our build, Nix should build the coreutils.drv. But since
    coreutils is already in our nix store, no build is needed, it's already
    there with out path
    <filename>/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-<emphasis
    role="strong">coreutils-8.29</emphasis></filename>.
  </para>

</section>
<section>
  <title>When is the derivation built</title>

  <para>
    Nix does not build derivations <emphasis role="strong">during
    evaluation</emphasis> of Nix expressions. In fact, that's why we have to do
    ":b drv" in <literal>nix repl</literal>, or use nix-store -r in the first place.
  </para>

  <para>
    An important separation is made in Nix:
  </para>

  <itemizedlist mark='bullet'>
    <listitem>
      <para>
        <emphasis role="strong">Instantiate/Evaluation time</emphasis>: the Nix
        expression is parsed, interpreted and finally returns a derivation set.
        During evaluation, you can refer to other derivations because Nix will
        create .drv files and we will know out paths beforehand. This is
        achieved with <link
        xlink:href="https://nixos.org/manual/nix/stable/command-ref/nix-instantiate.html">nix-instantiate</link>.
      </para>
    </listitem>
    <listitem>
      <para>
        <emphasis role="strong">Realise/Build time</emphasis>: the .drv from the
        derivation set is built, first building .drv inputs (build
        dependencies). This is achieved with <link
        xlink:href="https://nixos.org/manual/nix/stable/command-ref/nix-store.html#operation---realise">nix-store
        -r</link>.
      </para>
    </listitem>
  </itemizedlist>

  <para>
    Think of it as of compile time and link time like with C/C++ projects. You
    first compile all source files to object files. Then link object files in a
    single executable.
  </para>

  <para>
    In Nix, first the Nix expression (usually in a .nix file) is compiled to
    .drv, then each .drv is built and the product is installed in the relative
    out paths.
  </para>

</section>
<section>
  <title>Conclusion</title>

  <para>
    Is it that complicated to create a package for Nix? No, it's not.
  </para>

  <para>
    We're walking through the fundamentals of Nix derivations, to understand how
    they work, how they are represented. Packaging in Nix is certainly easier
    than that, but we're not there yet in this post. More Nix pills are needed.
  </para>

  <para>
    With the derivation function we provide a set of information on how to build
    a package, and we get back the information about where the package was
    built. Nix converts a set to a string when there's an <code>outPath</code>; 
    that's very convenient. With that, it's easy to refer to other derivations.
  </para>

  <para>
    When Nix builds a derivation, it first creates a .drv file from a derivation
    expression, and uses it to build the output. It does so recursively for all
    the dependencies (inputs). It "executes" the .drv files like a machine. Not
    much magic after all.
  </para>


</section>
<section>
  <title>Next pill</title>

  <para>
    ...we will finally write our first <emphasis
    role="strong">working</emphasis> derivation. Yes, this post is about "our
    first derivation", but I never said it was a working one ;)
  </para>

</section>

</chapter>