nix-pills/pills/13-callpackage-design-pattern.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         version="5.0"
         xml:id="callpackage-design-pattern">

  <title>Callpackage Design Pattern</title>
  <para>
    Welcome to the 13th Nix pill. In the previous <link linkend="inputs-design-pattern">12th pill</link> we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged <package>graphviz</package> to have at least another package for our little repository.
  </para>
  <para>
    The next design pattern worth noting is what I'd like to call the <literal>callPackage</literal> pattern. This technique is extensively used in <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, it's the current standard for importing packages in a repository.
  </para>
  <section>
    <title>The callPackage convenience</title>
    <para>
      In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments.
    </para>
    <para>
      However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice. With package management, we often see common patterns. In the case of <literal>nixpkgs</literal> it's the following.
    </para>
    <para>
      Some package derivation:
    </para>
    <screen><xi:include href="./13/package-derivation.txt" parse="text" /></screen>
    <para>
      Repository derivation:
    </para>
    <screen><xi:include href="./13/repository-derivation.txt" parse="text" /></screen>
    <para>
      Where inputs may even be packages in the repository itself (note the rec keyword). The pattern here is clear, often inputs have the same name of the attributes in the repository itself. Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument).
    </para>
    <para>
      To achieve this, we will define a <literal>callPackage</literal> function with the following synopsis:
    </para>
    <screen><xi:include href="./13/callpackage-function-call.txt" parse="text" /></screen>
    <para>
      What should it do?
      <itemizedlist>
        <listitem><para>Import the given expression, which in turn returns a function.</para></listitem>
        <listitem><para>Determine the name of its arguments.</para></listitem>
        <listitem><para>Pass default arguments from the repository set, and let us override those arguments.</para></listitem>
      </itemizedlist>
    </para>
  </section>
  <section>
    <title>Implementing callPackage</title>
    <para>
      First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments.
    </para>
    <para>
      Then <literal>callPackage</literal> requires access to the whole packages set, because it needs to find the packages to pass automatically.
    </para>
    <para>
      We start off simple with <command></command>:
    </para>
    <screen><xi:include href="./13/get-args-function.txt" parse="text" /></screen>
    <para>
      Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names.
    </para>
    <para>
      Now we need a set with all the <literal>values</literal>, let's call it <literal>values</literal>.  And a way to intersect the attributes of values with the function arguments:
    </para>
    <screen><xi:include href="./13/intersect-attr-values.txt" parse="text" /></screen>
    <para>
      Perfect, note from the example above that the <literal>intersectAttrs</literal> returns a set whose names are the intersection, and the attribute values are taken from the second set.
    </para>
    <para>
      We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of <literal>callPackage</literal>:
    </para>
    <screen><xi:include href="./13/callpackage-function.txt" parse="text" /></screen>
    <para>
      Clearing up the syntax:
      <itemizedlist>
        <listitem><para>We define a <literal>callPackage</literal> variable which is a function.</para></listitem>
        <listitem><para>The second parameter is the function to "autocall".</para></listitem>
        <listitem><para>We take the argument names of the function and intersect with the set of all values.</para></listitem>
        <listitem><para>Finally we call the passed function <literal>f</literal> with the resulting intersection.</para></listitem>
      </itemizedlist>
    </para>
    <para>
      In the code above, I've also shown that the <literal>callPackage</literal> call is equivalent to directly calling <literal>add a b</literal>.
    </para>
    <para>
      We achieved what we wanted. Automatically call functions given a set of possible arguments. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs <literal>...</literal> as explained in the <link linkend="functions-and-imports">5th pill</link>).
    </para>
    <para>
      Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set. Then we add a further parameter, which takes a set of overrides:
    </para>
    <screen><xi:include href="./13/callpackage-function-overrides.txt" parse="text" /></screen>
    <para>
      Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set.
    </para>
  </section>
  <section>
    <title>Use callPackage to simplify the repository</title>
    <para>
      Given our brand new tool, we can simplify the repository expression (default.nix).
    </para>
    <para>
      Let me write it down first:
    </para>
    <screen><xi:include href="./13/callpackage-usage.txt" parse="text" /></screen>
    <para>
      Wow, there's a lot to say here:
      <itemizedlist>
        <listitem><para>We renamed the old <literal>pkgs</literal> of the previous pill to <literal>nixpkgs</literal>. Our package set is now instead named <literal>pkgs</literal>. Sorry for the confusion.</para></listitem>
        <listitem><para>We needed a way to pass pkgs to <literal>callPackage</literal> somehow. Instead of returning the set of packages directly from <filename>default.nix</filename>, we first assign it to a <literal>let</literal> variable and reuse it in <literal>callPackage</literal>.</para></listitem>
        <listitem><para>For convenience, in <literal>callPackage</literal> we first import the file, instead of calling it directly. Otherwise for each package we would have to write the <literal>import</literal>.</para></listitem>
        <listitem><para>Since our expressions use packages from <literal>nixpkgs</literal>, in <literal>callPackage</literal> we use <literal>allPkgs</literal>, which is the union of <literal>nixpkgs</literal> and our packages.</para></listitem>
        <listitem><para>We moved <literal>mkDerivation</literal> into <literal>pkgs</literal> itself, so that it also gets passed automatically.</para></listitem>
      </itemizedlist>
    </para>
    <para>
      Note how easy is to override arguments in the case of <package>graphviz</package> without <package>gd</package>. But most importantly, how easy it was to merge two repositories: <literal>nixpkgs</literal> and our <literal>pkgs</literal>!
    </para>
    <para>
      The reader should notice a magic thing happening. We're defining <literal>pkgs</literal> in terms of <literal>callPackage</literal>, and <literal>callPackage</literal> in terms of <literal>pkgs</literal>. That magic is possible thanks to lazy evaluation.
    </para>
  </section>
  <section>
    <title>Conclusion</title>
    <para>
      The "<literal>callPackage</literal>" pattern has simplified a lot our repository. We're able to import packages that require some named arguments and call them automatically, given the set of all packages.
    </para>
    <para>
      We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. That's why they are not documented in the <link xlink:href="http://nixos.org/nix/manual/">nix manual</link>.
    </para>
    <para>
      Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions.
    </para>
  </section>
  <section>
    <title>Next pill</title>
    <para>
      ...we will talk about the "<literal>override</literal>" design pattern. The <literal>graphvizCore</literal> seems straightforward. It starts from <filename>graphviz.nix</filename> and builds it without <package>gd</package>. Now I want to give you another point of view: what if we instead wanted to start from <literal>pkgs.graphviz</literal> and disable <package>gd</package>?
    </para>
  </section>
</chapter>