nix-pills/pills/13-callpackage-design-pattern.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         version="5.0"
         xml:id="callpackage-design-pattern">

  <title>Callpackage Design Pattern</title>
  <para>
    Welcome to the 13th Nix pill. In the previous <link linkend="inputs-design-pattern">12th
    pill</link>, we introduced the first basic design pattern for organizing a repository of
    software. In addition, we packaged <package>graphviz</package> so that we had two packages
    to bundle into an example repository.
  </para>
  <para>
    The next design pattern we will examine is called the <literal>callPackage</literal>
    pattern. This technique is extensively used in <link
    xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, and it's the current
    de facto standard for importing packages in a repository. Its purpose is to reduce
    the duplication of identifiers between package derivation inputs and repository
    derivations.
  </para>
  <section>
    <title>The callPackage convenience</title>
    <para>
      In the previous pill, we demonstrated how the <literal>inputs</literal>
      pattern decouples packages from the repository. This allowed us to
      manually pass the inputs to the derivation; the derivation declares
      its inputs, and the caller passes the arguments.
    </para>
    <para>
      However, as with usual programming languages, there is some duplication of work:
      we declare parameter names and then we pass arguments, typically with the same name.
      For example, if we define a package derivation using the <literal>inputs</literal>
      pattern such as:
    </para>
    <screen><xi:include href="./13/package-derivation.txt" parse="text" /></screen>
    <para>
      we would likely want to bundle that package derivation into a repository via a
      an attribute set defined as something like:
    </para>
    <screen><xi:include href="./13/repository-derivation.txt" parse="text" /></screen>
    <para>
      There are two things to note. First, that inputs often have the same name as
      attributes in the repository itself. Second, that (due to the <code>rec</code>
      keyword), the inputs to a package derivation may be other packages in the
      repository itself.
    </para>
    <para>
      Rather than passing the inputs twice, we would prefer to pass those inputs from
      the repository automatically and allow for manually overriding defaults.
    </para>
    <para>
      To achieve this, we will define a <literal>callPackage</literal> function with
      the following calling convention:
    </para>
    <screen><xi:include href="./13/callpackage-function-call.txt" parse="text" /></screen>
    <para>
      We want <code>callPackage</code> to be a function of two arguments, with the
      following behavior:
      <itemizedlist>
        <listitem><para>
          Import the given expression contained in the file of the first argument,
          and return a function. This function returns a package derivation that
          uses the inputs pattern.
        </para></listitem>
        <listitem><para>
          Determine the name of the arguments to the function (i.e., the names
          of the inputs to the package derivation).
        </para></listitem>
        <listitem><para>
          Pass default arguments from the repository set, and let us override those
          arguments if we wish to customize the package derivation.
        </para></listitem>
      </itemizedlist>
    </para>
  </section>
  <section>
    <title>Implementing <code>callPackage</code></title>
    <para>
      In this section, we will build up the <code>callPackages</code> pattern
      from scratch. To start, we need a way to obtain the argument names
      of a function (in this case, the function that takes "inputs" and produces
      a package derivation) at runtime. This is because we want to automatically pass
      such arguments.
    </para>
    <para>
      Nix provides a builtin function to do this:
    </para>
    <screen><xi:include href="./13/get-args-function.txt" parse="text" /></screen>
    <para>
      In addition to returning the argument names, the attribute set returned by
      <code>functionArgs</code> indicates whether or not the argument has a default value.
      For our purposes, we are only interested in the argument names; we do not care
      about the default values right now.
    </para>

    <para>
      The next step is to make <code>callPackage</code> automatically pass inputs to our
      package derivations based on the argument names we've just obtained with
      <code>functionArgs</code>.
    </para>
    <para>
      To do this, we need two things:
      <itemizedlist>
        <listitem><para>
          A package repository set containing package derivations that match the arguments
          names we've obtained
        </para></listitem>
        <listitem><para>
          A way to obtain an auto-populated attribute set combining the package repository
          and the return value of <code>functionArgs</code>.
        </para></listitem>
      </itemizedlist>
    </para>
    <para>
      The former is easy: we just have to set our package derivation's inputs
      to be package names in a repository, such as <code>nixpkgs</code>. For
      the latter, Nix provides another builtin function:
    </para>
    <screen><xi:include href="./13/intersect-attr-values.txt" parse="text" /></screen>
    <para>
      The <literal>intersectAttrs</literal> returns an attribute set whose names are
      the intersection of both arguments' attribute names, with the attribute
      values taken from the second argument.
    </para>
    <para>
      This is all we need to do: we have obtained the argument names from a function,
      and populated these with an existing set of attributes. This is our simple
      implementation of <literal>callPackage</literal>:
    </para>
    <screen><xi:include href="./13/callpackage-function.txt" parse="text" /></screen>
    <para>
      Let's dissect the above snippet:
      <itemizedlist>
        <listitem><para>
          We define a <literal>callPackage</literal> variable which is a
          function.
        </para></listitem>
        <listitem><para>
          The first parameter to the <literal>callPackage</literal> function
          is a set of name-value pairs that may appear in the argument set of
          the function we wish to "autocall".
        </para></listitem>
        <listitem><para>
          The second parameter is the function to "autocall"
        </para></listitem>
        <listitem><para>
          We take the argument names of the function and intersect with the set of all
          values.
        </para></listitem>
        <listitem><para>
          Finally, we call the passed function <literal>f</literal> with the resulting
          intersection.
        </para></listitem>
      </itemizedlist>
    </para>
    <para>
      In the snippet above, we've also demonstrated that the <literal>callPackage</literal>
      call is equivalent to directly calling <literal>add a b</literal>.
    </para>
    <para>
      We achieved most of what we wanted: to automatically call functions given a set of
      possible arguments. If an argument is not found within the set we used to call the
      function, then we receive an error (unless the function has variadic arguments
      denoted with <literal>...</literal>,  as explained in the <link
      linkend="functions-and-imports">5th pill</link>).
    </para>
    <para>
      The last missing piece is allowing users to override some of the parameters.
      We may not want to always call functions with values taken from the big set.
      Thus, we add a third parameter which takes a set of overrides:
    </para>
    <screen><xi:include href="./13/callpackage-function-overrides.txt" parse="text" /></screen>
    <para>
      Apart from the increasing number of parentheses, it should be clear that we simply
      take a set union between the default arguments and the overriding set.
    </para>
  </section>
  <section>
    <title>Using callPackage to simplify the repository</title>
    <para>
      Given our <literal>callPackages</literal>, we can simplify the repository expression
      in <filename>default.nix</filename>:
    </para>
    <screen><xi:include href="./13/callpackage-usage.txt" parse="text" /></screen>
    <para>
      Let's examine this in detail:
      <itemizedlist>
        <listitem><para>
          The expression above defines our own package repository, which we call
          <literal>pkgs</literal>, that contains <literal>hello</literal> along
          with our two variants of <literal>graphviz</literal>.
        </para></listitem>
        <listitem><para>
          In the <code>let</code> expression, we import <literal>nixpkgs</literal>.
          Note that previously, we referred to this import with the variable
          <literal>pkgs</literal>, but now that name is taken by the repository
          we are creating ourselves.
        </para></listitem>
        <listitem><para>
          We needed a way to pass <literal>pkgs</literal> to <literal>callPackage</literal>
          somehow. Instead of returning the set of packages directly from
          <filename>default.nix</filename>, we first assign it to a <literal>let</literal>
          variable and reuse it in <literal>callPackage</literal>.
        </para></listitem>
        <listitem><para>
          For convenience, in <literal>callPackage</literal> we first
          import the file instead of calling it directly. Otherwise we would have to
          write the <literal>import</literal> for each package.
        </para></listitem>
        <listitem><para>
          Since our expressions use packages from <literal>nixpkgs</literal>, in
          <literal>callPackage</literal> we use <literal>allPkgs</literal>, which
          is the union of <literal>nixpkgs</literal> and our packages.
        </para></listitem>
        <listitem><para>
          We moved <literal>mkDerivation</literal> into <literal>pkgs</literal> itself,
          so that it also gets passed automatically.</para></listitem>
      </itemizedlist>
    </para>
    <para>
      Note how easily we overrode arguments in the case of <literal>graphviz</literal>
      without <literal>gd</literal>. In addition, note how easy it was to merge
      two repositories: <literal>nixpkgs</literal> and our <literal>pkgs</literal>!
    </para>
    <para>
      The reader should notice a magic thing happening. We're defining
      <literal>pkgs</literal> in terms of <literal>callPackage</literal>, and
      <literal>callPackage</literal> in terms of <literal>pkgs</literal>. That magic is
      possible thanks to lazy evaluation: <literal>builtins.intersectAttrs</literal> doesn't
      need to know the values in <literal>allPkgs</literal> in order to perform intersection,
      only the keys that do not require <literal>callPackage</literal> evaluation.
    </para>
  </section>
  <section>
    <title>Conclusion</title>
    <para>
      The "<literal>callPackage</literal>" pattern has simplified our repository
      considerably. We were able to import packages that require named arguments
      and call them automatically, given the set of all packages sourced from
      <literal>nixpkgs</literal>.
    </para>
    <para>
      We've also introduced some useful builtin functions that allows us to introspect Nix
      functions and manipulate attributes. These builtin functions are not usually used when
      packaging software, but rather act as tools for packaging. They are documented in the
      <link xlink:href="https://nixos.org/manual/nix/stable/expressions/builtins.html">Nix
      manual</link>.
    </para>
    <para>
      Writing a repository in Nix is an evolution of writing convenient functions for
      combining the packages. This pill demonstrates how Nix can be a generic tool
      to build and deploy software, and how suitable it is to create software
      repositories with our own conventions.
    </para>
  </section>
  <section>
    <title>Next pill</title>
    <para>
      In the next pill, we will talk about the "<literal>override</literal>" design
      pattern. The <literal>graphvizCore</literal> seems straightforward. It starts from
      <filename>graphviz.nix</filename> and builds it without <package>gd</package>.
      In the next pill, we will consider another point of view: starting from
      <literal>pkgs.graphviz</literal> and disabling <package>gd</package>?
    </para>
  </section>
</chapter>