2017-08-11 21:41:14 -04:00
<chapter xmlns= "http://docbook.org/ns/docbook"
2017-08-22 08:19:43 -04:00
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="inputs-design-pattern">
2017-08-11 18:22:51 -04:00
2024-02-09 06:47:48 -05:00
<title > Package Repositories and the Inputs Design Pattern</title>
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
Welcome to the 12th Nix pill. In the previous <link linkend= "garbage-collector" > 11th
pill</link> , we stopped packaging and cleaned up the system with the garbage collector.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
This time, we will resume packaging and improve different aspects of it. We will also
demonstrate how to create a repository of multiple packages.
2017-08-17 23:54:26 -04:00
</para>
<section >
<title > Repositories in Nix</title>
<para >
2024-02-20 00:14:06 -05:00
Package repositories in Nix arose naturally from the need to organize packages.
2024-02-09 06:47:48 -05:00
There is no preset directory structure or packaging policy prescribed by Nix itself;
Nix, as a full, functional programming language, is powerful enough to support
multiple different repository formats.
2017-08-17 23:54:26 -04:00
</para>
2024-02-09 06:47:48 -05:00
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
Over time, the <literal > nixpkgs</literal> repository evolved a particular
structure. This structure reflects the history of Nix as well as the design
2024-02-20 00:14:06 -05:00
patterns adopted by its users as useful tools in building and organizing
2024-02-09 06:47:48 -05:00
packages. Below, we will examine some of these patterns in detail.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
<title > The single repository pattern</title>
<para >
2024-02-09 06:47:48 -05:00
Different operating system distributions have different opinions about how
2024-02-20 00:14:06 -05:00
package repositories should be organized. Systems like Debian scatter packages
2024-02-09 06:47:48 -05:00
in several small repositories (which tends to make tracking interdependent
changes more difficult, and hinders contributions to the repositories),
while systems like Gentoo put all package descriptions in a single repository.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
Nix follows the "single repository" pattern by placing all descriptions of all
packages into <link xlink:href= "https://github.com/NixOS/nixpkgs" > nixpkgs</link> .
This approach has proven natural and attractive for new contributions.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
For the rest of this pill, we will adopt the single repository pattern. The
natural implementation in Nix is to create a top-level Nix expression, followed
by one expression for each package. The top-level expression imports and combines
all package expressions in an attribute set mapping names to packages.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
In some programming languages, such an approach -- including every possible
package description in a single data structure -- would be untenable due
to the language needing to load the entire data structure into memory before
operating on it. Nix, however, is a lazy language and only evaluates what is
needed.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
2024-02-09 06:47:48 -05:00
<title > Packaging <code > graphviz</code> </title>
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
We have already packaged GNU <code > hello</code> . Next, we will package a
graph-drawing program called <code > graphviz</code> so that we can
create a repository containing multiple packages. The <code > graphviz</code>
package was selected because it uses the standard autotools build system and
requires no patching. It also has optional dependencies, which will give us
an opportunity to illustrate a technique to configure builds to a particular
situation.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
First, we download <code > graphviz</code> from <link xlink:href= "https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.3/graphviz-2.49.3.tar.gz" > gitlab</link> . The <filename > graphviz.nix</filename> expression is straightforward:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/graphviz-derivation.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
If we build the project with <command > nix-build graphviz.nix</command> , we will get runnable binaries under <filename > result/bin</filename> . Notice how we reused the same <filename > autotools.nix</filename> of <filename > hello.nix.</filename>
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
By default, <code > graphviz</code> does not compile with the ability to produce
<code > png</code> files. Thus, the derivation above will build a binary
supporting only the native output formats, as we see below:
2017-08-17 23:54:26 -04:00
</para>
2024-02-09 06:47:48 -05:00
<screen > <xi:include href= "./12/simple-png.txt" parse= "text" /> </screen>
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
If we want to produce a <code > png</code> file with <code > graphviz</code> , we
must add it to our derivation. The place to do so is
in <filename > autotools.nix</filename> , where we created a
<literal > buildInputs</literal> variable that gets concatenated to
<literal > baseInputs</literal> . This is the exact reason for this variable: to
allow users of <filename > autotools.nix</filename> to add additional inputs
from package expressions.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
Version 2.49 of <code > graphviz</code> has several plugins to output
<code > png</code> . For simplicity, we will use <code > libgd</code> .
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
2024-02-09 06:47:48 -05:00
<title > Passing library information to <command > pkg-config</command> via environment
variables</title>
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
The <code > graphviz</code> configuration script uses <command > pkg-config</command>
to specify which flags are passed to the compiler. Since there is no global location
for libraries, we need to tell <command > pkg-config</command> where to find
2024-02-20 00:14:06 -05:00
its description files, which tell the configuration script where to find
2024-02-09 06:47:48 -05:00
headers and libraries.
2017-08-17 23:54:26 -04:00
</para>
<para >
2021-12-02 13:27:44 -05:00
In classic POSIX systems, <command > pkg-config</command> just finds the
<filename > .pc</filename> files of all installed libraries in system folders
2024-02-09 06:47:48 -05:00
like <filename > /usr/lib/pkgconfig</filename> . However, these files
are not present in the isolated environments presented to Nix.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
As an alternative, we can inform <command > pkg-config</command> about
the location of libraries via the <varname > PKG_CONFIG_PATH</varname>
environment variable. We can populate this environment variable
using the same trick we used for <varname > PATH</varname> :
automatically filling the variables from <literal > buildInputs</literal> .
This is the relevant snippet of <filename > setup.sh</filename> :
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/setup-sh.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
Now if we add derivations to <literal > buildInputs</literal> , their
<filename > lib/pkgconfig</filename> and <filename > bin</filename> paths
are automatically added in <filename > setup.sh</filename> .
2017-08-17 23:54:26 -04:00
</para>
</section>
2024-02-09 06:47:48 -05:00
2017-08-17 23:54:26 -04:00
<section >
2024-02-09 06:47:48 -05:00
<title > Completing graphviz with <code > gd</code> </title>
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
Below, we finish the expression for <code > graphviz</code> with <code > gd</code> support.
Note the use of the <literal > with</literal> expression in <literal > buildInputs</literal> to avoid repeating <literal > pkgs</literal> :
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/graphviz-gd-derivation.txt" parse= "text" /> </screen>
<para >
2021-12-02 13:27:44 -05:00
We add <command > pkg-config</command> to the derivation to make this tool
2024-02-09 06:47:48 -05:00
available for the configure script. As <code > gd</code> is a package
2021-12-02 13:27:44 -05:00
with <link xlink:href= "https://nixos.org/manual/nixpkgs/stable/#sec-multiple-outputs-" > split outputs</link> ,
2024-02-09 06:47:48 -05:00
we need to add both the library and development outputs.
2021-12-02 13:27:44 -05:00
</para>
<para >
2024-02-09 06:47:48 -05:00
After building, <code > graphviz</code> can now create <code > png</code> s.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
<title > The repository expression</title>
<para >
2024-02-09 06:47:48 -05:00
Now that we have two packages, we want to combine them into a single repository.
To do so, we'll mimic what <literal > nixpkgs</literal> does: we will create
a single attribute set containing derivations. This attribute set can
2024-02-20 00:14:06 -05:00
then be imported, and derivations can be selected by accessing the
2024-02-09 06:47:48 -05:00
top-level attribute set.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
Using this technique we are able to abstract from the file names.
Instead of referring to a package by <filename > REPO/some/sub/dir/package.nix</filename> ,
this technique allows us to select a derivation as
<literal > importedRepo.package</literal> (or <literal > pkgs.package</literal>
in our examples).
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
To begin, create a default.nix in the current directory:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/repository.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
This file is ready to use with <command > nix repl</command> :
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/repository-test-nix-repl.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
With <command > nix-build</command> , we can pass the <arg > -A</arg> option to
access an attribute of the set from the given <filename > .nix</filename> expression:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/repository-test-nix-build.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
The <filename > default.nix</filename> file is special. When a directory
2024-02-20 00:14:06 -05:00
contains a <filename > default.nix</filename> file, it is used as the implicit
nix expression of the directory. This, for example, allows us to run
2024-02-09 06:47:48 -05:00
<command > nix-build -A hello</command> without specifying
<filename > default.nix</filename> explicitly.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
We can now use <command > nix-env</command> to install the package into our
user environment:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/nix-env-install-graphviz.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
Taking a closer look at the above command, we see the following options:
<itemizedlist >
<listitem > <para >
The <arg > -f</arg> option is used to specify the expression to use. In this case,
the expression is the <filename > ./default.nix</filename> of the current directory.
</para> </listitem>
<listitem > <para >
The <arg > -i</arg> option stands for "installation".
</para> </listitem>
<listitem > <para >
The <arg > -A</arg> is the same as above for <command > nix-build</command> .
</para> </listitem>
</itemizedlist>
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
We reproduced the very basic behavior of <literal > nixpkgs</literal> : combining
multiple derivations into a single, top-level attribute set.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
<title > The inputs pattern</title>
<para >
2024-02-09 06:47:48 -05:00
The approach we've taken so far has a few problems:
2017-08-17 23:54:26 -04:00
<itemizedlist >
2024-02-09 06:47:48 -05:00
<listitem > <para >
First, <filename > hello.nix</filename> and <filename > graphviz.nix</filename> are
dependent on <literal > nixpkgs</literal> , which they import directly.
A better approach would be to pass in <literal > nixpkgs</literal> as an argument,
as we did in <filename > autotools.nix</filename> .
</para> </listitem>
<listitem > <para >
Second, we don't have a straightforward way to compile different variants
of the same software, such as <code > graphviz</code> with or without
<code > libgd</code> support.
</para> </listitem>
<listitem > <para >
Third, we don't have a way to test <code > graphviz</code>
with a particular <code > libgd</code> version.
</para> </listitem>
2017-08-17 23:54:26 -04:00
</itemizedlist>
</para>
<para >
2024-02-09 06:47:48 -05:00
Until now, our approach to addressing the above problems has been inadequate
and required changing the nix expression to match our needs. With the
<literal > inputs</literal> pattern, we provide another answer: let the user
change the <literal > inputs</literal> of the expression.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
When we talk about "the inputs of an expression", we are referring to the
set of derivations needed to build that expression. In this case:
2017-08-17 23:54:26 -04:00
<itemizedlist >
2024-02-09 06:47:48 -05:00
<listitem > <para >
<literal > mkDerivation</literal> from <code > autotools</code> . Recall
that <literal > mkDerivation</literal> has an implicit dependency on
the toolchain.
</para> </listitem>
<listitem > <para >
<code > libgd</code> and its dependencies.
</para> </listitem>
2017-08-17 23:54:26 -04:00
</itemizedlist>
</para>
<para >
2024-02-09 06:47:48 -05:00
The <filename > ./src</filename> directory is also an input,
but we wouldn't change the source from the caller.
In <literal > nixpkgs</literal> we prefer to write another expression
for version bumps (e.g. because patches or different inputs are needed).
2017-08-17 23:54:26 -04:00
</para>
2024-02-09 06:47:48 -05:00
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
Our goal is to make package expressions independent of the repository. To
achieve this, we use functions to declare inputs for a derivation. For example,
with <filename > graphviz.nix</filename> , we make the following changes to make
the derivation independent of the repository and customizable:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/graphviz-mkderivation.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
Recall that "<literal > {...}: ...</literal> " is the syntax for defining functions
accepting an attribute set as argument; the above snippet just defines a function.
2017-08-17 23:54:26 -04:00
</para>
2024-02-09 06:47:48 -05:00
2017-08-17 23:54:26 -04:00
<para >
2024-02-09 06:47:48 -05:00
We made <code > gd</code> and its dependencies optional. If <literal > gdSupport</literal>
is true (which it is by default), we will fill <literal > buildInputs</literal> and
<code > graphviz</code> will be built with <code > gd</code> support. Otherwise, if
an attribute set is passed with <code > gdSupport = false;</code> , the build
will be completed without <code > gd</code> support.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
Going back to back to <filename > default.nix</filename> , we modify our expression
to utilize the inputs pattern:
2017-08-17 23:54:26 -04:00
</para>
<screen > <xi:include href= "./12/repository-mkderivation.txt" parse= "text" /> </screen>
<para >
2024-02-09 06:47:48 -05:00
We factorized the import of <literal > nixpkgs</literal> and
<literal > mkDerivation</literal> , and also added a variant of <code > graphviz</code>
with <code > gd</code> support disabled. The result is that both
<filename > hello.nix</filename> (left as an exercise for the reader) and
<filename > graphviz.nix</filename> are independent of the repository and
customizable by passing specific inputs.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
If we wanted to build <code > graphviz</code> with a specific version of
<code > gd</code> , it would suffice to pass <literal > gd = ...;</literal> .
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-20 00:14:06 -05:00
If we wanted to change the toolchain, we would simply pass a different
2024-02-09 06:47:48 -05:00
<literal > mkDerivation</literal> function.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
Let's talk a closer look at the snippet and dissect the syntax:
2017-08-17 23:54:26 -04:00
<itemizedlist >
2024-02-09 06:47:48 -05:00
<listitem > <para > The entire expression in <filename > default.nix</filename>
returns an attribute set with the keys <code > hello</code> ,
<code > graphviz</code> , and <code > graphvizCore</code> .
</para> </listitem>
<listitem > <para >
With "<literal > let</literal> ", we define some local variables.
</para> </listitem>
<listitem > <para >
We bring <literal > pkgs</literal> into the scope when defining the
package set. This saves us from having to type
<literal > pkgs</literal> " repeatedly.
</para> </listitem>
<listitem > <para >
We import <filename > hello.nix</filename> and <filename > graphviz.nix</filename> ,
which each return a function. We call the functions with a set of inputs to
get back the derivation.
</para> </listitem>
<listitem > <para >
The "<literal > inherit x</literal> " syntax is equivalent to
"<literal > x = x</literal> ". This means that the "<literal > inherit gd</literal> "
here, combined with the above "<literal > with pkgs;</literal> ",
is equivalent to "<literal > gd = pkgs.gd</literal> ".
</para> </listitem>
2017-08-17 23:54:26 -04:00
</itemizedlist>
</para>
<para >
2024-02-09 06:47:48 -05:00
The entire repository of this can be found at the <link xlink:href= "https://gist.github.com/tfc/ca800a444b029e85a14e530c25f8e872" > pill 12</link> gist.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
<title > Conclusion</title>
<para >
2024-02-09 06:47:48 -05:00
The "<literal > inputs</literal> " pattern allows our expressions to be easily
customizable through a set of arguments. These arguments could be flags,
derivations, or any other customizations enabled by the nix language.
Our package expressions are simply functions: there is no extra magic present.
2017-08-17 23:54:26 -04:00
</para>
<para >
2024-02-09 06:47:48 -05:00
The "<literal > inputs</literal> " pattern also makes the expressions
independent of the repository. Given that we pass all needed information
through arguments, it is possible to use these expressions in any other context.
2017-08-17 23:54:26 -04:00
</para>
</section>
<section >
<title > Next pill</title>
<para >
2024-02-09 06:47:48 -05:00
In the next pill, we will talk about the "<literal > callPackage</literal> " design
pattern. This removes the tedium of specifying the names of the inputs twice:
once in the top-level <filename > default.nix</filename> , and once in the package
expression. With <literal > callPackage</literal> , we will
implicitly pass the necessary inputs from the top-level expression.
2017-08-17 23:54:26 -04:00
</para>
</section>
2017-08-11 21:41:14 -04:00
</chapter>