mirror of
https://github.com/NixOS/nix-pills
synced 2024-09-19 04:00:13 -04:00
362 lines
17 KiB
XML
362 lines
17 KiB
XML
<chapter xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
version="5.0"
|
|
xml:id="inputs-design-pattern">
|
|
|
|
<title>Package Repositories and the Inputs Design Pattern</title>
|
|
<para>
|
|
Welcome to the 12th Nix pill. In the previous <link linkend="garbage-collector">11th
|
|
pill</link>, we stopped packaging and cleaned up the system with the garbage collector.
|
|
</para>
|
|
<para>
|
|
This time, we will resume packaging and improve different aspects of it. We will also
|
|
demonstrate how to create a repository of multiple packages.
|
|
</para>
|
|
<section>
|
|
<title>Repositories in Nix</title>
|
|
<para>
|
|
Package repositories in Nix arose naturally from the need to organize packages.
|
|
There is no preset directory structure or packaging policy prescribed by Nix itself;
|
|
Nix, as a full, functional programming language, is powerful enough to support
|
|
multiple different repository formats.
|
|
</para>
|
|
|
|
<para>
|
|
Over time, the <literal>nixpkgs</literal> repository evolved a particular
|
|
structure. This structure reflects the history of Nix as well as the design
|
|
patterns adopted by its users as useful tools in building and organizing
|
|
packages. Below, we will examine some of these patterns in detail.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>The single repository pattern</title>
|
|
<para>
|
|
Different operating system distributions have different opinions about how
|
|
package repositories should be organized. Systems like Debian scatter packages
|
|
in several small repositories (which tends to make tracking interdependent
|
|
changes more difficult, and hinders contributions to the repositories),
|
|
while systems like Gentoo put all package descriptions in a single repository.
|
|
</para>
|
|
<para>
|
|
Nix follows the "single repository" pattern by placing all descriptions of all
|
|
packages into <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>.
|
|
This approach has proven natural and attractive for new contributions.
|
|
</para>
|
|
<para>
|
|
For the rest of this pill, we will adopt the single repository pattern. The
|
|
natural implementation in Nix is to create a top-level Nix expression, followed
|
|
by one expression for each package. The top-level expression imports and combines
|
|
all package expressions in an attribute set mapping names to packages.
|
|
</para>
|
|
<para>
|
|
In some programming languages, such an approach -- including every possible
|
|
package description in a single data structure -- would be untenable due
|
|
to the language needing to load the entire data structure into memory before
|
|
operating on it. Nix, however, is a lazy language and only evaluates what is
|
|
needed.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>Packaging <code>graphviz</code></title>
|
|
<para>
|
|
We have already packaged GNU <code>hello</code>. Next, we will package a
|
|
graph-drawing program called <code>graphviz</code> so that we can
|
|
create a repository containing multiple packages. The <code>graphviz</code>
|
|
package was selected because it uses the standard autotools build system and
|
|
requires no patching. It also has optional dependencies, which will give us
|
|
an opportunity to illustrate a technique to configure builds to a particular
|
|
situation.
|
|
</para>
|
|
<para>
|
|
First, we download <code>graphviz</code> from <link xlink:href="https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.3/graphviz-2.49.3.tar.gz">gitlab</link>. The <filename>graphviz.nix</filename> expression is straightforward:
|
|
</para>
|
|
<screen><xi:include href="./12/graphviz-derivation.txt" parse="text" /></screen>
|
|
<para>
|
|
If we build the project with <command>nix-build graphviz.nix</command>, we will get runnable binaries under <filename>result/bin</filename>. Notice how we reused the same <filename>autotools.nix</filename> of <filename>hello.nix.</filename>
|
|
</para>
|
|
<para>
|
|
By default, <code>graphviz</code> does not compile with the ability to produce
|
|
<code>png</code> files. Thus, the derivation above will build a binary
|
|
supporting only the native output formats, as we see below:
|
|
</para>
|
|
<screen><xi:include href="./12/simple-png.txt" parse="text" /></screen>
|
|
<para>
|
|
If we want to produce a <code>png</code> file with <code>graphviz</code>, we
|
|
must add it to our derivation. The place to do so is
|
|
in <filename>autotools.nix</filename>, where we created a
|
|
<literal>buildInputs</literal> variable that gets concatenated to
|
|
<literal>baseInputs</literal>. This is the exact reason for this variable: to
|
|
allow users of <filename>autotools.nix</filename> to add additional inputs
|
|
from package expressions.
|
|
</para>
|
|
<para>
|
|
Version 2.49 of <code>graphviz</code> has several plugins to output
|
|
<code>png</code>. For simplicity, we will use <code>libgd</code>.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>Passing library information to <command>pkg-config</command> via environment
|
|
variables</title>
|
|
<para>
|
|
The <code>graphviz</code> configuration script uses <command>pkg-config</command>
|
|
to specify which flags are passed to the compiler. Since there is no global location
|
|
for libraries, we need to tell <command>pkg-config</command> where to find
|
|
its description files, which tell the configuration script where to find
|
|
headers and libraries.
|
|
</para>
|
|
<para>
|
|
In classic POSIX systems, <command>pkg-config</command> just finds the
|
|
<filename>.pc</filename> files of all installed libraries in system folders
|
|
like <filename>/usr/lib/pkgconfig</filename>. However, these files
|
|
are not present in the isolated environments presented to Nix.
|
|
</para>
|
|
<para>
|
|
As an alternative, we can inform <command>pkg-config</command> about
|
|
the location of libraries via the <varname>PKG_CONFIG_PATH</varname>
|
|
environment variable. We can populate this environment variable
|
|
using the same trick we used for <varname>PATH</varname>:
|
|
automatically filling the variables from <literal>buildInputs</literal>.
|
|
This is the relevant snippet of <filename>setup.sh</filename>:
|
|
</para>
|
|
<screen><xi:include href="./12/setup-sh.txt" parse="text" /></screen>
|
|
<para>
|
|
Now if we add derivations to <literal>buildInputs</literal>, their
|
|
<filename>lib/pkgconfig</filename> and <filename>bin</filename> paths
|
|
are automatically added in <filename>setup.sh</filename>.
|
|
</para>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Completing graphviz with <code>gd</code></title>
|
|
<para>
|
|
Below, we finish the expression for <code>graphviz</code> with <code>gd</code> support.
|
|
Note the use of the <literal>with</literal> expression in <literal>buildInputs</literal> to avoid repeating <literal>pkgs</literal>:
|
|
</para>
|
|
<screen><xi:include href="./12/graphviz-gd-derivation.txt" parse="text" /></screen>
|
|
<para>
|
|
We add <command>pkg-config</command> to the derivation to make this tool
|
|
available for the configure script. As <code>gd</code> is a package
|
|
with <link xlink:href="https://nixos.org/manual/nixpkgs/stable/#sec-multiple-outputs-">split outputs</link>,
|
|
we need to add both the library and development outputs.
|
|
</para>
|
|
<para>
|
|
After building, <code>graphviz</code> can now create <code>png</code>s.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>The repository expression</title>
|
|
<para>
|
|
Now that we have two packages, we want to combine them into a single repository.
|
|
To do so, we'll mimic what <literal>nixpkgs</literal> does: we will create
|
|
a single attribute set containing derivations. This attribute set can
|
|
then be imported, and derivations can be selected by accessing the
|
|
top-level attribute set.
|
|
</para>
|
|
<para>
|
|
Using this technique we are able to abstract from the file names.
|
|
Instead of referring to a package by <filename>REPO/some/sub/dir/package.nix</filename>,
|
|
this technique allows us to select a derivation as
|
|
<literal>importedRepo.package</literal> (or <literal>pkgs.package</literal>
|
|
in our examples).
|
|
</para>
|
|
<para>
|
|
To begin, create a default.nix in the current directory:
|
|
</para>
|
|
<screen><xi:include href="./12/repository.txt" parse="text" /></screen>
|
|
<para>
|
|
This file is ready to use with <command>nix repl</command>:
|
|
</para>
|
|
<screen><xi:include href="./12/repository-test-nix-repl.txt" parse="text" /></screen>
|
|
<para>
|
|
With <command>nix-build</command>, we can pass the <arg>-A</arg> option to
|
|
access an attribute of the set from the given <filename>.nix</filename> expression:
|
|
</para>
|
|
<screen><xi:include href="./12/repository-test-nix-build.txt" parse="text" /></screen>
|
|
<para>
|
|
The <filename>default.nix</filename> file is special. When a directory
|
|
contains a <filename>default.nix</filename> file, it is used as the implicit
|
|
nix expression of the directory. This, for example, allows us to run
|
|
<command>nix-build -A hello</command> without specifying
|
|
<filename>default.nix</filename> explicitly.
|
|
</para>
|
|
<para>
|
|
We can now use <command>nix-env</command> to install the package into our
|
|
user environment:
|
|
</para>
|
|
<screen><xi:include href="./12/nix-env-install-graphviz.txt" parse="text" /></screen>
|
|
<para>
|
|
Taking a closer look at the above command, we see the following options:
|
|
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
The <arg>-f</arg> option is used to specify the expression to use. In this case,
|
|
the expression is the <filename>./default.nix</filename> of the current directory.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
The <arg>-i</arg> option stands for "installation".
|
|
</para></listitem>
|
|
<listitem><para>
|
|
The <arg>-A</arg> is the same as above for <command>nix-build</command>.
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
We reproduced the very basic behavior of <literal>nixpkgs</literal>: combining
|
|
multiple derivations into a single, top-level attribute set.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>The inputs pattern</title>
|
|
<para>
|
|
The approach we've taken so far has a few problems:
|
|
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
First, <filename>hello.nix</filename> and <filename>graphviz.nix</filename> are
|
|
dependent on <literal>nixpkgs</literal>, which they import directly.
|
|
A better approach would be to pass in <literal>nixpkgs</literal> as an argument,
|
|
as we did in <filename>autotools.nix</filename>.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
Second, we don't have a straightforward way to compile different variants
|
|
of the same software, such as <code>graphviz</code> with or without
|
|
<code>libgd</code> support.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
Third, we don't have a way to test <code>graphviz</code>
|
|
with a particular <code>libgd</code> version.
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
Until now, our approach to addressing the above problems has been inadequate
|
|
and required changing the nix expression to match our needs. With the
|
|
<literal>inputs</literal> pattern, we provide another answer: let the user
|
|
change the <literal>inputs</literal> of the expression.
|
|
</para>
|
|
<para>
|
|
When we talk about "the inputs of an expression", we are referring to the
|
|
set of derivations needed to build that expression. In this case:
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
<literal>mkDerivation</literal> from <code>autotools</code>. Recall
|
|
that <literal>mkDerivation</literal> has an implicit dependency on
|
|
the toolchain.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
<code>libgd</code> and its dependencies.
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
The <filename>./src</filename> directory is also an input,
|
|
but we wouldn't change the source from the caller.
|
|
In <literal>nixpkgs</literal> we prefer to write another expression
|
|
for version bumps (e.g. because patches or different inputs are needed).
|
|
</para>
|
|
|
|
<para>
|
|
Our goal is to make package expressions independent of the repository. To
|
|
achieve this, we use functions to declare inputs for a derivation. For example,
|
|
with <filename>graphviz.nix</filename>, we make the following changes to make
|
|
the derivation independent of the repository and customizable:
|
|
</para>
|
|
<screen><xi:include href="./12/graphviz-mkderivation.txt" parse="text" /></screen>
|
|
<para>
|
|
Recall that "<literal>{...}: ...</literal>" is the syntax for defining functions
|
|
accepting an attribute set as argument; the above snippet just defines a function.
|
|
</para>
|
|
|
|
<para>
|
|
We made <code>gd</code> and its dependencies optional. If <literal>gdSupport</literal>
|
|
is true (which it is by default), we will fill <literal>buildInputs</literal> and
|
|
<code>graphviz</code> will be built with <code>gd</code> support. Otherwise, if
|
|
an attribute set is passed with <code>gdSupport = false;</code>, the build
|
|
will be completed without <code>gd</code> support.
|
|
</para>
|
|
<para>
|
|
Going back to back to <filename>default.nix</filename>, we modify our expression
|
|
to utilize the inputs pattern:
|
|
</para>
|
|
<screen><xi:include href="./12/repository-mkderivation.txt" parse="text" /></screen>
|
|
<para>
|
|
We factorized the import of <literal>nixpkgs</literal> and
|
|
<literal>mkDerivation</literal>, and also added a variant of <code>graphviz</code>
|
|
with <code>gd</code> support disabled. The result is that both
|
|
<filename>hello.nix</filename> (left as an exercise for the reader) and
|
|
<filename>graphviz.nix</filename> are independent of the repository and
|
|
customizable by passing specific inputs.
|
|
</para>
|
|
<para>
|
|
If we wanted to build <code>graphviz</code> with a specific version of
|
|
<code>gd</code>, it would suffice to pass <literal>gd = ...;</literal>.
|
|
</para>
|
|
<para>
|
|
If we wanted to change the toolchain, we would simply pass a different
|
|
<literal>mkDerivation</literal> function.
|
|
</para>
|
|
<para>
|
|
Let's talk a closer look at the snippet and dissect the syntax:
|
|
<itemizedlist>
|
|
<listitem><para>The entire expression in <filename>default.nix</filename>
|
|
returns an attribute set with the keys <code>hello</code>,
|
|
<code>graphviz</code>, and <code>graphvizCore</code>.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
With "<literal>let</literal>", we define some local variables.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
We bring <literal>pkgs</literal> into the scope when defining the
|
|
package set. This saves us from having to type
|
|
<literal>pkgs</literal>" repeatedly.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
We import <filename>hello.nix</filename> and <filename>graphviz.nix</filename>,
|
|
which each return a function. We call the functions with a set of inputs to
|
|
get back the derivation.
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
The "<literal>inherit x</literal>" syntax is equivalent to
|
|
"<literal>x = x</literal>". This means that the "<literal>inherit gd</literal>"
|
|
here, combined with the above "<literal>with pkgs;</literal>",
|
|
is equivalent to "<literal>gd = pkgs.gd</literal>".
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
The entire repository of this can be found at the <link xlink:href="https://gist.github.com/tfc/ca800a444b029e85a14e530c25f8e872">pill 12</link> gist.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>Conclusion</title>
|
|
<para>
|
|
The "<literal>inputs</literal>" pattern allows our expressions to be easily
|
|
customizable through a set of arguments. These arguments could be flags,
|
|
derivations, or any other customizations enabled by the nix language.
|
|
Our package expressions are simply functions: there is no extra magic present.
|
|
</para>
|
|
<para>
|
|
The "<literal>inputs</literal>" pattern also makes the expressions
|
|
independent of the repository. Given that we pass all needed information
|
|
through arguments, it is possible to use these expressions in any other context.
|
|
</para>
|
|
</section>
|
|
<section>
|
|
<title>Next pill</title>
|
|
<para>
|
|
In the next pill, we will talk about the "<literal>callPackage</literal>" design
|
|
pattern. This removes the tedium of specifying the names of the inputs twice:
|
|
once in the top-level <filename>default.nix</filename>, and once in the package
|
|
expression. With <literal>callPackage</literal>, we will
|
|
implicitly pass the necessary inputs from the top-level expression.
|
|
</para>
|
|
</section>
|
|
</chapter>
|