NixOS/nix-pills
NixOS/nix-pills
1
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-09-18 03:50:12 -04:00
Code Activity

Non-content edits to pills 9-14 (#195)

Browse source 
* grammar and readability edits

Co-authored-by: Henrik <i97henka@gmail.com>
This commit is contained in:
Peter Dragos 2024-02-09 13:47:48 +02:00 committed by GitHub
parent 778512e183
commit a66fd074cf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 827 additions and 511 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

206
pills/09-automatic-runtime.xml
View file

@ -9,13 +9,13 @@
<para>
Welcome to the 9th Nix pill. In the previous
<link linkend="generic-builders">8th pill</link> we wrote a generic builder
for autotools projects. We feed build dependencies, a source tarball, and
we get a Nix derivation as a result.
for autotools projects. We fed in build dependencies and a source tarball, and
we received a Nix derivation as a result.
</para>
<para>
Today we stop by the GNU hello world program to analyze build and runtime
dependencies, and enhance the builder in order to avoid unnecessary runtime
Today we stop by the GNU <code>hello</code> program to analyze build and runtime
dependencies, and we enhance our builder to eliminate unnecessary runtime
dependencies.
</para>
@ -23,26 +23,26 @@
<title>Build dependencies</title>
<para>
Let's start analyzing build dependencies for our GNU hello world package:
Let's start analyzing build dependencies for our GNU <code>hello</code> package:
</para>
<screen><xi:include href="./09/instantiate.txt" parse="text" /></screen>
<para>
It has exactly the derivations referenced in the <code>derivation</code>
function, nothing more, nothing less. Some of them might not be used at
all, however given that our generic mkDerivation function always pulls
It has precisely the derivations referenced in the <code>derivation</code> function;
nothing more, nothing less. Of course, we may not use some of them at all.
However, given that our generic <code>mkDerivation</code> function always pulls
such dependencies (think of it like
<link xlink:href="https://packages.debian.org/unstable/build-essential">build-essential</link>
of Debian), for every package you build from now on, you will have these
packages in the nix store.
from Debian), we will already have these packages in the nix store for any future packages that
need them.
</para>
<para>
Why are we looking at .drv files? Because the hello.drv file is the
representation of the build action to perform in order to build the hello
out path, and as such it also contains the input derivations needed to be
built before building hello.
Why are we looking at <code>.drv</code> files? Because the <code>hello.drv</code>
file is the representation of the build action that builds the <code>hello</code>
out path. As such, it contains the input derivations needed before building
<code>hello</code>.
</para>
</section>
@ -50,23 +50,29 @@
<title>Digression about NAR files</title>
<para>
NAR is the Nix ARchive. First question: why not tar? Why another archiver?
Because commonly used archivers are not deterministic. They add padding,
they do not sort files, they add timestamps, etc.. Hence NAR, a very
simple deterministic archive format being used by Nix for deployment.
NARs are also used extensively within Nix itself as we'll see below.
The <code>NAR</code> format is the "Nix ARchive". This format was designed due to
existing archive formats, such as <code>tar</code>, being insufficient.
Nix benefits from deterministic build tools, but commonly used archivers
lack this property: they add padding, they do not sort files, they add timestamps,
and so on. This can result in directories containing bit-identical files turning into
non-bit-identical archives, which leads to different hashes.
</para>
<para>
For the rationale and implementation details you can find more in the
Thus the <code>NAR</code> format was developed as a simple, deterministic
archive format. <code>NAR</code>s are used extensively within Nix, as we will
see below.
</para>
<para>
For more rationale and implementation details behind <code>NAR</code> see
<link xlink:href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Dolstra's PhD Thesis</link>.
</para>
<para>
To create NAR archives, it's possible to use
To create NAR archives from store paths, we can use
<command>nix-store --dump</command> and
<command>nix-store --restore</command>. Those two commands work
regardless of <filename>/nix/store</filename>.
<command>nix-store --restore</command>.
</para>
</section>
@ -74,83 +80,72 @@
<title>Runtime dependencies</title>
<para>
Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in
any <code>derivation</code> call, but we never specify what are the
runtime dependencies for a derivation.
We now note that Nix automatically recognized build dependencies once our
<code>derivation</code> call referred to them, but we never specified the
runtime dependencies.
</para>
<para>
There's really no black magic involved. It's something that at first glance
makes you think "no, this can't work in the long term", but at the same
time it works so well that a whole operating system is built on top of
this magic.
</para>
<para>
In other words, Nix automatically computes all the runtime dependencies
of a derivation, and it's possible thanks to the hash of the store paths.
</para>
<para>
Steps:
Nix handles runtime dependencies for us automatically. The technique it uses
to do so may seem fragile at first glance, but it works so well that the NixOS
operating system is built off of it. The underlying mechanism relies on the
hash of the store paths. It proceeds in three steps:
</para>
<orderedlist>
<listitem>
<para>
Dump the derivation as NAR, a serialization of the derivation output.
Works fine whether it's a single file or a directory.
Dump the derivation as a NAR. Recall that this is a serialization of
the derivation output -- meaning this works fine whether the output
is a single file or a directory.
</para>
</listitem>
<listitem>
<para>
For each build dependency .drv and its relative out path, search the
contents of the NAR for this out path.
For each build dependency <code>.drv</code> and it's relative out path,
search the contents of the NAR for this out path.
</para>
</listitem>
<listitem>
<para>
If found, then it's a runtime dependency.
If the path is found, then it's a runtime dependency.
</para>
</listitem>
</orderedlist>
<para>
You really do get all the runtime dependencies; that's why Nix
deployments are so easy.
The snippet below shows the dependencies for <code>hello</code>.
</para>
<screen><xi:include href="./09/instantiate-hello.txt" parse="text" /></screen>
<para>
Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
We see that <code>glibc</code> and <code>gcc</code> are runtime dependencies.
Intuitively, <code>gcc</code> shouldn't be in this list! Displaying the
printable strings in the <code>hello</code> binary shows that the out path
of <code>gcc</code> does indeed appear:
</para>
<screen><xi:include href="./09/strings.txt" parse="text" /></screen>
<para>
Oh Nix added gcc because its out path is mentioned in the "hello" binary.
Why is that? That's the
<link xlink:href="http://en.wikipedia.org/wiki/Rpath">ld rpath</link>.
It's the list of directories where libraries can be found at runtime. In
other distributions, this is usually not abused. But in Nix, we have to
refer to particular versions of libraries, thus the rpath has an
important role.
This is why Nix added <code>gcc</code>. But why is that path present in the
first place? The answer is that it is the <link xlink:href="http://en.wikipedia.org/wiki/Rpath">ld rpath</link>: the list of
directories where libraries can be found at runtime. In other distributions,
this is usually not abused. But in Nix, we have to refer to particular versions
of libraries, and thus the rpath has an important role.
</para>
<para>
The build process adds that gcc lib path thinking it may be useful at
runtime, but really it's not. How do we get rid of it? Nix authors have
written another magical tool called
<link xlink:href="https://github.com/NixOS/patchelf">patchelf</link>, which
is able to reduce the rpath to the paths that are really used by the
binary.
The build process adds the <code>gcc</code> lib path thinking it may be useful
at runtime, but this isn't necessary. To address issues like these, Nix provides
a tool called <link xlink:href="https://nixos.org/patchelf.html">patchelf</link>,
which reduces the rpath to the paths that are actually used by the binary.
</para>
<para>
Even after reducing the rpath, the hello binary would still
depend upon gcc because of some debugging information. This
Even after reducing the rpath, the <code>hello</code> binary would still
depend upon <code>gcc</code> because of some debugging information. This
unnecesarily increases the size of our runtime
dependencies. We'll explore how <command><link
xlink:href="https://linux.die.net/man/1/strip">strip</link>
@ -162,48 +157,49 @@
<title>Another phase in the builder</title>
<para>
We will add a new phase to our autotools builder. The builder has these
We will add a new phase to our autotools builder. The builder has six
phases already:
</para>
<orderedlist>
<listitem>
<para>
First the environment is set up
The "environment setup" phase
</para>
</listitem>
<listitem>
<para>
Unpack phase: we unpack the sources in the current directory
(remember, Nix changes dir to a temporary directory first)
The "unpack phase": we unpack the sources in the current directory
(remember, Nix changes to a temporary directory first)
</para>
</listitem>
<listitem>
<para>
Change source root to the directory that has been unpacked
The "change directory" phase, where we change source root to the
directory that has been unpacked
</para>
</listitem>
<listitem>
<para>
Configure phase: <command>./configure</command>
The "configure" phase: <command>./configure</command>
</para>
</listitem>
<listitem>
<para>
Build phase: <command>make</command>
The "build" phase: <command>make</command>
</para>
</listitem>
<listitem>
<para>
Install phase: <command>make install</command>
The "install" phase: <command>make install</command>
</para>
</listitem>
</orderedlist>
<para>
We add a new phase after the installation phase, which we call
<emphasis role="bold">fixup</emphasis> phase. At the end of the
<filename>builder.sh</filename> follows:
Now we will add a new phase after the installation phase, which we call
the "fixup" phase. At the end of the
<filename>builder.sh</filename>, we append:
</para>
<screen><xi:include href="./09/find.txt" parse="text" /></screen>
@ -211,36 +207,40 @@
<para>
That is, for each file we run <command>patchelf --shrink-rpath</command>
and <command>strip</command>. Note that we used two new commands here,
<command>find</command> and <command>patchelf</command>.
<emphasis role="bold">Exercise:</emphasis> These two
deserve a place in <code>baseInputs</code> of
<filename>autotools.nix</filename> as <command>findutils</command> and
<command>patchelf</command>.
<command>find</command> and <command>patchelf</command>. These must be
added to our derivation.
</para>
<para>
Rebuild <filename>hello.nix</filename> and...:
<emphasis role="bold">Exercise:</emphasis> Add <code>findutils</code>
and <code>patchelf</code> to the <code>baseInputs</code> of
<filename>autotools.nix</filename>.
</para>
<para>
Now, we rebuild <filename>hello.nix</filename>:nd...:
</para>
<screen><xi:include href="./09/build-hello-nix.txt" parse="text" /></screen>
<para>
...only glibc is the runtime dependency. Exactly what we wanted.
and we see that <code>glibc</code> is a runtime dependency. This is
exactly what we wanted.
</para>
<para>
The package is self-contained, copy its closure on another machine and
you will be able to run it. Remember, only a very few components under
the <filename>/nix/store</filename> are required to
The package is self-contained. This means that we can copy its closure onto
another machine and we will be able to run it. Remember, only a very few
components under the <filename>/nix/store</filename> are required to
<link linkend="install-on-your-running-system">run nix</link>.
The hello binary will use that exact version of glibc library and
interpreter, not the system one:
The <code>hello</code> binary will use the exact version of <code>glibc</code>
library and interpreter referred to in the binary, rather than the system one:
</para>
<screen><xi:include href="./09/ldd-hello.txt" parse="text" /></screen>
<para>
Of course, the executable runs fine as long as everything is under the
Of course, the executable will run fine as long as everything is under the
<filename>/nix/store</filename> path.
</para>
</section>
@ -249,21 +249,19 @@
<title>Conclusion</title>
<para>
Short post compared to previous ones as I'm still on vacation, but I hope
you enjoyed it. Nix provides tools with cool features. In particular, Nix
is able to compute all runtime dependencies automatically for us. This is
not limited to only shared libraries, but also referenced executables,
scripts, Python libraries etc..
We saw some of the tools Nix provides, along with their features.
In particular, we saw how Nix is able to compute runtime dependencies
automatically. This is not limited to only shared libraries,
but can also referenced executables, scripts, Python libraries, and so
forth.
</para>
<para>
This makes packages self-contained, ensuring (apart data and
configuration) that copying the runtime closure on another machine is
sufficient to run the program. That's what allows running programs without
installation using <code>nix-shell</code>
or
Approaching builds in this way makes packages self-contained, ensuring
(apart from data and configuration) that copying the runtime closure onto
another machine is sufficient to run the program. This enables us to run programs
without installation using <command>nix-shell</command>, and forms the basis for
<link xlink:href="https://nixos.org/manual/nix/stable/introduction.html">reliable deployment in the cloud</link>.
All with one tool.
</para>
</section>
@ -271,11 +269,13 @@
<title>Next pill</title>
<para>
...we will introduce nix-shell. With nix-build we always build
derivations from scratch: the source gets unpacked, configured, built
and installed. But this may take a long time, think of WebKit. What if we
want to apply some small changes and compile incrementally instead, yet
keeping a self-contained environment similar to nix-build?
The next pill will introduce <command>nix-shell</command>. With
<command>nix-build</command>, we've always built derivations from
scratch: the source gets unpacked, configured, built, and installed.
But this can take a long time for large packages. What if we want to
apply some small changes and compile incrementally instead, yet still
want to keep a self-contained environment similar to <command>nix-build</command>?
<command>nix-shell</command> enables this.
</para>
</section>
</chapter>

168
pills/10-developing-with-nix-shell.xml
View file

@ -9,85 +9,101 @@
<para>
Welcome to the 10th Nix pill. In the previous
<link linkend="automatic-runtime-dependencies">9th pill</link> we saw
one of the powerful features of nix, automatic discovery of runtime
dependencies and finalized the GNU hello world package.
one of the powerful features of Nix: automatic discovery of runtime
dependencies. We also finalized the GNU <code>hello</code> package.
</para>
<para>
Having returned from vacation, we want to hack a little the GNU hello
world program. The nix-build tool allows for an isolated environment
while building the derivation. Additionally, we'd like the same
isolation in order to modify some source files of the project.
In this pill, we will introduce the <command>nix-shell</command> tool
and use it to hack on the GNU <code>hello</code> program. We will
see how <command>nix-shell</command> gives us an isolated environment
while we modify the source files of the project, similar to how
<command>nix-build</command> gave us an isolated environment while building
the derivation.
</para>
<para>
Finally, we will modify our builder to work more ergonomically
with a <command>nix-shell</command>-focused workflow.
</para>
<section>
<title>What's nix-shell</title>
<title>What is <command>nix-shell</command>?</title>
<para>
The <link
xlink:href="https://nixos.org/manual/nix/stable/command-ref/nix-shell.html">nix-shell</link>
tool drops us in a shell by setting up the necessary environment
variables to hack on a derivation. It does not build the derivation, it
tool drops us in a shell after setting up the environment variables necessary
to hack on a derivation. It does not build the derivation; it
only serves as a preparation so that we can run the build steps manually.
</para>
<para>
I remind you, in a nix environment you don't have access to libraries and
programs unless you install them with nix-env. However installing
libraries with nix-env is not good practice. We prefer to have isolated
environments for development.
Recall that in a nix environment, we don't have access to libraries or
programs unless they have been installed with <command>nix-env</command>.
However, installing libraries with <command>nix-env</command> is not
good practice. We prefer to have isolated environments for development, which
<command>nix-shell</command> provides for us.
</para>
<para>
We can call <command>nix-shell</command> on any Nix expression which
returns a derivation, but the resulting <code>bash</code> shell's
<code>PATH</code> does not have the utilities we want:
</para>
<screen><xi:include href="./10/nix-shell-hello.txt" parse="text" /></screen>
<para>
First thing to notice, we call <command>nix-shell</command> on a nix
expression which returns a derivation. We then enter a new bash shell,
but it's really useless. We expected to have the GNU hello world build
inputs available in PATH, including GNU make, but it's not the case.
This shell is rather useless. It would be reasonable to expect that the GNU
<code>hello</code> build inputs are available in <code>PATH</code>, including
GNU <code>make</code>, but this is not the case.
</para>
<para>
But, we have the environment variables that we set in the derivation,
However, we do have the environment variables that we set in the derivation,
like <code>$baseInputs</code>, <code>$buildInputs</code>,
<code>$src</code> and so on.
<code>$src</code>, and so on.
</para>
<para>
That means we can source our <filename>builder.sh</filename>, and it will
build the derivation. You may get an error in the installation phase,
because the user may not have the permission to write to
<filename>/nix/store</filename>:
This means the we can <command>source</command> our
<filename>builder.sh</filename>, and it will build the derivation.
You may get an error in the installation phase, because your user may
not have the permission to write to <filename>/nix/store</filename>:
</para>
<screen><xi:include href="./10/source-builder.txt" parse="text" /></screen>
<para>
It didn't install, but it built. Things to notice:
The derivation didn't install, but it did build. Note the following:
</para>
<itemizedlist>
<listitem>
<para>
We sourced builder.sh, therefore it ran all the steps including
setting up the PATH for us.
We sourced <filename>builder.sh</filename> and it ran all of the build
steps, including setting up the <code>PATH</code> for us.
</para>
</listitem>
<listitem>
<para>
The working directory is no more a temp directory created by nix-build, but the current directory. Therefore, hello-2.10 has been unpacked there.
The working directory is no longer a temp directory created by
<command>nix-build</command>, but is instead the directory in which
we entered the shell. Therefore, <filename>hello-2.10</filename> has
been unpacked in the current directory.
</para>
</listitem>
</itemizedlist>
<para>
We're able to <command>cd</command> into hello-2.10 and type
<command>make</command>, because now it's available.
We are able to <command>cd</command> into <filename>hello-2.10</filename> and type
<command>make</command>, because <command>make</command> is now available.
</para>
<para>
In other words, <command>nix-shell</command> drops us in a shell with the
same (or almost) environment used to run the builder!
The take-away is that <command>nix-shell</command> drops us in a shell with the
same (or very similar) environment used to run the builder.
</para>
</section>
@ -95,44 +111,50 @@
<title>A builder for nix-shell</title>
<para>
The previous steps are a bit annoying of course, but we can improve our
builder to be more nix-shell friendly.
The previous steps require some manual commands to be run and are not
optimized for a workflow centered on <command>nix-shell</command>. We
will now improve our builder to be more <command>nix-shell</command> friendly.
</para>
<para>
First of all, we were able to source <filename>builder.sh</filename>
because it was in our current directory, but that's not nice. We want the
<filename>builder.sh</filename> that is stored in the nix store, the one
that would be used by <command>nix-build</command>. To do so, the right
way is to pass the usual environment variable through the derivation.
There are a few things that we would like to change.
</para>
<para>
<emphasis role="underlined">Note</emphasis>: <code>$builder</code> is
already defined, but it's the bash executable, not our
First, when we <command>source</command>d the <filename>builder.sh</filename>
file, we obtained the file in the currenty directory. What we really wanted
was the <filename>builder.sh</filename> that is stored in the nix store,
as this is the file that would be used by <command>nix-build</command>.
To achieve this, the correct technique is to pass an environment variable
through the derivation. (Note that <code>$builder</code> is
already defined, but it points to the bash executable rather than our
<filename>builder.sh</filename>. Our <filename>builder.sh</filename> is
an argument to bash.
passed as an argument to bash.)
</para>
<para>
Second, we don't want to run the whole builder, we only want it to setup
the necessary environment for manually building the project. So we'll
write two files, one for setting up the environment, and the real
<filename>builder.sh</filename> that runs with
<command>nix-build</command>.
Second, we don't want to run the whole builder: we only want to setup
the necessary environment for manually building the project. Thus, we
can break <filename>builder.sh</filename> into two files: a
<filename>setup.sh</filename> for setting up the environment, and
the real <filename>builder.sh</filename> that <command>nix-build</command>
expects.
</para>
<para>
Additionally, we'll wrap the phases in functions, it may be useful, and
move the <code>set -e</code> to the builder instead of the setup. The
<code>set -e</code> is annoying in <command>nix-shell</command>.
During our refactoring, we ill wrap the build phases in functions to
give more structure to our design. Additionally, we'll move the
<code>set -e</code> to the builder file instead of the setup file.
The <code>set -e</code> is annoying in <command>nix-shell</command>,
as it will terminate the shell if an error is encountered (such as
a mistyped command.)
</para>
<para>
Here is our modified <filename>autotools.nix</filename>.
Noteworthy is the <code>setup = ./setup.sh;</code> attribute in the
derivation, which adds <filename>setup.sh</filename> to the nix store and
as usual, adds a <code>$setup</code> environment variable in the builder.
correspondingly adds a <code>$setup</code> environment variable in the builder.
</para>
<programlisting><xi:include href="./10/autotools-nix.txt" parse="text" /></programlisting>
@ -140,25 +162,25 @@
<para>
Thanks to that, we can split <filename>builder.sh</filename> into
<filename>setup.sh</filename> and <filename>builder.sh</filename>. What
<filename>builder.sh</filename> does is sourcing <code>$setup</code> and
calling the <code>genericBuild</code> function. Everything else is just
some bash changes.
<filename>builder.sh</filename> does is <command>source</command>
<code>$setup</code> and call the <code>genericBuild</code> function.
Everything else is just some changes to the bash script.
</para>
<para>
Here is the modified <filename>builder.sh</filename>.
Here is the modified <filename>builder.sh</filename>:
</para>
<programlisting><xi:include href="./10/builder-sh.txt" parse="text" /></programlisting>
<para>
Here is the newly added <filename>setup.sh</filename>.
Here is the newly added <filename>setup.sh</filename>:
</para>
<programlisting><xi:include href="./10/setup-sh.txt" parse="text" /></programlisting>
<para>
Finally, here is <filename>hello.nix</filename>.
Finally, here is <filename>hello.nix</filename>:
</para>
<programlisting><xi:include href="./10/hello-nix.txt" parse="text" /></programlisting>
@ -170,17 +192,18 @@
<screen><xi:include href="./10/nix-shell-source.txt" parse="text" /></screen>
<para>
Now you can run, for example, <code>unpackPhase</code> which unpacks
Now, for example, you can run <code>unpackPhase</code> which unpacks
<code>$src</code> and enters the directory. And you can run commands
like <command>./configure</command>, <command>make</command> etc.
like <command>./configure</command>, <command>make</command>, and so forth
manually, or run phases with their respective functions.
</para>
<para>
It's that straightforward, <command>nix-shell</command> builds the .drv file
and its input dependencies, then drops into a shell by setting up the
environment variables necessary to build the .drv, in particular those
passed to the derivation function.
The process is that straightforward. <command>nix-shell</command> builds the
<code>.drv</code> file and its input dependencies, then drops into a shell
by setting up the environment variables necessary to build the <code>.drv</code>.
In particular, the environment variables in the shell match those passed
to the <code>derivation</code> function.
</para>
</section>
@ -188,13 +211,14 @@
<title>Conclusion</title>
<para>
With <command>nix-shell</command> we're able to drop into an isolated
environment for developing a project, with the necessary dependencies
just like <command>nix-build</command> does. Additionally, we can build and
debug the project manually, step by step like you would do in any other
operating system. Note that we never installed <command>gcc</command>,
<command>make</command>, etc. system-wide. These tools and libraries are
available per-build.
With <command>nix-shell</command> we are able to drop into an isolated
environment suitable for developing a project. This environment provides the necessary
dependencies for the development shell, similar to how
<command>nix-build</command> providesthe necessary dependencies to a builder.
Additionally, we can build and debug the project manually, executing step-by-step
like we would in any other operating system. Note that we never installed tools
such <command>gcc</command> or <command>make</command> system-wide; these tools
and libraries are isolated and available per-build.
</para>
</section>
@ -202,9 +226,9 @@
<title>Next pill</title>
<para>
...we will clean up the nix store. We wrote and built derivations, added
stuff to nix store, but until now we never worried about cleaning up the
used space in the store. It's time to collect some garbage.
In the next pill, we will clean up the nix store. We have written and built
derivations which add to the nix store, but until now we haven't worried
about cleaning up the used space in the store.
</para>
</section>
</chapter>

289
pills/11-garbage-collector.xml
View file

@ -4,84 +4,70 @@
version="5.0"
xml:id="garbage-collector">
<title>Garbage Collector</title>
<title>The Garbage Collector</title>
<para>
Welcome to the 11th Nix pill. In the previous
<link linkend="developing-with-nix-shell">10th pill</link> we managed to
obtain a self-contained development environment for a project. The concept
is that <command>nix-build</command> is able to build a derivation
in isolation, while <command>nix-shell</command> is able to drop us in a
shell with (almost) the same environment used by <command>nix-build</command>.
This allows us to debug, modify and manually build software.
<link linkend="developing-with-nix-shell">10th pill</link>, we drew a
a parallel between the isolated build environment provided by
<command>nix-build</command> and the isolated development shell provided by
<command>nix-shell</command>. Using <command>nix-shell</command> allowed us
to debug, modify, and manually build software using an environment that
is almost identical to the one provided by <command>nix-build</command>.
</para>
<para>
Today we stop packaging and look at a mandatory nix component, the garbage
collector. When using nix tools, often derivations are built. This include
both .drv files and out paths. These artifacts go in the nix store, but
we've never cared about deleting them until now.
Today, we will stop focusing on packaging and instead look at a critical
component of Nix: the garbage collector. When we use Nix tools, we are
often building derivations. This includes <code>.drv</code> files as well as
out paths. These artifacts go in the Nix store and take up space in our storage.
Eventually we may wish to free up some space by removing derivations we no longer
need. This is the focus of the 11th pill.
By default, Nix takes a relatively conservative approach when automatically
deciding which derivations are "needed". In this pill, we will also see
a technique to conduct more destructive upgrade and deletion operations.
</para>
<section>
<title>How does it work</title>
<title>How does garbage collection work?</title>
<para>
Other package managers, like <command>dpkg</command>, have ways of
removing unused software. Nix is much more precise in its garbage
collection compared to these other systems.
Programming languages with garbage collectors use the concept of a set of
"garbage collector (or 'GC') roots" to keep track of "live" objects.
A GC root is an object that is always considered "live" (unless explicitly
removed as GC root). The garbage collection process starts from the GC roots
and proceeds by recursively marking object references as "live". All other
objects can be collected and deleted.
</para>
<para>
I bet with <command>dpkg</command>, <command>rpm</command> or similar
traditional packaging systems, you end up having some unnecessary
packages installed or dangling files. With nix this does not happen.
</para>
<para>
How do we determine whether a store path is still needed? The same way
programming languages with a garbage collector decide whether an object
is still alive.
Instead of objects, Nix's garbage collection operates on store paths, <link xlink:href="https://nixos.org/manual/nix/stable/package-management/garbage-collector-roots.html">with the GC roots themselves being store paths</link>.
. This approach is much mode principled than traditional package
managers such as <code>dpkg</code> or <code>rpm</code>, which may
leave around unused packages or dangling files.
</para>
<para>
Programming languages with a garbage collector have an important concept
in order to keep track of live objects: GC roots. A GC root is an object
that is always alive (unless explicitly removed as GC root). All objects
recursively referred to by a GC root are live.
The implementation is very simple and transparent to the user. The primary
GC roots are stored under <filename>/nix/var/nix/gcroots</filename>. If there
is a symlink to a store path, then linked store path is a GC root.
</para>
<para>
Therefore, the garbage collection process starts from GC roots, and
recursively mark referenced objects as live. All other objects can be
collected and deleted.
Nix allows this directory to have subdirectories: it will simply recursively
traverse the subdirectories in search of symlinks to store paths. When
a symlink is encountered, it's target is added to the list of live store
paths.
</para>
<para>
In Nix there's this same concept. Instead of being objects, of course,
<link xlink:href="https://nixos.org/manual/nix/stable/package-management/garbage-collector-roots.html">GC roots are store paths</link>.
The implementation is very simple and transparent to the user. GC roots
are stored under <filename>/nix/var/nix/gcroots</filename>. If there's a
symlink to a store path, then that store path is a GC root.
</para>
<para>
Nix allows this directory to have subdirectories: it will simply recurse
directories in search of symlinks to store paths.
</para>
<para>
So we have a list of GC roots. At this point, deleting dead store paths
is as easy as you can imagine. We have the list of all live store paths,
hence the rest of the store paths are dead.
</para>
<para>
In particular, Nix first moves dead store paths to
<filename>/nix/store/trash</filename> which is an atomic operation.
Afterwards, the trash is emptied.
In summary, Nix maintains a list of GC roots. These roots can then be
used to compute a list of all live store paths. Any other store
paths are considered dead. Deleting these paths is now straightforward.
Nix first moves dead store paths to <filename>/nix/store/trash</filename>,
which is an atomic operation. Afterwards, the trash is emptied.
</para>
</section>
@ -89,50 +75,45 @@
<title>Playing with the GC</title>
<para>
Before playing with the GC, first run the
Before we begin we first run the
<link xlink:href="https://nixos.org/manual/nix/stable/command-ref/nix-collect-garbage.html">nix garbage collector</link>
once, so that we have a clean playground for our experiments:
so that we have a clean setup for our experiments:
</para>
<screen><xi:include href="./11/nix-collect-garbage.txt" parse="text" /></screen>
<para>
Perfect, if you run it again it won't find anything new to delete, as
expected.
If we run the garbage collector again it won't find anything new to delete,
as we expect. After running the garbage collector, the nix store only contains
paths with references from the GC roots.
</para>
<para>
What's left in the nix store is everything being referenced from the GC
roots.
</para>
<para>
Let's install for a moment bsd-games:
We now install a new program, <code>bsd-games</code>, inspect its
store path, and examine it's GC root. The <command>nix-store -q --roots</command>
command is used to query the GC roots that refer to a given derivation. In this
case, our current user environment refers to <code>bsd-games</code>:
</para>
<screen><xi:include href="./11/install-bsd-games.txt" parse="text" /></screen>
<para>
The nix-store command can be used to query the GC roots that refer to a
given derivation. In this case, our current user environment does refer
to bsd-games.
</para>
<para>
Now remove it, collect garbage and note that bsd-games is still in the nix
store:
Now we remove it and run the garbage collector, and note that <code>bsd-games</code>
is still in the nix store:
</para>
<screen><xi:include href="./11/remove-bsd-games.txt" parse="text" /></screen>
<para>
This is because the old generation is still in the nix store because it's a GC root. As we'll see below, all profiles and their generations are GC roots.
The old generation is still in the nix store because it is a GC root.
As we will see below, all profiles and their generations are automatically
GC roots.
</para>
<para>
Removing a GC root is simple. Let's try deleting the generation that
refers to bsd-games, collect garbage, and note that now bsd-games is no
longer in the nix store:
Removing a GC root is simple. In our case, we delete the generation that
refers to <code>bsd-games</code>, run the garbage collector, and note
that <code>bsd-games</code> is no longer in the nix store:
</para>
<screen><xi:include href="./11/remove-gen-9.txt" parse="text" /></screen>
@ -145,55 +126,55 @@
</para>
<para>
However we removed the link from
Note that we removed the link from
<filename>/nix/var/nix/profiles</filename>, not from
<filename>/nix/var/nix/gcroots</filename>. Turns out, Nix
also treats <filename>/nix/var/nix/profiles</filename> as a GC root.
That is very handy. It means any profile and its generations are GC roots.
There are other paths that are taken into account as well, for example <filename>/run/booted-system</filename> on NixOS.
The command <command>nix-store --gc --print-roots</command> prints all the paths considered before performing a GC.
<filename>/nix/var/nix/gcroots</filename>. In addition to the latter,
Nix treats <filename>/nix/var/nix/profiles</filename> as a GC root.
This is useful because it means that any profile and its generations
are GC roots. Other path are considered GC roots as well; for example,
<filename>/run/booted-system</filename> on NixOS.
The command <command>nix-store --gc --print-roots</command> prints all
paths considered as GC roots when running the garbage collector.
</para>
<para>
It's as simple as that, anything under
<filename>/nix/var/nix/gcroots</filename> is a GC root. And anything not
being garbage collected is because it's referred from one of the GC roots.
</para>
</section>
<section>
<title>Indirect roots</title>
<para>
Remember that building the GNU hello world package with
Recall that building the GNU <code>hello</code> package with
<command>nix-build</command> produces a <filename>result</filename>
symlink in the current directory. Despite the collected garbage done
above, the <command>hello</command> program is still working: therefore
it has not been garbage collected. Clearly, since there's no other
derivation that depends upon the GNU hello world package, it must be a
symlink in the current directory. Despite the garbage collection done
above, the <command>hello</command> program is still working. Therefore,
it has not been garbage collected. Since there is no other
derivation that depends upon the GNU <code>hello</code> package, it must be a
GC root.
</para>
<para>
In fact, <command>nix-build</command> automatically adds the result
symlink as a GC root. Yes, not the built derivation, but the symlink.
These GC roots are added under
<filename>/nix/var/nix/gcroots/auto</filename>.
In fact, <command>nix-build</command> automatically adds the
<filename>result</filename> symlink as a GC root. Note that this
is not the built derivation, but the symlink itself. These GC roots
are added under <filename>/nix/var/nix/gcroots/auto</filename>.
</para>
<screen><xi:include href="./11/ls-gcroots-auto.txt" parse="text" /></screen>
<para>
Don't care about the name of the symlink. What's important is that a
symlink exists that point to <filename>/home/nix/result</filename>. This
is called an <emphasis role="bold">indirect GC root</emphasis>. That is,
the GC root is effectively specified outside of
<filename>/nix/var/nix/gcroots</filename>. Whatever
<filename>result</filename> points to, it will not be garbage collected.
The name of the GC root symlink is not important to us at this time.
What is important is that such a symlink exists and points to
<filename>/home/nix/result</filename>. This is called an
<emphasis role="bold">indirect GC root</emphasis>. A GC root is
considred indirect if it's specification is outside of
<filename>/nix/var/nix/gcroots</filename>. In this case, this means
that the target of the <filename>result</filename> symlink will
not be garbage collected.
</para>
<para>
How do we remove the derivation then? There are two possibilities:
To remove a deriviation considered "live" by an indirect GC root,
there are two possibilities:
</para>
<itemizedlist>
@ -211,15 +192,15 @@
</itemizedlist>
<para>
In the first case, the derivation will be deleted from the nix store, and
<filename>result</filename> becomes a dangling symlink. In the second
case, the derivation is removed as well as the indirect root in
In the first case, the derivation will be deleted from the nix store during
garbage collection, and <filename>result</filename> becomes a dangling symlink.
In the second case, the derivation is removed as well as the indirect root in
<filename>/nix/var/nix/gcroots/auto</filename>.
</para>
<para>
Running <command>nix-collect-garbage</command> after deleting the GC root
or the indirect GC root, will remove the derivation from the store.
or the indirect GC root will remove the derivation from the store.
</para>
</section>
@ -227,49 +208,64 @@
<title>Cleanup everything</title>
<para>
What's the main source of software duplication in the nix store? Clearly,
GC roots due to <command>nix-build</command> and profile generations.
Doing a <command>nix-build</command> results in a GC root for a build
that somehow will refer to a specific version of <package>glibc</package>,
and other libraries. After an upgrade, if that build is not deleted by
the user, it will not be garbage collected. Thus the old dependencies
referred to by the build will not be deleted either.
The main source of software duplication in the nix store comes from
GC roots, due to <command>nix-build</command> and profile generations.
Running <command>nix-build</command> results in a GC root for the build
that refers to a specific version of specific libaries, such as
<package>glibc</package>. After an upgrade, we must delete the previous build
if we want the garbage collector to remove the corresponding derivation,
as well as if we want old dependencies cleaned up.
</para>
<para>
Same goes for profiles. Manipulating the <command>nix-env</command>
The same holds for profiles. Manipulating the <command>nix-env</command>
profile will create further generations. Old generations refer to old
software, thus increasing duplication in the nix store after an upgrade.
</para>
<para>
What are the basic steps for upgrading and removing everything old,
including old generations? In other words, do an upgrade similar to other
systems, where they forget everything about the older state:
Other systems typically "forget" everything about their previous state after
an upgrade. With Nix, we can perform this type of upgrade (having Nix remove
all old derivations, including old generations), but we do so manually.
There are four steps to doing this:
<itemizedlist>
<listitem>
<para>
First, we download a new version of the nixpkgs channel, which holds the
description of all the software. This is done via
<command>nix-channel --update</command>.
</para>
</listitem>
<listitem>
<para>
Then we upgrade our installed packages with <command>nix-env -u</command>.
This will bring us into a new generation with updated software.
</para>
</listitem>
<listitem>
<para>
Then we remove all the indirect roots generated by
<command>nix-build</command>: beware, as this will result in dangling
symlinks. A smarter strategy would also remove the target of those symlinks.
</para>
</listitem>
<listitem>
<para>
Finally, the <command>-d</command> option of
<command>nix-collect-garbage</command> is used to delete old generations
of all profiles, then collect garbage. After this, you lose the ability
to rollback to any previous generation. It is important to ensure the new
generation is working well before running this command.
</para>
</listitem>
</itemizedlist>
The four steps are shown below:
</para>
<screen><xi:include href="./11/channel-update.txt" parse="text" /></screen>
<para>
First, we download a new version of the nixpkgs channel, which holds the
description of all the software. Then we upgrade our installed packages
with <command>nix-env -u</command>. That will bring us into a fresh new
generation with all updated software.
</para>
<para>
Then we remove all the indirect roots generated by
<command>nix-build</command>: beware, this will result in dangling
symlinks. You may be smarter and also remove the target of those symlinks.
</para>
<para>
Finally, the <command>-d</command> option of
<command>nix-collect-garbage</command> is used to delete old generations
of all profiles, then collect garbage. After this, you lose the ability
to rollback to any previous generation. So make sure the new generation
is working well before running the command.
</para>
</section>
<section>
@ -277,15 +273,10 @@
<para>
Garbage collection in Nix is a powerful mechanism to cleanup your system.
The nix-store commands allow us to know why a certain derivation is in
the nix store.
</para>
<para>
Cleaning up everything down to the oldest bit of software after an
upgrade seems a bit contrived, but that's the price of having multiple
generations, multiple profiles, multiple versions of software, thus
rollbacks etc.. The price of having many possibilities.
The <command>nix-store</command> commands allow us to know why a certain
derivation is present in the nix store, and whether or not it is eligible
for garbage collection. We also saw how to conduct more destructive deletion
and upgrade operations.
</para>
</section>
@ -293,8 +284,8 @@
<title>Next pill</title>
<para>
...we will package another project and introduce what I call the "inputs"
design pattern. We've only played with a single derivation until now,
In the next pill, we will package another project and introduce the "inputs"
design pattern. We've only played with a single derivation until now;
however we'd like to start organizing a small repository of software. The
"inputs" pattern is widely used in nixpkgs; it allows us to decouple
derivations from the repository itself and increase customization

322
pills/12-inputs-design-pattern.xml
View file

@ -4,236 +4,358 @@
version="5.0"
xml:id="inputs-design-pattern">
<title>Inputs Design Pattern</title>
<title>Package Repositories and the Inputs Design Pattern</title>
<para>
Welcome to the 12th Nix pill. In the previous <link linkend="garbage-collector">11th pill</link> we stopped packaging and cleaned up the system with the garbage collector.
Welcome to the 12th Nix pill. In the previous <link linkend="garbage-collector">11th
pill</link>, we stopped packaging and cleaned up the system with the garbage collector.
</para>
<para>
We'll be resuming packaging, and improving different aspects of it. We've only packaged a hello world program so far, but what if we want to create a repository of multiple packages?
This time, we will resume packaging and improve different aspects of it. We will also
demonstrate how to create a repository of multiple packages.
</para>
<section>
<title>Repositories in Nix</title>
<para>
Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. It's more like a consequence due to the need of organizing packages.
Package repositories in Nix arose naturally from the need to organize pacakges.
There is no preset directory structure or packaging policy prescribed by Nix itself;
Nix, as a full, functional programming language, is powerful enough to support
multiple different repository formats.
</para>
<para>
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.
</para>
<para>
There is no preset directory structure or preset packaging policy. It's all about you and Nix.
</para>
<para>
The <literal>nixpkgs</literal> repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.
</para>
<para>
Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.
Over time, the <literal>nixpkgs</literal> repository evolved a particular
structure. This structure reflects the history of Nix as well as the design
patterns adopted by it's users as useful tools in building and organizing
packages. Below, we will examine some of these patterns in detail.
</para>
</section>
<section>
<title>The single repository pattern</title>
<para>
Before introducing the "<literal>inputs</literal>" pattern, we can start talking about another pattern first which I'd like to call "<literal>single repository</literal>" pattern.
Different operating system distributions have different opinions about how
package repositories should organized. Systems like Debian scatter packages
in several small repositories (which tends to make tracking interdependent
changes more difficult, and hinders contributions to the repositories),
while systems like Gentoo put all package descriptions in a single repository.
</para>
<para>
Systems like Debian scatter packages in several small repositories. This can make it hard to track interdependent changes and to contribute to new packages.
Nix follows the "single repository" pattern by placing all descriptions of all
packages into <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>.
This approach has proven natural and attractive for new contributions.
</para>
<para>
Alternatively, systems like Gentoo put package descriptions all in a single repository.
For the rest of this pill, we will adopt the single repository pattern. The
natural implementation in Nix is to create a top-level Nix expression, followed
by one expression for each package. The top-level expression imports and combines
all package expressions in an attribute set mapping names to packages.
</para>
<para>
The nix reference for packages is <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.
</para>
<para>
For the rest of this chapter, we will adopt the single repository technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs.
</para>
<para>
But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why <literal>nixpkgs</literal> is able to maintain such a big software repository in a giant attribute set.
In some programming languages, such an approach -- including every possible
package description in a single data structure -- would be untenable due
to the language needing to load the entire data structure into memory before
operating on it. Nix, however, is a lazy language and only evaluates what is
needed.
</para>
</section>
<section>
<title>Packaging graphviz</title>
<title>Packaging <code>graphviz</code></title>
<para>
We have packaged <package>GNU hello world</package>, imagine you would like to package something else for creating at least a repository of two projects :-) . I chose <package>graphviz</package>, which uses the standard autotools build system, requires no patching and dependencies are optional.
We have already packaged GNU <code>hello</code>. Next, we will package a
graph-drawing program called <code>graphviz</code> so that we can
create a repository containing multiple packages. The <code>graphviz</code>
package was selected because it uses the standard autotools build system and
requires no patching. It also has optional dependencies, which will give us
an opportunity to illustrate a technique to configure builds to a particular
situation.
</para>
<para>
Download <package>graphviz</package> from <link xlink:href="https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.3/graphviz-2.49.3.tar.gz">here</link>. The <filename>graphviz.nix</filename> expression is straightforward:
First, we download <code>graphviz</code> from <link xlink:href="https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.3/graphviz-2.49.3.tar.gz">gitlab</link>. The <filename>graphviz.nix</filename> expression is straightforward:
</para>
<screen><xi:include href="./12/graphviz-derivation.txt" parse="text" /></screen>
<para>
Build with <command>nix-build graphviz.nix</command> and you will get runnable binaries under <filename>result/bin</filename>. Notice how we reused the same <filename>autotools.nix</filename> of <filename>hello.nix.</filename> Let's create a simple png:
If we build the project with <command>nix-build graphviz.nix</command>, we will get runnable binaries under <filename>result/bin</filename>. Notice how we reused the same <filename>autotools.nix</filename> of <filename>hello.nix.</filename>
</para>
<para>
By default, <code>graphviz</code> does not compile with the ability to produce
<code>png</code> files. Thus, the derivation above will build a binary
supporting only the native output formats, as we see below:
</para>
<screen><xi:include href="./12/simple-png.txt" parse="text" /></screen>
<para>
Oh of course... <package>graphviz</package> doesn't know about png. It built only the output formats it supports natively, without using any extra library.
If we want to produce a <code>png</code> file with <code>graphviz</code>, we
must add it to our derivation. The place to do so is
in <filename>autotools.nix</filename>, where we created a
<literal>buildInputs</literal> variable that gets concatenated to
<literal>baseInputs</literal>. This is the exact reason for this variable: to
allow users of <filename>autotools.nix</filename> to add additional inputs
from package expressions.
</para>
<para>
Remember, in <filename>autotools.nix</filename> there's a <literal>buildInputs</literal> variable which gets concatenated to <literal>baseInputs</literal>. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.
</para>
<para>
This 2.49 version of <package>graphviz</package> has several plugins to output png. For simplicity, we will use <package>libgd</package>.
Version 2.49 of <code>graphviz</code> has several plugins to output
<code>png</code>. For simplicity, we will use <code>libgd</code>.
</para>
</section>
<section>
<title>Digression about gcc and ld wrappers</title>
<title>Passing library information to <command>pkg-config</command> via environment
variables</title>
<para>
The <package>graphviz</package> configure script uses <command>pkg-config</command> to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell <command>pkg-config</command> where to find pkg-config description files, in order to enable it to tell the configure script where to find headers and libraries.
The <code>graphviz</code> configuration script uses <command>pkg-config</command>
to specify which flags are passed to the compiler. Since there is no global location
for libraries, we need to tell <command>pkg-config</command> where to find
it's description files, which tell the configuration script where to find
headers and libraries.
</para>
<para>
In classic POSIX systems, <command>pkg-config</command> just finds the
<filename>.pc</filename> files of all installed libraries in system folders
like <filename>/usr/lib/pkgconfig</filename>, but we don't have that in
the nix sandbox. The nix way to educate <command>pkg-config</command>
about the existence of libraries is the environment variable
<varname>PKG_CONFIG_PATH</varname>.
like <filename>/usr/lib/pkgconfig</filename>. However, these files
are not present in the isolated environments presented to Nix.
</para>
<para>
What can we do about it? We can employ the same trick we did for <varname>PATH</varname>: automatically filling the variables from <literal>buildInputs</literal>. This is the relevant snippet of <filename>setup.sh</filename>:
As an alternative, we can inform <command>pkg-config</command> about
the location of libraries via the <varname>PKG_CONFIG_PATH</varname>
environment variable. We can populate this environment variable
using the same trick we used for <varname>PATH</varname>:
automatically filling the variables from <literal>buildInputs</literal>.
This is the relevant snippet of <filename>setup.sh</filename>:
</para>
<screen><xi:include href="./12/setup-sh.txt" parse="text" /></screen>
<para>
Now adding derivations to <literal>buildInputs</literal> will add their <filename>lib/pkgconfig</filename> and <filename>bin</filename> paths automatically in <filename>setup.sh</filename>.
Now if we add derivations to <literal>buildInputs</literal>, their
<filename>lib/pkgconfig</filename> and <filename>bin</filename> paths
are automatically added in <filename>setup.sh</filename>.
</para>
</section>
<section>
<title>Completing graphviz with gd</title>
<title>Completing graphviz with <code>gd</code></title>
<para>
Finish the expression for <package>graphviz</package> with <package>gd</package> support (note the use of the <literal>with</literal> expression in <literal>buildInputs</literal> to avoid repeating <literal>pkgs</literal>):
Below, we finish the expression for <code>graphviz</code> with <code>gd</code> support.
Note the use of the <literal>with</literal> expression in <literal>buildInputs</literal> to avoid repeating <literal>pkgs</literal>:
</para>
<screen><xi:include href="./12/graphviz-gd-derivation.txt" parse="text" /></screen>
<para>
We add <command>pkg-config</command> to the derivation to make this tool
available for the configure script. As <package>gd</package> is a package
available for the configure script. As <code>gd</code> is a package
with <link xlink:href="https://nixos.org/manual/nixpkgs/stable/#sec-multiple-outputs-">split outputs</link>,
we need to add both the library- and development outputs.
we need to add both the library and development outputs.
</para>
<para>
Now you can create the png!
After building, <code>graphviz</code> can now create <code>png</code>s.
</para>
</section>
<section>
<title>The repository expression</title>
<para>
Now that we have two packages, what's a good way to put them together in a single repository? We'll do something like <literal>nixpkgs</literal> does. With <literal>nixpkgs</literal>, we <literal>import</literal> it and then we pick derivations by accessing the giant attribute set.
Now that we have two packages, we want to combine them into a single repository.
To do so, we'll mimic what <literal>nixpkgs</literal> does: we will create
a single attribute set containing derivations. This attribute set can
then be imported, and deriviations can be selected by accessing the
top-level attribute set.
</para>
<para>
For us nixers, this is a good technique, because it abstracts from the file names. We don't refer to a package by <filename>REPO/some/sub/dir/package.nix</filename> but by <literal>importedRepo.package</literal> (or <literal>pkgs.package</literal> in our examples).
Using this technique we are able to abstract from the file names.
Instead of referring to a package by <filename>REPO/some/sub/dir/package.nix</filename>,
this technique allows us to select a derivation as
<literal>importedRepo.package</literal> (or <literal>pkgs.package</literal>
in our examples).
</para>
<para>
Create a default.nix in the current directory:
To begin, create a default.nix in the current directory:
</para>
<screen><xi:include href="./12/repository.txt" parse="text" /></screen>
<para>
Ready to use! Try it with <command>nix repl</command>:
This file is ready to use with <command>nix repl</command>:
</para>
<screen><xi:include href="./12/repository-test-nix-repl.txt" parse="text" /></screen>
<para>
With <command>nix-build</command>:
With <command>nix-build</command>, we can pass the <arg>-A</arg> option to
access an attribute of the set from the given <filename>.nix</filename> expression:
</para>
<screen><xi:include href="./12/repository-test-nix-build.txt" parse="text" /></screen>
<para>
The <arg>-A</arg> argument is used to access an attribute of the set from the given .nix expression.
The <filename>default.nix</filename> file is special. When a directory
contains a <filename>default.nix</filename> file, it is used as the implict
nix expression of the directory. This, for example, allows use to run
<command>nix-build -A hello</command> without specifying
<filename>default.nix</filename> explicitly.
</para>
<para>
<emphasis role="bold">Important:</emphasis> why did we choose the <filename>default.nix</filename>? Because when a directory (by default the current directory) has a <filename>default.nix</filename>, that <filename>default.nix</filename> will be used (see <literal>import</literal> <link xlink:href="https://nixos.org/manual/nix/stable/expressions/builtins.html">here</link>). In fact you can run <command>nix-build -A hello</command> without specifying <filename>default.nix</filename>.
</para>
<para>
For pythoners, it is similar to <filename>__init__.py</filename>.
</para>
<para>
With <command>nix-env</command>, install the package into your user environment:
We can now use <command>nix-env</command> to install the package into our
user environment:
</para>
<screen><xi:include href="./12/nix-env-install-graphviz.txt" parse="text" /></screen>
<para>
The <arg>-f</arg> option is used to specify the expression to use, in this case the current directory, therefore <filename>./default.nix</filename>.
Taking a closer look at the above command, we see the following options:
<itemizedlist>
<listitem><para>
The <arg>-f</arg> option is used to specify the expression to use. In this case,
the expression is the <filename>./default.nix</filename> of the current directory.
</para></listitem>
<listitem><para>
The <arg>-i</arg> option stands for "installation".
</para></listitem>
<listitem><para>
The <arg>-A</arg> is the same as above for <command>nix-build</command>.
</para></listitem>
</itemizedlist>
</para>
<para>
The <arg>-i</arg> stands for installation.
</para>
<para>
The <arg>-A</arg> is the same as above for <command>nix-build</command>.
</para>
<para>
We reproduced the very basic behavior of <literal>nixpkgs</literal>.
We reproduced the very basic behavior of <literal>nixpkgs</literal>: combining
multiple derivations into a single, top-level attribute set.
</para>
</section>
<section>
<title>The inputs pattern</title>
<para>
After a long preparation, we finally arrived. I know you're having a big doubt in this moment. It's about the <filename>hello.nix</filename> and <filename>graphviz.nix</filename>. They are very, very dependent on <literal>nixpkgs</literal>:
The approach we've taken so far has a few problems:
<itemizedlist>
<listitem><para>First big problem: they import <literal>nixpkgs</literal> directly. In <filename>autotools.nix</filename> instead we pass <literal>nixpkgs</literal> as an argument. That's a much better approach.</para></listitem>
<listitem><para>Second problem: what if we want a variant of <package>graphviz</package> without <package>libgd</package> support?</para></listitem>
<listitem><para>Third problem: what if we want to test <package>graphviz</package> with a particular <package>libgd</package> version?</para></listitem>
<listitem><para>
First, <filename>hello.nix</filename> and <filename>graphviz.nix</filename> are
dependent on <literal>nixpkgs</literal>, which they import directly.
A better approach would be to pass in <literal>nixpkgs</literal> as an argument,
as we did in <filename>autotools.nix</filename>.
</para></listitem>
<listitem><para>
Second, we don't have a straightforward way to compile different variants
of the same software, such as <code>graphviz</code> with or without
<code>libgd</code> support.
</para></listitem>
<listitem><para>
Third, we don't have a way to test <code>graphviz</code>
with a particular <code>libgd</code> version.
</para></listitem>
</itemizedlist>
</para>
<para>
The current answers to the above questions are: change the expression to match your needs (or change the callee to match your needs).
Until now, our approach to addressing the above problems has been inadequate
and required changing the nix expression to match our needs. With the
<literal>inputs</literal> pattern, we provide another answer: let the user
change the <literal>inputs</literal> of the expression.
</para>
<para>
With the <literal>inputs</literal> pattern, we decided to provide another answer: let the user change the <literal>inputs</literal> of the expression (or change the caller to pass different inputs).
</para>
<para>
By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:
When we talk about "the inputs of an expression", we are referring to the
set of derivations needed to build that expression. In this case:
<itemizedlist>
<listitem><para><literal>mkDerivation</literal> from <package>autotools</package>. Recall that <literal>mkDerivation</literal> has an implicit dependency on the toolchain.</para></listitem>
<listitem><para><package>libgd</package> and its dependencies.</para></listitem>
<listitem><para>
<literal>mkDerivation</literal> from <code>autotools</code>. Recall
that <literal>mkDerivation</literal> has an implicit dependency on
the toolchain.
</para></listitem>
<listitem><para>
<code>libgd</code> and its dependencies.
</para></listitem>
</itemizedlist>
</para>
<para>
The src is also an input but it's pointless to change the source from the caller. For version bumps, in <literal>nixpkgs</literal> we prefer to write another expression (e.g. because patches are needed or different inputs are needed).
The <filename>./src</filename> directory is also an input,
but we wouldn't change the source from the caller.
In <literal>nixpkgs</literal> we prefer to write another expression
for version bumps (e.g. because patches or different inputs are needed).
</para>
<para>
<emphasis role="underline">Goal:</emphasis> make package expressions independent of the repository.
</para>
<para>
How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for <filename>graphviz.nix</filename>, will make the derivation independent of the repository and customizable:
Our goal is to make package expressions independent of the repository. To
achieve this, we use functions to declare inputs for a derivation. For example,
with <filename>graphviz.nix</filename>, we make the following changes to make
the derivation independent of the repository and customizable:
</para>
<screen><xi:include href="./12/graphviz-mkderivation.txt" parse="text" /></screen>
<para>
I recall that "<literal>{...}: ...</literal>" is the syntax for defining functions accepting an attribute set as argument.
Recall that "<literal>{...}: ...</literal>" is the syntax for defining functions
accepting an attribute set as argument; the above snippet just defines a function.
</para>
<para>
We made <code>gd</code> and its dependencies optional. If <literal>gdSupport</literal>
is true (which it is by default), we will fill <literal>buildInputs</literal> and
<code>graphviz</code> will be built with <code>gd</code> support. Otherwise, if
an attribute set is passed with <code>gdSupport = false;</code>, the build
will be completed without <code>gd</code> support.
</para>
<para>
We made <package>gd</package> and its dependencies optional. If <literal>gdSupport</literal> is true (by default), we will fill <literal>buildInputs</literal> and thus <package>graphviz</package> will be built with <package>gd</package> support, otherwise it won't.
</para>
<para>
Now back to default.nix:
Going back to back to <filename>default.nix</filename>, we modify our expression
to utilize the inputs pattern:
</para>
<screen><xi:include href="./12/repository-mkderivation.txt" parse="text" /></screen>
<para>
So we factorized the import of <literal>nixpkgs</literal> and <literal>mkDerivation</literal>, and also added a variant of <package>graphviz</package> with <package>gd</package> support disabled. The result is that both <filename>hello.nix</filename> (exercise for the reader) and <filename>graphviz.nix</filename> are independent of the repository and customizable by passing specific inputs.
We factorized the import of <literal>nixpkgs</literal> and
<literal>mkDerivation</literal>, and also added a variant of <code>graphviz</code>
with <code>gd</code> support disabled. The result is that both
<filename>hello.nix</filename> (left as an exercise for the reader) and
<filename>graphviz.nix</filename> are independent of the repository and
customizable by passing specific inputs.
</para>
<para>
If you wanted to build <package>graphviz</package> with a specific version of <package>gd</package>, it would suffice to pass <literal>gd = ...;</literal>.
If we wanted to build <code>graphviz</code> with a specific version of
<code>gd</code>, it would suffice to pass <literal>gd = ...;</literal>.
</para>
<para>
If you wanted to change the toolchain, you may pass a different <literal>mkDerivation</literal> function.
If we wanted to change the toolchain, we simply pass a different
<literal>mkDerivation</literal> function.
</para>
<para>
Clearing up the syntax:
Let's talk a closer look at the snippet and dissect the syntax:
<itemizedlist>
<listitem><para>In the end we return an attribute set from <filename>default.nix</filename>. With "<literal>let</literal>" we define some local variables.</para></listitem>
<listitem><para>We bring <literal>pkgs</literal> into the scope when defining the packages set, which is very convenient instead of typing everytime "<literal>pkgs</literal>".</para></listitem>
<listitem><para>We import <filename>hello.nix</filename> and <filename>graphviz.nix</filename>, which will return a function, and call it with a set of inputs to get back the derivation.</para></listitem>
<listitem><para>The "<literal>inherit x</literal>" syntax is equivalent to "<literal>x = x</literal>". So "<literal>inherit gd</literal>" here, combined to the above "<literal>with pkgs;</literal>" is equivalent to "<literal>gd = pkgs.gd</literal>".</para></listitem>
<listitem><para>The entire expression in <filename>default.nix</filename>
returns an attribute set with the keys <code>hello</code>,
<code>graphviz</code>, and <code>graphvizCore</code>.
</para></listitem>
<listitem><para>
With "<literal>let</literal>", we define some local variables.
</para></listitem>
<listitem><para>
We bring <literal>pkgs</literal> into the scope when defining the
package set. This saves us from having to type
<literal>pkgs</literal>" repeatedly.
</para></listitem>
<listitem><para>
We import <filename>hello.nix</filename> and <filename>graphviz.nix</filename>,
which each return a function. We call the functions with a set of inputs to
get back the derivation.
</para></listitem>
<listitem><para>
The "<literal>inherit x</literal>" syntax is equivalent to
"<literal>x = x</literal>". This means that the "<literal>inherit gd</literal>"
here, combined with the above "<literal>with pkgs;</literal>",
is equivalent to "<literal>gd = pkgs.gd</literal>".
</para></listitem>
</itemizedlist>
</para>
<para>
You can find the whole repository at the <link xlink:href="https://gist.github.com/tfc/ca800a444b029e85a14e530c25f8e872">pill 12</link> gist.
The entire repository of this can be found at the <link xlink:href="https://gist.github.com/tfc/ca800a444b029e85a14e530c25f8e872">pill 12</link> gist.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>inputs</literal>" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatever else. Our package expressions are functions, don't think there's any magic in there.
The "<literal>inputs</literal>" pattern allows our expressions to be easily
customizable through a set of arguments. These arguments could be flags,
derivations, or any other customizations enabled by the nix language.
Our package expressions are simply functions: there is no extra magic present.
</para>
<para>
It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.
The "<literal>inputs</literal>" pattern also makes the expressions
independent of the repository. Given that we pass all needed information
through arguments, it is possible to use these expressions in any other context.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about the "<literal>callPackage</literal>" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level <filename>default.nix</filename>, and once in the package expression. With <literal>callPackage</literal>, we will implicitly pass the necessary inputs from the top-level expression.
In the next pill, we will talk about the "<literal>callPackage</literal>" design
pattern. This removes the tedium of specifying the names of the inputs twice:
once in the top-level <filename>default.nix</filename>, and once in the package
expression. With <literal>callPackage</literal>, we will
implicitly pass the necessary inputs from the top-level expression.
</para>
</section>
</chapter>

228
pills/13-callpackage-design-pattern.xml
View file

@ -6,134 +6,262 @@
<title>Callpackage Design Pattern</title>
<para>
Welcome to the 13th Nix pill. In the previous <link linkend="inputs-design-pattern">12th pill</link> we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged <package>graphviz</package> to have at least another package for our little repository.
Welcome to the 13th Nix pill. In the previous <link linkend="inputs-design-pattern">12th
pill</link>, we introduced the first basic design pattern for organizing a repository of
software. In addition, we packaged <package>graphviz</package> so that we had two packages
to bundle into an example repository.
</para>
<para>
The next design pattern worth noting is what I'd like to call the <literal>callPackage</literal> pattern. This technique is extensively used in <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, it's the current standard for importing packages in a repository.
The next design pattern we will examine is called the <literal>callPackage</literal>
pattern. This technique is extensively used in <link
xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, and it's the current
de facto standard for importing packages in a repository. It's purpose is to reduce
the duplication of identifiers between package derivation inputs and repository
derivations.
</para>
<section>
<title>The callPackage convenience</title>
<para>
In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments.
In the previous pill, we demonstrated how the <literal>inputs</literal>
pattern decouples packages from the repository. This allowed us to
manually pass the inputs to the derivation; the derivation declares
its inputs, and the caller passes the arguments.
</para>
<para>
However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice. With package management, we often see common patterns. In the case of <literal>nixpkgs</literal> it's the following.
</para>
<para>
Some package derivation:
However, as with usual programming languages, there is some duplication of work:
we declare parameter names and then we pass arguments, typically with the same name.
For example, if we define a package derivation using the <literal>inputs</literal>
pattern such as:
</para>
<screen><xi:include href="./13/package-derivation.txt" parse="text" /></screen>
<para>
Repository derivation:
we would likely want to bundle that package derivation into a respository via a
an attribute set defined as something like:
</para>
<screen><xi:include href="./13/repository-derivation.txt" parse="text" /></screen>
<para>
Where inputs may even be packages in the repository itself (note the rec keyword). The pattern here is clear, often inputs have the same name of the attributes in the repository itself. Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument).
There are two things to note. First, that inputs often have the same name as
attributes in the respository itself. Second, that (due to the <code>rec</code>
keyword), the inputs to a package derivation may be other packages in the
repository itself.
</para>
<para>
To achieve this, we will define a <literal>callPackage</literal> function with the following synopsis:
Rather than passing the inputs twice, we would prefer to pass those inputs from
the respoistory automatically and allow for manually overriding defaults.
</para>
<para>
To achieve this, we will define a <literal>callPackage</literal> function with
the following calling convention:
</para>
<screen><xi:include href="./13/callpackage-function-call.txt" parse="text" /></screen>
<para>
What should it do?
We want <code>callPackage</code> to be a function of two arguments, with the
following behavior:
<itemizedlist>
<listitem><para>Import the given expression, which in turn returns a function.</para></listitem>
<listitem><para>Determine the name of its arguments.</para></listitem>
<listitem><para>Pass default arguments from the repository set, and let us override those arguments.</para></listitem>
<listitem><para>
Import the given expression contained in the file of the first argument,
and return a function. This function returns a package derivation that
uses the inputs pattern.
</para></listitem>
<listitem><para>
Determine the name of the arguments to the function (i.e., the names
of the inputs to the package derivation).
</para></listitem>
<listitem><para>
Pass default arguments from the repository set, and let us override those
arguments if we wish to customize the package derivation.
</para></listitem>
</itemizedlist>
</para>
</section>
<section>
<title>Implementing callPackage</title>
<title>Implementing <code>callPackage</code></title>
<para>
First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments.
In this section, we will build up the <code>callPackages</code> pattern
from scratch. To start, we need a way to obtain the argument names
of a function (in this case, the function that takes "inputs" and produces
a package derivation) at runtime. This is because we want to automatically pass
such arguments.
</para>
<para>
Then <literal>callPackage</literal> requires access to the whole packages set, because it needs to find the packages to pass automatically.
</para>
<para>
We start off simple with <command></command>:
Nix provides a builtin function to do this:
</para>
<screen><xi:include href="./13/get-args-function.txt" parse="text" /></screen>
<para>
Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names.
In addition to returning the argument names, the attribute set returned by
<code>functionArgs</code> indicates whether or not the argument has a default value.
For our purposes, we are only interested in the argument names; we do not care
about the default values right now.
</para>
<para>
The next step is to make <code>callPackage</code> automatically pass inputs to our
package derivations based on the argument names we've just obtained with
<code>functionArgs</code>.
</para>
<para>
Now we need a set with all the <literal>values</literal>, let's call it <literal>values</literal>. And a way to intersect the attributes of values with the function arguments:
To do this, we need two things:
<itemizedlist>
<listitem><para>
A package repository set containing package derivations that match the arguments
names we've obtained
</para></listitem>
<listitem><para>
A way to obtain an auto-populated attribute set combining the package repository
and the return value of <code>functionArgs</code>.
</para></listitem>
</itemizedlist>
</para>
<para>
The former is easy: we just have to set our package deriviation's inputs
to be package names in a repository, such as <code>nixpkgs</code>. For
the latter, Nix provides another builtin function:
</para>
<screen><xi:include href="./13/intersect-attr-values.txt" parse="text" /></screen>
<para>
Perfect, note from the example above that the <literal>intersectAttrs</literal> returns a set whose names are the intersection, and the attribute values are taken from the second set.
The <literal>intersectAttrs</literal> returns an attribute set whose names are
the intersection of both arguments' attribute names, with the attribute
values taken from the second argument.
</para>
<para>
We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of <literal>callPackage</literal>:
This is all we need to do: we have obtained the argument names from a function,
and populated these with an existing set of attributes. This is our simple
implementation of <literal>callPackage</literal>:
</para>
<screen><xi:include href="./13/callpackage-function.txt" parse="text" /></screen>
<para>
Clearing up the syntax:
Let's dissect the above snippet:
<itemizedlist>
<listitem><para>We define a <literal>callPackage</literal> variable which is a function.</para></listitem>
<listitem><para>The second parameter is the function to "autocall".</para></listitem>
<listitem><para>We take the argument names of the function and intersect with the set of all values.</para></listitem>
<listitem><para>Finally we call the passed function <literal>f</literal> with the resulting intersection.</para></listitem>
<listitem><para>
We define a <literal>callPackage</literal> variable which is a
function.
</para></listitem>
<listitem><para>
The first parameter to the <literal>callPackage</literal> function
is a set of name-value pairs that may appear in the argument set of
the function we wish to "autocall".
</para></listitem>
<listitem><para>
The second parameter is the function to "autocall"
</para></listitem>
<listitem><para>
We take the argument names of the function and intersect with the set of all
values.
</para></listitem>
<listitem><para>
Finally, we call the passed function <literal>f</literal> with the resulting
intersection.
</para></listitem>
</itemizedlist>
</para>
<para>
In the code above, I've also shown that the <literal>callPackage</literal> call is equivalent to directly calling <literal>add a b</literal>.
In the snippet above, we've also demonstrated that the <literal>callPackage</literal>
call is equivalent to directly calling <literal>add a b</literal>.
</para>
<para>
We achieved what we wanted. Automatically call functions given a set of possible arguments. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs <literal>...</literal> as explained in the <link linkend="functions-and-imports">5th pill</link>).
We achieved most of what we wanted: to automatically call functions given a set of
possible arguments. If an argument is not found within the set we used to call the
function, then we receive an error (unless the function has variadic arguments
denoted with <literal>...</literal>, as explained in the <link
linkend="functions-and-imports">5th pill</link>).
</para>
<para>
Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set. Then we add a further parameter, which takes a set of overrides:
The last missing piece is allowing users to override some of the parameters.
We may not want to always call functions with values taken from the big set.
Thus, we add a third parameter which takes a set of overrides:
</para>
<screen><xi:include href="./13/callpackage-function-overrides.txt" parse="text" /></screen>
<para>
Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set.
Apart from the increasing number of parentheses, it should be clear that we simply
take a set union between the default arguments and the overriding set.
</para>
</section>
<section>
<title>Use callPackage to simplify the repository</title>
<title>Using callPackage to simplify the repository</title>
<para>
Given our brand new tool, we can simplify the repository expression (default.nix).
</para>
<para>
Let me write it down first:
Given our <literal>callPackages</literal>, we can simplify the repository expression
in <filename>default.nix</filename>:
</para>
<screen><xi:include href="./13/callpackage-usage.txt" parse="text" /></screen>
<para>
Wow, there's a lot to say here:
Let's examine this in detail:
<itemizedlist>
<listitem><para>We renamed the old <literal>pkgs</literal> of the previous pill to <literal>nixpkgs</literal>. Our package set is now instead named <literal>pkgs</literal>. Sorry for the confusion.</para></listitem>
<listitem><para>We needed a way to pass pkgs to <literal>callPackage</literal> somehow. Instead of returning the set of packages directly from <filename>default.nix</filename>, we first assign it to a <literal>let</literal> variable and reuse it in <literal>callPackage</literal>.</para></listitem>
<listitem><para>For convenience, in <literal>callPackage</literal> we first import the file, instead of calling it directly. Otherwise for each package we would have to write the <literal>import</literal>.</para></listitem>
<listitem><para>Since our expressions use packages from <literal>nixpkgs</literal>, in <literal>callPackage</literal> we use <literal>allPkgs</literal>, which is the union of <literal>nixpkgs</literal> and our packages.</para></listitem>
<listitem><para>We moved <literal>mkDerivation</literal> into <literal>pkgs</literal> itself, so that it also gets passed automatically.</para></listitem>
<listitem><para>
The expression above defines our own package repository, which we call
<literal>pkgs</literal>, that contains <literal>hello</literal> along
with our two variants of <literal>graphviz</literal>.
</para></listitem>
<listitem><para>
In the <code>let</code> expression, we import <literal>nixpkgs</literal>.
Note that previously, we referred to this import with the variable
<literal>pkgs</literal>, but now that name is taken by the repository
we are creating ourselves.
</para></listitem>
<listitem><para>
We needed a way to pass <literal>pkgs</literal> to <literal>callPackage</literal>
somehow. Instead of returning the set of packages directly from
<filename>default.nix</filename>, we first assign it to a <literal>let</literal>
variable and reuse it in <literal>callPackage</literal>.
</para></listitem>
<listitem><para>
For convenience, in <literal>callPackage</literal> we first
import the file instead of calling it directly. Otherwise we would have to
write the <literal>import</literal> for each package.
</para></listitem>
<listitem><para>
Since our expressions use packages from <literal>nixpkgs</literal>, in
<literal>callPackage</literal> we use <literal>allPkgs</literal>, which
is the union of <literal>nixpkgs</literal> and our packages.
</para></listitem>
<listitem><para>
We moved <literal>mkDerivation</literal> into <literal>pkgs</literal> itself,
so that it also gets passed automatically.</para></listitem>
</itemizedlist>
</para>
<para>
Note how easy is to override arguments in the case of <package>graphviz</package> without <package>gd</package>. But most importantly, how easy it was to merge two repositories: <literal>nixpkgs</literal> and our <literal>pkgs</literal>!
Note how easily we overrode arguments in the case of <literal>graphviz</literal>
without <literal>gd</literal>. In addition, note how easy it was to merge
two repositories: <literal>nixpkgs</literal> and our <literal>pkgs</literal>!
</para>
<para>
The reader should notice a magic thing happening. We're defining <literal>pkgs</literal> in terms of <literal>callPackage</literal>, and <literal>callPackage</literal> in terms of <literal>pkgs</literal>. That magic is possible thanks to lazy evaluation: <literal>builtins.intersectAttrs</literal> doesn't need to know the values in <literal>allPkgs</literal> in order to perform intersection, only the keys that do not require <literal>callPackage</literal> evaluation.
The reader should notice a magic thing happening. We're defining
<literal>pkgs</literal> in terms of <literal>callPackage</literal>, and
<literal>callPackage</literal> in terms of <literal>pkgs</literal>. That magic is
possible thanks to lazy evaluation: <literal>builtins.intersectAttrs</literal> doesn't
need to know the values in <literal>allPkgs</literal> in order to perform intersection,
only the keys that do not require <literal>callPackage</literal> evaluation.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>callPackage</literal>" pattern has simplified our repository a lot. We're able to import packages that require some named arguments and call them automatically, given the set of all packages.
The "<literal>callPackage</literal>" pattern has simplified our repository
considerably. We were able to import packages that require named arguments
and call them automatically, given the set of all packages sourced from
<literal>nixpkgs</literal>.
</para>
<para>
We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. They are documented in the <link xlink:href="https://nixos.org/manual/nix/stable/expressions/builtins.html">Nix manual</link>.
We've also introduced some useful builtin functions that allows us to introspect Nix
functions and manipulate attributes. These builtin functions are not usually used when
packaging software, but rather act as tools for packaging. They are documented in the
<link xlink:href="https://nixos.org/manual/nix/stable/expressions/builtins.html">Nix
manual</link>.
</para>
<para>
Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions.
Writing a repository in Nix is an evolution of writing convenient functions for
combining the packages. This pills demonstrates how Nix can be a generic tool
to build and deploy software, and how suitable it is to create software
repositories with our own conventions.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about the "<literal>override</literal>" design pattern. The <literal>graphvizCore</literal> seems straightforward. It starts from <filename>graphviz.nix</filename> and builds it without <package>gd</package>. Now I want to give you another point of view: what if we instead wanted to start from <literal>pkgs.graphviz</literal> and disable <package>gd</package>?
In the next pill, we will talk about the "<literal>override</literal>" design
pattern. The <literal>graphvizCore</literal> seems straightforward. It starts from
<filename>graphviz.nix</filename> and builds it without <package>gd</package>.
In the next pill, we will consider another point of view: starting from
<literal>pkgs.graphviz</literal> and disabling <package>gd</package>?
</para>
</section>
</chapter>

125
pills/14-override-design-pattern.xml
View file

@ -6,139 +6,190 @@
<title>Override Design Pattern</title>
<para>
Welcome to the 14th Nix pill. In the previous <link linkend="callpackage-design-pattern">13th</link> pill we have introduced the <literal>callPackage</literal> pattern, used to simplify the composition of software in a repository.
Welcome to the 14th Nix pill. In the previous <link linkend="callpackage-design-pattern">13th</link> pill, we introduced the
<literal>callPackage</literal> pattern and used it to simplify the composition
of software in a repository.
</para>
<para>
The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.
The next design pattern is less necessary, but is useful in many cases and is
a good exercise to learn more about Nix.
</para>
<section>
<title>About composability</title>
<para>
Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.
Functional languages are known for being able to compose functions. In particular,
these languages gain expressivity from functions that manipulate an original
value into a new value having the same structure. This allows us to compose
multiple functions to perform the desired modifications.
</para>
<para>
In Nix we mostly talk about <emphasis role="bold">functions</emphasis> that accept inputs in order to return <emphasis role="bold">derivations</emphasis>. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.
In Nix, we mostly talk about <emphasis role="bold">functions</emphasis>
that accept inputs in order to return <emphasis role="bold">derivations</emphasis>.
In our world, we want utility functions that are able to manipulate those structures.
These utilities add some useful properties to the original value, and we'd like to be
able to apply more utilities on top of the result.
</para>
<para>
For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:
For example, let's say we have an initial derivation <literal>drv</literal> and
we want to transform it into a <literal>drv</literal> with debugging information and
custom patches:
</para>
<screen>debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)</screen>
<para>
The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.
The final result should be the original derivation with some changes.
This is both interesting and very different from other packaging approaches,
which is a consequence of using a functional language to describe packages.
</para>
<para>
Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.
Designing such utilities is not trivial in a functional language without static
typing, because understanding what can or cannot be composed is difficult.
But we try to do our best.
</para>
</section>
<section>
<title>The override pattern</title>
<para>
In the <link linkend="inputs-design-pattern">pill 12</link> we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.
In <link linkend="inputs-design-pattern">pill 12</link> we introduced the inputs
design pattern. We do not return a derivation picking dependencies directly from the
repository; rather we declare the inputs and let the callers pass the necessary
arguments.
</para>
<para>
In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the <package>graphviz</package> attribute:
In our repository we have a set of attributes that import the expressions of the
packages and pass these arguments, getting back a derivation. Let's take for example
the <package>graphviz</package> attribute:
</para>
<screen>graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };</screen>
<para>
If we wanted to produce a derivation of <package>graphviz</package> with a customized <package>gd</package> version, we would have to repeat most of the above plus specifying an alternative <package>gd</package>:
If we wanted to produce a derivation of <package>graphviz</package> with a customized
<package>gd</package> version, we would have to repeat most of the above plus
specifying an alternative <package>gd</package>:
</para>
<screen><xi:include href="./14/mygraphviz.txt" parse="text" /></screen>
<para>
That's hard to maintain. Using callPackage it would be easier:
That's hard to maintain. Using <code>callPackage</code> would be easier:
</para>
<screen>mygraphviz = callPackage ./graphviz.nix { gd = customgd; };</screen>
<para>
But we may still be diverging from the original <package>graphviz</package> in the repository.
</para>
<para>
We would like to avoid specifying the nix expression again, instead reuse the original <package>graphviz</package> attribute in the repository and add our overrides like this:
We would like to avoid specifying the nix expression again. Instead, we would
like to reuse the original <package>graphviz</package> attribute in the
repository and add our overrides like so:
</para>
<screen>mygraphviz = graphviz.override { gd = customgd; };</screen>
<para>
The difference is obvious, as well as the advantages of this approach.
</para>
<para>
<emphasis role="underline">Note:</emphasis> that <literal>.override</literal> is not a "method" in the OO sense as you may think. Nix is a functional language. That <literal>.override</literal> is simply an attribute of a set.
<emphasis role="underline">Note:</emphasis> that <literal>.override</literal> is
not a "method" in the OO sense as you may think. Nix is a functional language.
The<literal>.override</literal> is simply an attribute of a set.
</para>
</section>
<section>
<title>The override implementation</title>
<para>
I remind you, the <package>graphviz</package> attribute in the repository is the derivation returned by the function imported from <filename>graphviz.nix</filename>. We would like to add a further attribute named "<literal>override</literal>" to the returned set.
Recall that the <package>graphviz</package> attribute in the repository is
the derivation returned by the function imported from
<filename>graphviz.nix</filename>. We would like to add a further attribute
named "<literal>override</literal>" to the returned set.
</para>
<para>
Let's start simple by first creating a function "<literal>makeOverridable</literal>" that takes a function and a set of original arguments to be passed to the function.
Let's start by first creating a function "<literal>makeOverridable</literal>".
This function will take two arguments: a function (that must return a set)
and the set of original arguments to be passed to the function.
</para>
<para>
<emphasis role="underline">Contract:</emphasis> the wrapped function must return a set.
</para>
<para>
Let's write a lib.nix:
We will put this function in a <filename>lib.nix</filename>:
</para>
<screen><xi:include href="./14/make-overridable-lib.txt" parse="text" /></screen>
<para>
So <literal>makeOverridable</literal> takes a function and a set of original arguments. It returns the original returned set, plus a new <literal>override</literal> attribute.
<literal>makeOverridable</literal> takes a function and a set of original arguments.
It returns the original returned set, plus a new <literal>override</literal> attribute.
</para>
<para>
This <literal>override</literal> attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.
This <literal>override</literal> attribute is a function taking a set of new
arguments, and returns the result of the original function called with the
original arguments unified with the new arguments. This is admittedly somewhat
confusing, but the examples below should make it clear.
</para>
<para>
Let's try it with <literal>nix repl</literal>:
</para>
<screen><xi:include href="./14/nix-repl-make-overridable-test.txt" parse="text" /></screen>
<para>
Note that the function <literal>f</literal> does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)
Note that, as we specified above, the function <literal>f</literal> does not return
the plain sum. Instead, it returns a set with the sum bound to the name
<literal>result</literal>.
</para>
<para>
The variable <literal>res</literal> is the result of the function call without any override. It's easy to see in the definition of <literal>makeOverridable</literal>. In addition you can see the new <literal>override</literal> attribute being a function.
The variable <literal>res</literal> contains the result of the function call without
any override. It's easy to see in the definition of <literal>makeOverridable</literal>.
In addition, you can see that new <literal>override</literal> attribute is a function.
</para>
<para>
Calling that <literal>.override</literal> with a set will invoke the original function with the overrides, as expected.
Calling <literal>res.override</literal> with a set will invoke the original function
with the overrides, as expected.
</para>
<para>
But: we can't override again! Because the returned set with result 15 does not have an <literal>override</literal> attribute!
This is a good start, but we can't override again! This is because the returned
set (with <literal>result = 15</literal>) does not have an <literal>override</literal>
attribute of it's own. This is bad; it breaks further composition.
</para>
<para>
That's bad, it breaks further compositions.
</para>
<para>
The solution is simple, the <literal>.override</literal> function should make the result overridable again:
The solution is simple: the <literal>.override</literal> function should make the
result overridable again:
</para>
<screen><xi:include href="./14/rec-make-overridable.txt" parse="text" /></screen>
<para>
Please note the <literal>rec</literal> keyword. It's necessary so that we can refer to <literal>makeOverridable</literal> from <literal>makeOverridable</literal> itself.
Please note the <literal>rec</literal> keyword. It's necessary so that we can refer
to <literal>makeOverridable</literal> from <literal>makeOverridable</literal> itself.
</para>
<para>
Now let's try overriding twice:
</para>
<screen><xi:include href="./14/nix-repl-make-overridable-twice.txt" parse="text" /></screen>
<para>
Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20.
Success! The result is 30 (as expected) because <literal>a</literal> is overridden
to 10 in the first override, and <literal>b</literal> is overridden to 20 in the
second.
</para>
<para>
Now it would be nice if <literal>callPackage</literal> made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.
Now it would be nice if <literal>callPackage</literal> made our
derivations overridable. This is an exercise for the reader.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>override</literal>" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like <literal>nixpkgs</literal>, and defining overrides on our local machine without even modifying the original package.
The "<literal>override</literal>" pattern simplifies the way we customize packages
starting from an existing set of packages. This opens a world of possibilities for
using a central repository like <literal>nixpkgs</literal> and defining overrides
on our local machine without modifying the original package.
</para>
<para>
Dream of a custom isolated <command>nix-shell</command> environment for testing <package>graphviz</package> with a custom <package>gd</package>:
We can dream of a custom, isolated <command>nix-shell</command> environment for
testing <package>graphviz</package> with a custom <package>gd</package>:
</para>
<screen>debugVersion (graphviz.override { gd = customgd; })</screen>
<para>
Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.
Once a new version of the overridden package comes out in the repository, the
customized package will make use of it automatically.
</para>
<para>
The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.
The key in Nix is to find powerful yet simple abstractions in order to let the user
customize their environment with highest consistency and lowest maintenance time, by
using predefined composable components.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <literal>&lt;nixpkgs&gt;</literal> come from?
In the next pill, we will talk about Nix search paths. By "search path", we mean a
place in the file system where Nix looks for expressions. This answers the
question of where <literal>&lt;nixpkgs&gt;</literal> comes from.
</para>
</section>
</chapter>