NixOS/nix-pills
NixOS/nix-pills
1
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-09-19 04:00:13 -04:00
Code Activity

port pill #12

Browse source
This commit is contained in:
Samuel Leathers 2017-08-17 23:54:26 -04:00
parent 9977703fa8
commit b014776dc8
11 changed files with 300 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

241
pills/12-inputs-design-pattern.xml
View file

@ -1,8 +1,239 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="inputs-design-pattern">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="inputs-design-pattern">
<title>inputs design pattern</title>
<title>inputs design pattern</title>
<para>
Welcome to the 12th Nix pill. In the previous <link linkend="garbage-collector">11th pill</link> we stopped packaging and cleaned up the system with the garbage collector.
</para>
<para>
We restart our packaging, but we will improve a different aspect. We only packaged an hello world program so far, what if we want to create a repository of multiple packages?
</para>
<section>
<title>Repositories in Nix</title>
<para>
Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. See it more like a consequence due to the need of organizing packages.
</para>
<para>
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.
</para>
<para>
There is no preset directory structure or preset packaging policy. It's all about you and Nix.
</para>
<para>
The <literal>nixpkgs</literal> repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.
</para>
<para>
Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.
</para>
</section>
<section>
<title>The single repository pattern</title>
<para>
Before introducing the "<literal>inputs</literal>" pattern, we can start talking about another pattern first which I'd like to call "<literal>single repository</literal>" pattern.
</para>
<para>
Systems like Debian scatter packages in several small repositories. Personally, this makes it hard to track interdependent changes and to contribute to new packages.
</para>
<para>
Systems like Gentoo instead, put package descriptions all in a single repository.
</para>
<para>
The nix reference for packages is <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.
</para>
<para>
From now on, we will adopt this technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs.
</para>
<para>
But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why <literal>nixpkgs</literal> is able to maintain such a big software repository in a giant attribute set.
</para>
</section>
<section>
<title>Packaging graphviz</title>
<para>
We have packaged <package>GNU hello worl</package>, I guess you would like to package something else for creating at least a repository of two projects :-) . I chose <package>graphviz</package>, which uses the standard autotools build system, requires no patching and dependencies are optional.
</para>
<para>
Download <package>graphviz</package> from here. The <filename>graphviz.nix</filename> expression is straightforward:
</para>
<screen><xi:include href="./12/graphviz-derivation.txt" parse="text" /></screen>
<para>
Build with <command>nix-build graphviz.nix</command> and you will get runnable binaries under <filename>result/bin</filename>. Notice how we did reuse the same <filename>autotools.nix</filename> of <filename>hello.nix.</filename> Let's create a simple png:
</para>
<screen><xi:include href="./12/simple-png.txt" parse="text" /></screen>
<para>
Oh of course... <package>graphviz</package> can't know about png. It built only the output formats it supports natively, without using any extra library.
</para>
<para>
I remind you, in <filename>autotools.nix</filename> there's a <literal>buildInputs</literal> variable which gets concatenated to <literal>baseInputs</literal>. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.
</para>
<para>
This 2.38 version of <package>graphviz</package> has several plugins to output png. For simplicity, we will use <package>libgd</package>.
</para>
</section>
<section>
<title>Digression about gcc and ld wrappers</title>
<para>
The <package>gd</package>, <package>jpeg</package>, <package>fontconfig</package> and <package>bzip2</package> libraries (dependencies of <package>gd</package>) don't use <command>pkg-config</command> to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell <command>gcc</command> and <command>ld</command> where to find includes and libs.
</para>
<para>
The <literal>nixpkgs</literal> provides <package>gcc</package> and <package>binutils</package>, and we are using them for our packaging. Not only, it also <link xlink:href="http://nixos.org/nixpkgs/manual/#ssec-setup-hooks">provides wrappers</link> for them which allow passing extra arguments to <command>gcc</command> and <command>ld</command>, bypassing the project build systems:
<itemizedlist>
<listitem><para><varname>NIX_CFLAGS_COMPILE</varname>: extra flags to <command>gcc</command> at compile time</para></listitem>
<listitem><para><varname>NIX_LDFLAGS</varname>: extra flags to <command>ld</command></para></listitem>
</itemizedlist>
</para>
<para>
What can we do about it? We can employ the same trick we did for <varname>PATH</varname>: automatically filling the variables from <literal>buildInputs</literal>. This is the relevant snippet of <filename>setup.sh</filename>:
</para>
<screen><xi:include href="./12/setup-sh.txt" parse="text" /></screen>
<para>
Now by adding derivations to <literal>buildInputs</literal>, will add the <filename>lib</filename>, <filename>include</filename> and <filename>bin</filename> paths automatically in <filename>setup.sh</filename>.
</para>
<para>
The <arg>-rpath</arg> flag in <command>ld</command> is needed because at runtime, the executable must use exactly that version of the library.
</para>
<para>
If unneeded paths are specified, the <literal>fixup</literal> phase will shrink the <literal>rpath</literal> for us!
</para>
</section>
<section>
<title>Completing graphviz with gd</title>
<para>
Finish the expression for <package>graphviz</package> with <package>gd</package> support (note the use of the <literal>with</literal> expression in <literal>buildInputs</literal> to avoid repeating <literal>pkgs</literal>):
</para>
<screen><xi:include href="./12/graphviz-gd-derivation.txt" parse="text" /></screen>
<para>
Now you can create the png! Ignore any error from <package>fontconfig</package>, especially if you are in a <literal>chroot</literal>.
</para>
</section>
<section>
<title>The repository expression</title>
<para>
Now that we have two packages, what's a good way to put them together in a single repository? We do something like <literal>nixpkgs</literal> does. With <literal>nixpkgs</literal>, we <literal>import</literal> it and then we peek derivations by accessing the giant attribute set.
</para>
<para>
For us nixers, this a good technique, because it abstracts from the file names. We don't refer to a package by <filename>REPO/some/sub/dir/package.nix</filename> but by <literal>importedRepo.package</literal> (or <literal>pkgs.package</literal> in our examples).
</para>
<para>
Create a default.nix in the current directory:
</para>
<screen><xi:include href="./12/repository.txt" parse="text" /></screen>
<para>
Ready to use! Try it with <command>nix-repl</command>:
</para>
<screen><xi:include href="./12/repository-test-nix-repl.txt" parse="text" /></screen>
<para>
With <command>nix-build</command>:
</para>
<screen><xi:include href="./12/repository-test-nix-build.txt" parse="text" /></screen>
<para>
The <arg>-A</arg> argument is used to access an attribute of the set from the given .nix expression.
</para>
<para>
<emphasis role="bold">Important:</emphasis> why did we choose the <filename>default.nix</filename>? Because when a directory (by default the current directory) has a <filename>default.nix</filename>, that <filename>default.nix</filename> will be used (see <literal>import</literal> <link xlink:href="http://nixos.org/nix/manual/#ssec-builtins">here</link>). In fact you can run <command>nix-build -A hello</command> without specifying <filename>default.nix</filename>.
</para>
<para>
For pythoners, it is similar to <filename>__init__.py</filename>.
</para>
<para>
With <command>nix-env</command>, to install the package in your user environment:
</para>
<screen><xi:include href="./12/nix-env-install-graphviz.txt" parse="text" /></screen>
<para>
The <arg>-f</arg> option is used to specify the expression to use, in this case the current directory, therefore <filename>./default.nix</filename>.
</para>
<para>
The <arg>-i</arg> stands for installation.
</para>
<para>
The <arg>-A</arg> is the same as above for <command>nix-build</command>.
</para>
<para>
We reproduced the very basic behavior of <literal>nixpkgs</literal>.
</para>
</section>
<section>
<title>The inputs pattern</title>
<para>
After a long preparation, we finally arrived. I know you have a big doubt in this moment. It's about the <filename>hello.nix</filename> and <filename>graphviz.nix</filename>. They are very, very dependent on <literal>nixpkgs</literal>:
<itemizedlist>
<listitem><para>First big problem: they import <literal>nixpkgs</literal> directly. In <filename>autotools.nix</filename> instead we pass <literal>nixpkgs</literal> as an argument. That's a much better approach.</para></listitem>
<listitem><para>Second problem: what if we want a variant of <package>graphviz</package> without <package>libgd</package> support?</para></listitem>
<listitem><para>Third problem: what if we want to test <package>graphviz</package> with a particular <package>libgd</package> version?</para></listitem>
</itemizedlist>
</para>
<para>
The current answer to the above questions is: change the expression to match your needs (or change the callee to match your needs).
</para>
<para>
With the <literal>inputs</literal> pattern, we choose to give another answer: let the user change the <literal>inputs</literal> of the expression (or change the caller to pass different inputs).
</para>
<para>
By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:
<itemizedlist>
<listitem><para><literal>mkDerivation</literal> from <package>autotools</package>. Recall that <literal>mkDerivation</literal> has an implicit dependency on the toolchain.</para></listitem>
<listitem><para><package>libgd</package> and its dependencies.</para></listitem>
</itemizedlist>
</para>
<para>
The src is also an input but it's pointless to change the source from the caller. For version bumps, in <literal>nixpkgs</literal> we prefer to write another expression (e.g. because patches are needed or different inputs are needed).
</para>
<para>
<emphasis role="underline">Goal:</emphasis> make package expressions independent of the repository.
</para>
<para>
How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for <filename>graphviz.nix</filename>, will make the derivation independent of the repository and customizable:
</para>
<screen><xi:include href="./12/graphviz-mkderivation.txt" parse="text" /></screen>
<para>
I recall that "<literal>{...}: ...</literal>" is the syntax for defining functions accepting an attribute set as argument.
</para>
<para>
We made <package>gd</package> and its dependencies optional. If <literal>gdSupport</literal> is true (by default), we will fill <literal>buildInputs</literal> and thus <package>graphviz</package> will be built with <package>gd</package> support, otherwise it won't.
</para>
<para>
Now back to default.nix:
</para>
<screen><xi:include href="./12/repository-mkderivation.txt" parse="text" /></screen>
<para>
So we factorized the import of <literal>nixpkgs</literal> and <literal>mkDerivation</literal>, and also added a variant of <package>graphviz</package> with <package>gd</package> support disabled. The result is that both <filename>hello.nix</filename> (exercise for the reader) and <filename>graphviz.nix</filename> are independent of the repository and customizable by passing specific inputs.
</para>
<para>
If you wanted to build <package>graphviz</package> with a specific version of <package>gd</package>, it would suffice to pass <literal>gd = ...;</literal>.
</para>
<para>
If you wanted to change the toolchain, you may pass a different <literal>mkDerivation</literal> function.
</para>
<para>
Clearing up the syntax:
<itemizedlist>
<listitem><para>In the end we return an attribute set from <filename>default.nix</filename>. With "<literal>let</literal>" we define some local variables.</para></listitem>
<listitem><para>We bring <literal>pkgs</literal> into the scope when defining the packages set, which is very convenient instead of typing everytime "<literal>pkgs</literal>".</para></listitem>
<listitem><para>We import <filename>hello.nix</filename> and <filename>graphviz.nix</filename>, which will return a function, and call it with a set of inputs to get back the derivation.</para></listitem>
<listitem><para>The "<literal>inherit x</literal>" syntax is equivalent to "<literal>x = x</literal>". So "<literal>inherit gd</literal>" here, combined to the above "<literal>with pkgs;</literal>" is equivalent to "<literal>x = pkgs.gd</literal>".</para></listitem>
</itemizedlist>
</para>
<para>
You can find the whole repository at the <link xlink:href="https://gist.github.com/lethalman/734b168a0258b8a38ca2">pill 12</link> gist.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>inputs</literal>" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatelse. Our package expressions are functions, don't think there's any magic in there.
</para>
<para>
It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about the "<literal>callPackage</literal>" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level <filename>default.nix</filename>, and once in the package expression. With <literal>callPackage</literal>, we will implicitly pass the necessary inputs from the top-level expression.
</para>
</section>
</chapter>

7
pills/12/graphviz-derivation.txt Normal file
View file

@ -0,0 +1,7 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
}

8
pills/12/graphviz-gd-derivation.txt Normal file
View file

@ -0,0 +1,8 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}

7
pills/12/graphviz-mkderivation.txt Normal file
View file

@ -0,0 +1,7 @@
{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:
mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}

3
pills/12/nix-env-install-graphviz.txt Normal file
View file

@ -0,0 +1,3 @@
$ nix-env -f . -iA graphviz
[...]
$ dot -V

11
pills/12/repository-mkderivation.txt Normal file
View file

@ -0,0 +1,11 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
hello = import ./hello.nix { inherit mkDerivation; };
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
graphvizCore = import ./graphviz.nix {
inherit mkDerivation gd fontconfig libjpeg bzip2;
gdSupport = false;
};
}

4
pills/12/repository-test-nix-build.txt Normal file
View file

@ -0,0 +1,4 @@
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!

7
pills/12/repository-test-nix-repl.txt Normal file
View file

@ -0,0 +1,7 @@
$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»

4
pills/12/repository.txt Normal file
View file

@ -0,0 +1,4 @@
{
hello = import ./hello.nix;
graphviz = import ./graphviz.nix;
}

11
pills/12/setup-sh.txt Normal file
View file

@ -0,0 +1,11 @@
for p in $baseInputs $buildInputs; do
if [ -d $p/bin ]; then
export PATH="$p/bin${PATH:+:}$PATH"
fi
if [ -d $p/include ]; then
export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
fi
if [ -d $p/lib ]; then
export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
fi
done

2
pills/12/simple-png.txt Normal file
View file

@ -0,0 +1,2 @@
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]