NixOS/nix-pills
NixOS/nix-pills
1
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-09-19 04:00:13 -04:00
Code Activity

Pill 20: Init according 19's teaser

Browse source
This commit is contained in:
John Ericson 2018-02-23 17:27:25 -05:00
parent aca2170ce0
commit 4c513daccd
15 changed files with 338 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
book.xml
View file

@ -57,4 +57,5 @@
<xi:include href="pills/17-nixpkgs-overriding-packages.xml" />
<xi:include href="pills/18-nix-store-paths.xml" />
<xi:include href="pills/19-fundamentals-of-stdenv.xml" />
<xi:include href="pills/20-basic-dependencies-and-hooks.xml" />
</book>

3
pills/19-fundamentals-of-stdenv.xml
View file

@ -179,7 +179,8 @@
<title>Next pill...</title>
<para>
...we will talk about how to add dependencies to our packages, <literal>buildInputs</literal>, <literal>propagatedBuildInputs</literal> and <literal>setup</literal> hooks. These three concepts are at the base of the current <literal>nixpkgs</literal> packages composition.
...we will talk about how to add dependencies to our packages with <literal>buildInputs</literal> and <literal>propagatedBuildInputs</literal>, and influence downstream builds with <firstterm>setup hooks</firstterm> and <firstterm>env hooks</firstterm>.
These concepts are crucial to how <literal>nixpkgs</literal> packages are composed.
</para>
</section>

179
pills/20-basic-dependencies-and-hooks.xml Normal file
View file

@ -0,0 +1,179 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="basic-dependencies-and-hooks">
<title>Basic Dependencies and Hooks</title>
<para>
Welcome to the 20th Nix pill.
In the previous <link linkend="fundamentals-of-stdenv">19th</link> pill we introduced Nixpkgs' Stdenv, including <filename>setup.sh</filename> script, <filename>default-builder.sh</filename> helper script, and <literal>stdenv.mkDerivation</literal> builder.
We focused on how stdenv is put together, and how it's used, an a bit about the phases of <function>genericBuild</function>.
</para>
<para>
This time, we'll focus on the interaction of packages built with <literal>stdenv.mkDerivation</literal>.
Packages need to depend on each other, of course.
For this we have <varname>buildInputs</varname> and <varname>propagatedBuildInputs</varname> attributes.
We've also found that dependencies sometimes need to influence their dependents in ways the dependents can't or shouldn't predict.
For this we have <firstterm>setup hooks</firstterm> and <firstterm>env hooks</firstterm>.
Together, these 4 concepts support almost all build-time package interactions.
</para>
<para>
This Nix pill is going to teach Nixpkgs a bit differently.
The Nix pills before this one have used the most recent version of Nixpkgs when discussing it.
But the dependencies and hooks infrastructure has for a few years been increasingly complicated to support cross compilation, with a final boost of complexity in recently in 2017.
While the added mechanism shouldn't be too confusing after the core concepts are taught, they just get in the way before, so this Nix Pill will go all the way back to 2009 with commit <link xlink:href="https://github.com/nixos/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25">6675f0a5</link>.
This is the last version of Stdenv before extra dependency and hook types for cross compilation were created.
</para>
<section>
<title>The <varname>buildInputs</varname> Attribute</title>
<para>
For the simplest dependencies where the current package directly needs another, we use the <varname>buildInputs</varname> attribute.
This is exactly the pattern in taught with our builder in <link linkend="generic-builders">Pill 8</link>.
To demo this, lets build GNU Hello, and then another package which provides a shell script that <command>exec</command>s it.
<screen><xi:include href="./20/two-hellos.txt" parse="text" /></screen>
</para>
<para>
This works because stdenv contains something like:
<screen><xi:include href="./20/build-inputs-0.bash" parse="text" /></screen>
where <function>findInputs</function> is defined like:
<screen><xi:include href="./20/build-inputs-1.bash" parse="text" /></screen>
then after this is run:
<screen><xi:include href="./20/build-inputs-2.bash" parse="text" /></screen>
where <function>addToEnv</function> is defined like:
<screen><xi:include href="./20/build-inputs-3.bash" parse="text" /></screen>
The <function>addToSearchPath</function> call adds <literal>$1/bin</literal> to <envar>_PATH</envar> if the former exists (code <link xlink:href="https://github.com/NixOS/nixpkgs/blob/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/stdenv/generic/setup.sh#L60-L73">here</link>).
With the real <command>hello</command> on the path, the <function>installPhase</function> should hopefully make sense.
</para>
</section>
<section>
<title>The <varname>propagatedBuildInputs</varname> Attribute</title>
<para>
The <varname>buildInputs</varname> covers direct dependencies, but what about indirect dependencies where one package needs a second package which needs a third?
Nix itself handles this just fine, understanding various dependency <firstterm>closures</firstterm> as covered in previous builds.
But what about the conveniences that <varname>buildInputs</varname> provides, namely accumulating in <envar>pkgs</envar> environment variable and inclusion of <filename><replaceable>pkg</replaceable>/bin</filename> directories on the <envar>PATH</envar>?
For this, Stdenv provides the <varname>propagatedBuildInputs</varname>:
<screen><xi:include href="./20/two-hellos.txt" parse="text" /></screen>
See how the intermediate package has a <varname>propagatedBuildInputs</varname> dependency, but the wrapper only needs a <varname>buildInputs</varname> dependency on the intermediary.
</para>
<para>
How does this work?
You might think we do something in Nix, but actually its done not at eval time but at build time in bash.
lets look at part of the <function>fixupPhase</function> of Stdenv:
<screen><xi:include href="./20/propagated-build-inputs-0.bash" parse="text" /></screen>
This dumps the propagated build inputs in a so-named file in <filename>$out/nix-support/</filename>.
Then, back in <function>findInputs</function> look at the lines at the bottom we elided before:
<screen><xi:include href="./20/propagated-build-inputs-1.bash" parse="text" /></screen>
See how <function>findInputs</function> is actually recursive, looking at the propagated build inputs of each dependency, and those dependencies' propagated build inputs, etc.
</para>
<para>
We actually simplified the <function>findInputs</function> call site from before; <envar>propagatedBuildInputs</envar> is also looped over in reality:
<screen><xi:include href="./20/propagated-build-inputs-2.bash" parse="text" /></screen>
This demonstrates an important point. For the <emphasis>current</emphasis> package alone, it doesn't matter whether a dependency is propagated or not.
It will be processed the same way: called with <function>findInputs</function> and <function>addToEnv</function>.
(The packages discovered by <function>findInputs</function>, which are also accumulated in <envar>pkgs</envar> and passed to <function>addToEnv</function>, are also the same in both cases.)
Downstream however, it certainly does matter because only the propagated immediate dependencies are put in the <filename>$out/nix-support/propagated-build-inputs</filename>.
</para>
</section>
<section>
<title>Setup Hooks</title>
<para>
As we mentioned above, sometimes dependencies need to influence the packages that use them in ways other than just <emphasis>being</emphasis> a dependency.
<footnote>
<para>
We can now be precise and consider what <function>addToEnv</function> does alone the minimal treatment of a dependency:
i.e. a package that is <emphasis>just</emphasis> a dependency would <emphasis>only</emphasis> have <function>addToEnv</function> applied to it.
</para>
</footnote>
<varname>propagatedBuildInputs</varname> can actually be seen as an example of this:
packages using that are effectively "injecting" those dependencies as extra <varname>buildInputs</varname> in their downstream dependents.
But in general, a dependency might to affect the packages it depends on in arbitrary ways.
<emphasis>arbitrary</emphasis> is the key word here
We could teach <filename>setup.sh</filename> things about upstream packages like <filename><replaceable>pkg</replaceable>/nix-support/propagated-build-inputs</filename>, but not arbitrary interactions.
</para>
<para>
<firstterm>Setup hooks</firstterm> are the basic building block we have for this.
In nixpkgs, a "hook" is basically a bash callback, and a setup hook is no exception.
Let's look at the last part of <function>findInputs</function> we haven't covered:
<screen><xi:include href="./20/setup-hooks-0.bash" parse="text" /></screen>
If a package includes the path <filename><replaceable>pkg</replaceable>/nix-support/setup-hook</filename>, it will be sourced by any Stdenv-based build including that as a dependency.
</para>
<para>
This is strictly more general than any of the other mechanisms introduced in this chapter.
For example, try writing a setup hook that has the same effect as a <emphasis>propagatedBuildInputs</emphasis> entry.
One can almost think of this as an escape hatch around Nix's normal isolation guarantees, and the principle that dependencies are immutable and inert.
We're not actually doing something unsafe or modifying dependencies, but we are allowing arbitrary ad-hoc behavior.
For this reason, setup-hooks should only be used as a last resort.
</para>
</section>
<section>
<title>Environment Hooks</title>
<para>
As a final convenience, we have environment hooks.
Recall in <link linkend="inputs-design-pattern">Pill 12</link> how we created <envar>NIX_CFLAGS_COMPILE</envar> for <literal>-I</literal> flags and <envar>NIX_LDFLAGS</envar> for <literal>-L</literal> flags, in a similar manner to how we prepared the <envar>PATH</envar>.
One point of ugliness was how anti-modular this was.
It makes sense to build the <envar>PATH</envar> in generic builder, because the <envar>PATH</envar> is used by the shell, and the generic builder is intrinsically tied to the shell.
But <literal>-I</literal> and <literal>-L</literal> flags are only relevant to the C compiler.
The stdenv isn't wedded to including a C compiler (though it does by default), and there are other compilers too which may take completely different flags.
</para>
<para>
As a first step, we can move that logic to a setup hook on the C compiler;
indeed that's just what we do in CC Wrapper.
<footnote>
<para>
<link xlink:href="https://github.com/NixOS/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/build-support/gcc-wrapper" />. It was called GCC Wrapper in the version of nixpkgs we're using in this pill; Darwin and Clang support hadn't yet motivated the rename.
</para>
</footnote>
But this pattern comes up fairly often, so somebody decided to add some helper support to reduce boilerplate.
</para>
<para>
The other half of <function>addToEnv</function> is:
<screen><xi:include href="./20/env-hooks-0.bash" parse="text" /></screen>
See how for every package we iterate through the contents of <varname>envHooks</varname>, and apply every function within to the package in question.
The idea is one can write a setup hook like:
<screen><xi:include href="./20/env-hooks-1.bash" parse="text" /></screen>
and if one dependency has that setup hook then all of them will be so <command>echo</command>ed.
Allowing dependencies to learn about their <emphasis>sibling</emphasis> dependencies is exactly what compiler need.
</para>
</section>
<section>
<title>Next pill...</title>
<para>
...I'm not sure!
We could talk the additional dependency types and hooks which cross compilation necessitates, building on our knowledge here to cover stdenv as it works today.
We could talk about how nixpkgs is bootstrapped.
Or we could talk about how <varname>localSystem</varname> and <varname>crossSystem</varname> are elaborated into the <varname>buildPlatform</varname>, <varname>hostPlatform</varname>, and <varname>targetPlatform</varname> each bootstrapping stage receives.
Let us know which most interests you!
</para>
</section>
</chapter>

4
pills/20/build-inputs-0.bash Normal file
View file

@ -0,0 +1,4 @@
pkgs=""
for i in $buildInputs; do
findInputs $i
done

14
pills/20/build-inputs-1.bash Normal file
View file

@ -0,0 +1,14 @@
findInputs() {
local pkg=$1
## Don't need to repeat already processed package
case $pkgs in
*\ $pkg\ *)
return 0
;;
esac
pkgs="$pkgs $pkg "
## More goes here in reality that we can ignore for now.
}

3
pills/20/build-inputs-2.bash Normal file
View file

@ -0,0 +1,3 @@
for i in $pkgs; do
addToEnv $i
done

9
pills/20/build-inputs-3.bash Normal file
View file

@ -0,0 +1,9 @@
addToEnv() {
local pkg=$1
if test -d $1/bin; then
addToSearchPath _PATH $1/bin
fi
## More goes here in reality that we can ignore for now.
}

10
pills/20/env-hooks-0.bash Normal file
View file

@ -0,0 +1,10 @@
addToEnv() {
local pkg=$1
## More goes here in reality that we can ignore for now.
# Run the package-specific hooks set by the setup-hook scripts.
for i in "${envHooks[@]}"; do
$i $pkg
done
}

7
pills/20/env-hooks-1.bash Normal file
View file

@ -0,0 +1,7 @@
anEnvHook() {
local pkg=$1
echo "I'm depending on \"$pkg\""
}
envHooks+=(anEnvHook)

12
pills/20/propagated-build-inputs-0.bash Normal file
View file

@ -0,0 +1,12 @@
fixupPhase() {
## Elided
if test -n "$propagatedBuildInputs"; then
ensureDir "$out/nix-support"
echo "$propagatedBuildInputs" > "$out/nix-support/propagated-build-inputs"
fi
## Elided
}

11
pills/20/propagated-build-inputs-1.bash Normal file
View file

@ -0,0 +1,11 @@
findInputs() {
local pkg=$1
## More goes here in reality that we can ignore for now.
if test -f $pkg/nix-support/propagated-build-inputs; then
for i in $(cat $pkg/nix-support/propagated-build-inputs); do
findInputs $i
done
fi
}

4
pills/20/propagated-build-inputs-3.bash Normal file
View file

@ -0,0 +1,4 @@
pkgs=""
for i in $buildInputs $propagatedBuildInputs; do
findInputs $i
done

12
pills/20/setup-hooks-0.bash Normal file
View file

@ -0,0 +1,12 @@
findInputs() {
local pkg=$1
## More goes here in reality that we can ignore for now.
if test -f $pkg/nix-support/setup-hook; then
source $pkg/nix-support/setup-hook
fi
## More goes here in reality that we can ignore for now.
}

40
pills/20/three-hellos.nix Normal file
View file

@ -0,0 +1,40 @@
let
nixpkgs = import (builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/6675f0a52c0962042a1000c7f20e887d0d26ae25.tar.gz";
}) {};
inherit (nixpkgs) stdenv fetchurl;
actualHello = stdenv.mkDerivation {
name = "hello-2.3";
src = fetchurl {
url = mirror://gnu/hello/hello-2.3.tar.bz2;
sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1";
};
};
intermediary = stdenv.mkDerivation {
name = "middle-man";
propagatedBuildInputs = [ actualHello ];
installPhase = ''
mkdir -p "$out"
'';
};
wrappedHello = stdenv.mkDerivation {
name = "hello-wrapper";
buildInputs = [ intermediary ];
installPhase = ''
mkdir -p "$out/bin"
echo "#! ${stdenv.shell}" >> "$out/bin/hello"
echo "exec $(which hello)" >> "$out/bin/hello"
'';
};
in wrappedHello

30
pills/20/two-hellos.nix Normal file
View file

@ -0,0 +1,30 @@
let
nixpkgs = import (builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/6675f0a52c0962042a1000c7f20e887d0d26ae25.tar.gz";
}) {};
inherit (nixpkgs) stdenv fetchurl;
actualHello = stdenv.mkDerivation {
name = "hello-2.3";
src = fetchurl {
url = mirror://gnu/hello/hello-2.3.tar.bz2;
sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1";
};
};
wrappedHello = stdenv.mkDerivation {
name = "hello-wrapper";
buildInputs = [ actualHello ];
installPhase = ''
mkdir -p "$out/bin"
echo "#! ${stdenv.shell}" >> "$out/bin/hello"
echo "exec $(which hello)" >> "$out/bin/hello"
'';
};
in wrappedHello