working derivation
+ working derivation
+
+
+ Introduction
+
+ Welcome to the seventh nix pill. In the previous
+ sixth pill we introduced the
+ notion of derivation in the Nix language — how to define a raw derivation
+ and how to (try to) build it.
+
+
+ In this post we continue along the path, by creating a derivation that
+ actually builds something. Then, we try to package a real program: we
+ compile a simple C file and create a derivation out of it, given a blessed
+ toolchain.
+
+
+
+ I remind you how to enter the Nix environment:
+ source ~/.nix-profile/etc/profile.d/nix.sh
+
+
+
+
+ Using a script as a builder
+
+
+ What's the easiest way to run a sequence of commands for building
+ something? A bash script. We write a custom bash script, and we want it to
+ be our builder. Given a builder.sh, we want the
+ derivation to run bash builder.sh.
+
+
+
+ We don't use hash bangs in builder.sh, because at the
+ time we are writing it we do not know the path to
+ bash in the nix store. Yes, even bash is in the
+ nix store, everything is there.
+
+
+
+ We don't even use /usr/bin/env, because then we
+ lose the cool stateless property of Nix. Not to mention that
+ PATH gets cleared when building, so it wouldn't find
+ bash anyway.
+
+
+
+ In summary, we want the builder to be bash, and
+ pass it an argument, builder.sh. Turns out the
+ derivation function accepts an optional
+ args attribute which is used to pass arguments to
+ the builder executable.
+
+
+
+ First of all, let's write our builder.sh in the
+ current directory:
+
+
+
+ As we covered in the previous pill, Nix computes the output path of the
+ derivation. The resulting .drv file contains a list of
+ environment variables passed to the builder. One of these is
+ $out.
+
+
+ What we have to do is create something in the path
+ $out, be it a file or a directory. In this case we are
+ creating a file.
+
+
+
+ In addition, we print out the environment variables during the build
+ process. We cannot use env for this, because
+ env is part of
+ coreutils and we don't have a dependency to it
+ yet. We only have bash for now.
+
+
+
+ Like for coreutils in the previous pill, we get a blessed bash for free
+ from our magic nixpkgs stuff:
+
+
+
+ So with the usual trick, we can refer to
+ bin/bash and create our derivation:
+
+
+
+ We did it! The contents of
+ /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo
+ is really foo. We've built our first derivation.
+
+
+ Note that we used ./builder.sh and not
+ "./builder.sh". This way, it is parsed as a path, and Nix
+ performs some magic which we will cover later. Try using the string
+ version and you will find that it cannot find
+ builder.sh. This is because it tries to find it
+ relative to the temporary build directory.
+
+
+
+
+ The builder environment
+
+ Let's inspect those environment variables printed during the build process.
+
+
+ $HOME is not your home directory, and
+ /homeless-shelter doesn't exist at all. We force
+ packages not to depend on $HOME during the build
+ process.
+
+
+ $PATH plays the same game as $HOME
+
+
+ $NIX_BUILD_CORES and $NIX_STORE are
+ nix
+ configuration options
+
+
+ $PWD and $TMP clearly show that nix
+ created a temporary build directory
+
+
+ Then $builder, $name,
+ $out, and $system are variables set due
+ to the .drv file's contents.
+
+
+
+
+ And that's how we were able to use $out in our derivation
+ and put stuff in it. It's like Nix reserved a slot in the nix store for
+ us, and we must fill it.
+
+
+
+ In terms of autotools, $out will be the
+ path. Yes, not the make
+ , but the . That's the
+ essence of stateless packaging. You don't install the package in a global
+ common path under /, you install it in a local
+ isolated path under your nix store slot.
+
+
+
+ The .drv contents
+
+ We added something else to the derivation this time: the args attribute.
+ Let's see how this changed the .drv compared to the previous pill:
+
+
+
+ Much like the usual .drv, except that there's a list of arguments in there
+ passed to the builder (bash) with
+ builder.sh… In the nix store..? Nix automatically
+ copies files or directories needed for the build into the store to ensure
+ that they are not changed during the build process and that the deployment
+ is stateless and independent of the building machine.
+ builder.sh is not only in the arguments passed to the
+ builder, it's also in the input derivations.
+
+
+ Given that builder.sh is a plain file, it has no .drv
+ associated with it. The store path is computed based on the filename and
+ on the hash of its contents. Store paths are covered in detail in a later pill.
+
+
+
+ Packaging a simple C program
+
+ Start off by writing a simple C program:
+
+
+
+ And its simple_builder.sh:
+
+
+
+ Don't worry too much about where those variables come from yet; let's
+ write the derivation and build it:
+
+
+
+ Now you can run
+ /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple
+ in your shell.
+
+
+
+ Explanation
+
+ We added two new attributes to the derivation call, gcc
+ and coreutils. In gcc = gcc;, the name on
+ the left is the name in the derivation set, and the name on the right
+ refers to the gcc derivation from nixpkgs. The same applies for coreutils.
+
+
+ We also added the src attribute, nothing magical — it's
+ just a name, to which the path ./simple.c is
+ assigned. Like simple-builder.sh,
+ simple.c will be added to the store.
+
+
+ The trick: every attribute in the set passed to
+ derivation will be converted to a string and passed
+ to the builder as an environment variable. This is how the builder gains
+ access to coreutils and
+ gcc: when converted to strings, the derivations
+ evaluate to their output paths, and appending /bin to
+ these leads us to their binaries.
+
+
+ The same goes for the src variable. $src
+ is the path to simple.c in the nix store. As an
+ exercise, pretty print the .drv file. You'll see
+ simple_builder.sh and simple.c
+ listed in the input derivations, along with
+ bash, gcc and
+ coreutils .drv files. The newly added
+ environment variables described above will also appear.
+
+
+ In simple_builder.sh we set the PATH
+ for gcc and
+ coreutils binaries, so that our build script
+ can find the necessary utilities like mkdir and
+ gcc.
+
+
+ We then create $out as a directory and place the binary
+ inside it. Note that gcc is found via the
+ PATH environment variable, but it could equivalently be
+ referenced explicitly using $gcc/bin/gcc.
+
+
+
+ Enough of nix-repl
+
+ Drop out of nix-repl and write a file
+ simple.nix:
+
+
+
+ Now you can build it with nix-build simple.nix. This
+ will create a symlink result in the current
+ directory, pointing to the out path of the derivation.
+
+
+ nix-build does two jobs:
+
+
+
+ nix-instantiate
+ : parse and evaluate simple.nix and return
+ the .drv file corresponding to the parsed derivation set
+
+
+
+ nix-store -r
+ : realise the .drv file, which actually builds it.
+
+
+ Finally, it creates the symlink.
+
+
+ In the first line of simple.nix, we have an
+ import function call nested in a with
+ statement. Recall that import accepts one argument, a
+ nix file to load. In this case, the contents of the file evaluated to a
+ function.
+
+
+ Afterwards, we call the function with the empty set. We saw this already
+ in the fifth pill. To
+ reiterate: import <nixpkgs> {} is calling two functions,
+ not one. Reading it as (import <nixpkgs> {} makes this
+ clearer.
+
+
+ The value returned by the nixpkgs function is a set. More specifically,
+ it's a set of derivations. Using the with expression we bring
+ them into scope. This is the same as what :l does in
+ nix-repl, so we can easily access derivations
+ such as bash, gcc, and
+ coreutils.
+
+
+ Then we meet the
+ inherit keyword.
+ inherit foo; is equivalent to foo = foo;;
+ inherit foo bar; is equivalent to foo = foo; bar = bar;.
+
+
+ This syntax only makes sense inside sets. There's no magic involved, it's
+ simply a convenience to avoid repeating the same name for both the
+ attribute name and the value in scope.
+
+
+
+ Next pill
+
+ We will generalize the builder. You may have noticed that we wrote two
+ separate builder.sh scripts in this post. We would
+ like to have a generic builder script instead, especially since each build
+ script goes in the nix store: a bit of a waste.
+
+
+ Is it really that hard to package stuff in Nix? No,
+ here we're studying the fundamentals of Nix.
+
+
diff --git a/pills/07/bash.xml b/pills/07/bash.xml
new file mode 100644
index 0000000..e288773
--- /dev/null
+++ b/pills/07/bash.xml
@@ -0,0 +1,5 @@
+nix-repl> :l <nixpkgs>
+Added 3950 variables.
+nix-repl> "${bash}"
+"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45"
+
diff --git a/pills/07/builder.sh.txt b/pills/07/builder.sh.txt
new file mode 100644
index 0000000..19459b7
--- /dev/null
+++ b/pills/07/builder.sh.txt
@@ -0,0 +1,2 @@
+declare -xp
+echo foo > $out
diff --git a/pills/07/c-program-derivation.xml b/pills/07/c-program-derivation.xml
new file mode 100644
index 0000000..30684b6
--- /dev/null
+++ b/pills/07/c-program-derivation.xml
@@ -0,0 +1,7 @@
+nix-repl> :l <nixpkgs>
+nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem;
+nix-repl> :b simple
+this derivation produced the following outputs:
+
+ out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple
+
diff --git a/pills/07/foo.drv.xml b/pills/07/foo.drv.xml
new file mode 100644
index 0000000..c446c5d
--- /dev/null
+++ b/pills/07/foo.drv.xml
@@ -0,0 +1,14 @@
+$ pp-aterm -i /nix/store/g6jj1mjzq68i66rbqyb3gpx3k0x606af-foo.drv
+Derive(
+ [("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo", "", "")]
+, [("/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv", ["out"])]
+, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
+, "x86_64-linux"
+, "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash"
+, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
+, [ ("builder", "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash")
+ , ("name", "foo")
+ , ("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo")
+ , ("system", "x86_64-linux")
+ ]
+)
diff --git a/pills/07/simple-derivation.xml b/pills/07/simple-derivation.xml
new file mode 100644
index 0000000..541266e
--- /dev/null
+++ b/pills/07/simple-derivation.xml
@@ -0,0 +1,10 @@
+nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; }
+nix-repl> :b d
+these derivations will be built:
+ /nix/store/ybnysdh5k6cjznhg4afjgbhr6czbwb4s-foo.drv
+building path(s) `/nix/store/72v14vk4li47n8sx3z2ibd802ihpqyvx-foo'
+these derivations will be built:
+ /nix/store/ibwr68l3rjlx02kgz61dkkkrlpgljfxd-simple.drv
+[...]
+this derivation produced the following outputs:
+ out -> /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo
diff --git a/pills/07/simple.c.txt b/pills/07/simple.c.txt
new file mode 100644
index 0000000..c0a3532
--- /dev/null
+++ b/pills/07/simple.c.txt
@@ -0,0 +1,3 @@
+void main() {
+ puts("Simple!");
+}
diff --git a/pills/07/simple.txt b/pills/07/simple.txt
new file mode 100644
index 0000000..b975e95
--- /dev/null
+++ b/pills/07/simple.txt
@@ -0,0 +1,9 @@
+with (import {});
+derivation {
+ name = "simple";
+ builder = "${bash}/bin/bash";
+ args = [ ./simple_builder.sh ];
+ inherit gcc coreutils;
+ src = ./simple.c;
+ system = builtins.currentSystem;
+}
diff --git a/pills/07/simple_builder.sh.txt b/pills/07/simple_builder.sh.txt
new file mode 100644
index 0000000..8a38571
--- /dev/null
+++ b/pills/07/simple_builder.sh.txt
@@ -0,0 +1,3 @@
+export PATH="$coreutils/bin:$gcc/bin"
+mkdir $out
+gcc -o $out/simple $src