From d20e2c1746f3e87e7f073ea571535aa1e4b02a35 Mon Sep 17 00:00:00 2001 From: Linus Heckemann Date: Sun, 13 Aug 2017 10:56:53 +0100 Subject: [PATCH] Transcribe pill 7 --- pills/07-working-derivation.xml | 321 +++++++++++++++++++++++++++++- pills/07/bash.xml | 5 + pills/07/builder.sh.txt | 2 + pills/07/c-program-derivation.xml | 7 + pills/07/foo.drv.xml | 14 ++ pills/07/simple-derivation.xml | 10 + pills/07/simple.c.txt | 3 + pills/07/simple.txt | 9 + pills/07/simple_builder.sh.txt | 3 + 9 files changed, 373 insertions(+), 1 deletion(-) create mode 100644 pills/07/bash.xml create mode 100644 pills/07/builder.sh.txt create mode 100644 pills/07/c-program-derivation.xml create mode 100644 pills/07/foo.drv.xml create mode 100644 pills/07/simple-derivation.xml create mode 100644 pills/07/simple.c.txt create mode 100644 pills/07/simple.txt create mode 100644 pills/07/simple_builder.sh.txt diff --git a/pills/07-working-derivation.xml b/pills/07-working-derivation.xml index d962973..b14b3b8 100644 --- a/pills/07-working-derivation.xml +++ b/pills/07-working-derivation.xml @@ -4,5 +4,324 @@ version="5.0" xml:id="working-derivation"> -working derivation + working derivation + +
+ Introduction + + Welcome to the seventh nix pill. In the previous + sixth pill we introduced the + notion of derivation in the Nix language — how to define a raw derivation + and how to (try to) build it. + + + In this post we continue along the path, by creating a derivation that + actually builds something. Then, we try to package a real program: we + compile a simple C file and create a derivation out of it, given a blessed + toolchain. + + + + I remind you how to enter the Nix environment: + source ~/.nix-profile/etc/profile.d/nix.sh + +
+ +
+ Using a script as a builder + + + What's the easiest way to run a sequence of commands for building + something? A bash script. We write a custom bash script, and we want it to + be our builder. Given a builder.sh, we want the + derivation to run bash builder.sh. + + + + We don't use hash bangs in builder.sh, because at the + time we are writing it we do not know the path to + bash in the nix store. Yes, even bash is in the + nix store, everything is there. + + + + We don't even use /usr/bin/env, because then we + lose the cool stateless property of Nix. Not to mention that + PATH gets cleared when building, so it wouldn't find + bash anyway. + + + + In summary, we want the builder to be bash, and + pass it an argument, builder.sh. Turns out the + derivation function accepts an optional + args attribute which is used to pass arguments to + the builder executable. + + + + First of all, let's write our builder.sh in the + current directory: + + + + As we covered in the previous pill, Nix computes the output path of the + derivation. The resulting .drv file contains a list of + environment variables passed to the builder. One of these is + $out. + + + What we have to do is create something in the path + $out, be it a file or a directory. In this case we are + creating a file. + + + + In addition, we print out the environment variables during the build + process. We cannot use env for this, because + env is part of + coreutils and we don't have a dependency to it + yet. We only have bash for now. + + + + Like for coreutils in the previous pill, we get a blessed bash for free + from our magic nixpkgs stuff: + + + + So with the usual trick, we can refer to + bin/bash and create our derivation: + + + + We did it! The contents of + /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo + is really foo. We've built our first derivation. + + + Note that we used ./builder.sh and not + "./builder.sh". This way, it is parsed as a path, and Nix + performs some magic which we will cover later. Try using the string + version and you will find that it cannot find + builder.sh. This is because it tries to find it + relative to the temporary build directory. + +
+ +
+ The builder environment + + Let's inspect those environment variables printed during the build process. + + + $HOME is not your home directory, and + /homeless-shelter doesn't exist at all. We force + packages not to depend on $HOME during the build + process. + + + $PATH plays the same game as $HOME + + + $NIX_BUILD_CORES and $NIX_STORE are + nix + configuration options + + + $PWD and $TMP clearly show that nix + created a temporary build directory + + + Then $builder, $name, + $out, and $system are variables set due + to the .drv file's contents. + + + + + And that's how we were able to use $out in our derivation + and put stuff in it. It's like Nix reserved a slot in the nix store for + us, and we must fill it. + + + + In terms of autotools, $out will be the + path. Yes, not the make + , but the . That's the + essence of stateless packaging. You don't install the package in a global + common path under /, you install it in a local + isolated path under your nix store slot. + +
+
+ The .drv contents + + We added something else to the derivation this time: the args attribute. + Let's see how this changed the .drv compared to the previous pill: + + + + Much like the usual .drv, except that there's a list of arguments in there + passed to the builder (bash) with + builder.sh… In the nix store..? Nix automatically + copies files or directories needed for the build into the store to ensure + that they are not changed during the build process and that the deployment + is stateless and independent of the building machine. + builder.sh is not only in the arguments passed to the + builder, it's also in the input derivations. + + + Given that builder.sh is a plain file, it has no .drv + associated with it. The store path is computed based on the filename and + on the hash of its contents. Store paths are covered in detail in a later pill. + +
+
+ Packaging a simple C program + + Start off by writing a simple C program: + + + + And its simple_builder.sh: + + + + Don't worry too much about where those variables come from yet; let's + write the derivation and build it: + + + + Now you can run + /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple + in your shell. + +
+
+ Explanation + + We added two new attributes to the derivation call, gcc + and coreutils. In gcc = gcc;, the name on + the left is the name in the derivation set, and the name on the right + refers to the gcc derivation from nixpkgs. The same applies for coreutils. + + + We also added the src attribute, nothing magical — it's + just a name, to which the path ./simple.c is + assigned. Like simple-builder.sh, + simple.c will be added to the store. + + + The trick: every attribute in the set passed to + derivation will be converted to a string and passed + to the builder as an environment variable. This is how the builder gains + access to coreutils and + gcc: when converted to strings, the derivations + evaluate to their output paths, and appending /bin to + these leads us to their binaries. + + + The same goes for the src variable. $src + is the path to simple.c in the nix store. As an + exercise, pretty print the .drv file. You'll see + simple_builder.sh and simple.c + listed in the input derivations, along with + bash, gcc and + coreutils .drv files. The newly added + environment variables described above will also appear. + + + In simple_builder.sh we set the PATH + for gcc and + coreutils binaries, so that our build script + can find the necessary utilities like mkdir and + gcc. + + + We then create $out as a directory and place the binary + inside it. Note that gcc is found via the + PATH environment variable, but it could equivalently be + referenced explicitly using $gcc/bin/gcc. + +
+
+ Enough of nix-repl + + Drop out of nix-repl and write a file + simple.nix: + + + + Now you can build it with nix-build simple.nix. This + will create a symlink result in the current + directory, pointing to the out path of the derivation. + + + nix-build does two jobs: + + + + nix-instantiate + : parse and evaluate simple.nix and return + the .drv file corresponding to the parsed derivation set + + + + nix-store -r + : realise the .drv file, which actually builds it. + + + Finally, it creates the symlink. + + + In the first line of simple.nix, we have an + import function call nested in a with + statement. Recall that import accepts one argument, a + nix file to load. In this case, the contents of the file evaluated to a + function. + + + Afterwards, we call the function with the empty set. We saw this already + in the fifth pill. To + reiterate: import <nixpkgs> {} is calling two functions, + not one. Reading it as (import <nixpkgs> {} makes this + clearer. + + + The value returned by the nixpkgs function is a set. More specifically, + it's a set of derivations. Using the with expression we bring + them into scope. This is the same as what :l does in + nix-repl, so we can easily access derivations + such as bash, gcc, and + coreutils. + + + Then we meet the + inherit keyword. + inherit foo; is equivalent to foo = foo;; + inherit foo bar; is equivalent to foo = foo; bar = bar;. + + + This syntax only makes sense inside sets. There's no magic involved, it's + simply a convenience to avoid repeating the same name for both the + attribute name and the value in scope. + +
+
+ Next pill + + We will generalize the builder. You may have noticed that we wrote two + separate builder.sh scripts in this post. We would + like to have a generic builder script instead, especially since each build + script goes in the nix store: a bit of a waste. + + + Is it really that hard to package stuff in Nix? No, + here we're studying the fundamentals of Nix. + +
diff --git a/pills/07/bash.xml b/pills/07/bash.xml new file mode 100644 index 0000000..e288773 --- /dev/null +++ b/pills/07/bash.xml @@ -0,0 +1,5 @@ +nix-repl> :l <nixpkgs> +Added 3950 variables. +nix-repl> "${bash}" +"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45" + diff --git a/pills/07/builder.sh.txt b/pills/07/builder.sh.txt new file mode 100644 index 0000000..19459b7 --- /dev/null +++ b/pills/07/builder.sh.txt @@ -0,0 +1,2 @@ +declare -xp +echo foo > $out diff --git a/pills/07/c-program-derivation.xml b/pills/07/c-program-derivation.xml new file mode 100644 index 0000000..30684b6 --- /dev/null +++ b/pills/07/c-program-derivation.xml @@ -0,0 +1,7 @@ +nix-repl> :l <nixpkgs> +nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem; +nix-repl> :b simple +this derivation produced the following outputs: + + out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple + diff --git a/pills/07/foo.drv.xml b/pills/07/foo.drv.xml new file mode 100644 index 0000000..c446c5d --- /dev/null +++ b/pills/07/foo.drv.xml @@ -0,0 +1,14 @@ +$ pp-aterm -i /nix/store/g6jj1mjzq68i66rbqyb3gpx3k0x606af-foo.drv +Derive( + [("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo", "", "")] +, [("/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv", ["out"])] +, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"] +, "x86_64-linux" +, "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash" +, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"] +, [ ("builder", "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash") + , ("name", "foo") + , ("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo") + , ("system", "x86_64-linux") + ] +) diff --git a/pills/07/simple-derivation.xml b/pills/07/simple-derivation.xml new file mode 100644 index 0000000..541266e --- /dev/null +++ b/pills/07/simple-derivation.xml @@ -0,0 +1,10 @@ +nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; } +nix-repl> :b d +these derivations will be built: + /nix/store/ybnysdh5k6cjznhg4afjgbhr6czbwb4s-foo.drv +building path(s) `/nix/store/72v14vk4li47n8sx3z2ibd802ihpqyvx-foo' +these derivations will be built: + /nix/store/ibwr68l3rjlx02kgz61dkkkrlpgljfxd-simple.drv +[...] +this derivation produced the following outputs: + out -> /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo diff --git a/pills/07/simple.c.txt b/pills/07/simple.c.txt new file mode 100644 index 0000000..c0a3532 --- /dev/null +++ b/pills/07/simple.c.txt @@ -0,0 +1,3 @@ +void main() { + puts("Simple!"); +} diff --git a/pills/07/simple.txt b/pills/07/simple.txt new file mode 100644 index 0000000..b975e95 --- /dev/null +++ b/pills/07/simple.txt @@ -0,0 +1,9 @@ +with (import {}); +derivation { + name = "simple"; + builder = "${bash}/bin/bash"; + args = [ ./simple_builder.sh ]; + inherit gcc coreutils; + src = ./simple.c; + system = builtins.currentSystem; +} diff --git a/pills/07/simple_builder.sh.txt b/pills/07/simple_builder.sh.txt new file mode 100644 index 0000000..8a38571 --- /dev/null +++ b/pills/07/simple_builder.sh.txt @@ -0,0 +1,3 @@ +export PATH="$coreutils/bin:$gcc/bin" +mkdir $out +gcc -o $out/simple $src