From efb0f37e61c0dcbefc0356394f52a417f767274b Mon Sep 17 00:00:00 2001 From: Samuel Leathers Date: Wed, 16 Aug 2017 13:11:45 -0400 Subject: [PATCH] port pill #18 --- pills/18-nix-store-paths.xml | 145 +++++++++++++++++- pills/18/bar-derivation.txt | 4 + pills/18/derivation-simple-contents.txt | 6 + pills/18/derivation-simple.txt | 3 + .../mycontent-string-representation-hash.txt | 3 + pills/18/mycontent-string-representation.txt | 4 + pills/18/mycontent.txt | 5 + pills/18/myfile-final-hash.txt | 2 + pills/18/myfile-hash-alternate.txt | 2 + pills/18/myfile-hash.txt | 2 + pills/18/myfile-string-hash.txt | 3 + pills/18/myout-drv-hash.txt | 5 + pills/18/output-path-replace-empty.txt | 2 + pills/19-fundamentals-of-stdenv.xml | 10 +- 14 files changed, 186 insertions(+), 10 deletions(-) create mode 100644 pills/18/bar-derivation.txt create mode 100644 pills/18/derivation-simple-contents.txt create mode 100644 pills/18/derivation-simple.txt create mode 100644 pills/18/mycontent-string-representation-hash.txt create mode 100644 pills/18/mycontent-string-representation.txt create mode 100644 pills/18/mycontent.txt create mode 100644 pills/18/myfile-final-hash.txt create mode 100644 pills/18/myfile-hash-alternate.txt create mode 100644 pills/18/myfile-hash.txt create mode 100644 pills/18/myfile-string-hash.txt create mode 100644 pills/18/myout-drv-hash.txt create mode 100644 pills/18/output-path-replace-empty.txt diff --git a/pills/18-nix-store-paths.xml b/pills/18-nix-store-paths.xml index ba1e209..81c2d51 100644 --- a/pills/18-nix-store-paths.xml +++ b/pills/18-nix-store-paths.xml @@ -1,8 +1,143 @@ + xmlns:xlink="http://www.w3.org/1999/xlink" + xmlns:xi="http://www.w3.org/2001/XInclude" + version="5.0" + xml:id="nix-store-paths"> -nix store paths + nix store paths + + Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides. + + + Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs. + + + The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code. + +
+ Source paths + + Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is ./myfile gets stored into /nix/store/....... We want to understand how is the store path generated for such a file: + + $ echo mycontent > myfile + + I remind you, the simplest derivation you can write has a name, a builder and the system: + + + + Now inspect the .drv to see where is ./myfile being stored: + + + + Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck ? Keep looking at the nix comments. + + + Note: doing nix-store --add myfile will store the file in the same store path. + + +
+ Step 1, compute the hash of the file + + The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways: + + + + Or: + + + + In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything. + +
+
+ Step 2, build the string description + + Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file: + + $ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str +
+
+ Step 3, compute the final hash + + Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string: + + +
+
+
+ Output paths + + Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path hs0yi5n5nw6micqhy8l1igkbhqdkzqa1. This is because the out path only depends on inputs. + + + It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is output:out. In case of multiple outputs, we may have different output:<id>. + + + At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string: + + + + The myout.drv is the .drv state in which nix is when computing the out path for our derivation: + + + + Then nix puts that out path in the .drv, and that's it. + + + In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash. + + + In other words, you get a final .drv where every other .drv path is replaced by its hash. + +
+
+ Fixed-output paths + + Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs. + + + A derivation can take three special attributes: outputHashMode, outputHash and outputHashAlgo which are well documented in the nix manual. + + + The builder must create the out path and make sure its hash is the same as the one declared with outputHash. + + + Let's say our builder should create a file whose contents is mycontent: + + + + Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples: + + + + It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash. + + + What nix does is to create an intermediate string representation of the fixed-output content: + + + + Then proceed as it was a normal derivation output path: + + + + Hence, the store path only depends on the declared fixed-output hash. + +
+
+ Conclusion + + There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string. + + + Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs. + +
+
+ Next pill + + ...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems. + + +
diff --git a/pills/18/bar-derivation.txt b/pills/18/bar-derivation.txt new file mode 100644 index 0000000..31a45e7 --- /dev/null +++ b/pills/18/bar-derivation.txt @@ -0,0 +1,4 @@ +$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv +Derive( + [("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")] +... diff --git a/pills/18/derivation-simple-contents.txt b/pills/18/derivation-simple-contents.txt new file mode 100644 index 0000000..1257c67 --- /dev/null +++ b/pills/18/derivation-simple-contents.txt @@ -0,0 +1,6 @@ +./myfile./myfileterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv +Derive( + [("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")] +, [] +, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"] +, "x86_64-linux" diff --git a/pills/18/derivation-simple.txt b/pills/18/derivation-simple.txt new file mode 100644 index 0000000..0f5cff8 --- /dev/null +++ b/pills/18/derivation-simple.txt @@ -0,0 +1,3 @@ +$ nix-repl +nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; } +«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv» diff --git a/pills/18/mycontent-string-representation-hash.txt b/pills/18/mycontent-string-representation-hash.txt new file mode 100644 index 0000000..e7a5d09 --- /dev/null +++ b/pills/18/mycontent-string-representation-hash.txt @@ -0,0 +1,3 @@ +$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str +$ nix-hash --type sha256 --truncate --base32 --flat myfile.str +a00d5f71k0vp5a6klkls0mvr1f7sx6ch diff --git a/pills/18/mycontent-string-representation.txt b/pills/18/mycontent-string-representation.txt new file mode 100644 index 0000000..d30da2d --- /dev/null +++ b/pills/18/mycontent-string-representation.txt @@ -0,0 +1,4 @@ +$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str +$ sha256sum mycontent.str +423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str + diff --git a/pills/18/mycontent.txt b/pills/18/mycontent.txt new file mode 100644 index 0000000..7d13e80 --- /dev/null +++ b/pills/18/mycontent.txt @@ -0,0 +1,5 @@ +$ echo mycontent > myfile +$ sha256sum myfile +f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile +nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; } +«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv» diff --git a/pills/18/myfile-final-hash.txt b/pills/18/myfile-final-hash.txt new file mode 100644 index 0000000..a5b6c10 --- /dev/null +++ b/pills/18/myfile-final-hash.txt @@ -0,0 +1,2 @@ +$ nix-hash --type sha256 --truncate --base32 --flat myfile.str +xv2iccirbrvklck36f1g7vldn5v58vck diff --git a/pills/18/myfile-hash-alternate.txt b/pills/18/myfile-hash-alternate.txt new file mode 100644 index 0000000..a5dca21 --- /dev/null +++ b/pills/18/myfile-hash-alternate.txt @@ -0,0 +1,2 @@ +$ nix-store --dump myfile|sha256sum +2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 diff --git a/pills/18/myfile-hash.txt b/pills/18/myfile-hash.txt new file mode 100644 index 0000000..cbaf8f1 --- /dev/null +++ b/pills/18/myfile-hash.txt @@ -0,0 +1,2 @@ +$ nix-hash --type sha256 myfile +2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 diff --git a/pills/18/myfile-string-hash.txt b/pills/18/myfile-string-hash.txt new file mode 100644 index 0000000..e7a5d09 --- /dev/null +++ b/pills/18/myfile-string-hash.txt @@ -0,0 +1,3 @@ +$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str +$ nix-hash --type sha256 --truncate --base32 --flat myfile.str +a00d5f71k0vp5a6klkls0mvr1f7sx6ch diff --git a/pills/18/myout-drv-hash.txt b/pills/18/myout-drv-hash.txt new file mode 100644 index 0000000..fa9d2a4 --- /dev/null +++ b/pills/18/myout-drv-hash.txt @@ -0,0 +1,5 @@ +$ sha256sum myout.drv +1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv +$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str +$ nix-hash --type sha256 --truncate --base32 --flat myout.str +hs0yi5n5nw6micqhy8l1igkbhqdkzqa1 diff --git a/pills/18/output-path-replace-empty.txt b/pills/18/output-path-replace-empty.txt new file mode 100644 index 0000000..6a7eebf --- /dev/null +++ b/pills/18/output-path-replace-empty.txt @@ -0,0 +1,2 @@ +$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv +$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv diff --git a/pills/19-fundamentals-of-stdenv.xml b/pills/19-fundamentals-of-stdenv.xml index 8a9626f..7f7e9a7 100644 --- a/pills/19-fundamentals-of-stdenv.xml +++ b/pills/19-fundamentals-of-stdenv.xml @@ -80,7 +80,7 @@ So we ran the configurePhase function and buildPhase function and they worked. These bash functions should be self-explanatory, you can read the code in the setup file. - +
How is the setup file built @@ -108,7 +108,7 @@ - +
The stdenv.mkDerivation builder @@ -145,7 +145,7 @@ Last bit, the unpackPhase in the setup is used to unpack the sources and enter the directory, again like we did in our old builder. - +
@@ -172,7 +172,7 @@ Really, take your time to read that file. Don't forget that juicy docs are also available in the nixpkgs manual. - +
Next pill... @@ -180,6 +180,6 @@ ...we will talk about how to add dependencies to our packages, buildInputs, propagatedBuildInputs and setup hooks. These three concepts are at the base of the current nixpkgs packages composition. - +