NixOS/nix-pills
NixOS/nix-pills
1
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-09-19 04:00:13 -04:00
Code Activity

port pill #18

Browse source
This commit is contained in:
Samuel Leathers 2017-08-16 13:11:45 -04:00
parent cbce4aa202
commit efb0f37e61
14 changed files with 186 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

135
pills/18-nix-store-paths.xml
View file

@ -5,4 +5,139 @@
xml:id="nix-store-paths">
<title>nix store paths</title>
<para>
Welcome to the 18th Nix pill. In the previous <link linkend="nixpkgs-overriding-packages">17th</link> pill we have scratched the surface of the <literal>nixpkgs</literal> repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.
</para>
<para>
Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.
</para>
<para>
The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the <link xlink:href="http://lxr.devzen.net/source/xref/nix/src/libstore/store-api.cc#97">Nix source code</link>.
</para>
<section>
<title>Source paths</title>
<para>
Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is <filename>./myfile</filename> gets stored into <filename>/nix/store/.......</filename> We want to understand how is the store path generated for such a file:
</para>
<screen>$ echo mycontent > myfile</screen>
<para>
I remind you, the simplest derivation you can write has a <literal>name</literal>, a <literal>builder</literal> and the <literal>system</literal>:
</para>
<screen><xi:include href="./18/derivation-simple.txt" parse="text" /></screen>
<para>
Now inspect the .drv to see where is <filename>./myfile</filename> being stored:
</para>
<screen><xi:include href="./18/derivation-simple-contents.txt" parse="text" /></screen>
<para>
Great, how did nix decide to use <literal>xv2iccirbrvklck36f1g7vldn5v58vck</literal> ? Keep looking at the nix comments.
</para>
<para>
<emphasis role="bold">Note:</emphasis> doing <command>nix-store --add myfile</command> will store the file in the same store path.
</para>
<section>
<title>Step 1, compute the hash of the file</title>
<para>
The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
</para>
<screen><xi:include href="./18/myfile-hash.txt" parse="text" /></screen>
<para>
Or:
</para>
<screen><xi:include href="./18/myfile-hash-alternate.txt" parse="text" /></screen>
<para>
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.
</para>
</section>
<section>
<title>Step 2, build the string description</title>
<para>
Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
</para>
<screen>$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str</screen>
</section>
<section>
<title>Step 3, compute the final hash</title>
<para>
Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
</para>
<screen><xi:include href="./18/myfile-final-hash.txt" parse="text" /></screen>
</section>
</section>
<section>
<title>Output paths</title>
<para>
Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path <literal>hs0yi5n5nw6micqhy8l1igkbhqdkzqa1</literal>. This is because the out path only depends on inputs.
</para>
<para>
It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is <literal>output:out</literal>. In case of multiple outputs, we may have different <literal>output:&lt;id&gt;</literal>.
</para>
<para>
At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
</para>
<screen><xi:include href="./18/output-path-replace-empty.txt" parse="text" /></screen>
<para>
The <literal>myout.drv</literal> is the .drv state in which nix is when computing the out path for our derivation:
</para>
<screen><xi:include href="./18/myout-drv-hash.txt" parse="text" /></screen>
<para>
Then nix puts that out path in the .drv, and that's it.
</para>
<para>
In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.
</para>
<para>
In other words, you get a final .drv where every other .drv path is replaced by its hash.
</para>
</section>
<section>
<title>Fixed-output paths</title>
<para>
Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.
</para>
<para>
A derivation can take three special attributes: <literal>outputHashMode</literal>, <literal>outputHash</literal> and <literal>outputHashAlgo</literal> which are well documented in the <link xlink:href="http://nixos.org/nix/manual/#ssec-derivation">nix manual</link>.
</para>
<para>
The builder must create the out path and make sure its hash is the same as the one declared with <literal>outputHash</literal>.
</para>
<para>
Let's say our builder should create a file whose contents is <literal>mycontent</literal>:
</para>
<screen><xi:include href="./18/mycontent.txt" parse="text" /></screen>
<para>
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
</para>
<screen><xi:include href="./18/bar-derivation.txt" parse="text" /></screen>
<para>
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
</para>
<para>
What nix does is to create an intermediate string representation of the fixed-output content:
</para>
<screen><xi:include href="./18/mycontent-string-representation.txt" parse="text" /></screen>
<para>
Then proceed as it was a normal derivation output path:
</para>
<screen><xi:include href="./18/myfile-string-hash.txt" parse="text" /></screen>
<para>
Hence, the store path only depends on the declared fixed-output hash.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.
</para>
<para>
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will introduce <literal>stdenv</literal>. In the previous pills we rolled our own <literal>mkDerivation</literal> convenience function for wrapping the builtin derivation, but the <literal>nixpkgs</literal> repository also has its own convenience functions for dealing with <package>autotools</package> projects and other build systems.
</para>
</section>
</chapter>

4
pills/18/bar-derivation.txt Normal file
View file

@ -0,0 +1,4 @@
$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
[("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...

6
pills/18/derivation-simple-contents.txt Normal file
View file

@ -0,0 +1,6 @@
./myfile./myfileterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
[("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"

3
pills/18/derivation-simple.txt Normal file
View file

@ -0,0 +1,3 @@
$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»

3
pills/18/mycontent-string-representation-hash.txt Normal file
View file

@ -0,0 +1,3 @@
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch

4
pills/18/mycontent-string-representation.txt Normal file
View file

@ -0,0 +1,4 @@
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str

5
pills/18/mycontent.txt Normal file
View file

@ -0,0 +1,5 @@
$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»

2
pills/18/myfile-final-hash.txt Normal file
View file

@ -0,0 +1,2 @@
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck

2
pills/18/myfile-hash-alternate.txt Normal file
View file

@ -0,0 +1,2 @@
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3

2
pills/18/myfile-hash.txt Normal file
View file

@ -0,0 +1,2 @@
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3

3
pills/18/myfile-string-hash.txt Normal file
View file

@ -0,0 +1,3 @@
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch

5
pills/18/myout-drv-hash.txt Normal file
View file

@ -0,0 +1,5 @@
$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1

2
pills/18/output-path-replace-empty.txt Normal file
View file

@ -0,0 +1,2 @@
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv