mirror of
https://github.com/NixOS/nix-pills
synced 2024-09-19 04:00:13 -04:00
port pill #18
This commit is contained in:
parent
cbce4aa202
commit
efb0f37e61
|
@ -1,8 +1,143 @@
|
||||||
<chapter xmlns="http://docbook.org/ns/docbook"
|
<chapter xmlns="http://docbook.org/ns/docbook"
|
||||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||||||
version="5.0"
|
version="5.0"
|
||||||
xml:id="nix-store-paths">
|
xml:id="nix-store-paths">
|
||||||
|
|
||||||
<title>nix store paths</title>
|
<title>nix store paths</title>
|
||||||
|
<para>
|
||||||
|
Welcome to the 18th Nix pill. In the previous <link linkend="nixpkgs-overriding-packages">17th</link> pill we have scratched the surface of the <literal>nixpkgs</literal> repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the <link xlink:href="http://lxr.devzen.net/source/xref/nix/src/libstore/store-api.cc#97">Nix source code</link>.
|
||||||
|
</para>
|
||||||
|
<section>
|
||||||
|
<title>Source paths</title>
|
||||||
|
<para>
|
||||||
|
Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is <filename>./myfile</filename> gets stored into <filename>/nix/store/.......</filename> We want to understand how is the store path generated for such a file:
|
||||||
|
</para>
|
||||||
|
<screen>$ echo mycontent > myfile</screen>
|
||||||
|
<para>
|
||||||
|
I remind you, the simplest derivation you can write has a <literal>name</literal>, a <literal>builder</literal> and the <literal>system</literal>:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/derivation-simple.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Now inspect the .drv to see where is <filename>./myfile</filename> being stored:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/derivation-simple-contents.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Great, how did nix decide to use <literal>xv2iccirbrvklck36f1g7vldn5v58vck</literal> ? Keep looking at the nix comments.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
<emphasis role="bold">Note:</emphasis> doing <command>nix-store --add myfile</command> will store the file in the same store path.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<section>
|
||||||
|
<title>Step 1, compute the hash of the file</title>
|
||||||
|
<para>
|
||||||
|
The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/myfile-hash.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Or:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/myfile-hash-alternate.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Step 2, build the string description</title>
|
||||||
|
<para>
|
||||||
|
Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
|
||||||
|
</para>
|
||||||
|
<screen>$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str</screen>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Step 3, compute the final hash</title>
|
||||||
|
<para>
|
||||||
|
Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/myfile-final-hash.txt" parse="text" /></screen>
|
||||||
|
</section>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Output paths</title>
|
||||||
|
<para>
|
||||||
|
Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path <literal>hs0yi5n5nw6micqhy8l1igkbhqdkzqa1</literal>. This is because the out path only depends on inputs.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is <literal>output:out</literal>. In case of multiple outputs, we may have different <literal>output:<id></literal>.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/output-path-replace-empty.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
The <literal>myout.drv</literal> is the .drv state in which nix is when computing the out path for our derivation:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/myout-drv-hash.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Then nix puts that out path in the .drv, and that's it.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
In other words, you get a final .drv where every other .drv path is replaced by its hash.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Fixed-output paths</title>
|
||||||
|
<para>
|
||||||
|
Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
A derivation can take three special attributes: <literal>outputHashMode</literal>, <literal>outputHash</literal> and <literal>outputHashAlgo</literal> which are well documented in the <link xlink:href="http://nixos.org/nix/manual/#ssec-derivation">nix manual</link>.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
The builder must create the out path and make sure its hash is the same as the one declared with <literal>outputHash</literal>.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
Let's say our builder should create a file whose contents is <literal>mycontent</literal>:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/mycontent.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/bar-derivation.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
What nix does is to create an intermediate string representation of the fixed-output content:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/mycontent-string-representation.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Then proceed as it was a normal derivation output path:
|
||||||
|
</para>
|
||||||
|
<screen><xi:include href="./18/myfile-string-hash.txt" parse="text" /></screen>
|
||||||
|
<para>
|
||||||
|
Hence, the store path only depends on the declared fixed-output hash.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Conclusion</title>
|
||||||
|
<para>
|
||||||
|
There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.
|
||||||
|
</para>
|
||||||
|
<para>
|
||||||
|
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<title>Next pill</title>
|
||||||
|
<para>
|
||||||
|
...we will introduce <literal>stdenv</literal>. In the previous pills we rolled our own <literal>mkDerivation</literal> convenience function for wrapping the builtin derivation, but the <literal>nixpkgs</literal> repository also has its own convenience functions for dealing with <package>autotools</package> projects and other build systems.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
</section>
|
||||||
</chapter>
|
</chapter>
|
||||||
|
|
4
pills/18/bar-derivation.txt
Normal file
4
pills/18/bar-derivation.txt
Normal file
|
@ -0,0 +1,4 @@
|
||||||
|
$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
|
||||||
|
Derive(
|
||||||
|
[("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
|
||||||
|
...
|
6
pills/18/derivation-simple-contents.txt
Normal file
6
pills/18/derivation-simple-contents.txt
Normal file
|
@ -0,0 +1,6 @@
|
||||||
|
./myfile./myfileterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
|
||||||
|
Derive(
|
||||||
|
[("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
|
||||||
|
, []
|
||||||
|
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
|
||||||
|
, "x86_64-linux"
|
3
pills/18/derivation-simple.txt
Normal file
3
pills/18/derivation-simple.txt
Normal file
|
@ -0,0 +1,3 @@
|
||||||
|
$ nix-repl
|
||||||
|
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
|
||||||
|
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
|
3
pills/18/mycontent-string-representation-hash.txt
Normal file
3
pills/18/mycontent-string-representation-hash.txt
Normal file
|
@ -0,0 +1,3 @@
|
||||||
|
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
|
||||||
|
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
|
||||||
|
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
|
4
pills/18/mycontent-string-representation.txt
Normal file
4
pills/18/mycontent-string-representation.txt
Normal file
|
@ -0,0 +1,4 @@
|
||||||
|
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
|
||||||
|
$ sha256sum mycontent.str
|
||||||
|
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str
|
||||||
|
|
5
pills/18/mycontent.txt
Normal file
5
pills/18/mycontent.txt
Normal file
|
@ -0,0 +1,5 @@
|
||||||
|
$ echo mycontent > myfile
|
||||||
|
$ sha256sum myfile
|
||||||
|
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
|
||||||
|
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
|
||||||
|
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
|
2
pills/18/myfile-final-hash.txt
Normal file
2
pills/18/myfile-final-hash.txt
Normal file
|
@ -0,0 +1,2 @@
|
||||||
|
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
|
||||||
|
xv2iccirbrvklck36f1g7vldn5v58vck
|
2
pills/18/myfile-hash-alternate.txt
Normal file
2
pills/18/myfile-hash-alternate.txt
Normal file
|
@ -0,0 +1,2 @@
|
||||||
|
$ nix-store --dump myfile|sha256sum
|
||||||
|
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
|
2
pills/18/myfile-hash.txt
Normal file
2
pills/18/myfile-hash.txt
Normal file
|
@ -0,0 +1,2 @@
|
||||||
|
$ nix-hash --type sha256 myfile
|
||||||
|
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
|
3
pills/18/myfile-string-hash.txt
Normal file
3
pills/18/myfile-string-hash.txt
Normal file
|
@ -0,0 +1,3 @@
|
||||||
|
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
|
||||||
|
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
|
||||||
|
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
|
5
pills/18/myout-drv-hash.txt
Normal file
5
pills/18/myout-drv-hash.txt
Normal file
|
@ -0,0 +1,5 @@
|
||||||
|
$ sha256sum myout.drv
|
||||||
|
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
|
||||||
|
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
|
||||||
|
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
|
||||||
|
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
|
2
pills/18/output-path-replace-empty.txt
Normal file
2
pills/18/output-path-replace-empty.txt
Normal file
|
@ -0,0 +1,2 @@
|
||||||
|
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
|
||||||
|
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
|
|
@ -80,7 +80,7 @@
|
||||||
<para>
|
<para>
|
||||||
So we ran the <literal>configurePhase</literal> function and <literal>buildPhase</literal> function and they worked. These bash functions should be self-explanatory, you can read the code in the <filename>setup</filename> file.
|
So we ran the <literal>configurePhase</literal> function and <literal>buildPhase</literal> function and they worked. These bash functions should be self-explanatory, you can read the code in the <filename>setup</filename> file.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
</section>
|
</section>
|
||||||
<section>
|
<section>
|
||||||
<title>How is the setup file built</title>
|
<title>How is the setup file built</title>
|
||||||
|
@ -108,7 +108,7 @@
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
<screen><xi:include href="./19/stdenv-hello-build.txt" parse="text" /></screen>
|
<screen><xi:include href="./19/stdenv-hello-build.txt" parse="text" /></screen>
|
||||||
|
|
||||||
</section>
|
</section>
|
||||||
<section>
|
<section>
|
||||||
<title>The stdenv.mkDerivation builder</title>
|
<title>The stdenv.mkDerivation builder</title>
|
||||||
|
@ -145,7 +145,7 @@
|
||||||
<para>
|
<para>
|
||||||
Last bit, the <literal>unpackPhase</literal> in the setup is used to unpack the sources and enter the directory, again like we did in our old builder.
|
Last bit, the <literal>unpackPhase</literal> in the setup is used to unpack the sources and enter the directory, again like we did in our old builder.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section>
|
<section>
|
||||||
|
@ -172,7 +172,7 @@
|
||||||
<para>
|
<para>
|
||||||
Really, take your time to read that file. Don't forget that juicy docs are also available in the <link xlink:href="http://nixos.org/nixpkgs/manual/#chap-stdenv">nixpkgs manual</link>.
|
Really, take your time to read that file. Don't forget that juicy docs are also available in the <link xlink:href="http://nixos.org/nixpkgs/manual/#chap-stdenv">nixpkgs manual</link>.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
</section>
|
</section>
|
||||||
<section>
|
<section>
|
||||||
<title>Next pill...</title>
|
<title>Next pill...</title>
|
||||||
|
@ -180,6 +180,6 @@
|
||||||
<para>
|
<para>
|
||||||
...we will talk about how to add dependencies to our packages, <literal>buildInputs</literal>, <literal>propagatedBuildInputs</literal> and <literal>setup</literal> hooks. These three concepts are at the base of the current <literal>nixpkgs</literal> packages composition.
|
...we will talk about how to add dependencies to our packages, <literal>buildInputs</literal>, <literal>propagatedBuildInputs</literal> and <literal>setup</literal> hooks. These three concepts are at the base of the current <literal>nixpkgs</literal> packages composition.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
</section>
|
</section>
|
||||||
</chapter>
|
</chapter>
|
||||||
|
|
Loading…
Reference in a new issue