From a66fd074cf4b69ea7105ab8dea28903ecc7a8447 Mon Sep 17 00:00:00 2001 From: Peter Dragos Date: Fri, 9 Feb 2024 13:47:48 +0200 Subject: [PATCH] Non-content edits to pills 9-14 (#195) * grammar and readability edits Co-authored-by: Henrik --- pills/09-automatic-runtime.xml | 206 +++++++-------- pills/10-developing-with-nix-shell.xml | 168 +++++++------ pills/11-garbage-collector.xml | 289 +++++++++++---------- pills/12-inputs-design-pattern.xml | 322 ++++++++++++++++-------- pills/13-callpackage-design-pattern.xml | 228 +++++++++++++---- pills/14-override-design-pattern.xml | 125 ++++++--- 6 files changed, 827 insertions(+), 511 deletions(-) diff --git a/pills/09-automatic-runtime.xml b/pills/09-automatic-runtime.xml index f236cc5..a8d370d 100644 --- a/pills/09-automatic-runtime.xml +++ b/pills/09-automatic-runtime.xml @@ -9,13 +9,13 @@ Welcome to the 9th Nix pill. In the previous 8th pill we wrote a generic builder - for autotools projects. We feed build dependencies, a source tarball, and - we get a Nix derivation as a result. + for autotools projects. We fed in build dependencies and a source tarball, and + we received a Nix derivation as a result. - Today we stop by the GNU hello world program to analyze build and runtime - dependencies, and enhance the builder in order to avoid unnecessary runtime + Today we stop by the GNU hello program to analyze build and runtime + dependencies, and we enhance our builder to eliminate unnecessary runtime dependencies. @@ -23,26 +23,26 @@ Build dependencies - Let's start analyzing build dependencies for our GNU hello world package: + Let's start analyzing build dependencies for our GNU hello package: - It has exactly the derivations referenced in the derivation - function, nothing more, nothing less. Some of them might not be used at - all, however given that our generic mkDerivation function always pulls + It has precisely the derivations referenced in the derivation function; + nothing more, nothing less. Of course, we may not use some of them at all. + However, given that our generic mkDerivation function always pulls such dependencies (think of it like build-essential - of Debian), for every package you build from now on, you will have these - packages in the nix store. + from Debian), we will already have these packages in the nix store for any future packages that + need them. - Why are we looking at .drv files? Because the hello.drv file is the - representation of the build action to perform in order to build the hello - out path, and as such it also contains the input derivations needed to be - built before building hello. + Why are we looking at .drv files? Because the hello.drv + file is the representation of the build action that builds the hello + out path. As such, it contains the input derivations needed before building + hello. @@ -50,23 +50,29 @@ Digression about NAR files - NAR is the Nix ARchive. First question: why not tar? Why another archiver? - Because commonly used archivers are not deterministic. They add padding, - they do not sort files, they add timestamps, etc.. Hence NAR, a very - simple deterministic archive format being used by Nix for deployment. - NARs are also used extensively within Nix itself as we'll see below. + The NAR format is the "Nix ARchive". This format was designed due to + existing archive formats, such as tar, being insufficient. + Nix benefits from deterministic build tools, but commonly used archivers + lack this property: they add padding, they do not sort files, they add timestamps, + and so on. This can result in directories containing bit-identical files turning into + non-bit-identical archives, which leads to different hashes. - For the rationale and implementation details you can find more in the + Thus the NAR format was developed as a simple, deterministic + archive format. NARs are used extensively within Nix, as we will + see below. + + + + For more rationale and implementation details behind NAR see Dolstra's PhD Thesis. - To create NAR archives, it's possible to use + To create NAR archives from store paths, we can use nix-store --dump and - nix-store --restore. Those two commands work - regardless of /nix/store. + nix-store --restore. @@ -74,83 +80,72 @@ Runtime dependencies - Something is different for runtime dependencies however. Build - dependencies are automatically recognized by Nix once they are used in - any derivation call, but we never specify what are the - runtime dependencies for a derivation. + We now note that Nix automatically recognized build dependencies once our + derivation call referred to them, but we never specified the + runtime dependencies. - There's really no black magic involved. It's something that at first glance - makes you think "no, this can't work in the long term", but at the same - time it works so well that a whole operating system is built on top of - this magic. - - - - In other words, Nix automatically computes all the runtime dependencies - of a derivation, and it's possible thanks to the hash of the store paths. - - - - Steps: + Nix handles runtime dependencies for us automatically. The technique it uses + to do so may seem fragile at first glance, but it works so well that the NixOS + operating system is built off of it. The underlying mechanism relies on the + hash of the store paths. It proceeds in three steps: - Dump the derivation as NAR, a serialization of the derivation output. - Works fine whether it's a single file or a directory. + Dump the derivation as a NAR. Recall that this is a serialization of + the derivation output -- meaning this works fine whether the output + is a single file or a directory. - For each build dependency .drv and its relative out path, search the - contents of the NAR for this out path. + For each build dependency .drv and it's relative out path, + search the contents of the NAR for this out path. - If found, then it's a runtime dependency. + If the path is found, then it's a runtime dependency. - You really do get all the runtime dependencies; that's why Nix - deployments are so easy. + The snippet below shows the dependencies for hello. - Ok glibc and gcc. Well, gcc really should not be a runtime dependency! + We see that glibc and gcc are runtime dependencies. + Intuitively, gcc shouldn't be in this list! Displaying the + printable strings in the hello binary shows that the out path + of gcc does indeed appear: - Oh Nix added gcc because its out path is mentioned in the "hello" binary. - Why is that? That's the - ld rpath. - It's the list of directories where libraries can be found at runtime. In - other distributions, this is usually not abused. But in Nix, we have to - refer to particular versions of libraries, thus the rpath has an - important role. + This is why Nix added gcc. But why is that path present in the + first place? The answer is that it is the ld rpath: the list of + directories where libraries can be found at runtime. In other distributions, + this is usually not abused. But in Nix, we have to refer to particular versions + of libraries, and thus the rpath has an important role. - The build process adds that gcc lib path thinking it may be useful at - runtime, but really it's not. How do we get rid of it? Nix authors have - written another magical tool called - patchelf, which - is able to reduce the rpath to the paths that are really used by the - binary. + The build process adds the gcc lib path thinking it may be useful + at runtime, but this isn't necessary. To address issues like these, Nix provides + a tool called patchelf, + which reduces the rpath to the paths that are actually used by the binary. - Even after reducing the rpath, the hello binary would still - depend upon gcc because of some debugging information. This + Even after reducing the rpath, the hello binary would still + depend upon gcc because of some debugging information. This unnecesarily increases the size of our runtime dependencies. We'll explore how strip @@ -162,48 +157,49 @@ Another phase in the builder - We will add a new phase to our autotools builder. The builder has these + We will add a new phase to our autotools builder. The builder has six phases already: - First the environment is set up + The "environment setup" phase - Unpack phase: we unpack the sources in the current directory - (remember, Nix changes dir to a temporary directory first) + The "unpack phase": we unpack the sources in the current directory + (remember, Nix changes to a temporary directory first) - Change source root to the directory that has been unpacked + The "change directory" phase, where we change source root to the + directory that has been unpacked - Configure phase: ./configure + The "configure" phase: ./configure - Build phase: make + The "build" phase: make - Install phase: make install + The "install" phase: make install - We add a new phase after the installation phase, which we call - fixup phase. At the end of the - builder.sh follows: + Now we will add a new phase after the installation phase, which we call + the "fixup" phase. At the end of the + builder.sh, we append: @@ -211,36 +207,40 @@ That is, for each file we run patchelf --shrink-rpath and strip. Note that we used two new commands here, - find and patchelf. - Exercise: These two - deserve a place in baseInputs of - autotools.nix as findutils and - patchelf. + find and patchelf. These must be + added to our derivation. - Rebuild hello.nix and...: + Exercise: Add findutils + and patchelf to the baseInputs of + autotools.nix. + + + + Now, we rebuild hello.nix:nd...: - ...only glibc is the runtime dependency. Exactly what we wanted. + and we see that glibc is a runtime dependency. This is + exactly what we wanted. - The package is self-contained, copy its closure on another machine and - you will be able to run it. Remember, only a very few components under - the /nix/store are required to + The package is self-contained. This means that we can copy its closure onto + another machine and we will be able to run it. Remember, only a very few + components under the /nix/store are required to run nix. - The hello binary will use that exact version of glibc library and - interpreter, not the system one: + The hello binary will use the exact version of glibc + library and interpreter referred to in the binary, rather than the system one: - Of course, the executable runs fine as long as everything is under the + Of course, the executable will run fine as long as everything is under the /nix/store path. @@ -249,21 +249,19 @@ Conclusion - Short post compared to previous ones as I'm still on vacation, but I hope - you enjoyed it. Nix provides tools with cool features. In particular, Nix - is able to compute all runtime dependencies automatically for us. This is - not limited to only shared libraries, but also referenced executables, - scripts, Python libraries etc.. + We saw some of the tools Nix provides, along with their features. + In particular, we saw how Nix is able to compute runtime dependencies + automatically. This is not limited to only shared libraries, + but can also referenced executables, scripts, Python libraries, and so + forth. - This makes packages self-contained, ensuring (apart data and - configuration) that copying the runtime closure on another machine is - sufficient to run the program. That's what allows running programs without - installation using nix-shell - or + Approaching builds in this way makes packages self-contained, ensuring + (apart from data and configuration) that copying the runtime closure onto + another machine is sufficient to run the program. This enables us to run programs + without installation using nix-shell, and forms the basis for reliable deployment in the cloud. - All with one tool. @@ -271,11 +269,13 @@ Next pill - ...we will introduce nix-shell. With nix-build we always build - derivations from scratch: the source gets unpacked, configured, built - and installed. But this may take a long time, think of WebKit. What if we - want to apply some small changes and compile incrementally instead, yet - keeping a self-contained environment similar to nix-build? + The next pill will introduce nix-shell. With + nix-build, we've always built derivations from + scratch: the source gets unpacked, configured, built, and installed. + But this can take a long time for large packages. What if we want to + apply some small changes and compile incrementally instead, yet still + want to keep a self-contained environment similar to nix-build? + nix-shell enables this. diff --git a/pills/10-developing-with-nix-shell.xml b/pills/10-developing-with-nix-shell.xml index 19c119d..7c19da7 100644 --- a/pills/10-developing-with-nix-shell.xml +++ b/pills/10-developing-with-nix-shell.xml @@ -9,85 +9,101 @@ Welcome to the 10th Nix pill. In the previous 9th pill we saw - one of the powerful features of nix, automatic discovery of runtime - dependencies and finalized the GNU hello world package. + one of the powerful features of Nix: automatic discovery of runtime + dependencies. We also finalized the GNU hello package. - Having returned from vacation, we want to hack a little the GNU hello - world program. The nix-build tool allows for an isolated environment - while building the derivation. Additionally, we'd like the same - isolation in order to modify some source files of the project. + In this pill, we will introduce the nix-shell tool + and use it to hack on the GNU hello program. We will + see how nix-shell gives us an isolated environment + while we modify the source files of the project, similar to how + nix-build gave us an isolated environment while building + the derivation. + + + + Finally, we will modify our builder to work more ergonomically + with a nix-shell-focused workflow.
- What's nix-shell + What is <command>nix-shell</command>? The nix-shell - tool drops us in a shell by setting up the necessary environment - variables to hack on a derivation. It does not build the derivation, it + tool drops us in a shell after setting up the environment variables necessary + to hack on a derivation. It does not build the derivation; it only serves as a preparation so that we can run the build steps manually. - I remind you, in a nix environment you don't have access to libraries and - programs unless you install them with nix-env. However installing - libraries with nix-env is not good practice. We prefer to have isolated - environments for development. + Recall that in a nix environment, we don't have access to libraries or + programs unless they have been installed with nix-env. + However, installing libraries with nix-env is not + good practice. We prefer to have isolated environments for development, which + nix-shell provides for us. + + + + We can call nix-shell on any Nix expression which + returns a derivation, but the resulting bash shell's + PATH does not have the utilities we want: - First thing to notice, we call nix-shell on a nix - expression which returns a derivation. We then enter a new bash shell, - but it's really useless. We expected to have the GNU hello world build - inputs available in PATH, including GNU make, but it's not the case. + This shell is rather useless. It would be reasonable to expect that the GNU + hello build inputs are available in PATH, including + GNU make, but this is not the case. - But, we have the environment variables that we set in the derivation, + However, we do have the environment variables that we set in the derivation, like $baseInputs, $buildInputs, - $src and so on. + $src, and so on. - That means we can source our builder.sh, and it will - build the derivation. You may get an error in the installation phase, - because the user may not have the permission to write to - /nix/store: + This means the we can source our + builder.sh, and it will build the derivation. + You may get an error in the installation phase, because your user may + not have the permission to write to /nix/store: - It didn't install, but it built. Things to notice: + The derivation didn't install, but it did build. Note the following: - We sourced builder.sh, therefore it ran all the steps including - setting up the PATH for us. + We sourced builder.sh and it ran all of the build + steps, including setting up the PATH for us. - The working directory is no more a temp directory created by nix-build, but the current directory. Therefore, hello-2.10 has been unpacked there. + The working directory is no longer a temp directory created by + nix-build, but is instead the directory in which + we entered the shell. Therefore, hello-2.10 has + been unpacked in the current directory. - We're able to cd into hello-2.10 and type - make, because now it's available. + We are able to cd into hello-2.10 and type + make, because make is now available. - In other words, nix-shell drops us in a shell with the - same (or almost) environment used to run the builder! + The take-away is that nix-shell drops us in a shell with the + same (or very similar) environment used to run the builder.
@@ -95,44 +111,50 @@ A builder for nix-shell - The previous steps are a bit annoying of course, but we can improve our - builder to be more nix-shell friendly. + The previous steps require some manual commands to be run and are not + optimized for a workflow centered on nix-shell. We + will now improve our builder to be more nix-shell friendly. - First of all, we were able to source builder.sh - because it was in our current directory, but that's not nice. We want the - builder.sh that is stored in the nix store, the one - that would be used by nix-build. To do so, the right - way is to pass the usual environment variable through the derivation. + There are a few things that we would like to change. - Note: $builder is - already defined, but it's the bash executable, not our + First, when we sourced the builder.sh + file, we obtained the file in the currenty directory. What we really wanted + was the builder.sh that is stored in the nix store, + as this is the file that would be used by nix-build. + To achieve this, the correct technique is to pass an environment variable + through the derivation. (Note that $builder is + already defined, but it points to the bash executable rather than our builder.sh. Our builder.sh is - an argument to bash. + passed as an argument to bash.) - Second, we don't want to run the whole builder, we only want it to setup - the necessary environment for manually building the project. So we'll - write two files, one for setting up the environment, and the real - builder.sh that runs with - nix-build. + Second, we don't want to run the whole builder: we only want to setup + the necessary environment for manually building the project. Thus, we + can break builder.sh into two files: a + setup.sh for setting up the environment, and + the real builder.sh that nix-build + expects. - Additionally, we'll wrap the phases in functions, it may be useful, and - move the set -e to the builder instead of the setup. The - set -e is annoying in nix-shell. + During our refactoring, we ill wrap the build phases in functions to + give more structure to our design. Additionally, we'll move the + set -e to the builder file instead of the setup file. + The set -e is annoying in nix-shell, + as it will terminate the shell if an error is encountered (such as + a mistyped command.) Here is our modified autotools.nix. Noteworthy is the setup = ./setup.sh; attribute in the derivation, which adds setup.sh to the nix store and - as usual, adds a $setup environment variable in the builder. + correspondingly adds a $setup environment variable in the builder. @@ -140,25 +162,25 @@ Thanks to that, we can split builder.sh into setup.sh and builder.sh. What - builder.sh does is sourcing $setup and - calling the genericBuild function. Everything else is just - some bash changes. + builder.sh does is source + $setup and call the genericBuild function. + Everything else is just some changes to the bash script. - Here is the modified builder.sh. + Here is the modified builder.sh: - Here is the newly added setup.sh. + Here is the newly added setup.sh: - Finally, here is hello.nix. + Finally, here is hello.nix: @@ -170,17 +192,18 @@ - Now you can run, for example, unpackPhase which unpacks + Now, for example, you can run unpackPhase which unpacks $src and enters the directory. And you can run commands - like ./configure, make etc. + like ./configure, make, and so forth manually, or run phases with their respective functions. - It's that straightforward, nix-shell builds the .drv file - and its input dependencies, then drops into a shell by setting up the - environment variables necessary to build the .drv, in particular those - passed to the derivation function. + The process is that straightforward. nix-shell builds the + .drv file and its input dependencies, then drops into a shell + by setting up the environment variables necessary to build the .drv. + In particular, the environment variables in the shell match those passed + to the derivation function. @@ -188,13 +211,14 @@ Conclusion - With nix-shell we're able to drop into an isolated - environment for developing a project, with the necessary dependencies - just like nix-build does. Additionally, we can build and - debug the project manually, step by step like you would do in any other - operating system. Note that we never installed gcc, - make, etc. system-wide. These tools and libraries are - available per-build. + With nix-shell we are able to drop into an isolated + environment suitable for developing a project. This environment provides the necessary + dependencies for the development shell, similar to how + nix-build providesthe necessary dependencies to a builder. + Additionally, we can build and debug the project manually, executing step-by-step + like we would in any other operating system. Note that we never installed tools + such gcc or make system-wide; these tools + and libraries are isolated and available per-build. @@ -202,9 +226,9 @@ Next pill - ...we will clean up the nix store. We wrote and built derivations, added - stuff to nix store, but until now we never worried about cleaning up the - used space in the store. It's time to collect some garbage. + In the next pill, we will clean up the nix store. We have written and built + derivations which add to the nix store, but until now we haven't worried + about cleaning up the used space in the store. diff --git a/pills/11-garbage-collector.xml b/pills/11-garbage-collector.xml index 9f08d85..9698094 100644 --- a/pills/11-garbage-collector.xml +++ b/pills/11-garbage-collector.xml @@ -4,84 +4,70 @@ version="5.0" xml:id="garbage-collector"> - Garbage Collector + The Garbage Collector Welcome to the 11th Nix pill. In the previous - 10th pill we managed to - obtain a self-contained development environment for a project. The concept - is that nix-build is able to build a derivation - in isolation, while nix-shell is able to drop us in a - shell with (almost) the same environment used by nix-build. - This allows us to debug, modify and manually build software. + 10th pill, we drew a + a parallel between the isolated build environment provided by + nix-build and the isolated development shell provided by + nix-shell. Using nix-shell allowed us + to debug, modify, and manually build software using an environment that + is almost identical to the one provided by nix-build. - Today we stop packaging and look at a mandatory nix component, the garbage - collector. When using nix tools, often derivations are built. This include - both .drv files and out paths. These artifacts go in the nix store, but - we've never cared about deleting them until now. + Today, we will stop focusing on packaging and instead look at a critical + component of Nix: the garbage collector. When we use Nix tools, we are + often building derivations. This includes .drv files as well as + out paths. These artifacts go in the Nix store and take up space in our storage. + Eventually we may wish to free up some space by removing derivations we no longer + need. This is the focus of the 11th pill. + + By default, Nix takes a relatively conservative approach when automatically + deciding which derivations are "needed". In this pill, we will also see + a technique to conduct more destructive upgrade and deletion operations.
- How does it work + How does garbage collection work? - Other package managers, like dpkg, have ways of - removing unused software. Nix is much more precise in its garbage - collection compared to these other systems. + Programming languages with garbage collectors use the concept of a set of + "garbage collector (or 'GC') roots" to keep track of "live" objects. + A GC root is an object that is always considered "live" (unless explicitly + removed as GC root). The garbage collection process starts from the GC roots + and proceeds by recursively marking object references as "live". All other + objects can be collected and deleted. - I bet with dpkg, rpm or similar - traditional packaging systems, you end up having some unnecessary - packages installed or dangling files. With nix this does not happen. - - - - - How do we determine whether a store path is still needed? The same way - programming languages with a garbage collector decide whether an object - is still alive. + Instead of objects, Nix's garbage collection operates on store paths, with the GC roots themselves being store paths. +. This approach is much mode principled than traditional package + managers such as dpkg or rpm, which may + leave around unused packages or dangling files. - Programming languages with a garbage collector have an important concept - in order to keep track of live objects: GC roots. A GC root is an object - that is always alive (unless explicitly removed as GC root). All objects - recursively referred to by a GC root are live. + The implementation is very simple and transparent to the user. The primary + GC roots are stored under /nix/var/nix/gcroots. If there + is a symlink to a store path, then linked store path is a GC root. - Therefore, the garbage collection process starts from GC roots, and - recursively mark referenced objects as live. All other objects can be - collected and deleted. + Nix allows this directory to have subdirectories: it will simply recursively + traverse the subdirectories in search of symlinks to store paths. When + a symlink is encountered, it's target is added to the list of live store + paths. - In Nix there's this same concept. Instead of being objects, of course, - GC roots are store paths. - The implementation is very simple and transparent to the user. GC roots - are stored under /nix/var/nix/gcroots. If there's a - symlink to a store path, then that store path is a GC root. - - - - Nix allows this directory to have subdirectories: it will simply recurse - directories in search of symlinks to store paths. - - - - So we have a list of GC roots. At this point, deleting dead store paths - is as easy as you can imagine. We have the list of all live store paths, - hence the rest of the store paths are dead. - - - - In particular, Nix first moves dead store paths to - /nix/store/trash which is an atomic operation. - Afterwards, the trash is emptied. + In summary, Nix maintains a list of GC roots. These roots can then be + used to compute a list of all live store paths. Any other store + paths are considered dead. Deleting these paths is now straightforward. + Nix first moves dead store paths to /nix/store/trash, + which is an atomic operation. Afterwards, the trash is emptied.
@@ -89,50 +75,45 @@ Playing with the GC - Before playing with the GC, first run the + Before we begin we first run the nix garbage collector - once, so that we have a clean playground for our experiments: + so that we have a clean setup for our experiments: - Perfect, if you run it again it won't find anything new to delete, as - expected. + If we run the garbage collector again it won't find anything new to delete, + as we expect. After running the garbage collector, the nix store only contains + paths with references from the GC roots. - What's left in the nix store is everything being referenced from the GC - roots. - - - - Let's install for a moment bsd-games: + We now install a new program, bsd-games, inspect its + store path, and examine it's GC root. The nix-store -q --roots + command is used to query the GC roots that refer to a given derivation. In this + case, our current user environment refers to bsd-games: - The nix-store command can be used to query the GC roots that refer to a - given derivation. In this case, our current user environment does refer - to bsd-games. - - - - Now remove it, collect garbage and note that bsd-games is still in the nix - store: + Now we remove it and run the garbage collector, and note that bsd-games + is still in the nix store: - This is because the old generation is still in the nix store because it's a GC root. As we'll see below, all profiles and their generations are GC roots. + The old generation is still in the nix store because it is a GC root. + As we will see below, all profiles and their generations are automatically + GC roots. - Removing a GC root is simple. Let's try deleting the generation that - refers to bsd-games, collect garbage, and note that now bsd-games is no - longer in the nix store: + Removing a GC root is simple. In our case, we delete the generation that + refers to bsd-games, run the garbage collector, and note + that bsd-games is no longer in the nix store: @@ -145,55 +126,55 @@ - However we removed the link from + Note that we removed the link from /nix/var/nix/profiles, not from - /nix/var/nix/gcroots. Turns out, Nix - also treats /nix/var/nix/profiles as a GC root. - That is very handy. It means any profile and its generations are GC roots. - There are other paths that are taken into account as well, for example /run/booted-system on NixOS. - The command nix-store --gc --print-roots prints all the paths considered before performing a GC. + /nix/var/nix/gcroots. In addition to the latter, + Nix treats /nix/var/nix/profiles as a GC root. + This is useful because it means that any profile and its generations + are GC roots. Other path are considered GC roots as well; for example, + /run/booted-system on NixOS. + The command nix-store --gc --print-roots prints all + paths considered as GC roots when running the garbage collector. - - It's as simple as that, anything under - /nix/var/nix/gcroots is a GC root. And anything not - being garbage collected is because it's referred from one of the GC roots. -
Indirect roots - Remember that building the GNU hello world package with + Recall that building the GNU hello package with nix-build produces a result - symlink in the current directory. Despite the collected garbage done - above, the hello program is still working: therefore - it has not been garbage collected. Clearly, since there's no other - derivation that depends upon the GNU hello world package, it must be a + symlink in the current directory. Despite the garbage collection done + above, the hello program is still working. Therefore, + it has not been garbage collected. Since there is no other + derivation that depends upon the GNU hello package, it must be a GC root. - In fact, nix-build automatically adds the result - symlink as a GC root. Yes, not the built derivation, but the symlink. - These GC roots are added under - /nix/var/nix/gcroots/auto. + In fact, nix-build automatically adds the + result symlink as a GC root. Note that this + is not the built derivation, but the symlink itself. These GC roots + are added under /nix/var/nix/gcroots/auto. - Don't care about the name of the symlink. What's important is that a - symlink exists that point to /home/nix/result. This - is called an indirect GC root. That is, - the GC root is effectively specified outside of - /nix/var/nix/gcroots. Whatever - result points to, it will not be garbage collected. + The name of the GC root symlink is not important to us at this time. + What is important is that such a symlink exists and points to + /home/nix/result. This is called an + indirect GC root. A GC root is + considred indirect if it's specification is outside of + /nix/var/nix/gcroots. In this case, this means + that the target of the result symlink will + not be garbage collected. - How do we remove the derivation then? There are two possibilities: + To remove a deriviation considered "live" by an indirect GC root, + there are two possibilities: @@ -211,15 +192,15 @@ - In the first case, the derivation will be deleted from the nix store, and - result becomes a dangling symlink. In the second - case, the derivation is removed as well as the indirect root in + In the first case, the derivation will be deleted from the nix store during + garbage collection, and result becomes a dangling symlink. + In the second case, the derivation is removed as well as the indirect root in /nix/var/nix/gcroots/auto. Running nix-collect-garbage after deleting the GC root - or the indirect GC root, will remove the derivation from the store. + or the indirect GC root will remove the derivation from the store.
@@ -227,49 +208,64 @@ Cleanup everything - What's the main source of software duplication in the nix store? Clearly, - GC roots due to nix-build and profile generations. - Doing a nix-build results in a GC root for a build - that somehow will refer to a specific version of glibc, - and other libraries. After an upgrade, if that build is not deleted by - the user, it will not be garbage collected. Thus the old dependencies - referred to by the build will not be deleted either. + The main source of software duplication in the nix store comes from + GC roots, due to nix-build and profile generations. + Running nix-build results in a GC root for the build + that refers to a specific version of specific libaries, such as + glibc. After an upgrade, we must delete the previous build + if we want the garbage collector to remove the corresponding derivation, + as well as if we want old dependencies cleaned up. - Same goes for profiles. Manipulating the nix-env + The same holds for profiles. Manipulating the nix-env profile will create further generations. Old generations refer to old software, thus increasing duplication in the nix store after an upgrade. - What are the basic steps for upgrading and removing everything old, - including old generations? In other words, do an upgrade similar to other - systems, where they forget everything about the older state: + Other systems typically "forget" everything about their previous state after + an upgrade. With Nix, we can perform this type of upgrade (having Nix remove + all old derivations, including old generations), but we do so manually. + There are four steps to doing this: + + + + + First, we download a new version of the nixpkgs channel, which holds the + description of all the software. This is done via + nix-channel --update. + + + + + Then we upgrade our installed packages with nix-env -u. + This will bring us into a new generation with updated software. + + + + + Then we remove all the indirect roots generated by + nix-build: beware, as this will result in dangling + symlinks. A smarter strategy would also remove the target of those symlinks. + + + + + Finally, the -d option of + nix-collect-garbage is used to delete old generations + of all profiles, then collect garbage. After this, you lose the ability + to rollback to any previous generation. It is important to ensure the new + generation is working well before running this command. + + + + + The four steps are shown below: - - First, we download a new version of the nixpkgs channel, which holds the - description of all the software. Then we upgrade our installed packages - with nix-env -u. That will bring us into a fresh new - generation with all updated software. - - - - Then we remove all the indirect roots generated by - nix-build: beware, this will result in dangling - symlinks. You may be smarter and also remove the target of those symlinks. - - - - Finally, the -d option of - nix-collect-garbage is used to delete old generations - of all profiles, then collect garbage. After this, you lose the ability - to rollback to any previous generation. So make sure the new generation - is working well before running the command. -
@@ -277,15 +273,10 @@ Garbage collection in Nix is a powerful mechanism to cleanup your system. - The nix-store commands allow us to know why a certain derivation is in - the nix store. - - - - Cleaning up everything down to the oldest bit of software after an - upgrade seems a bit contrived, but that's the price of having multiple - generations, multiple profiles, multiple versions of software, thus - rollbacks etc.. The price of having many possibilities. + The nix-store commands allow us to know why a certain + derivation is present in the nix store, and whether or not it is eligible + for garbage collection. We also saw how to conduct more destructive deletion + and upgrade operations.
@@ -293,8 +284,8 @@ Next pill - ...we will package another project and introduce what I call the "inputs" - design pattern. We've only played with a single derivation until now, + In the next pill, we will package another project and introduce the "inputs" + design pattern. We've only played with a single derivation until now; however we'd like to start organizing a small repository of software. The "inputs" pattern is widely used in nixpkgs; it allows us to decouple derivations from the repository itself and increase customization diff --git a/pills/12-inputs-design-pattern.xml b/pills/12-inputs-design-pattern.xml index 97b9f08..3911467 100644 --- a/pills/12-inputs-design-pattern.xml +++ b/pills/12-inputs-design-pattern.xml @@ -4,236 +4,358 @@ version="5.0" xml:id="inputs-design-pattern"> - Inputs Design Pattern + Package Repositories and the Inputs Design Pattern - Welcome to the 12th Nix pill. In the previous 11th pill we stopped packaging and cleaned up the system with the garbage collector. + Welcome to the 12th Nix pill. In the previous 11th + pill, we stopped packaging and cleaned up the system with the garbage collector. - We'll be resuming packaging, and improving different aspects of it. We've only packaged a hello world program so far, but what if we want to create a repository of multiple packages? + This time, we will resume packaging and improve different aspects of it. We will also + demonstrate how to create a repository of multiple packages.
Repositories in Nix - Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. It's more like a consequence due to the need of organizing packages. + Package repositories in Nix arose naturally from the need to organize pacakges. + There is no preset directory structure or packaging policy prescribed by Nix itself; + Nix, as a full, functional programming language, is powerful enough to support + multiple different repository formats. + - Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional. - - - There is no preset directory structure or preset packaging policy. It's all about you and Nix. - - - The nixpkgs repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software. - - - Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course. + Over time, the nixpkgs repository evolved a particular + structure. This structure reflects the history of Nix as well as the design + patterns adopted by it's users as useful tools in building and organizing + packages. Below, we will examine some of these patterns in detail.
The single repository pattern - Before introducing the "inputs" pattern, we can start talking about another pattern first which I'd like to call "single repository" pattern. + Different operating system distributions have different opinions about how + package repositories should organized. Systems like Debian scatter packages + in several small repositories (which tends to make tracking interdependent + changes more difficult, and hinders contributions to the repositories), + while systems like Gentoo put all package descriptions in a single repository. - Systems like Debian scatter packages in several small repositories. This can make it hard to track interdependent changes and to contribute to new packages. + Nix follows the "single repository" pattern by placing all descriptions of all + packages into nixpkgs. + This approach has proven natural and attractive for new contributions. - Alternatively, systems like Gentoo put package descriptions all in a single repository. + For the rest of this pill, we will adopt the single repository pattern. The + natural implementation in Nix is to create a top-level Nix expression, followed + by one expression for each package. The top-level expression imports and combines + all package expressions in an attribute set mapping names to packages. - The nix reference for packages is nixpkgs, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions. - - - For the rest of this chapter, we will adopt the single repository technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs. - - - But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why nixpkgs is able to maintain such a big software repository in a giant attribute set. + In some programming languages, such an approach -- including every possible + package description in a single data structure -- would be untenable due + to the language needing to load the entire data structure into memory before + operating on it. Nix, however, is a lazy language and only evaluates what is + needed.
- Packaging graphviz + Packaging <code>graphviz</code> - We have packaged GNU hello world, imagine you would like to package something else for creating at least a repository of two projects :-) . I chose graphviz, which uses the standard autotools build system, requires no patching and dependencies are optional. + We have already packaged GNU hello. Next, we will package a + graph-drawing program called graphviz so that we can + create a repository containing multiple packages. The graphviz + package was selected because it uses the standard autotools build system and + requires no patching. It also has optional dependencies, which will give us + an opportunity to illustrate a technique to configure builds to a particular + situation. - Download graphviz from here. The graphviz.nix expression is straightforward: + First, we download graphviz from gitlab. The graphviz.nix expression is straightforward: - Build with nix-build graphviz.nix and you will get runnable binaries under result/bin. Notice how we reused the same autotools.nix of hello.nix. Let's create a simple png: + If we build the project with nix-build graphviz.nix, we will get runnable binaries under result/bin. Notice how we reused the same autotools.nix of hello.nix. + + + By default, graphviz does not compile with the ability to produce + png files. Thus, the derivation above will build a binary + supporting only the native output formats, as we see below: - Oh of course... graphviz doesn't know about png. It built only the output formats it supports natively, without using any extra library. + If we want to produce a png file with graphviz, we + must add it to our derivation. The place to do so is + in autotools.nix, where we created a + buildInputs variable that gets concatenated to + baseInputs. This is the exact reason for this variable: to + allow users of autotools.nix to add additional inputs + from package expressions. - Remember, in autotools.nix there's a buildInputs variable which gets concatenated to baseInputs. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions. - - - This 2.49 version of graphviz has several plugins to output png. For simplicity, we will use libgd. + Version 2.49 of graphviz has several plugins to output + png. For simplicity, we will use libgd.
- Digression about gcc and ld wrappers + Passing library information to <command>pkg-config</command> via environment + variables - The graphviz configure script uses pkg-config to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell pkg-config where to find pkg-config description files, in order to enable it to tell the configure script where to find headers and libraries. + The graphviz configuration script uses pkg-config + to specify which flags are passed to the compiler. Since there is no global location + for libraries, we need to tell pkg-config where to find + it's description files, which tell the configuration script where to find + headers and libraries. In classic POSIX systems, pkg-config just finds the .pc files of all installed libraries in system folders - like /usr/lib/pkgconfig, but we don't have that in - the nix sandbox. The nix way to educate pkg-config - about the existence of libraries is the environment variable - PKG_CONFIG_PATH. + like /usr/lib/pkgconfig. However, these files + are not present in the isolated environments presented to Nix. - What can we do about it? We can employ the same trick we did for PATH: automatically filling the variables from buildInputs. This is the relevant snippet of setup.sh: + As an alternative, we can inform pkg-config about + the location of libraries via the PKG_CONFIG_PATH + environment variable. We can populate this environment variable + using the same trick we used for PATH: + automatically filling the variables from buildInputs. + This is the relevant snippet of setup.sh: - Now adding derivations to buildInputs will add their lib/pkgconfig and bin paths automatically in setup.sh. + Now if we add derivations to buildInputs, their + lib/pkgconfig and bin paths + are automatically added in setup.sh.
+
- Completing graphviz with gd + Completing graphviz with <code>gd</code> - Finish the expression for graphviz with gd support (note the use of the with expression in buildInputs to avoid repeating pkgs): + Below, we finish the expression for graphviz with gd support. + Note the use of the with expression in buildInputs to avoid repeating pkgs: We add pkg-config to the derivation to make this tool - available for the configure script. As gd is a package + available for the configure script. As gd is a package with split outputs, - we need to add both the library- and development outputs. + we need to add both the library and development outputs. - Now you can create the png! + After building, graphviz can now create pngs.
The repository expression - Now that we have two packages, what's a good way to put them together in a single repository? We'll do something like nixpkgs does. With nixpkgs, we import it and then we pick derivations by accessing the giant attribute set. + Now that we have two packages, we want to combine them into a single repository. + To do so, we'll mimic what nixpkgs does: we will create + a single attribute set containing derivations. This attribute set can + then be imported, and deriviations can be selected by accessing the + top-level attribute set. - For us nixers, this is a good technique, because it abstracts from the file names. We don't refer to a package by REPO/some/sub/dir/package.nix but by importedRepo.package (or pkgs.package in our examples). + Using this technique we are able to abstract from the file names. + Instead of referring to a package by REPO/some/sub/dir/package.nix, + this technique allows us to select a derivation as + importedRepo.package (or pkgs.package + in our examples). - Create a default.nix in the current directory: + To begin, create a default.nix in the current directory: - Ready to use! Try it with nix repl: + This file is ready to use with nix repl: - With nix-build: + With nix-build, we can pass the -A option to + access an attribute of the set from the given .nix expression: - The -A argument is used to access an attribute of the set from the given .nix expression. + The default.nix file is special. When a directory + contains a default.nix file, it is used as the implict + nix expression of the directory. This, for example, allows use to run + nix-build -A hello without specifying + default.nix explicitly. - Important: why did we choose the default.nix? Because when a directory (by default the current directory) has a default.nix, that default.nix will be used (see import here). In fact you can run nix-build -A hello without specifying default.nix. - - - For pythoners, it is similar to __init__.py. - - - With nix-env, install the package into your user environment: + We can now use nix-env to install the package into our + user environment: - The -f option is used to specify the expression to use, in this case the current directory, therefore ./default.nix. + Taking a closer look at the above command, we see the following options: + + + + The -f option is used to specify the expression to use. In this case, + the expression is the ./default.nix of the current directory. + + + The -i option stands for "installation". + + + The -A is the same as above for nix-build. + + - The -i stands for installation. - - - The -A is the same as above for nix-build. - - - We reproduced the very basic behavior of nixpkgs. + We reproduced the very basic behavior of nixpkgs: combining + multiple derivations into a single, top-level attribute set.
The inputs pattern - After a long preparation, we finally arrived. I know you're having a big doubt in this moment. It's about the hello.nix and graphviz.nix. They are very, very dependent on nixpkgs: + The approach we've taken so far has a few problems: + - First big problem: they import nixpkgs directly. In autotools.nix instead we pass nixpkgs as an argument. That's a much better approach. - Second problem: what if we want a variant of graphviz without libgd support? - Third problem: what if we want to test graphviz with a particular libgd version? + + First, hello.nix and graphviz.nix are + dependent on nixpkgs, which they import directly. + A better approach would be to pass in nixpkgs as an argument, + as we did in autotools.nix. + + + + Second, we don't have a straightforward way to compile different variants + of the same software, such as graphviz with or without + libgd support. + + + + Third, we don't have a way to test graphviz + with a particular libgd version. + - The current answers to the above questions are: change the expression to match your needs (or change the callee to match your needs). + Until now, our approach to addressing the above problems has been inadequate + and required changing the nix expression to match our needs. With the + inputs pattern, we provide another answer: let the user + change the inputs of the expression. - With the inputs pattern, we decided to provide another answer: let the user change the inputs of the expression (or change the caller to pass different inputs). - - - By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case: + When we talk about "the inputs of an expression", we are referring to the + set of derivations needed to build that expression. In this case: - mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain. - libgd and its dependencies. + + mkDerivation from autotools. Recall + that mkDerivation has an implicit dependency on + the toolchain. + + + libgd and its dependencies. + - The src is also an input but it's pointless to change the source from the caller. For version bumps, in nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed). + The ./src directory is also an input, + but we wouldn't change the source from the caller. + In nixpkgs we prefer to write another expression + for version bumps (e.g. because patches or different inputs are needed). + - Goal: make package expressions independent of the repository. - - - How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for graphviz.nix, will make the derivation independent of the repository and customizable: + Our goal is to make package expressions independent of the repository. To + achieve this, we use functions to declare inputs for a derivation. For example, + with graphviz.nix, we make the following changes to make + the derivation independent of the repository and customizable: - I recall that "{...}: ..." is the syntax for defining functions accepting an attribute set as argument. + Recall that "{...}: ..." is the syntax for defining functions + accepting an attribute set as argument; the above snippet just defines a function. + + + + We made gd and its dependencies optional. If gdSupport + is true (which it is by default), we will fill buildInputs and + graphviz will be built with gd support. Otherwise, if + an attribute set is passed with gdSupport = false;, the build + will be completed without gd support. - We made gd and its dependencies optional. If gdSupport is true (by default), we will fill buildInputs and thus graphviz will be built with gd support, otherwise it won't. - - - Now back to default.nix: + Going back to back to default.nix, we modify our expression + to utilize the inputs pattern: - So we factorized the import of nixpkgs and mkDerivation, and also added a variant of graphviz with gd support disabled. The result is that both hello.nix (exercise for the reader) and graphviz.nix are independent of the repository and customizable by passing specific inputs. + We factorized the import of nixpkgs and + mkDerivation, and also added a variant of graphviz + with gd support disabled. The result is that both + hello.nix (left as an exercise for the reader) and + graphviz.nix are independent of the repository and + customizable by passing specific inputs. - If you wanted to build graphviz with a specific version of gd, it would suffice to pass gd = ...;. + If we wanted to build graphviz with a specific version of + gd, it would suffice to pass gd = ...;. - If you wanted to change the toolchain, you may pass a different mkDerivation function. + If we wanted to change the toolchain, we simply pass a different + mkDerivation function. - Clearing up the syntax: + Let's talk a closer look at the snippet and dissect the syntax: - In the end we return an attribute set from default.nix. With "let" we define some local variables. - We bring pkgs into the scope when defining the packages set, which is very convenient instead of typing everytime "pkgs". - We import hello.nix and graphviz.nix, which will return a function, and call it with a set of inputs to get back the derivation. - The "inherit x" syntax is equivalent to "x = x". So "inherit gd" here, combined to the above "with pkgs;" is equivalent to "gd = pkgs.gd". + The entire expression in default.nix + returns an attribute set with the keys hello, + graphviz, and graphvizCore. + + + + With "let", we define some local variables. + + + + We bring pkgs into the scope when defining the + package set. This saves us from having to type + pkgs" repeatedly. + + + + We import hello.nix and graphviz.nix, + which each return a function. We call the functions with a set of inputs to + get back the derivation. + + + + The "inherit x" syntax is equivalent to + "x = x". This means that the "inherit gd" + here, combined with the above "with pkgs;", + is equivalent to "gd = pkgs.gd". + - You can find the whole repository at the pill 12 gist. + The entire repository of this can be found at the pill 12 gist.
Conclusion - The "inputs" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatever else. Our package expressions are functions, don't think there's any magic in there. + The "inputs" pattern allows our expressions to be easily + customizable through a set of arguments. These arguments could be flags, + derivations, or any other customizations enabled by the nix language. + Our package expressions are simply functions: there is no extra magic present. - It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context. + The "inputs" pattern also makes the expressions + independent of the repository. Given that we pass all needed information + through arguments, it is possible to use these expressions in any other context.
Next pill - ...we will talk about the "callPackage" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level default.nix, and once in the package expression. With callPackage, we will implicitly pass the necessary inputs from the top-level expression. + In the next pill, we will talk about the "callPackage" design + pattern. This removes the tedium of specifying the names of the inputs twice: + once in the top-level default.nix, and once in the package + expression. With callPackage, we will + implicitly pass the necessary inputs from the top-level expression.
diff --git a/pills/13-callpackage-design-pattern.xml b/pills/13-callpackage-design-pattern.xml index fac7873..f8443a3 100644 --- a/pills/13-callpackage-design-pattern.xml +++ b/pills/13-callpackage-design-pattern.xml @@ -6,134 +6,262 @@ Callpackage Design Pattern - Welcome to the 13th Nix pill. In the previous 12th pill we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged graphviz to have at least another package for our little repository. + Welcome to the 13th Nix pill. In the previous 12th + pill, we introduced the first basic design pattern for organizing a repository of + software. In addition, we packaged graphviz so that we had two packages + to bundle into an example repository. - The next design pattern worth noting is what I'd like to call the callPackage pattern. This technique is extensively used in nixpkgs, it's the current standard for importing packages in a repository. + The next design pattern we will examine is called the callPackage + pattern. This technique is extensively used in nixpkgs, and it's the current + de facto standard for importing packages in a repository. It's purpose is to reduce + the duplication of identifiers between package derivation inputs and repository + derivations.
The callPackage convenience - In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments. + In the previous pill, we demonstrated how the inputs + pattern decouples packages from the repository. This allowed us to + manually pass the inputs to the derivation; the derivation declares + its inputs, and the caller passes the arguments. - However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice. With package management, we often see common patterns. In the case of nixpkgs it's the following. - - - Some package derivation: + However, as with usual programming languages, there is some duplication of work: + we declare parameter names and then we pass arguments, typically with the same name. + For example, if we define a package derivation using the inputs + pattern such as: - Repository derivation: + we would likely want to bundle that package derivation into a respository via a + an attribute set defined as something like: - Where inputs may even be packages in the repository itself (note the rec keyword). The pattern here is clear, often inputs have the same name of the attributes in the repository itself. Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument). + There are two things to note. First, that inputs often have the same name as + attributes in the respository itself. Second, that (due to the rec + keyword), the inputs to a package derivation may be other packages in the + repository itself. - To achieve this, we will define a callPackage function with the following synopsis: + Rather than passing the inputs twice, we would prefer to pass those inputs from + the respoistory automatically and allow for manually overriding defaults. + + + To achieve this, we will define a callPackage function with + the following calling convention: - What should it do? + We want callPackage to be a function of two arguments, with the + following behavior: - Import the given expression, which in turn returns a function. - Determine the name of its arguments. - Pass default arguments from the repository set, and let us override those arguments. + + Import the given expression contained in the file of the first argument, + and return a function. This function returns a package derivation that + uses the inputs pattern. + + + Determine the name of the arguments to the function (i.e., the names + of the inputs to the package derivation). + + + Pass default arguments from the repository set, and let us override those + arguments if we wish to customize the package derivation. +
- Implementing callPackage + Implementing <code>callPackage</code> - First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments. + In this section, we will build up the callPackages pattern + from scratch. To start, we need a way to obtain the argument names + of a function (in this case, the function that takes "inputs" and produces + a package derivation) at runtime. This is because we want to automatically pass + such arguments. - Then callPackage requires access to the whole packages set, because it needs to find the packages to pass automatically. - - - We start off simple with : + Nix provides a builtin function to do this: - Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names. + In addition to returning the argument names, the attribute set returned by + functionArgs indicates whether or not the argument has a default value. + For our purposes, we are only interested in the argument names; we do not care + about the default values right now. + + + + The next step is to make callPackage automatically pass inputs to our + package derivations based on the argument names we've just obtained with + functionArgs. - Now we need a set with all the values, let's call it values. And a way to intersect the attributes of values with the function arguments: + To do this, we need two things: + + + A package repository set containing package derivations that match the arguments + names we've obtained + + + A way to obtain an auto-populated attribute set combining the package repository + and the return value of functionArgs. + + + + + The former is easy: we just have to set our package deriviation's inputs + to be package names in a repository, such as nixpkgs. For + the latter, Nix provides another builtin function: - Perfect, note from the example above that the intersectAttrs returns a set whose names are the intersection, and the attribute values are taken from the second set. + The intersectAttrs returns an attribute set whose names are + the intersection of both arguments' attribute names, with the attribute + values taken from the second argument. - We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of callPackage: + This is all we need to do: we have obtained the argument names from a function, + and populated these with an existing set of attributes. This is our simple + implementation of callPackage: - Clearing up the syntax: + Let's dissect the above snippet: - We define a callPackage variable which is a function. - The second parameter is the function to "autocall". - We take the argument names of the function and intersect with the set of all values. - Finally we call the passed function f with the resulting intersection. + + We define a callPackage variable which is a + function. + + + The first parameter to the callPackage function + is a set of name-value pairs that may appear in the argument set of + the function we wish to "autocall". + + + The second parameter is the function to "autocall" + + + We take the argument names of the function and intersect with the set of all + values. + + + Finally, we call the passed function f with the resulting + intersection. + - In the code above, I've also shown that the callPackage call is equivalent to directly calling add a b. + In the snippet above, we've also demonstrated that the callPackage + call is equivalent to directly calling add a b. - We achieved what we wanted. Automatically call functions given a set of possible arguments. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs ... as explained in the 5th pill). + We achieved most of what we wanted: to automatically call functions given a set of + possible arguments. If an argument is not found within the set we used to call the + function, then we receive an error (unless the function has variadic arguments + denoted with ..., as explained in the 5th pill). - Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set. Then we add a further parameter, which takes a set of overrides: + The last missing piece is allowing users to override some of the parameters. + We may not want to always call functions with values taken from the big set. + Thus, we add a third parameter which takes a set of overrides: - Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set. + Apart from the increasing number of parentheses, it should be clear that we simply + take a set union between the default arguments and the overriding set.
- Use callPackage to simplify the repository + Using callPackage to simplify the repository - Given our brand new tool, we can simplify the repository expression (default.nix). - - - Let me write it down first: + Given our callPackages, we can simplify the repository expression + in default.nix: - Wow, there's a lot to say here: + Let's examine this in detail: - We renamed the old pkgs of the previous pill to nixpkgs. Our package set is now instead named pkgs. Sorry for the confusion. - We needed a way to pass pkgs to callPackage somehow. Instead of returning the set of packages directly from default.nix, we first assign it to a let variable and reuse it in callPackage. - For convenience, in callPackage we first import the file, instead of calling it directly. Otherwise for each package we would have to write the import. - Since our expressions use packages from nixpkgs, in callPackage we use allPkgs, which is the union of nixpkgs and our packages. - We moved mkDerivation into pkgs itself, so that it also gets passed automatically. + + The expression above defines our own package repository, which we call + pkgs, that contains hello along + with our two variants of graphviz. + + + In the let expression, we import nixpkgs. + Note that previously, we referred to this import with the variable + pkgs, but now that name is taken by the repository + we are creating ourselves. + + + We needed a way to pass pkgs to callPackage + somehow. Instead of returning the set of packages directly from + default.nix, we first assign it to a let + variable and reuse it in callPackage. + + + For convenience, in callPackage we first + import the file instead of calling it directly. Otherwise we would have to + write the import for each package. + + + Since our expressions use packages from nixpkgs, in + callPackage we use allPkgs, which + is the union of nixpkgs and our packages. + + + We moved mkDerivation into pkgs itself, + so that it also gets passed automatically. - Note how easy is to override arguments in the case of graphviz without gd. But most importantly, how easy it was to merge two repositories: nixpkgs and our pkgs! + Note how easily we overrode arguments in the case of graphviz + without gd. In addition, note how easy it was to merge + two repositories: nixpkgs and our pkgs! - The reader should notice a magic thing happening. We're defining pkgs in terms of callPackage, and callPackage in terms of pkgs. That magic is possible thanks to lazy evaluation: builtins.intersectAttrs doesn't need to know the values in allPkgs in order to perform intersection, only the keys that do not require callPackage evaluation. + The reader should notice a magic thing happening. We're defining + pkgs in terms of callPackage, and + callPackage in terms of pkgs. That magic is + possible thanks to lazy evaluation: builtins.intersectAttrs doesn't + need to know the values in allPkgs in order to perform intersection, + only the keys that do not require callPackage evaluation.
Conclusion - The "callPackage" pattern has simplified our repository a lot. We're able to import packages that require some named arguments and call them automatically, given the set of all packages. + The "callPackage" pattern has simplified our repository + considerably. We were able to import packages that require named arguments + and call them automatically, given the set of all packages sourced from + nixpkgs. - We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. They are documented in the Nix manual. + We've also introduced some useful builtin functions that allows us to introspect Nix + functions and manipulate attributes. These builtin functions are not usually used when + packaging software, but rather act as tools for packaging. They are documented in the + Nix + manual. - Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions. + Writing a repository in Nix is an evolution of writing convenient functions for + combining the packages. This pills demonstrates how Nix can be a generic tool + to build and deploy software, and how suitable it is to create software + repositories with our own conventions.
Next pill - ...we will talk about the "override" design pattern. The graphvizCore seems straightforward. It starts from graphviz.nix and builds it without gd. Now I want to give you another point of view: what if we instead wanted to start from pkgs.graphviz and disable gd? + In the next pill, we will talk about the "override" design + pattern. The graphvizCore seems straightforward. It starts from + graphviz.nix and builds it without gd. + In the next pill, we will consider another point of view: starting from + pkgs.graphviz and disabling gd?
diff --git a/pills/14-override-design-pattern.xml b/pills/14-override-design-pattern.xml index 81dd1af..1d66737 100644 --- a/pills/14-override-design-pattern.xml +++ b/pills/14-override-design-pattern.xml @@ -6,139 +6,190 @@ Override Design Pattern - Welcome to the 14th Nix pill. In the previous 13th pill we have introduced the callPackage pattern, used to simplify the composition of software in a repository. + Welcome to the 14th Nix pill. In the previous 13th pill, we introduced the + callPackage pattern and used it to simplify the composition + of software in a repository. - The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix. + The next design pattern is less necessary, but is useful in many cases and is + a good exercise to learn more about Nix.
About composability - Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications. + Functional languages are known for being able to compose functions. In particular, + these languages gain expressivity from functions that manipulate an original + value into a new value having the same structure. This allows us to compose + multiple functions to perform the desired modifications. - In Nix we mostly talk about functions that accept inputs in order to return derivations. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it. + In Nix, we mostly talk about functions + that accept inputs in order to return derivations. + In our world, we want utility functions that are able to manipulate those structures. + These utilities add some useful properties to the original value, and we'd like to be + able to apply more utilities on top of the result. - For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches: + For example, let's say we have an initial derivation drv and + we want to transform it into a drv with debugging information and + custom patches: debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv) - The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages. + The final result should be the original derivation with some changes. + This is both interesting and very different from other packaging approaches, + which is a consequence of using a functional language to describe packages. - Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best. + Designing such utilities is not trivial in a functional language without static + typing, because understanding what can or cannot be composed is difficult. + But we try to do our best.
The override pattern - In the pill 12 we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments. + In pill 12 we introduced the inputs + design pattern. We do not return a derivation picking dependencies directly from the + repository; rather we declare the inputs and let the callers pass the necessary + arguments. - In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute: + In our repository we have a set of attributes that import the expressions of the + packages and pass these arguments, getting back a derivation. Let's take for example + the graphviz attribute: graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; }; - If we wanted to produce a derivation of graphviz with a customized gd version, we would have to repeat most of the above plus specifying an alternative gd: + If we wanted to produce a derivation of graphviz with a customized + gd version, we would have to repeat most of the above plus + specifying an alternative gd: - That's hard to maintain. Using callPackage it would be easier: + That's hard to maintain. Using callPackage would be easier: mygraphviz = callPackage ./graphviz.nix { gd = customgd; }; But we may still be diverging from the original graphviz in the repository. - We would like to avoid specifying the nix expression again, instead reuse the original graphviz attribute in the repository and add our overrides like this: + We would like to avoid specifying the nix expression again. Instead, we would + like to reuse the original graphviz attribute in the + repository and add our overrides like so: mygraphviz = graphviz.override { gd = customgd; }; The difference is obvious, as well as the advantages of this approach. - Note: that .override is not a "method" in the OO sense as you may think. Nix is a functional language. That .override is simply an attribute of a set. + Note: that .override is + not a "method" in the OO sense as you may think. Nix is a functional language. + The.override is simply an attribute of a set.
The override implementation - I remind you, the graphviz attribute in the repository is the derivation returned by the function imported from graphviz.nix. We would like to add a further attribute named "override" to the returned set. + Recall that the graphviz attribute in the repository is + the derivation returned by the function imported from + graphviz.nix. We would like to add a further attribute + named "override" to the returned set. - Let's start simple by first creating a function "makeOverridable" that takes a function and a set of original arguments to be passed to the function. + Let's start by first creating a function "makeOverridable". + This function will take two arguments: a function (that must return a set) + and the set of original arguments to be passed to the function. - Contract: the wrapped function must return a set. - - - Let's write a lib.nix: + We will put this function in a lib.nix: - So makeOverridable takes a function and a set of original arguments. It returns the original returned set, plus a new override attribute. + makeOverridable takes a function and a set of original arguments. + It returns the original returned set, plus a new override attribute. - This override attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess. + This override attribute is a function taking a set of new + arguments, and returns the result of the original function called with the + original arguments unified with the new arguments. This is admittedly somewhat + confusing, but the examples below should make it clear. Let's try it with nix repl: - Note that the function f does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-) + Note that, as we specified above, the function f does not return + the plain sum. Instead, it returns a set with the sum bound to the name + result. - The variable res is the result of the function call without any override. It's easy to see in the definition of makeOverridable. In addition you can see the new override attribute being a function. + The variable res contains the result of the function call without + any override. It's easy to see in the definition of makeOverridable. + In addition, you can see that new override attribute is a function. - Calling that .override with a set will invoke the original function with the overrides, as expected. + Calling res.override with a set will invoke the original function + with the overrides, as expected. - But: we can't override again! Because the returned set with result 15 does not have an override attribute! + This is a good start, but we can't override again! This is because the returned + set (with result = 15) does not have an override + attribute of it's own. This is bad; it breaks further composition. - That's bad, it breaks further compositions. - - - The solution is simple, the .override function should make the result overridable again: + The solution is simple: the .override function should make the + result overridable again: - Please note the rec keyword. It's necessary so that we can refer to makeOverridable from makeOverridable itself. + Please note the rec keyword. It's necessary so that we can refer + to makeOverridable from makeOverridable itself. Now let's try overriding twice: - Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20. + Success! The result is 30 (as expected) because a is overridden + to 10 in the first override, and b is overridden to 20 in the + second. - Now it would be nice if callPackage made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader. + Now it would be nice if callPackage made our + derivations overridable. This is an exercise for the reader.
Conclusion - The "override" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like nixpkgs, and defining overrides on our local machine without even modifying the original package. + The "override" pattern simplifies the way we customize packages + starting from an existing set of packages. This opens a world of possibilities for + using a central repository like nixpkgs and defining overrides + on our local machine without modifying the original package. - Dream of a custom isolated nix-shell environment for testing graphviz with a custom gd: + We can dream of a custom, isolated nix-shell environment for + testing graphviz with a custom gd: debugVersion (graphviz.override { gd = customgd; }) - Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically. + Once a new version of the overridden package comes out in the repository, the + customized package will make use of it automatically. - The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components. + The key in Nix is to find powerful yet simple abstractions in order to let the user + customize their environment with highest consistency and lowest maintenance time, by + using predefined composable components.
Next pill - ...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <nixpkgs> come from? + In the next pill, we will talk about Nix search paths. By "search path", we mean a + place in the file system where Nix looks for expressions. This answers the + question of where <nixpkgs> comes from.