"description lang=en": "Packaging Existing Software With Nix"
"keywords": "Nix, packaging"
---
(packaging-existing-software)=
# Packaging Existing Software With Nix
One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like *managing dependencies*.
In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding.
If you haven't already read the tutorial on making a derivation, please go do so before reading this!
In this tutorial, we'll see how to create Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages.
We'll begin by considering `hello`, a feature-complete implementation of the famous "hello world", which requires no external dependencies.
Then we'll move to progressively more complex packages with their own separate dependencies, leading us to use additional derivation features.
Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go.
Before we proceed, an important point of clarification: we conventionally use the term "package" by analogy to other systems, although this term is not a proper concept in Nix.
For the purposes of this tutorial, by "package" we mean "a Nix function which takes an attribute set of 'dependencies' and produces a derivation", where "dependencies" could be other packages or configuration parameters.
Since GNU Hello is a popular (in a certain sense) package from the GNU Project, we can easily access its source code [from GNU's FTP](https://ftp.gnu.org/gnu/hello/).
In this case, we'll download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello`, currently `2.12.1`.
Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions.
Here is our first iterative debugging technique: we can't actually know the hash until after we've downloaded and unpacked the tarball, but Nix will complain at us if the hash we supplied was incorrect, so we can just supply a fake one with `lib.fakeSha256` and change our derivation after Nix informs us of the correct hash:
... and immediately run into a problem: every derivation is a *function*, and functions need *arguments*!
### Your New Favorite Command
In order to pass the `pkgs` argument to our derivation, we'll need to import `nixpkgs` in another Nix expression. The `nix-build` command lets us pass whole expressions as an argument following the `-E/--expr` flag.
We'll use the following expression:
```console
with import <nixpkgs> {}; callPackage ./hello.nix {}
`callPackage` automatically passes attributes from `pkgs` to the given function, if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `pkgs`, `lib`, and `stdenv`.
Progress! The new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; we successfully resolved the previous error by importing `nixpkgs` in the expression we passed to `nix-build`.
### Naming a Derivation
Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist.
at /builtin/derivation.nix:9:12: (source not available)
… while evaluating derivation 'hello'
whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7
… while evaluating attribute 'src' of derivation 'hello'
As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our `hello.nix` file, replacing `lib.fakeSha256`:
We can see from the console output that `configure` was called, which produced a `Makefile` that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in `stdenv` is based on `autoconf`.
We can check our working directory for the result:
```console
$ ls
hello.nix result
```
This result is a symbolic link to a Nix store location containing the built binary; we can call `./result/bin/hello` to execute this program:
```console
$ ./result/bin/hello
Hello, world!
```
We've successfully packaged our first program with Nix! The experience was a little bit *too* magical though, so up next we'll package another piece of software which has external dependencies and a different means of building, which will require us to lean more on `mkDerivation`.
## Something Bigger
The `hello` program is a simple and common place to start packaging, but it's not very useful or interesting, so we can't stop there.
Now, we'll look at packaging a somewhat more complicated program, `icat`, which allows us to render images in our terminal.
We'll start by copying `hello.nix` from the previous section to a new file, `icat.nix`. Then we'll make several changes, starting with the `name` attribute:
Now we'll download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so we should slightly modify our previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `builtins.fetchTarball`, we'll use `pkgs.fetchFromGitHub`:
While `fetchTarball` required `url` and `sha256` arguments, we'll need more than that for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub).
The source we want is hosted on GitHub at `https://github.com/atextor/icat`, which already gives us the first two arguments:
-`owner`: the name of the account controlling the repository; `owner = "atextor"`
-`repo`: the name of the repository we want to fetch; `repo = "icat"`
We can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release we want to fetch. In this case, the latest release tag is `v0.5`.
As in the `hello` example, we also need to supply a hash. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, we'll fetch the correct hash in the first place with the `nix-prefetch-url` command. We want the SHA256 hash of the *contents* of the tarball, so we need to pass the `--unpack` and `--type sha256` arguments too:
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
> from icat.c:31:
> /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but is missing a dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`.
We can add this package to our build environment by either
- adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or
- adding `pkgs.imlib2` to the `buildInputs` directly, since `pkgs` is already in-scope.
Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as well as in our `nix-build` invocation, the first approach is the one currently favored, and we'll use it here:
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
We can see a few warnings which should be corrected in the upstream code, but the important bit for our purposes is `fatal error: X11/Xlib.h: No such file or directory`: we're missing another dependency.
In addition to the widespread practice of prefixing a project name with `lib` to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given [derivation](https://nixos.org/manual/nix/stable/language/derivations.html).
:::{note}
Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords, in addition to checking discussion platforms like [the official NixOS Discourse](https://discourse.nixos.org).
:::
We need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. We'll add this to our derivation's input attribute set and to `buildInputs`:
We only added the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev` which would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way doesn't actually include all of `xorg` into our build context.
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;]
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> | ^~~~~~~
> icat.c: In function 'main':
> icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;]
The `stdenv` is automatically working with the `Makefile` that comes with `icat`: we can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to us.
To add this step to our derivation, we use the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to perform the installation.
Because the `make` step completes successfully, the `icat` executable is available in the build directory, and we only need to copy it from there to the output directory. In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; we'll create a `bin` directory within that and copy our `icat` binary there:
buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ];
installPhase = ''
mkdir -p $out/bin
cp icat $out/bin
'';
}
```
### Phases and Hooks
Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process.
We saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, we defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it.
During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations.
It's good practice when packaging software with Nix to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits.
We should now adjust our `installPhase` to call the appropriate hooks:
Running our `nix-build` command once more will finally do what we want, and more safely than before; we can `ls` in the local directory to find a `result` symlink to a location in the Nix store:
