24 KiB
myst | ||||||
---|---|---|---|---|---|---|
|
(packaging-existing-software)=
Packaging Existing Software With Nix
One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like managing dependencies.
In the long term, Nix helps tremendously in alleviating that stress, but when first (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding.
If you haven't already read the tutorial on making a derivation, please go do so before reading this!
In this tutorial, we'll see how to create Nix derivations to package C/C++ software, taking advantage of the nixpkgs
stdenv
which automates much of the work of building self-contained C/C++ packages.
We'll begin by considering hello
, a feature-complete implementation of the famous "hello world", which requires no external dependencies.
Then we'll move to progressively more complex packages with their own separate dependencies, leading us to use additional derivation features.
Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go.
Packages in Nix
Before we proceed, an important point of clarification: we conventionally use the term "package" by analogy to other systems, although this term is not a proper concept in Nix.
For the purposes of this tutorial, by "package" we mean "a Nix function which takes an attribute set of 'dependencies' and produces a derivation", where "dependencies" could be other packages or configuration parameters.
A Simple Project
To start, we'll write a skeleton derivation, updating this as we go:
{ pkgs
, stdenv
}:
stdenv.mkDerivation { };
Hello, World!
Since GNU Hello is a popular (in a certain sense) package from the GNU Project, we can easily access its source code from GNU's FTP.
In this case, we'll download the latest version of hello
, currently 2.12.1
.
Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions.
In this case, we'll use fetchTarball
, which takes the URI path to the download file and a SHA256 hash of its contents.
Here is our first iterative debugging technique: we can't actually know the hash until after we've downloaded and unpacked the tarball, but Nix will complain at us if the hash we supplied was incorrect, so we can just supply a fake one with lib.fakeSha256
and change our derivation after Nix informs us of the correct hash:
# hello.nix
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
src = builtins.fetchTarball {
url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz";
sha256 = lib.fakeSha256;
};
}
Let's save this file to hello.nix
and try to build it. To do so, we'll use nix-build
...
$ nix-build hello.nix
error: cannot evaluate a function that has an argument without a value ('pkgs')
Nix attempted to evaluate a function as a top level expression; in
this case it must have its arguments supplied either by default
values, or passed explicitly with '--arg' or '--argstr'. See
https://nixos.org/manual/nix/stable/language/constructs.html#functions.
at /home/nix-user/hello.nix:2:3:
1| # hello.nix
2| { pkgs
| ^
3| , lib
... and immediately run into a problem: every derivation is a function, and functions need arguments!
Your New Favorite Command
In order to pass the pkgs
argument to our derivation, we'll need to import nixpkgs
in another Nix expression. The nix-build
command lets us pass whole expressions as an argument following the -E/--expr
flag.
We'll use the following expression:
with import <nixpkgs> {}; callPackage ./hello.nix {}
callPackage
automatically passes attributes from pkgs
to the given function, if they match attributes required by that function's argument attrset. Here, callPackage
will supply pkgs
, lib
, and stdenv
.
Let's run this now:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./hello.nix {}'
error: derivation name missing
Progress! The new failure occurs with the derivation, further down in the file than the initial error on line 2 about the pkgs
argument not having a value; we successfully resolved the previous error by importing nixpkgs
in the expression we passed to nix-build
.
Naming a Derivation
Every derivation needs a name
attribute, which must either be set directly or constructed by mkDerivation
from pname
and version
attributes, if they exist.
Let's update the file again to add a name
:
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
name = "hello";
src = builtins.fetchTarball {
url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz";
sha256 = lib.fakeSha256;
};
}
and then re-run the command:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./hello.nix {}'
error:
… while calling the 'derivationStrict' builtin
at /builtin/derivation.nix:9:12: (source not available)
… while evaluating derivation 'hello'
whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7
… while evaluating attribute 'src' of derivation 'hello'
at /home/nix-user/hello.nix:5:3:
4| name = "hello";
5| src = builtins.fetchTarball {
| ^
6| url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz";
error: hash mismatch in file downloaded from 'https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz':
specified: sha256:0000000000000000000000000000000000000000000000000000
got: sha256:0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn
Finding The File Hash
As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our hello.nix
file, replacing lib.fakeSha256
:
# hello.nix
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
name = "hello";
src = builtins.fetchTarball {
url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz";
sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn";
};
}
Now let's run that command again:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./hello.nix {}'
this derivation will be built:
/nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv
building '/nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv'...
unpacking sources
unpacking source archive /nix/store/xdbysilxxgbs55rrdxniglqg9m1v61h4-source
source root is source
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/y55w1djfnxkl2jk9w0liancp83zqb7ki-hello
...
configure: creating ./config.status
config.status: creating Makefile
...
building
build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash
... <many more lines omitted>
Great news: our derivation built successfully!
We can see from the console output that configure
was called, which produced a Makefile
that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in stdenv
is based on autoconf
.
Build Result
We can check our working directory for the result:
$ ls
hello.nix result
This result is a symbolic link to a Nix store location containing the built binary; we can call ./result/bin/hello
to execute this program:
$ ./result/bin/hello
Hello, world!
We've successfully packaged our first program with Nix! The experience was a little bit too magical though, so up next we'll package another piece of software which has external dependencies and a different means of building, which will require us to lean more on mkDerivation
.
Something Bigger
The hello
program is a simple and common place to start packaging, but it's not very useful or interesting, so we can't stop there.
Now, we'll look at packaging a somewhat more complicated program, icat
, which allows us to render images in our terminal.
We'll start by copying hello.nix
from the previous section to a new file, icat.nix
. Then we'll make several changes, starting with the name
attribute:
# icat.nix
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
name = "icat";
src = builtins.fetchTarball {
...
};
}
Now we'll download the source code. icat
's upstream repository is hosted on GitHub, so we should slightly modify our previous source fetcher: instead of builtins.fetchTarball
, we'll use pkgs.fetchFromGitHub
:
# icat.nix
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
name = "icat";
src = pkgs.fetchFromGitHub {
...
};
}
Fetching Source from GitHub
While fetchTarball
required url
and sha256
arguments, we'll need more than that for fetchFromGitHub
.
The source we want is hosted on GitHub at https://github.com/atextor/icat
, which already gives us the first two arguments:
owner
: the name of the account controlling the repository;owner = "atextor"
repo
: the name of the repository we want to fetch;repo = "icat"
We can navigate to the project's Releases page to find a suitable rev
, such as the git commit hash or tag (e.g. v1.0
) corresponding to the release we want to fetch. In this case, the latest release tag is v0.5
.
As in the hello
example, we also need to supply a hash. This time, instead of using lib.fakeSha256
and letting nix-build
report the correct one in an error, we'll fetch the correct hash in the first place with the nix-prefetch-url
command. We want the SHA256 hash of the contents of the tarball, so we need to pass the --unpack
and --type sha256
arguments too:
$ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz --type sha256
path is '/nix/store/p8jl1jlqxcsc7ryiazbpm7c1mqb6848b-v0.5.tar.gz'
0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka
Now we can supply the correct hash to fetchFromGitHub
:
# icat.nix
{ pkgs
, lib
, stdenv
}:
stdenv.mkDerivation {
name = "icat";
src = pkgs.fetchFromGitHub {
owner = "atextor";
repo = "icat";
rev = "v0.5";
sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka";
};
}
Missing Dependencies
Now we run into an entirely new issue:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./icat.nix {}'
this derivation will be built:
/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv
building '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'...
unpacking sources
unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash
gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
| ^~~~~~~
icat.c:39:10: fatal error: Imlib2.h: No such file or directory
39 | #include <Imlib2.h>
| ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:16: icat.o] Error 1
error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed with exit code 2;
last 10 log lines:
> from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
> from icat.c:31:
> /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> | ^~~~~~~
> icat.c:39:10: fatal error: Imlib2.h: No such file or directory
> 39 | #include <Imlib2.h>
> | ^~~~~~~~~~
> compilation terminated.
> make: *** [Makefile:16: icat.o] Error 1
For full logs, run 'nix log /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'.
Finally, a compiler error! We've successfully pulled the icat
source from GitHub, and Nix tried to build what it found, but is missing a dependency: the imlib2
header. If we search for imlib2
on search.nixos.org, we'll find that imlib2
is already in nixpkgs
.
We can add this package to our build environment by either
- adding
imlib2
to the set of inputs to the expression inicat.nix
, and then addingimlib2
to the list ofbuildInputs
instdenv.mkDerivation
, or - adding
pkgs.imlib2
to thebuildInputs
directly, sincepkgs
is already in-scope.
Because callPackage
is used to provide all necessary inputs in nixpkgs
as well as in our nix-build
invocation, the first approach is the one currently favored, and we'll use it here:
# icat.nix
{ pkgs
, lib
, stdenv
, imlib2
}:
stdenv.mkDerivation {
name = "icat";
src = pkgs.fetchFromGitHub {
owner = "atextor";
repo = "icat";
rev = "v0.5";
sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka";
};
buildInputs = [ imlib2 ];
}
Another error, but we get further this time:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./icat.nix {}'
this derivation will be built:
/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv
building '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'...
unpacking sources
unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash
gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
| ^~~~~~~
In file included from icat.c:39:
/nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory
45 | #include <X11/Xlib.h>
| ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:16: icat.o] Error 1
error: builder for '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv' failed with exit code 2;
last 10 log lines:
> from icat.c:31:
> /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> | ^~~~~~~
> In file included from icat.c:39:
> /nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory
> 45 | #include <X11/Xlib.h>
> | ^~~~~~~~~~~~
> compilation terminated.
> make: *** [Makefile:16: icat.o] Error 1
For full logs, run 'nix log /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'.
We can see a few warnings which should be corrected in the upstream code, but the important bit for our purposes is fatal error: X11/Xlib.h: No such file or directory
: we're missing another dependency.
In addition to the widespread practice of prefixing a project name with lib
to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given derivation.
:::{note}
Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the nixpkgs
source for keywords, in addition to checking discussion platforms like the official NixOS Discourse.
:::
We need the Xlib.h
headers from the X11
C package, the Nixpkgs derivation for which is libX11
, available in the xorg
package set. The Xlib
headers in turn live in the dev
output of xorg.libX11
. We'll add this to our derivation's input attribute set and to buildInputs
:
# icat.nix
{ pkgs
, lib
, stdenv
, imlib2
, xorg
}:
stdenv.mkDerivation {
name = "icat";
src = pkgs.fetchFromGitHub {
owner = "atextor";
repo = "icat";
rev = "v0.5";
sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka";
};
buildInputs = [ imlib2 xorg.libX11.dev ];
}
:::{note}
We only added the top-level xorg
derivation to the input attrset, rather than the full xorg.libX11.dev
which would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way doesn't actually include all of xorg
into our build context.
:::
buildInputs
and nativeBuildInputs
Running our favorite command again:
$ nix-build -E 'with import <nixpkgs> {}; callPackage ./icat.nix {}'
this derivation will be built:
/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv
building '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'...
unpacking sources
unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash
gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c
In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33,
from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27,
from icat.c:31:
/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;]
195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
| ^~~~~~~
icat.c: In function 'main':
icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;]
319 | write(tempfile, &buf, 1);
| ^~~~~~~~~~~~~~~~~~~~~~~~
gcc -o icat icat.o -lImlib2
installing
install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install
make: *** No rule to make target 'install'. Stop.
error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed with exit code 2;
last 10 log lines:
> 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
> | ^~~~~~~
> icat.c: In function 'main':
> icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;]
> 319 | write(tempfile, &buf, 1);
> | ^~~~~~~~~~~~~~~~~~~~~~~~
> gcc -o icat icat.o -lImlib2
> installing
> install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install
> make: *** No rule to make target 'install'. Stop.
For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'.
installPhase
In order to make packages available for other packages to depend on, Nix copies everything to the Nix store (at /nix/store
), and symlinks them from there into build contexts and development environments.
The Makefile
doesn't provide an installation step, so we must produce one for our derivation, using the installPhase
attribute, which contains a list of command strings to execute to accomplish the installation.
The icat
executable is only used at runtime, and isn't a compile-time input for anything else at this point, so we only need to concern ourselves with the bin
output. In Nix, the result of a build is copied to a location stored in the $out
variable accessible in the derivation's component scripts; we'll create a bin
directory within that, and then copy our icat
executable there:
# icat.nix
{ pkgs
, lib
, stdenv
, imlib2
, xorg
}:
stdenv.mkDerivation {
name = "icat";
src = pkgs.fetchFromGitHub {
owner = "atextor";
repo = "icat";
rev = "v0.5";
sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka";
};
buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ];
installPhase = ''
mkdir -p $out/bin
cp icat $out/bin
'';
}
Phases and Hooks
Nix package derivations are separated into phases, each of which is intended to control some aspect of the build process.
We saw earlier how the stdenv
expected the project's Makefile
to have an install
target, and failed when it didn't. To fix this, we defined a custom installPhase
, containing instructions for copying the icat
binary to the correct output location, in effect installing it.
Up to that point, the stdenv
automatically determined the buildPhase
information for our icat
package.
During derivation realisation, there are a number of shell functions ("hooks", in nixpkgs
) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations.
It's good practice when packaging software with Nix to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy overriding of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits.
We should now adjust our installPhase
to call the appropriate hooks:
# icat.nix
...
installPhase = ''
runHook preInstall
mkdir -p $out/bin
cp icat $out/bin
runHook postInstall
'';
...
Running our nix-build
command once more will finally do what we want, and more safely than before; we can ls
in the local directory to find a result
symlink to a location in the Nix store:
$ ls
hello.nix icat.nix result
result/bin/icat
is the executable we built previously. Success!