1
0
Fork 0
mirror of https://github.com/NixOS/nix synced 2024-10-18 14:32:45 -04:00
Commit graph

11143 commits

Author SHA1 Message Date
Robert Hensing 3f37785afd NIX_REMOTE_SYSTEMS: actually support multiple :-separated entries
Bug not reported in 6 years, but here you go.

Also it is safe to switch to normal concatStringsSep behavior
because tokenizeString does not produce empty items.
2024-07-13 03:06:24 +02:00
Robert Hensing 39878c8979 dropEmptyInitThenConcatStringsSep -> concatStringSep: preserve empty attr
The empty attribute name should not be dropped from attribute paths.
Rendering attribute paths with concatStringsSep is lossy and wrong,
but this is just a first improvement while dealing with the
dropEmptyInitThenConcatStringsSep problem.
2024-07-13 03:06:24 +02:00
Robert Hensing ea966a70fc dropEmptyInitThenConcatStringsSep -> concatStringSep: diagnostics and docs
These are non-critical, so their behavior is ok to change.
Dropping empty items is not needed and usually not expected.
2024-07-13 03:06:24 +02:00
Robert Hensing a681d354e7 Add fresh concatStringsSep without bug
The buggy version was previously renamed to
dropEmptyInitThenConcatStringsSep
2024-07-13 03:06:24 +02:00
Robert Hensing 79eb0adf9d dropEmptyInitThenConcatStringSep: Check that we don't drop...
... initial empty strings.

The tests pass, which is encouraging.
2024-07-13 03:05:50 +02:00
Robert Hensing 1a8defd06f Refactor: rename C++ concatStringsSep -> dropEmptyInitThenConcatStringsSep 2024-07-13 03:05:50 +02:00
John Ericson bc83b9dc1f Remove comparator.hh and switch to <=> in a bunch of places
Known behavior changes:

- `MemorySourceAccessor`'s comparison operators no longer forget to
  compare the `SourceAccessor` base class.

Progress on #10832

What remains for that issue is hopefully much easier!
2024-07-12 14:54:18 -04:00
Lexi Mattick 6c4470ec2a Clean up cache for all commands 2024-07-12 11:54:12 -07:00
Lexi Mattick e764ed31f6 Eval cache: fix cache regressions
- Fix eval cache not being persisted in `nix develop` (since #10570)
- Don't attempt to commit cache transaction if there is no active transaction, which will spew errors in edge cases
- Drive-by: trivial typo fix
2024-07-12 09:45:35 -07:00
Eelco Dolstra 2a95a2d780
Merge pull request #11091 from NixOS/darwin-no-map-spaceship
Remove unused `operator<=>`'s that darwin can't generate
2024-07-12 18:40:20 +02:00
Eelco Dolstra cdc23b67a6 Provide std::hash<SourcePath> 2024-07-12 17:54:27 +02:00
Robert Hensing 11a6db5993 Remove unused operator<=>'s that darwin can't generate
It was complaining *a lot*, with dozens of MB of logs.
2024-07-12 17:37:27 +02:00
John Ericson 337a5a23b7
Merge pull request #11089 from NixOS/warnings-includes
Fix warnings and optimize includes
2024-07-12 10:29:26 -04:00
Robert Hensing 27eaeebc41 nar-accessor.cc: Silence unused variable warning 2024-07-12 15:38:17 +02:00
Robert Hensing 8df041cbc6 Solve unused header warnings reported by clangd 2024-07-12 15:37:54 +02:00
John Ericson 3fc77f281e No global settings in libnixfetchers and libnixflake
Progress on #5638

There are still a global fetcher and eval settings, but they are pushed
down into `libnixcmd`, which is a lot less bad a place for this sort of
thing.

Continuing process pioneered in
52bfccf8d8.
2024-07-12 08:50:28 -04:00
Robert Hensing f070d68c32 Add BaseError assignment operators
The move assignment was implicitly generated and used in

    src/libstore/build/goal.cc:90:22:
       90 |             this->ex = std::move(*ex);

Clang warns about this generated method being deprecated, so making
them explicit fixes the warning.
2024-07-12 14:25:16 +02:00
Eelco Dolstra 61080554ab SymbolStr: Remove std::string conversion
This refactoring allows the symbol table to be stored as something
other than std::strings.
2024-07-11 17:43:10 +02:00
Eelco Dolstra 87323a5689 Remove unused InstallableFlake::getFlakeOutputs() 2024-07-11 16:21:27 +02:00
Robert Hensing bb312a7174 Edit CompatibilitySettings 2024-07-11 13:59:49 +02:00
Robert Hensing 56bf39e905 eqValues/assertEqValues: Clean up assertions
It's still paranoid, and probably a waste of words, but at least
now it's consistent and readily identifyable from a log.
2024-07-11 12:19:29 +02:00
Robert Hensing 4fd8f19ecf Fix build to use CanonPath in new FSO sinks 2024-07-11 12:14:48 +02:00
Robert Hensing 86420753ec Merge remote-tracking branch 'upstream/master' into support-hardlinks-in-tarballs 2024-07-11 11:43:02 +02:00
Robert Hensing 61577402ba Add EvalErrorBuilder::panic()
An nicer alternative to printError + abort, or assert(false /* foo */)
2024-07-11 11:35:58 +02:00
Robert Hensing c4e3e2dc27 Soft-deprecate the compatibility settings 2024-07-10 16:24:31 +02:00
Robert Hensing d942d505ca Merge remote-tracking branch 'upstream/master' into more-nix-shell 2024-07-10 16:19:18 +02:00
Robert Hensing 13522229a9 assertEqValues: clarify potential bug error message 2024-07-10 16:08:10 +02:00
John Ericson 7a6269ba7b Package the Nix CLI with Meson
Co-Authored-By: Qyriad <qyriad@qyriad.me>
2024-07-08 17:49:44 -04:00
John Ericson f1d88e0f27
Merge pull request #11064 from obsidiansystems/header-generater
Use a meson "generator" to deduplicate `.gen.hh` creation
2024-07-08 12:03:59 -04:00
John Ericson 1439de874f
Merge pull request #11063 from obsidiansystems/config-expr
Forgot to include `config-expr.hh` in two places
2024-07-08 11:27:57 -04:00
John Ericson 6e5cec292b Use a meson "generator" to deduplicate .gen.hh creation 2024-07-08 11:13:11 -04:00
John Ericson c5284a84f3 Forgot to include config-expr.hh in some places 2024-07-08 10:37:34 -04:00
Robert Hensing cfe3ee3de8
nix-shell: look up shell.nix when argument is a directory (#11057)
* Refactor: rename runEnv -> isNixShell

* Refactor: rename left -> remainingArgs

* nix-build.cc: Refactor: extract baseDir variable

* nix-build.cc: Refactor: extract sourcePath, resolvedPath variables

* nix-shell: Look for shell.nix when directory is specified

* Add legacy setting: nix-shell-always-looks-for-shell-nix

* rl-next: Add note about shell.nix lookups

* tests/functional/shell.nix: Implement runHook for dummy stdenv
2024-07-08 14:36:36 +02:00
John Ericson d8850618b6
Merge pull request #11059 from rhendric/rhendric/reference-manual
docs: merge builtin-constants into builtins
2024-07-07 21:47:33 -04:00
Ryan Hendrickson 95890b3e1d docs: merge builtin-constants into builtins 2024-07-07 15:57:23 -04:00
Robert Hensing a9592077fb
Merge pull request #11055 from NixOS/packaging-for-nixpkgs
Indirections for packaging meson-based granular build for Nixpkgs
2024-07-07 21:33:01 +02:00
Robert Hensing fd4b17ab2c
Merge pull request #11056 from romain-neil/s3-allow-use-of-proxy
Configure aws s3 lib to use system defined proxy, if existent
2024-07-07 02:47:24 +02:00
Robert Hensing 63262e78c7 Add opt-out: nix-shell-shebang-arguments-relative-to-script 2024-07-07 00:55:33 +02:00
Robert Hensing 4c59d6e9f5 Merge branch 'nix-shell-lookup-shell-nix' into more-nix-shell 2024-07-07 00:27:07 +02:00
Robert Hensing 6c6d5263e2 Add legacy setting: nix-shell-always-looks-for-shell-nix 2024-07-07 00:25:56 +02:00
Robert Hensing f5b59fbc64 Fix and extend nix-shell baseDir test 2024-07-07 00:23:22 +02:00
Robert Hensing 8838f5c746 Merge remote-tracking branch 'matthewbauer/nix-shell-relative-shebang' into more-nix-shell 2024-07-07 00:18:03 +02:00
Robert Hensing b865625a8e nix-shell: Look for shell.nix when directory is specified 2024-07-06 23:05:34 +02:00
Robert Hensing 32fb127b9c Add legacy setting: nix-shell-always-looks-for-shell-nix 2024-07-06 23:05:34 +02:00
Romain NEIL 514062c227 feat: configure aws s3 lib to use system defined proxy, if existent 2024-07-06 21:46:58 +02:00
Robert Hensing 76245ffbeb nix-build.cc: Refactor: extract sourcePath, resolvedPath variables 2024-07-06 20:55:27 +02:00
Robert Hensing e9479b272f nix-build.cc: Refactor: extract baseDir variable 2024-07-06 20:51:45 +02:00
Robert Hensing 5c367ece89 Refactor: rename left -> remainingArgs 2024-07-06 20:03:30 +02:00
Robert Hensing 13181356fc Refactor: rename runEnv -> isNixShell 2024-07-06 20:01:46 +02:00
Robert Hensing 0729f0a113 packaging: Pass version directly 2024-07-06 17:52:57 +02:00
Robert Hensing efd5f50f5e nix-perl: Add deps, use mkMesonDerivation 2024-07-06 17:52:57 +02:00
Robert Hensing 4c014e238b nix-main: Add openssl 2024-07-06 17:52:57 +02:00
Robert Hensing 4d0c55ae55 api docs: Use mkMesonDerivation 2024-07-06 17:52:57 +02:00
John Ericson 3acf3fc746 Package libnixmain and libnixcmd with Meson
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-07-05 16:40:55 -04:00
Robert Hensing 0b901e10ee
Merge pull request #11050 from hercules-ci/issue-10677
Explain when `man` is missing
2024-07-05 22:25:38 +02:00
John Ericson ff9b6d0e1f
Merge pull request #11037 from fricklerhandwerk/document-config-parsing
use self-descriptive name for config file parser, document
2024-07-05 15:21:26 -04:00
Eelco Dolstra d5461b9009
Merge pull request #11051 from Mic92/fix-prefetch
src/nix/prefetch: fix prefetch containing current directory instead o…
2024-07-05 20:33:05 +02:00
Robert Hensing ddff76f667
Merge pull request #10973 from NixOS/meson-libexpr
Meson build for libexpr libflake, external C API, unit tests
2024-07-05 20:27:12 +02:00
Jörg Thalheim 05381c0b30
Update src/nix/prefetch.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-07-05 19:45:03 +02:00
Jörg Thalheim 8cea1fbd97 src/nix/prefetch: fix prefetch containing current directory instead of tarball
When --unpack was used the nix would add the current directory to the
nix store instead of the content of unpacked.
The reason for this is that std::distance already consumes the iterator.
To fix this we re-instantiate the directory iterator in case the
directory only contains a single entry.
2024-07-05 19:18:49 +02:00
Robert Hensing 6ef00a503a Explain when man is missing
Have you seen this man?

Fixes #10677
2024-07-05 19:18:23 +02:00
Eelco Dolstra 8f280d72ff
Merge pull request #11019 from DeterminateSystems/fix-failed-to-open-archive
Tarball fetcher: Fix handling of cached tarballs
2024-07-05 17:10:02 +02:00
Robert Hensing a476383f46
Merge pull request #11031 from emilazy/push-xsrvoyspsvqx
libstore: fix sandboxed builds on macOS
2024-07-05 17:08:39 +02:00
Robert Hensing d63bd8295e assert: Report why values aren't equal 2024-07-05 16:43:48 +02:00
Eelco Dolstra 98bef7c38e
Merge pull request #11035 from siddhantk232/refactor
Factor duplicate code into util function `append`
2024-07-05 16:34:23 +02:00
Eelco Dolstra 61e1880847
Merge pull request #11041 from hercules-ci/trace-nix-env-attribute-names
getDerivations: add attributes to trace
2024-07-05 16:32:27 +02:00
Eelco Dolstra e1b6b3ce27
Merge pull request #11020 from DeterminateSystems/fix-tarball-caching
Tarball fetcher: Fix fetchToStore() and eval caching
2024-07-05 16:30:12 +02:00
Eelco Dolstra e7e070d36b Document 2024-07-05 16:29:16 +02:00
Robert Hensing 09763c7cad getDerivations: add attributes to trace
This improves the error message of nix-env -qa, among others, which
is crucial for understanding some ofborg eval error reports, such as
https://gist.github.com/GrahamcOfBorg/89101ca9c2c855d288178f1d3c78efef

After this change, it will report the same trace, but also start with

```
error:
       … while evaluating the attribute 'devShellTools'

       … while evaluating the attribute 'nixos'

       … while evaluating the attribute 'docker-tools-nix-shell'

       … while evaluating the attribute 'aarch64-darwin'

       … from call site
         at /home/user/h/nixpkgs/outpaths.nix:48:6:
           47|   tweak = lib.mapAttrs
           48|     (name: val:
             |      ^
           49|       if name == "recurseForDerivations" then true

<same>
```
2024-07-05 15:30:07 +02:00
John Ericson e4056b9afd
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-07-04 17:48:27 -04:00
Emily af2e1142b1 libstore: fix sandboxed builds on macOS
The recent fix for CVE-2024-38531 broke the sandbox on macOS
completely. As it’s not practical to use `chroot(2)` on
macOS, the build takes place in the main filesystem tree, and the
world‐unreadable wrapper directory prevents the build from accessing
its `$TMPDIR` at all.

The macOS sandbox probably shouldn’t be treated as any kind of a
security boundary in its current state, but this specific vulnerability
wasn’t possible to exploit on macOS anyway, as creating `set{u,g}id`
binaries is blocked by sandbox policy.

Locking down the build sandbox further may be a good idea in future,
but it already has significant compatibility issues. For now, restore
the previous status quo on macOS.

Thanks to @alois31 for helping me come to a better understanding of
the vulnerability.

Fixes: 1d3696f0fb
Closes: #11002
2024-07-04 16:28:37 +01:00
Emily 76e4adfaac libstore: clean up the build directory properly
After the fix for CVE-2024-38531, this was only removing the nested
build directory, rather than the top‐level temporary directory.

Fixes: 1d3696f0fb
2024-07-04 16:22:02 +01:00
Valentin Gagarin c66079f1e8 use self-descriptive name for config file parser, document
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-07-04 10:36:48 +02:00
siddhantCodes 976c05879f factor duplicate code into util function append 2024-07-04 11:09:23 +05:30
Eelco Dolstra 5b4102c3b2 Tarball fetcher: Include revCount/lastModified in the fingerprint
This can influence the evaluation result so they should be included in
the fingerprint.
2024-07-03 22:05:45 +02:00
kn 10ccdb7a41 Use proper struct sockpeercred for SO_PEERCRED for OpenBSD
getsockopt(2) documents this;  ucred is wrong ("cr_" member prefix, no pid).
2024-07-03 11:16:39 -04:00
John Ericson a09360400b Ident some CPP in nix daemon
Makes it easier for me to read.
2024-07-03 11:15:56 -04:00
Eelco Dolstra 1ff186fc6e nix flake metadata: Show flake fingerprint
This is useful for testing/debugging and maybe for sharing eval caches
(since it tells you what file in ~/.cache/nix/eval-cache-v5 to copy).
2024-07-03 17:00:30 +02:00
Eelco Dolstra 9d95c228ee Tarball fetcher: Fix fetchToStore() and eval caching 2024-07-03 16:50:45 +02:00
John Ericson 30de61f16d
Merge pull request #11018 from siddhantk232/canonpath-fs-sink
Use `CanonPath` in `fs-sink.hh`
2024-07-03 10:36:18 -04:00
Eelco Dolstra 79ed3df8f8 Tarball fetcher: Fix handling of cached tarballs
Fixes a regression introduced in
5a9e1c0d20 where downloading a cached
file causes the error "Failed to open archive (Unrecognized archive
format)".
2024-07-03 14:14:20 +02:00
siddhantCodes 2cf24a2df0 fix tests and minor changes
- use the iterator in `CanonPath` to count `level`
- use the `CanonPath::basename` method
- use `CanonPath::root` instead of `CanonPath{""}`
- remove `Path` and `PathView`, use `std::filesystem::path` directly
2024-07-03 17:43:55 +05:30
Valentin Gagarin a713476790
docs: split types from syntax (#11013)
move together all syntactic and semantic information into one
page, and add a page on data types, which in turn links to the syntax and
semantics.

also split out the note on scoping rules into its own page.

Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu>
2024-07-03 09:03:41 +02:00
John Ericson 451f8a8c19 Put back files for now
We'll revert this sometime later
2024-07-02 09:26:22 -04:00
John Ericson 11946817f0 fileset for store unit test data 2024-07-02 09:26:22 -04:00
John Ericson 4d6bc61b8d Fix things 2024-07-02 09:26:22 -04:00
John Ericson 11dab30be9 Update docs on the unit tests 2024-07-02 09:26:22 -04:00
John Ericson 224c6c3256 Fix test symlinks 2024-07-02 09:26:22 -04:00
John Ericson 496b4a9cd2 Move around unit test dirs to match new names 2024-07-02 09:26:22 -04:00
John Ericson 3ad39d2afb Fix library name 2024-07-02 09:26:22 -04:00
John Ericson 513f6b9718 meson: Prelink links to avoid missing C++ initializers
This is the same as what the old build system did in
7eca8a16ea, done for the same reasons.
2024-07-02 09:26:22 -04:00
John Ericson 912c517bc0 Fix build of unit tests 2024-07-02 09:26:22 -04:00
John Ericson f7ce10dbc1 Fix static build 2024-07-02 09:26:22 -04:00
John Ericson 874ff000d4 Fix format 2024-07-02 09:26:22 -04:00
John Ericson 7312d13acc Keep another test dir 2024-07-02 09:26:22 -04:00
John Ericson 479befa76d More fixes 2024-07-02 09:26:22 -04:00
John Ericson 6a0582d9fd Rename file to avoid reserved name 2024-07-02 09:26:22 -04:00
John Ericson 2c184f694b Ensure we have data dir for libexpr unit tests 2024-07-02 09:26:22 -04:00
John Ericson 46ec69a483 Everything builds in the dev shell now 2024-07-02 09:26:22 -04:00
John Ericson 429d6ae2b5 Add missing package.nix 2024-07-02 09:26:22 -04:00
John Ericson 0b539dea4a Improve boost hacks 2024-07-02 09:26:22 -04:00
John Ericson 8399bd6b8f Dedup 2024-07-02 09:26:21 -04:00
John Ericson 8198888bc4 More dedup 2024-07-02 09:23:25 -04:00
John Ericson d6f57f3260 More dedup 2024-07-02 09:23:25 -04:00
John Ericson c88f83b471 More dedup 2024-07-02 09:23:25 -04:00
John Ericson 4609ab318c Fix internal API docs 2024-07-02 09:23:25 -04:00
John Ericson d902481a36 Better org 2024-07-02 09:23:25 -04:00
John Ericson a81e319528 Deduplicating 2024-07-02 09:23:24 -04:00
John Ericson 17a8c2bfce Unit tests and external libraries 2024-07-02 09:23:24 -04:00
John Ericson 4fa8068b78 Mesonify other external API 2024-07-02 09:23:24 -04:00
John Ericson 31257009e1 Meson build for libexpr and libflake 2024-07-02 09:23:24 -04:00
John Ericson fbdc554908 Fix Nix shell for building Perl too 2024-07-02 09:23:24 -04:00
Jörg Thalheim 101915c9b7 enable -Werror=unused-result
Inspired by
010ff57ebb

From the original PR:

> We do not have any of these warnings appearing at the moment, but
> it seems like a good idea to enable [[nodiscard]] checking anyway.
> Once we start introducing more functions with must-use conditions we will
> need such checking, and the rust stdlib has proven them very useful.
2024-07-02 08:46:06 +02:00
Robert Hensing f0329568b5 GitFileSystemObjectSink: catch an overflow 2024-07-01 17:29:07 +02:00
Robert Hensing 1fac22b16e GitFileSystemObjectSink: Add path context to some messages 2024-07-01 17:29:07 +02:00
Robert Hensing 992912f3b4 test-support: Add TracingFileSystemObjectSink 2024-07-01 17:29:07 +02:00
Eelco Dolstra 10c9764c27
Merge pull request #10992 from hercules-ci/issue-10947-dont-cache-disallowed-ifd
Fix #10947; don't cache disallowed IFD
2024-07-01 11:20:27 +02:00
Robert Hensing 93b50857ed packaging: Restore .version value altering behavior 2024-06-30 19:44:36 +02:00
Robert Hensing 7dd938b228 libutil/package.nix: Remove .version symlink replacement solution 2024-06-30 19:44:04 +02:00
siddhantCodes 72bb530141 use CanonPath in fs-sink and its derivatives 2024-06-30 19:03:15 +05:30
Robert Hensing b2c7f09b0a Fix underflow in Printer::printList
Analogous to 9b88bf8adf / three commits back
2024-06-29 14:10:58 +02:00
Robert Hensing bfc5416240 Refactor: rename ValuePrinter::totalListItemsPrinted 2024-06-29 14:02:28 +02:00
Robert Hensing ce1dc87711 Refactor: rename ValuePrinter::totalAttrsPrinted
Make it more distinct from the attrs printed of any specific
attrset.
2024-06-29 14:01:15 +02:00
Robert Hensing 9b88bf8adf Fix underflow in Printer::printAttrs
The code that counts the number of elided attrs incorrectly used the
per-printer "global" attribute counter instead of a counter that
was relevant only to the current attribute set.

This bug flew under the radar because often the attribute sets aren't
nested, not big enough, or we wouldn't pay attention to the numbers.
I've noticed the issue because the difference underflowed.

Although this behavior is tested by the functional test
lang/eval-fail-bad-string-interpolation-4.nix, the underflow slipped
through review. A simpler reproducer would be as follows, but I
haven't added it to the test suite to keep it simple and marginally
faster.

```
$ nix run nix/2.23.1 -- eval --expr '"" + (let v = { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; b = 1; c = 1; }; }; in builtins.deepSeq v v)'
error:
       … while evaluating a path segment
         at «string»:1:6:
            1| "" + (let v = { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; b = 1; c = 1; }; }; in builtins.deepSeq v v)
             |      ^

       error: cannot coerce a set to a string: { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; «4294967289 attributes elided» }; }
```
2024-06-29 13:53:05 +02:00
Robert Hensing fd94b74ee5 Fix #10947; don't cache disallowed IFD 2024-06-29 13:19:04 +02:00
Cole Helbling d906d42330 Restore commit-lock-file-summary rename for consistency
It was originally renamed in https://github.com/NixOS/nix/pull/10691,
but https://github.com/NixOS/nix/pull/9063 accidentally removed the new
name and alias.
2024-06-28 14:53:40 -07:00
Winter 149d8eb8aa Stop vendoring toml11
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).

Closes https://git.lix.systems/lix-project/lix/issues/164

Change-Id: Ied071c841fc30b0dfb575151afd1e7f66970fdb9
(cherry picked from commit 80405d06264f0de1c16ee2646388ab501df20628)
2024-06-26 22:27:13 -04:00
John Ericson f002f85861 Avoid libmain header in libexpr
We just don't need it!
2024-06-26 22:26:45 -04:00
John Ericson 7181d1f4a1 Reformat
Factored out code is now elegible for formatting.
2024-06-26 19:56:21 -04:00
John Ericson 0084a486cc Split out a new libnixflake
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-26 19:56:21 -04:00
John Ericson 52730d38e2 Factor out flake:... lookup path from evaluator
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-26 19:56:21 -04:00
John Ericson 8a420162ab Merge branch 'master' into fix-sandbox-escape 2024-06-26 18:11:39 -04:00
Robert Hensing 85de5a60c7 Use lib instead of explicit fileset passing 2024-06-26 04:11:20 +02:00
Robert Hensing 6fe8fb967a libstore/worker.hh: Document Worker 2024-06-26 01:43:31 +02:00
Robert Hensing fd0b376c79 libstore/worker.cc: Remove outdated comment
It was added above this conditional

    Worker::Worker(LocalStore & store)
        : store(store)
    {
        /* Debugging: prevent recursive workers. */
        if (working) abort();
        working = true;

However, `working` has since been removed.

Source: 7f8e805c8e/src/libstore/build.cc (L2617)
2024-06-26 01:43:31 +02:00
John Ericson 65d7c80365
Merge pull request #10955 from NixOS/meson-nix-util-c
Build nix-util-c with meson + unit tests
2024-06-25 19:06:06 -04:00
Robert Hensing ac89828b5a Build nix-util-c with meson and unit test 2024-06-25 21:35:23 +02:00
Robert Hensing 7df9d6da65 Improve error messages for invalid derivation names 2024-06-25 19:41:29 +02:00
John Ericson 5f4f789144
Merge pull request #10954 from NixOS/ci-meson
ci.yml: Add meson_build
2024-06-25 09:02:33 -04:00
Robert Hensing 0674be8d49 nix-util: Fix build 2024-06-25 10:26:57 +02:00
Brian McKenna 5be44d235a Guard uses of lutimes, for portability 2024-06-24 17:35:34 -04:00
John Ericson 05580a373f Fix error in the no-GC build 2024-06-24 17:18:16 -04:00
Robert Hensing c66f1e7660
Merge pull request #10913 from NixOS/no-global-eval-settings-in-libexpr
No global eval settings in `libnixexpr`
2024-06-24 18:52:19 +02:00
John Ericson fda4c78921
Merge pull request #10951 from obsidiansystems/load-just-one-config
Small global config refactors
2024-06-24 12:38:04 -04:00
John Ericson 52bfccf8d8 No global eval settings in libnixexpr
Progress on #5638

There is still a global eval settings, but it pushed down into
`libnixcmd`, which is a lot less bad a place for this sort of thing.
2024-06-24 12:15:16 -04:00
John Ericson cb0c868da4 Allow loading config files into other config objects
This gives us some hope of moving away from global variables.
2024-06-24 12:07:56 -04:00
John Ericson b46e13840b Format config-global.{cc,hh}
Since the code is factored out, it is no longer avoding the formatter.
2024-06-24 12:07:56 -04:00
John Ericson 1620ad4587 Split out GlobalConfig into its own header
This makes it easier to understand the reach of global variables /
global state in the config system.
2024-06-24 11:36:21 -04:00
John Ericson b51e161af5 Cleanup ContentAddressMethod to match docs
The old `std::variant` is bad because we aren't adding a new case to
`FileIngestionMethod` so much as we are defining a separate concept ---
store object content addressing rather than file system object content
addressing. As such, it is more correct to just create a fresh
enumeration.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-24 10:24:06 -04:00
John Ericson 64e599ebe1 Rename Recursive -> NixArchive
For enums:

- `FileIngestionMethod`

- `FileSerialisationMethod`
2024-06-24 10:24:06 -04:00
Eelco Dolstra 903acc7c0f
Merge pull request #10873 from siddhantk232/rm-createdirs
use `std::filesystem::create_directories` for createDirs
2024-06-24 14:54:37 +02:00
Eelco Dolstra bc21c54565
Merge pull request #10943 from pineapplehunter/master
Accept response from gitlab api with more than one entry in json
2024-06-24 14:23:47 +02:00
Robert Hensing 6f64154eea
Merge pull request #10884 from tomberek/tomberek.warn_structuredAttrs_advanced
fix: warn and document when advanced attributes will have no impact d…
2024-06-24 07:56:26 +02:00
John Ericson df068734ac
Merge pull request #10769 from poweredbypie/mingw-spawn
Implement runProgram for Windows
2024-06-23 14:12:36 -04:00
Shogo Takata 0468061dd2
accept response from gitlab with more than one entry 2024-06-23 00:52:19 +09:00
Eelco Dolstra d54590fdf3 Fix --no-sandbox
When sandboxing is disabled, we cannot put $TMPDIR underneath an
inaccessible directory.
2024-06-21 17:06:19 +02:00
Eelco Dolstra 58b7b3fd15 Formatting 2024-06-21 17:06:19 +02:00
Eelco Dolstra ede95b1fc1 Put the chroot inside a directory that isn't group/world-accessible
Previously, the .chroot directory had permission 750 or 755 (depending
on the uid-range system feature) and was owned by root/nixbld. This
makes it possible for any nixbld user (if uid-range is disabled) or
any user (if uid-range is enabled) to inspect the contents of the
chroot of an active build and maybe interfere with it (e.g. via /tmp
in the chroot, which has 1777 permission).

To prevent this, the root is now a subdirectory of .chroot, which has
permission 700 and is owned by root/root.
2024-06-21 17:06:19 +02:00
Théophane Hufschmitt 1d3696f0fb Run the builds in a daemon-controled directory
Instead of running the builds under
`$TMPDIR/{unique-build-directory-owned-by-the-build-user}`, run them
under `$TMPDIR/{unique-build-directory-owned-by-the-daemon}/{subdir-owned-by-the-build-user}`
where the build directory is only readable and traversable by the daemon user.

This achieves two things:

1. It prevents builders from making their build directory world-readable
   (or even writeable), which would allow the outside world to interact
   with them.
2. It prevents external processes running as the build user (either
   because that somehow leaked, maybe as a consequence of 1., or because
   `build-users` isn't in use) from gaining access to the build
   directory.
2024-06-21 17:06:19 +02:00
siddhantCodes 85b7989764 fix: handle errors in nix::createDirs
the `std::filesystem::create_directories` can fail due to insufficient
permissions. We convert this error into a `SysError` and catch it
wherever required.
2024-06-20 19:53:25 +05:30
Valentin Gagarin 1c131ec2b7
Port C API docs to Meson (#10936)
* Port C API docs to Meson

* don't cross-compile the docs
2024-06-19 22:43:54 +02:00
PoweredByPie 8b81d083a7 Remove lookupPathForProgram and implement initial runProgram test
Apparently, CreateProcessW already searches path, so manual path search
isn't really necessary.
2024-06-18 01:01:52 -07:00
PoweredByPie fcb92b4fa4 Fix DWORD vs. int comparison warning 2024-06-17 22:14:38 -07:00
Mingye Wang ff1fc780d2
optimize-store.cc: Update macos exclusion comments
#2230 broadened the scope of macOS hardlink exclusion but did not change the comments. This was a little confusing for me, so I figured the comments should be updated.
2024-06-18 12:05:59 +08:00
PoweredByPie 4f6e3b9402 Implement tests for lookupPathForProgram and fix bugs caught by tests 2024-06-17 18:46:08 -07:00
PoweredByPie 4662e7d856 Implement windowsEscape 2024-06-17 14:57:57 -07:00
Tom Bereknyei 706edf26eb build: meson for libfetchers 2024-06-17 17:25:56 -04:00
PoweredByPie b11cf8166f Format runProgram declaration 2024-06-17 13:12:28 -07:00
Valentin Gagarin 6e34c68327 Convert the internal API doc build to Meson 2024-06-17 15:51:58 -04:00
John Ericson a83d95e26e Integrate perl with the other meson builds
One big dev shell!
2024-06-17 14:48:20 -04:00
PoweredByPie a58ca342ca Initial runProgram implementation for Windows
This is incomplete; proper shell escaping needs to be done
2024-06-17 11:13:22 -07:00
John Ericson a1bb668ccb Merge remote-tracking branch 'upstream/master' into rm-createdirs 2024-06-17 12:57:54 -04:00
Robert Hensing e48abec567
Merge pull request #10916 from jmbaur/read-only-no-chown
Don't chown when local-store is read-only
2024-06-17 13:49:19 +02:00
Eelco Dolstra 48d38b32d2
Merge pull request #10918 from andir/restrict-tarfile-formats
Restrict supported tarball formats to actual Tarballs
2024-06-17 13:20:23 +02:00
PoweredByPie b0cfac8f93 Fix compile error on windows 2024-06-17 00:03:50 -07:00
Jared Baur de639ceafe
Don't chown when local-store is read-only
If the local-store is using the read-only flag, the underlying
filesystem might be read-only, thus an attempt to `chown` would always
fail.
2024-06-16 23:03:33 -07:00
Andreas Rammhold 5a9e1c0d20 Restrict supported tarball formats to actual Tarballs
The documentation is clear about the supported formats (with at least
`builtins.fetchTarball`). The way the code was written previously it
supported all the formats that libarchive supported. That is a
surprisingly large amount of formats that are likely not on the radar
of the Nix developers and users. Before people end up relying on
this (or if they do) it is better to break it now before it becomes a
widespread "feature".

Zip file support has been retained as (at least to my knowledge)
historically that has been used to fetch nixpkgs in some shell
expressions *many* years back.

Fixes https://github.com/NixOS/nix/issues/10917
2024-06-15 14:28:20 +02:00
Robert Hensing 573e385a68
Merge pull request #10907 from hercules-ci/issue-10561
C API: Use opaque struct instead of void for `nix_value`
2024-06-15 10:12:13 +02:00
John Ericson 4728840ca8
Merge pull request #10908 from NixOS/meson-libstore
Build `nix-store` with Meson, adjust Perl bindings deps to integrate with other meson builds
2024-06-14 12:16:14 -04:00
Eelco Dolstra b3bd31c82d
Merge pull request #10878 from kirillrdy/kr/sysv
darwin: allow ipc-sysv* in sandbox
2024-06-14 17:11:20 +02:00
Robert Hensing 61381c9964 C API: Make nix_err_msg treat NIX_OK as having no message
The documentation "solved" this by specifying a precondition, but
let's just make it more robust, and not leak irrelevant messages
that might linger.
We don't clear the message when clearing the status, in order to
keep clearing fast; see last_err field doc.
2024-06-14 16:36:23 +02:00
Robert Hensing 2dc7598779 C API: Add nix_clear_err 2024-06-14 16:30:34 +02:00
John Ericson 81004a05c6 Build nix-store with Meson
Special thanks to everyone that has worked on a Meson port so far,
@p01arst0rm and @Qyriad in particular.

Co-Authored-By: p01arst0rm <polar@ever3st.com>
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Felix Uhl <felix.uhl@outlook.com>
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Co-Authored-By: Lunaphied <lunaphied@lunaphied.me>
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Pierre Bourdon <delroth@gmail.com>
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: Rebecca Turner <rbt@sent.as>
Co-Authored-By: Winter <winter@winter.cafe>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
Co-Authored-By: jade <lix@jade.fyi>
Co-Authored-By: julia <midnight@trainwit.ch>
Co-Authored-By: rebecca “wiggles” turner <rbt@sent.as>
Co-Authored-By: wiggles dog <rbt@sent.as>
Co-Authored-By: fricklerhandwerk <valentin@fricklerhandwerk.de>
Co-authored-by: Eli Schwartz <eschwartz93@gmail.com>
2024-06-14 10:25:14 -04:00
John Ericson ea8e49bea5 Force the cpuid option for libutil rather than relying on detection
This is more robust, and match's Nixpkgs policy to force enable flags
statically by default (a common distro thing).
2024-06-14 10:25:14 -04:00
John Ericson 2f5fdab06c
Merge pull request #10905 from obsidiansystems/platform-namespace
Put some file descriptor functions in unix and windows namespaces
2024-06-14 08:45:31 -04:00
Robert Hensing b94e1d6218 C API: Value -> nix_value
See issue https://github.com/NixOS/nix/issues/10434
2024-06-13 18:51:58 +02:00
Robert Hensing 5d8118d9cb C API: Docs 2024-06-13 18:23:21 +02:00
Robert Hensing c50db4e58c C API: Add nix_value_{inc,dec}ref
- Can be implemented more easily by more eval architectures.
- Better types in generated bindings remove some uncertainty and doubt.
2024-06-13 18:21:04 +02:00
Robert Hensing 0b56c98b1c C API: Value -> nix_value 2024-06-13 18:18:36 +02:00
John Ericson 56f0b5304f Document the nix-util Meson build system more extensively
I hope this will make it easier to maintain, and also make it easier for
others to assist with porting the rest of the build system to Meson.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-13 11:39:59 -04:00
John Ericson ff87c1a318 Put some file descriptor functions in unix and windows namespaces
It is misleading when platform-specific functions are in the overall
`nix` namespace. More namespaces also makes for nicer doxygen.
2024-06-13 11:18:59 -04:00
Robert Hensing afdd12be5e
Merge pull request #10900 from obsidiansystems/hash-order
hash: Compare hash algo second for back compat
2024-06-13 01:35:31 +02:00
John Ericson 25a9894943 hash: Compare hash algo second for back compat
Previously (in cfc18a7739), we forgot to
compare the algo at all. This means we keep the same ordering as before
by making the stuff we always have compared take priority.
2024-06-12 18:35:47 -04:00
John Ericson 28d2af4ea6 Build nix-util with Meson
The idea is two-fold:

- Replace autotools with Meson

- Build each library in its own derivation

The interaction of these two features is that Meson's "subprojects"
feature (https://mesonbuild.com/Subprojects) allows us to have single
dev shell for building all libraries still, while also building things
separately. This allows us to break up the build without a huge
productivity lost.

I tested the Linux native build, and NetBSD and Windows cross builds.

Also do some clean ups of the Flake in the process of supporting new
jobs.

Special thanks to everyone that has worked on a Meson port so far,
@p01arst0rm and @Qyriad in particular.

Co-Authored-By: p01arst0rm <polar@ever3st.com>
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Felix Uhl <felix.uhl@outlook.com>
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Co-Authored-By: Lunaphied <lunaphied@lunaphied.me>
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Pierre Bourdon <delroth@gmail.com>
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: Rebecca Turner <rbt@sent.as>
Co-Authored-By: Winter <winter@winter.cafe>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
Co-Authored-By: jade <lix@jade.fyi>
Co-Authored-By: julia <midnight@trainwit.ch>
Co-Authored-By: rebecca “wiggles” turner <rbt@sent.as>
Co-Authored-By: wiggles dog <rbt@sent.as>
Co-Authored-By: fricklerhandwerk <valentin@fricklerhandwerk.de>
Co-authored-By: Eli Schwartz <eschwartz93@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-12 18:31:02 -04:00
John Ericson 5b53d8fec3 Factor out GC initialization code
This is not really part of the evaluator: it is just an integration
between Boehm GC and Boost coroutines usable for any purpose. The
evaluator (merely) optionally uses it.
2024-06-12 16:00:03 -04:00
John Ericson 7c2981fc55 Fix FreeBSD build
This restores some CPP'd code that was added in
c18911602e and accidentally lost in
2477e4e3b8.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-06-12 15:59:54 -04:00
Eelco Dolstra efd4bf6533
Update src/libfetchers/git-utils.cc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-12 14:41:35 +02:00
Eelco Dolstra 35bdb9cee7 Support hard links in tarballs
Fixes #10395.
2024-06-11 16:05:57 +02:00
Tom Bereknyei 4809e59b7e fix: warn and document when advanced attributes will have no impact due to __structuredAttrs 2024-06-10 09:31:21 -04:00
Tom Bereknyei 1363f51bcb fix: remove usage of XDG_RUNTIME_DIR for TMP 2024-06-10 08:38:57 -04:00
Robert Hensing f91f34aa65 bdwgc 8.2.4 has sp_corrector
> Support client-defined stack pointer adjustment before thread stack push

-- https://github.com/ivmai/bdwgc/releases/tag/v8.2.4

This fixes an inaccuracy in cc6f315252,
in the update to Nixpkgs 24.05 https://github.com/NixOS/nix/pull/10835

After this fixup, the build log won't ask for an upgrade, and we'll be
able to collect when a coroutine exists, e.g. during filterSource.
2024-06-10 12:57:13 +02:00
Robert Hensing 0a09597790 Typo 2024-06-10 12:46:04 +02:00
Robert Hensing 4755e133c4 Fix warning 2024-06-10 12:46:04 +02:00
siddhantCodes 7a21432e77 fix: catch filesystem_error thrown by createDirs 2024-06-10 11:30:39 +05:30
Kirill Radzikhovskyy 372d5a441e darwin: allow ipc-sysv* in sandbox 2024-06-10 11:21:17 +10:00
siddhantCodes 25b0242ca6 std::filesystem::create_directories for createDirs
The implementation of `nix::createDirs` allows it to be a simple wrapper
around `std::filesystem::create_directories` as its return value is not
used anywhere.
2024-06-09 19:49:39 +05:30
Robert Hensing 0ab9369572
Merge pull request #10854 from DeterminateSystems/thread-safe-RegexCache
Make RegexCache thread-safe
2024-06-08 18:07:35 +02:00
Robert Hensing e1b3716d50
Merge pull request #10853 from DeterminateSystems/thread-safe-srcToStore
Make EvalState::srcToStore thread-safe
2024-06-06 19:31:46 +02:00
Eelco Dolstra fd9e49480a PosixSourceAccessor: Use SharedSync 2024-06-06 10:44:53 +02:00
Eelco Dolstra 2f39caf180 Sync: Add support for shared locks 2024-06-06 10:44:09 +02:00
Robert Hensing dadb6f86cf
Merge pull request #10865 from Mic92/better-cleanup
libutil: guard Finally against invalid exception throws
2024-06-05 23:59:51 +02:00
John Ericson 1a32490aca
Merge pull request #10835 from hercules-ci/update-nixpkgs-24.05
Update to Nixpkgs 24.05
2024-06-05 17:56:16 -04:00
eldritch horrors e291087747 libutil: guard Finally against invalid exception throws
throwing exceptions is fine, but throwing exceptions during exception
handling is hard enough to do correctly that we should just forbid it
entirely out of an overabundance of caution. in cases where terminate
is the correct answer the users of Finally must call it manually now.

Source: 6c777476c9
2024-06-05 22:45:37 +02:00
Eelco Dolstra edd445f745
Merge pull request #10861 from DeterminateSystems/fix-optional-deref
PackageInfo::queryDrvPath(): Don't dereference an empty optional
2024-06-05 16:48:09 +02:00
Eelco Dolstra 3e72ed9743 Typo 2024-06-05 16:19:01 +02:00
Eelco Dolstra d2eeabf3e6 PackageInfo::queryDrvPath(): Don't dereference an empty optional
Fixes a regression introduced in f923ed6b6a.

https://hydra.nixos.org/build/262267313
2024-06-05 16:17:24 +02:00
Enno Richter 80ba7778e7 flake check: Recognize well known homeModule/homeModules attributes 2024-06-05 07:38:20 +02:00
Jörg Thalheim 49c6f34911
docs: fixup description of builtins.importNative (#10810)
There was an argument missing and the fact that the imported function is
called.
2024-06-04 19:55:05 +00:00
Eelco Dolstra ef140c25d1
Merge pull request #10852 from siddhantk232/add-interrupts
add call to `checkInterrupt` in a bunch of places
2024-06-04 17:21:03 +02:00
Eelco Dolstra fbbca59453 Make RegexCache thread-safe 2024-06-04 16:59:30 +02:00
Eelco Dolstra 7d295c594e Make EvalState::srcToStore thread-safe 2024-06-04 16:56:06 +02:00
siddhantCodes 8f1a26667e add call to checkInterrupt in a bunch of places
This brings back the old behaviour. We check for interrupts in places
that may iterate over wide directories.
2024-06-04 19:35:40 +05:30
Eelco Dolstra 21be03b233
Merge pull request #10840 from obsidiansystems/libutil-pkg-config
Create and install a `nix-util.pc`
2024-06-04 12:33:37 +02:00
John Ericson 4e0d058fc3 eval.cc: Fix for Windows 2024-06-04 10:18:22 +02:00
Philipp 214051ba79
clarify not on nix_value_force (#10842)
* clarify not on `nix_value_force`

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-06-04 07:41:04 +00:00
John Ericson 06be6812a6 Create and install a nix-util.pc
Before, `-lnixutil` was just stuck in `nix-store.pc`, but that doesn't
seem so nice.

This prepares us to distribute `libnixutil` in a separate package if we
want, but it should be a good change either way. I suspect it wasn't
done before because libutil was an extra unstable interface, but I don't
think we need worry about that. *All* the C++ is less stable than the C
(or that's the goal at least).

For what it's worth, Lix also created this pkg-config file *en passant*
during their rename:
c97e17144e (diff-3c4f60cc44a0e35444c7f45331cfa50f76637118)
2024-06-03 14:14:40 -04:00
Robert Hensing 27f880c098 Format after clang-format update 2024-06-03 18:38:56 +02:00
Eelco Dolstra da92ad7dd2
Merge pull request #10592 from hercules-ci/builtins-warn
Add `builtins.warn`
2024-06-03 17:16:32 +02:00
Robert Hensing cc6f315252 nix: Disable GC during coroutine when bdwgc < 8.4
This re-enables support for older bwdgc versions without complicating
the code too much.
Coroutines generally only interfere with GC during source filtering,
so it's not too bad of a regression on older bdwgc.
This seems preferable over conditional compilation to enable the patch
etc; we've already spent a lot of complexity budget on this GC-coroutine
interaction...
2024-06-03 16:37:39 +02:00
Robert Hensing 2477e4e3b8 libexpr: Use GC_set_sp_corrector instead of patch
Manually tested by printing to stderr in both branches (sp in os
stack, or not), and triggering a GC in a filterSource function,
e.g.:

    let
      generateTree = n: if n == 0 then "ha" else { left = generateTree (n - 1); right = generateTree (n - 1); };
    in
      builtins.deepSeq (generateTree 18) ...

Note that the darwin still uses the strategy of disabling GC, despite
having an implementation that compiles. The proper solution will be
enabled and tested later.
2024-06-03 16:37:39 +02:00
Robert Hensing 70b1036224 builtins.warn: Use new EvalBaseError + "evaluation warning" 2024-06-03 16:24:21 +02:00
Robert Hensing 831d96d8d7 builtins.warn: Do not throw EvalError 2024-06-03 16:24:21 +02:00
Robert Hensing c07500e14d refactor: Extract EvalState::{runDebugRepl,canDebug} 2024-06-03 16:24:21 +02:00
Robert Hensing da82d67022 builtins.warn: Require string argument
... so that we may perhaps later extend the interface.
Note that Nixpkgs' lib.warn already requires a string coercible
argument, so this is reasonable. Also note that string coercible
values aren't all strings, but in practice, for warn, they are.
2024-06-03 16:24:21 +02:00
Robert Hensing 923cbea2af builtins.warn: Use logWarning
Constructing ErrorInfo is a little awkward for now, but this does
produce a richer log entry.
2024-06-03 16:24:21 +02:00
Robert Hensing 2d4c9d8f4a Add builtins.warn 2024-06-03 16:24:21 +02:00
Eelco Dolstra 54a9fbe5d6 Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 16:17:52 +02:00
Eelco Dolstra eb0d46fab6
Merge pull request #9897 from bryango/fix-submodule-subdir
libutil/url: fix git+file:./ parse error
2024-06-03 16:04:41 +02:00
Eelco Dolstra ac3e5d22e3
Merge pull request #10028 from DavHau/fetchTree-shallow-default
fetchTree: shallow git fetching by default
2024-06-03 16:02:34 +02:00
John Ericson 4e62629a2d
Merge pull request #10833 from obsidiansystems/hash-ordering
Modernize `Hash` ordering with C++20 `<=>`
2024-06-03 09:50:04 -04:00
Eelco Dolstra deac00c6d0 Rename large-path-warning-threshold -> warn-large-path-threshold 2024-06-03 15:49:15 +02:00
Eelco Dolstra 7f5b57d18f Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 15:32:27 +02:00
Eelco Dolstra ecfad6a828
Merge pull request #10564 from edolstra/remove-forceErrors
AttrCursor: Remove forceErrors
2024-06-03 15:30:01 +02:00
John Ericson cfc18a7739 Modernize Hash ordering with C++20 <=>
Progress on #10832

This doesn't switch to auto-deriving the fields, but by defining `<=>`
we allow deriving `<=>` in downstream types where `Hash` is used.
2024-06-03 09:24:33 -04:00
Eelco Dolstra d16fcaee21
Merge pull request #10782 from obsidiansystems/both-connections
Factor our connection code for worker proto like serve proto
2024-06-03 15:10:38 +02:00
John Ericson 84c65135a5 ValidPathInfo JSON format should use null not omit field
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-03 08:21:22 -04:00
John Ericson 213a7a87b4 Decouple within-build (structured attrs) and unstable CLI path info JSON
See code comment for details.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-03 08:21:22 -04:00
Robert Hensing b74a0df645
Merge pull request #10825 from tie/output-spec-assert
Fix empty outputsToInstall for InstallableAttrPath
2024-06-03 12:27:50 +02:00
Philipp Zander 25e2b1f7f7 improve note in nix_value_force documentation 2024-06-03 09:55:44 +02:00
Ivan Trubach 68090d7ff1 Fix empty outputsToInstall for InstallableAttrPath
Fixes assertion failure if outputsToInstall is empty by defaulting to the "out"
output. That is, behavior between the following commands should be consistent:

	$ nix build --no-link --json .#nothing-to-install-no-out
	error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'

	$ nix build --no-link --file default.nix --json nothing-to-install-no-out
	error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'

Real-world example of this issue:

	$ nix build --json .#.legacyPackages.aarch64-linux.texlive.pkgs.iwona
	error: derivation '/nix/store/dj0h6b0pnlnan5nidnhqa0bmzq4rv6sx-iwona-0.995b.drv' does not have wanted outputs 'out'

	$ git rev-parse HEAD
	eee33247cf6941daea8398c976bd2dda7962b125
	$ nix build --json --file . texlive.pkgs.iwona
	nix: src/libstore/outputs-spec.hh:46: nix::OutputsSpec::Names::Names(std::set<std::__cxx11::basic_string<char> >&&): Assertion `!empty()' failed.
	Aborted (core dumped)
2024-06-02 14:26:18 +03:00
John Ericson e0b159549b Misc Windows fixes
1. Fix build by making the legacy SSH Storey's secret `logFD` setting
   not a setting on Windows. (It doesn't make sense to specify `void *`
   handles by integer cross-proccess, I don't think.)

2. Move some files that don't need to be Unix-only anymore back to their
   original locations.
2024-06-01 19:19:35 -04:00
Robert Hensing 802b4e403b
Merge pull request #10814 from Mic92/fix-nix-edit
Fix nix edit
2024-05-31 13:30:24 +02:00
Robert Hensing 84e116379c
Merge pull request #10812 from Mic92/build-perf
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
2024-05-31 13:28:24 +02:00
Jade Lovelace 473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Robert Hensing 138aa2b0a7
Merge pull request #10807 from hercules-ci/issue-10504-nix-env-shell
Add `nix env shell`
2024-05-31 12:34:03 +02:00
Jörg Thalheim e1a817fb1b fix nix edit in pure mode
FilteringSourceAccessor was not delegating getPhysicalPath to its inner accessor.
2024-05-31 10:39:30 +02:00
Linus Heckemann a9031978da libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent

warning: unknown experimental feature 'repl-flake'
error:
       … while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the git ref really exist?

Adapted from: 3df013597d
2024-05-31 08:24:53 +02:00
Robert Hensing d93cc11491 Format 2024-05-30 19:41:58 +02:00
Robert Hensing c692f6af13 nix env shell: Move from nix shell, add shorthand alias 2024-05-30 19:41:58 +02:00
Robert Hensing 98b85b2166 nix/main: Add AliasStatus::{Deprecated,AcceptedShorthand} 2024-05-30 19:41:58 +02:00
Robert Hensing ef5c846e25
Merge pull request #10768 from obsidiansystems/legacy-ssh-expose-ssh-master-for-hydra
Create `CommonSSHStoreConfig::createSSHMaster`
2024-05-29 22:53:29 +02:00
Robert Hensing 1054ff0873
Merge pull request #10789 from nix-windows/windows-substitution-goal
More work on the scheduler for windows
2024-05-29 22:45:55 +02:00
Robert Hensing 154769544d
Merge pull request #10805 from hercules-ci/issue-10774
libcmd: Fix #10774
2024-05-29 22:39:00 +02:00
Robert Hensing 1c70eb8eee libcmd: Fix #10774 2024-05-29 21:42:53 +02:00
Qyriad 18ac6545fc print type and value in "flake attr is not a derivation" errors
This turns errors like:

error: flake output attribute 'hydraJobs' is not a derivation or path

into errors like:

error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }

This change affects all InstallableFlake commands.

Source: 20981461d4
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-29 20:51:32 +02:00
Jörg Thalheim 5786e1ae7c
docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation (#10803)
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation

Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.

Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-05-29 07:50:51 +00:00
Philipp 2bd66922ee
add empty line to documentation comments after @brief field (#10800)
* add empty line to documentation comments after `@brief` field

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-05-28 23:05:40 +00:00
John Ericson bcdee80a0d More work on the scheduler for windows
- Get a rump derivation goal: hook instance will come later, local
  derivation goal will come after that.

- Start cleaning up the channel / waiting code with an abstraction.
2024-05-28 11:39:49 -04:00
John Ericson 1d5d748fe4 Fix format
39b2a399ad passed CI but was landed after
the formatting change in 1d6c2316a9.
2024-05-27 22:32:52 -04:00
John Ericson d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal
Start building the scheduler for Windows
2024-05-27 17:47:29 -04:00
John Ericson 3e9c3738d3 Create CommonSSHStoreConfig::createSSHMaster
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.

Progress on #10766

Progress on https://github.com/NixOS/hydra/issues/1164
2024-05-27 16:12:53 -04:00
John Ericson 17964441d9
Merge pull request #10781 from obsidiansystems/build-mode-parse
Worker proto use proper serialiser for `BuildMode`
2024-05-27 16:07:54 -04:00
John Ericson 263905da4b
Merge pull request #10785 from mjoerg/fix-typos
fix typos
2024-05-27 10:24:53 -04:00
John Ericson 8527f4e7fa
Merge pull request #10754 from qwqawawow/master
nix repl: make runNix() isInteractive is true by default
2024-05-27 10:07:15 -04:00
Martin Joerg e7ea5591a2 fix typos 2024-05-27 15:56:52 +02:00
John Ericson aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion
repl: do not crash when tab-completing import errors
2024-05-27 09:54:29 -04:00
John Ericson f71b4da0b3 Factor our connection code for worker proto like serve proto
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.

Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.

The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.

Fixes #9584
2024-05-27 00:43:46 -04:00
John Ericson 8ebd99c74e Back in enum values for BuildMode serializer
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
2024-05-27 00:22:55 -04:00
John Ericson eeb89c28b0 Worker proto use proper serialiser for BuildMode
Do this instead of an unchecked cast

I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.

Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
2024-05-27 00:22:55 -04:00
Pierre Bourdon ffe6ba69d6 repl: do not crash when tab-completing import errors
File not found while importing is not currently caught by the tab-completion handler.

Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189

Example crash:

$ cat /tmp/foo.nix
{
  someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
2024-05-25 23:15:28 +02:00
Jörg Thalheim f97da4b11c libutil/source-accessor: custom error if source does not exist
This allows better error handling by catching this error in particular.
2024-05-25 23:06:57 +02:00
eldritch horrors 2de4589e46 libstore: remove unused copyPath function
Source: 47523944c5

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-25 22:28:41 +02:00
John Ericson 5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize
Implement `setStackSize` on Windows
2024-05-25 09:56:02 -04:00
John Ericson e0c94b91ee
Merge pull request #10757 from obsidiansystems/fix-4977
Require `drvPath` attribute to end with `.drv`
2024-05-24 12:14:59 -04:00
Eelco Dolstra eeb4c40867 Typo 2024-05-24 16:35:06 +02:00
Eelco Dolstra 2c88930ef2 AttrCursor: Remove forceErrors
Instead, force evaluation of the original value only if we need to
show the exception to the user.
2024-05-24 16:34:57 +02:00
Robert Hensing c90a763273
Merge pull request #10767 from hercules-ci/fix-c-api-primop-for-strict-initializers
C API: Fix custom primops
2024-05-24 08:32:02 +02:00
PoweredByPie 0b7da099d1 Commit more stack size in some windows binaries
This way we can commit the same amount of stack size (64 MB) without a conditional.
Includes nix, libnixexpr-tests, libnixfetchers-tests, libnixstore-tests, libnixutil-tests.
2024-05-23 17:42:55 -07:00
Robert Hensing 2497d10351 C API: Add nix_value_call_multi, NIX_VALUE_CALL
_multi can be implemented more efficiently.
NIX_VALUE_CALL is a convenient way to invoke it.
2024-05-23 21:24:06 +02:00
Robert Hensing 4bc4fb40ea C API: builtin -> custom function
Not all primops will be in `builtins`.
2024-05-23 21:23:14 +02:00
Robert Hensing 8ef6efc184 C API: Require non-thunk value from primop definition 2024-05-23 21:22:59 +02:00
Robert Hensing 8884227045 C API: Require initialized value from primop definition 2024-05-23 21:22:21 +02:00
Robert Hensing a942a34469 C API: Fix nix_c_primop_wrapper for strict initializers
https://github.com/NixOS/nix/pull/10555 added a check requiring
that output parameters always have an uninitialized Value as argument.

Unfortunately the output parameter of the primop callback received
a thunk instead.

See the comment for implementation considerations.
2024-05-23 18:32:49 +02:00
John Ericson 5384ceacc3 Document field being initialized in Machine constructor 2024-05-23 11:28:25 -04:00
PoweredByPie 5f68e6d69f Get max stack size in setStackSize to match Linux 2024-05-23 03:54:35 -07:00
John Ericson f2bcebc450 Restore exposing machine file parsing
This was accidentally removed in
e989c83b44. I restored it and also did a
few other cleanups:

- Make a static method for namespacing purposes

- Put the test files in the data dir with the other test data

- Avoid mutating globals in the machine config tests

This will be used by Hydra.
2024-05-23 00:03:52 -04:00
John Ericson 4a19f4a866
Merge pull request #9839 from obsidiansystems/more-machine-cleanup
Create `StoreReference` and use it in `Machine`
2024-05-22 17:01:57 -04:00
John Ericson dc7615dbbb tryUnshareFilesystem: Ignore ENOSYS too
Fixes #10747
2024-05-22 16:07:38 -04:00
John Ericson d5fdfdc592 unshareFilesystem: Do not assume caller 2024-05-22 16:07:34 -04:00
John Ericson f923ed6b6a Require drvPath attribute to end with .drv
Fixes #4977
2024-05-22 12:50:24 -04:00
John Ericson b3ebcc5aad Use the new StoreReference in Machine
This makes the remote builder abstract syntax more robust.
2024-05-22 09:20:15 -04:00
John Ericson b59a7a14c4 Add StoreReference::render
This will be needed for the next step.

Also allows us to write round trip tests.
2024-05-22 09:20:15 -04:00
John Ericson c036d75f9e Factor out abstract syntax for Store URIs
Need to decouple parsing from actually opening a store for Machine
configs.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-22 09:20:15 -04:00
John Ericson 1d6c2316a9 Slightly change formatting style
For long expressions, one argument or parameter per line is just easier.
2024-05-22 09:20:15 -04:00
Robert Hensing bd7a074636
Merge pull request #10724 from prednaz/libexpr-c_example
remove redundant and outdated example from `libexpr-c` documentation
2024-05-22 12:59:14 +02:00
John Ericson 470c0501eb Ensure all store types support "real" URIs
In particular `local://<path>` and `unix://` (without any path) now
work, and mean the same things as `local` and `daemon`, respectively. We
thus now have the opportunity to desguar `local` and `daemon` early.

This will allow me to make a change to
https://github.com/NixOS/nix/pull/9839 requested during review to
desugar those earlier.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2024-05-21 11:56:40 -04:00
eihqnh bb1a4ea21a nix repl: make runNix() isInteractive is true by default 2024-05-21 20:05:43 +08:00
Philipp Zander 142222030c remove redundant and outdated example from libexpr-c documentation 2024-05-21 14:00:34 +02:00
John Ericson 8b369f90fd Query path infos (plural) and handshake version minimum for hydra
1. Hydra currently queries for multiple path infos at once, so let us
   make a connection item for that.

2. The minimum of the two versions should always be used, see #9584.
   (The issue remains open because the daemon protocol needs to be
   likewise updated.)
2024-05-20 17:43:18 -04:00
Robert Hensing 9a2f21711d
Merge pull request #10494 from obsidiansystems/primop-better-method
Slight refactors in preparation for #10480
2024-05-20 16:03:15 +02:00
Robert Hensing 20ed0c02b8
Merge pull request #10688 from hraban/fix/develop-aliases
fix: don’t expand aliases in develop stdenv setup
2024-05-20 15:59:21 +02:00
Robert Hensing 67db9e0c64
Merge pull request #10722 from obsidiansystems/ca-obj-docs
Content addressing store objects
2024-05-20 15:58:29 +02:00
John Ericson 1c75af969a Document store object content addressing & improve JSON format
The JSON format no longer uses the legacy ATerm `r:` prefixing nonsese,
but separate fields.

Progress on #9866

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-20 09:34:35 -04:00
Robert Hensing e4be8abe42
Merge pull request #10701 from cole-h/nix-command-warn-unknown-settings
Warn on unknown settings when the first positional is an argument
2024-05-20 15:15:13 +02:00
John Ericson 43dc575fd7
Merge pull request #10738 from poweredbypie/mingw-windowSize
Implement `updateWindowSize` for Windows
2024-05-20 08:57:22 -04:00
John Ericson d48bbda2e7
Update the updateWindowSize documentation 2024-05-20 08:34:49 -04:00
Eelco Dolstra 927034e7ac value.hh: Shut up warning about useless const qualifier 2024-05-20 10:25:04 +02:00
Philipp e0bfa6c55f
small additions to the documentation of nix_store_open and nix_state_create (#10728) 2024-05-20 08:27:33 +02:00
PoweredByPie a41f4223de Use setStackSize on Windows 2024-05-18 16:19:01 -07:00
PoweredByPie 6a3f906382 Implement setStackSize for Windows 2024-05-18 16:19:01 -07:00
PoweredByPie e42d00c961 Change rlim_t to size_t in setStackSize in preparation of Windows impl 2024-05-18 16:18:50 -07:00
PoweredByPie 53f0c44d6c Implement updateWindowSize for Windows 2024-05-18 16:14:20 -07:00
John Ericson 1ab107b0bf
Merge pull request #10734 from Mic92/better-keepgoing
nix3-build: show all FOD errors with `--keep-going`
2024-05-17 18:46:57 -04:00
John Ericson d25e54d56a
Merge pull request #10733 from alyssais/nix-store-delete
Improve nix-store --delete failure message
2024-05-17 18:41:23 -04:00
Qyriad 05bc889b89 add and fix -Wignored-qualifiers
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-17 19:29:50 +02:00
Qyriad 52a16b7e59 add and fix -Wdeprecated-copy
*so* many warnings, from only two definitions

Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-17 19:25:35 +02:00
Maximilian Bosch 19720d733f nix3-build: show all FOD errors with --keep-going
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.

The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.

Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).

Closes https://git.lix.systems/lix-project/lix/issues/302

Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-17 18:48:52 +02:00
Alyssa Ross 979a019014
Improve nix-store --delete failure message
On several occasions I've found myself confused when trying to delete
a store path, because I am told it's still alive, but
nix-store --query --roots doesn't show anything.  Let's save future
users this confusion by mentioning that a path might be alive due to
having referrers, not just roots.
2024-05-17 18:20:22 +02:00
Philipp Zander f63292462c document nix_external_print's printer parameter to be an out parameter 2024-05-16 02:55:38 +02:00
Philipp Zander 359043ed0d add missing c api parameter names to documentation 2024-05-16 02:55:38 +02:00
Robert Hensing 303268bb71
Merge pull request #10479 from obsidiansystems/ca-fso-docs
Document file system object content addressing
2024-05-15 22:52:53 +02:00
John Ericson 043135a848 Document file system object content addressing
In addition:

- Take the opportunity to add a bunch more missing hyperlinks, too.

- Remove some glossary entries that are now subsumed by dedicated pages.
  We used to not be able to do this without breaking link fragments, but
  now we can, so pick up where we left off.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-15 16:28:48 -04:00
Eelco Dolstra 3026613893
Merge pull request #10675 from edolstra/zip-symlinks
Handle zip files containing symlinks
2024-05-15 22:07:22 +02:00
Cole Helbling 39a269657e libutil/args: warn on unknown settings after parsing all flags 2024-05-15 12:25:03 -07:00
Eelco Dolstra 39f7cbdc7c
Merge pull request #10691 from DeterminateSystems/commit-lock-file-summary
Rename commit-lockfile-summary to commit-lock-file-summary for consistency
2024-05-14 10:06:48 +02:00
Graham Christensen e1e041ed8f Rename commit-lockfile-summary to commit-lock-file-summary for consistency 2024-05-13 09:23:59 -04:00
siddhantCodes 39e8aad446 Merge branch 'master' of github.com:NixOS/nix 2024-05-13 18:43:12 +05:30
Eelco Dolstra 56abd341bb
Merge pull request #10685 from siddhantk232/fs-cleanup
inline the usage of `nix::renameFile`, `nix::getFileType` and `nix::copyFile`
2024-05-13 14:12:10 +02:00
siddhantCodes 62e1ea2f4b use path for from arg in nix::copyFile 2024-05-13 16:10:21 +05:30
Eelco Dolstra 5534682166 Update src/libutil/util.hh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-13 12:14:43 +02:00
siddhantCodes 4d0777ca69 fix: copy fileName before calling std::distance 2024-05-13 15:36:00 +05:30
Eelco Dolstra cf3b044b7e Make large path warnings human-readable 2024-05-13 11:52:38 +02:00
Eelco Dolstra 5314430437 Move printSize() into libutil
Also always include the unit (i.e. "MiB" instead of "M").
2024-05-13 11:52:38 +02:00
Eelco Dolstra dbe1b51580 Add setting to warn about copying/hashing large paths
This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:

   $ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
   warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
   warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
   warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
   warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
2024-05-13 11:52:27 +02:00
Eelco Dolstra d8559cad8d
Merge pull request #10686 from DeterminateSystems/long-commit-message
git putFile: support flake maximalists
2024-05-13 10:22:57 +02:00
Hraban Luyat 6bf7edb18b fix: don’t expand aliases in develop stdenv setup
This fixes https://github.com/NixOS/nixpkgs/pull/290775 by not expanding aliases
when sourcing the stdenv setup script. The way bash handles aliases is to expand
them when a function is defined, not when it is used. I.e.:

    $ alias echo="echo bar "
    $ echo foo
    bar foo
    $ xyzzy() { echo foo; }
    $ shopt -u expand_aliases
    $ xyzzy
    bar foo
    $ xyzzy2() { echo foo; }
    $ xyzzy2
    foo

The problem is that ~/.bashrc is sourced before the stdenv setup, and bashrc
commonly sets aliases for ‘cp’, ‘mv’ and ‘rm’ which you don’t want to take
effect in the stdenv derivation builders. The original commit introducing this
feature (5fd8cf7667) even mentioned this very
alias.

The only way to avoid this is to disable aliases entirely while sourcing the
stdenv setup, and reenable them afterwards.
2024-05-12 21:57:55 -04:00
Graham Christensen 8b5e8f4fba git putFile: support flake maximalists
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes.
2024-05-12 16:42:43 -04:00
Robert Hensing c940d11fb0
Merge pull request #10666 from tie/derivation-outputs-drv-path
Forbid drvPath in strictDerivation outputs attribute
2024-05-12 21:14:32 +02:00
siddhantCodes ccf94545db rename copy -> copyFile and remove old copyFile
the old `copyFile` was just a wrapper that was calling the `copy`
function. This wrapper function is removed and the `copy` function is
renamed to `copyFile`.
2024-05-12 19:20:17 +05:30
siddhantCodes d3b7367c80 inline usage of nix::getFileType and remove it 2024-05-12 18:58:05 +05:30
siddhantCodes 4537663740 inline the usage of nix::renameFile
use `std::filesystem::rename` everywhere and remove `nix::renameFile`
2024-05-12 18:40:16 +05:30
siddhantCodes 1db7d1b840 inline the usage of nix::readDirectory
`nix::readDirectory` is removed. `std::filesystem::directory_iterator`
is used directly in places that used this util.
2024-05-12 17:42:18 +05:30
John Ericson 39b2a399ad Start building the scheduler for Windows
Building derivations is a lot harder, but the downloading goals is
portable enough.

The "common channel" code is due to Volth. I wonder if there is a way we
can factor it out into separate functions / files to avoid some
within-function CPP.

Co-authored-by: volth <volth@volth.com>
2024-05-10 20:23:59 -04:00
John Ericson e0ff8da9d5 Build the local store on Windows
Fixes #10558

Co-Authored-By: Eugene Butler <eugene@eugene4.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-05-10 13:05:23 -04:00
John Ericson 0998a3ac01 Remove LocalStore::OptimiseStats::blocksFreed as it is dead code 2024-05-10 12:55:21 -04:00
Eelco Dolstra 9951e14ae0 Handle zip files containing symlinks
In streaming mode, libarchive doesn't handle symlinks in zip files
correctly. So write the entire file to disk so libarchive can access
it in random-access mode.

Fixes #10649. This was broken in cabee98152.
2024-05-09 19:33:09 +02:00
Eelco Dolstra de8c3c034c
Merge pull request #10668 from edolstra/unit-prefixes
Support unit prefixes in configuration settings
2024-05-09 19:29:36 +02:00
John Ericson b5605217ae
Document string context (#8595)
* Document string context

Now what we have enough primops, we can document how string contexts
work.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2024-05-08 23:14:00 +02:00
Eelco Dolstra 77a406a5a6 Fix warning 2024-05-08 21:16:53 +02:00
Eelco Dolstra 79c7d6205c Support unit prefixes in configuration settings
E.g. you can now say `--min-free 1G`.
2024-05-08 21:11:09 +02:00
Ivan Trubach 081faeda8c Forbid drvPath in strictDerivation outputs attribute
builtins.strictDerivation returns an attribute set with drvPath and
output paths. For some reason, current implementation forbids drv
instead of drvPath.
2024-05-08 18:20:37 +03:00
siddhantCodes ddea4c6deb rm isLink
isLink util is removed in favour of std::filesystem::is_symlink
2024-05-08 19:59:37 +05:30
Siddhant Kumar fcbc36cf78
Use std::filesystem::path in more places (#10657)
Progress on #9205

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>

* Get rid of `PathNG`, just use `std::filesystem::path`
2024-05-07 22:28:50 +00:00
John Ericson 9ae6455b0e
Merge pull request #10658 from nix-windows/more-std-filesystem
Use `std::filesystem` functions in more places
2024-05-07 16:52:37 -04:00
John Ericson 72a0d4b022 Try to fix macOS Nixpkgs lib test failure
Sometimes we read a directory with children we cannot stat. It's a pitty
we even try to stat at all (wasteful) in the `DT_UNKNOWN` case, but at
least this should get rid of the failure.
2024-05-07 16:21:02 -04:00
John Ericson a3c573950b Replace our DirEntry with std::filesystem's 2024-05-07 16:21:02 -04:00
Eelco Dolstra d641e8f717 builtin:fetchurl: Revert impureEnvVars attribute
This was changed in #10611, which caused the derivation paths of
anything using builtin:fetchurl to change (i.e. all of
Nixpkgs). However, impureEnvVars doesn't actually do anything for
builtin:fetchurl, so we can just set it to its historical value.
2024-05-07 11:25:07 +02:00
John Ericson c371070580 Use std::filesystem functions in more places
This makes for shorter and more portable code.

The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.

Good for windows portability; will help @siddhantk232 with his GSOC
project.
2024-05-07 00:16:54 -04:00
ramboman b4950404ba
Honor the same set of proxy environment variables (#10611)
Different parts of the project honor different sets of proxy environment
variables. With this commit all parts of the project will honor the same
set of proxy environment variables.

---------

Co-authored-by: Your Name <you@example.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2024-05-06 15:39:22 -04:00
Eelco Dolstra da3381d51f
Merge pull request #10465 from edolstra/remove-locked
Fetcher cache cleanups
2024-05-06 21:32:26 +02:00
Eelco Dolstra c7216a416f
Update src/libfetchers/cache.hh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-06 21:11:41 +02:00
John Ericson 1ad7b5451d
Merge pull request #10655 from edolstra/use-source-path
Use `SourcePath` in more places
2024-05-06 14:57:01 -04:00
John Ericson 5e189025ca Fix build failure with clang
A slight issue with feb1d10f60.
2024-05-06 13:39:21 -04:00
Eelco Dolstra ef28c7329c Rename makeFSSourceAccessor -> getFSSourceAccessor()
This makes it clearer that it returns a shared accessor object.
2024-05-06 19:16:52 +02:00
Eelco Dolstra eab2919119 Use SourcePath in more places
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
2024-05-06 19:05:42 +02:00
Eelco Dolstra 2926ef0e90
Merge pull request #10652 from tweag/check-additionalSandboxProfile
libstore: check additionalSandboxProfile
2024-05-06 17:59:25 +02:00
Théophane Hufschmitt 9bd1191fcc libstore: check additionalSandboxProfile
Make sure that `extraSandboxProfile` is set before we check whether it's
empty or not (in the `sandbox=true` case).

Also adds a test case for this.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2024-05-06 17:37:08 +02:00
Eelco Dolstra 709cd44d3e Rename remaining instances of "InputAccessor" to "SourceAccessor" 2024-05-06 17:29:03 +02:00
Eelco Dolstra ee2fa87a7e
Merge pull request #10642 from edolstra/remove-input-accessor
Merge InputAccessor into SourceAccessor
2024-05-06 15:48:25 +02:00
Charlie Moog 71c66de227
document store url trusted=true option behavior 2024-05-05 17:37:04 +00:00
Eelco Dolstra ffc280f27a Formatting 2024-05-03 15:41:03 +02:00
Eelco Dolstra 20558e0462 Remove FSInputAccessor 2024-05-03 12:30:28 +02:00
Eelco Dolstra ba5929c7be Merge InputAccessor into SourceAccessor
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
2024-05-03 12:14:01 +02:00
John Ericson 037c8d771d Fix format errors
Fix formatting violations, update blacklist to reflect moved files.

PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
2024-05-02 21:42:28 -04:00
John Ericson 840267491e
Merge pull request #10633 from hercules-ci/fix-eval-state-baseEnv-gc-root
libexpr: Add missing GC root for `baseEnv`
2024-05-02 10:10:54 -04:00
John Ericson 1948ec390c
Merge pull request #10556 from nix-windows/uds-remote-on-windows
Enable the `unix://` store on Windows
2024-05-02 09:53:00 -04:00
Robert Hensing f34b52b521 libexpr: Add missing GC root for baseEnv
This missing GC root wasn't much of a problem before, because the
heap would end up with a reference to the `baseEnv` pretty soon,
but when unit testing, the construction of `EvalState` doesn't
necessarily happen well before GC runs for the first time.

Found while unit testing the Rust bindings that currently reside
at https://github.com/nixops4/nixops4/tree/main/rust
2024-05-01 22:36:39 +02:00
Robert Hensing e17aad23d6
Merge pull request #10555 from jlesquembre/jl/c-api_check-init
Add isValid to nix::Value
2024-05-01 16:33:01 +02:00
Eelco Dolstra 4d99d07bc9 Whitespace 2024-04-30 15:34:35 +02:00
Sarah Brofeldt e5f509ef0b nix repl: hide progress bar during :edit 2024-04-29 10:03:34 +02:00
Eelco Dolstra ff107d9d03 Input::fetchToStore(): Don't try to substitute
Having a narHash doesn't mean that we have the other attributes
returned by the fetcher (such as lastModified or rev). For instance,

   $ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f
   Last modified: 2024-01-15 10:51:22

but

   $ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f?narHash=sha256-PPXqKY2hJng4DBVE0I4xshv/vGLUskL7jl53roB8UdU%3D
   (does not print a "Last modified")

The latter only happens if the store path already exists or is
substitutable, which made this impure behaviour unpredictable.

Fixes #10601.
2024-04-26 16:41:33 +02:00
Eelco Dolstra de51e5c335
Merge pull request #10570 from layus/shared_caches
Share evaluation caches across installables
2024-04-26 15:48:46 +02:00
John Ericson 4722b0c9e9
Merge pull request #10602 from haenoe/json-infra-tests-misc
Json infra tests misc
2024-04-25 16:53:51 -04:00
Eelco Dolstra 0353d6f79e
Merge pull request #10588 from hercules-ci/fix-nested-submodules
Fix fetchGit/fetchTree for nested submodules
2024-04-25 22:53:22 +02:00
Guillaume Maudoux 19cc50dcbf
fixup: Compute fingerprint only if needed 2024-04-25 00:47:46 +02:00
Guillaume Maudoux 1c4e392c64
Compute fingerprint only if needed
As per Eelco's review comments

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-04-25 00:44:47 +02:00
HaeNoe 943a877a6a
use default value in fetchers::PublicKey json deserialization 2024-04-24 18:23:04 +02:00
Guillaume Maudoux a60a1f09b2 Reuse eval caches and related values when possible 2024-04-22 20:32:41 +02:00
Dylan Green aa165301d1
Pathlocks Implementation for Windows (#10586)
Based on Volth's original port.

Co-authored-by: volth <volth@volth.com>
2024-04-22 15:08:10 +00:00
Robert Hensing 750bcaa330 Fix fetchGit nested submodules 2024-04-22 16:41:40 +02:00