diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc index 585aba5..d699102 100644 --- a/docs/modules/ROOT/nav.adoc +++ b/docs/modules/ROOT/nav.adoc @@ -1,6 +1,10 @@ * xref:index.adoc[] -* xref:go.adoc[] -* xref:zexp.adoc[] -* Code +* Core concepts +** xref:core/derivations.adoc[] +** xref:core/zexp.adoc[] +** xref:core/vfs.adoc[] +** xref:core/zilch-as-lib.adoc[] + +* Code reference include::generated:partial$nav.adoc[] diff --git a/docs/modules/ROOT/pages/core/derivations.adoc b/docs/modules/ROOT/pages/core/derivations.adoc new file mode 100644 index 0000000..9ed7a7a --- /dev/null +++ b/docs/modules/ROOT/pages/core/derivations.adoc @@ -0,0 +1,95 @@ += Content-addressed derivations and the Zilch build scheduler +:page-pagination: next + +Zilch implements its own content-addressed derivations, along with a subset of +the Nix build scheduler. This brings a few important features with it, such as +derivation fallbacks and post-build hooks. It also means that content-addressed +derivations can be experimented with inside Zilch without worrying about +changes to Nix breaking it, and without enabling any `experimental-features`. + +== Content-addressed derivations +Currently, the content-addressed derivation format used by Zilch closely +matches that of Nix, with a few subtleties. content-addressed derivations are +never written to the Nix store, as that would require Nix support them. + +If a Zilch CA derivation is built, it is translated to an input-addressed +derivation and built. Each output of this derivation is then queried and the +NAR hash is used to copy the output path to its final, content-addressed, +destination.footnote:[This means self-references aren't supported; though as +these are not correctly handled in Nix either this problem has been punted for +now.] These store paths are then used to substitute the placeholder in the +generated derivation for any derivation that depends on it. + +The output of these content-addressed derivations has the Deriver set properly, +so it's possible to use the Nix CLI to quickly query the (input-addressed +equivalent) derivation used to build them, using a shell script: + +[source,bash] +---- +nix derivation show "$(nix path-info --json /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello | jq -r '.[].deriver') +---- + + +== Build hooks +By implementing the content-addressed derivation logic itself, Zilch is able to +provide a few features Nix is unable to: Build hooks. When a content-addressed +derivation fails to build, it is possible to swap out the derivation, and +instead build a different derivation. This is done using xref:generated:zilch.magic.adoc#store-path-register-fallback[`store-path-register-fallback`], +which takes a thunk that is evaluated when the build fails, and returns a new +derivation to build. + +[source,scheme] +---- +(define fails-to-build + (cdar (store-path-for-ca-drv + "hello" "x86_64-linux" + '("/bin/sh" "-c" "echo i fail to build") + '() '("out"))) +;; # + +(define fallback + (cdar (store-path-for-ca-drv + "hello" "x86_64-linux" + '("/bin/sh" "-c" "echo hi > $out") + '() '("out"))) +;; # + +(store-path-register-fallback fails-to-build + (lambda () fallback)) +;; # + +(store-path-realised fails-to-build) +;> [..building "/nix/store/qmx86vqz7pl191vwpcc8zrpnh98rils8-hello.drv"] +;> [0/1 builds, 1 running] +;> i fail to build +;> +;> Error: daemon logger received error: (error "Error" 0 "builder for '\x1b[35;1m/nix/store/qmx86vqz7pl191vwpcc8zrpnh98r...") +;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"] +;> [0/1 builds, 1 running] +;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello" +---- + +The post-build hook works similarly, but calls the callback with an `alist` of output name to output store path: + +[source,scheme] +---- +(define example + (cdar (store-path-for-ca-drv + "hello" "x86_64-linux" + '("/bin/sh" "-c" "echo hi > $out") + '() '("out"))) +;; # + +(store-path-register-post-build example + (lambda (vals) (write vals) (newline))) +;; # + +(store-path-realised example) +;> (("out" . "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello")) +;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello" +---- diff --git a/docs/modules/ROOT/pages/core/vfs.adoc b/docs/modules/ROOT/pages/core/vfs.adoc new file mode 100644 index 0000000..e42b97c --- /dev/null +++ b/docs/modules/ROOT/pages/core/vfs.adoc @@ -0,0 +1,54 @@ += The Zilch Virtual Filesystem +:page-pagination: + +In many places, it's necessary to build up a structure of files and +directories. These structures are a subset of e.g. a project's source code. To +make these, Zilch has a `vfs` record. + +Core Zilch can create a VFS from two sources: an on-disk directory, as well as +a Nix store path: + +[,scheme,line-comment=;] +---- +(vfs-from-directory "/home/puck/example") +;; #> + +(define example-vfs + (vfs-from-store + (zdir "zero" (zsymlink "/dev/zero"))) +;; #> +---- + +These VFSes can be read out using `vfs-contents`: + +[,scheme,line-comment=;] +---- +(mapping->alist + (vfs-contents example-vfs)) +;; ((("" . "zero") +;; . # "/dev/zero">)) +---- + +VFS records store their contents as an SRFI 146 mapping, with a pair +`(dir . name)` as key, and the store path (or zexpr) to point at as its value. +Directories are indicated by the directory itself having a `'directory` symbol +as contents; these markers are used during serialization. + +`(zilch vfs)` contains a series of helpers to make standard changes to the +filesystem; such as filtering, and reading subtrees. + +== Serializing virtual filesystems to the store + +When a virtual filesystem is written to the store using `vfs-to-store`, a +xref:generated:zilch.file.adoc[`(zilch file)`]-based structure is created, +which can then be ``zexp-unquote``d safely inside a `zexp`. An important caveat +to this, however, is that the created structure uses symlinks to every file in +the vfs, rather than copying. This is chosen to both limit the expense of +copying the contents of a large file to the store when many VFSes contain it, +and a logistical limitation in the way Zilch interacts with Nix. It's possible +this restriction will be lifted in the future. + +== Language-specific + +Zilch's Go support has a special handler that can create a VFS from a `go.sum` +line. See the xref:go/library.adoc#vfs[Go] documentation for this procedure. diff --git a/docs/modules/ROOT/pages/core/zexp.adoc b/docs/modules/ROOT/pages/core/zexp.adoc new file mode 100644 index 0000000..cae42ad --- /dev/null +++ b/docs/modules/ROOT/pages/core/zexp.adoc @@ -0,0 +1,78 @@ += ``zexp``s, store paths, and context +:page-pagination: + +In Nix, strings carry around a "context", corresponding to a list of +dependencies that string has. For example, the `outPath` of a derivation in Nix +has a context that contains its derivation, plus the output name. When strings +are concatenated, their contexts are merged together. + +In Zilch, a similar concept exists; encapsulated in so-called "zexpressions". +A `zexpr` is a lazily evaluated thunk that, when evaluated, has access to other +``zexpr``s and closely related objects. When a `zexpr` is unwrapped, it returns +the resulting value, plus a list of derivations and outputs that are depended +on. + +[source,scheme] +---- +(define example (zexp foo)) +;; # + +(zexp-unwrap example) +;; # +---- + +The primary way a zexp that contains context is made, is by using +xref:generated:zilch.magic.adoc#store-path-for-drv[`store-path-for-drv`] and xref:generated:zilch.magic.adoc#store-path-for-ca-drv[`store-path-for-ca-drv`]. This returns a +``, which can be unwrapped like a `zexp`. + +[source,scheme] +---- +(define drv + (store-path-for-drv + "hello" "x86_64-linux" + '("/bin/sh" "-c" "echo hi > $out") '() + '("out")) + +;; (("out" +;; . #)) + +(zexp-unwrap (cdar drv)) +;; # "out")); +;; srcs: ()> +---- + +A `zexp` behaves similarly to the `quasiquote` macro, in that it is possible to +use `unquote` (or its reader syntax) to unquote values. Unquoting a `zexp`, +however, requires a special mechanism, and is executed before `unquote`. This +allows for full expressiveness when generating, for example, arguments or +environments for derivations. + +[source,scheme] +---- +(define complex + (zexp + (unquote + (string-append + "The path for the derivation's output is " + (zexp-unquote (cdar #3))))) + +(zexp-unwrap complex) +;; # "out")); +;; srcs: ()> +---- + +A store path, after being created by `store-path-for-(ca-)drv`, is not +guaranteed to exist in the store. To ensure it exists, xref:generated:zilch.magic.adoc#store-path-materialize[`store-path-materialize`] +is used; this is transparently done when a `store-path` is unwrapped inside a +`zexp`. + +== Reader syntax +For convenience, `+#~foo+` is used as reader syntax for `(zexp foo)`, while +`+#$foo+` and `+#$@foo+` are syntax for `(zexp-unwrap foo)` and `(zexp-unquote-splicing foo)` +respectively. + diff --git a/docs/modules/ROOT/pages/core/zilch-as-lib.adoc b/docs/modules/ROOT/pages/core/zilch-as-lib.adoc new file mode 100644 index 0000000..3ec1a5a --- /dev/null +++ b/docs/modules/ROOT/pages/core/zilch-as-lib.adoc @@ -0,0 +1,193 @@ += Using Zilch as a library +:page-pagination: prev + +Zilch is able to be used as a library; provided you have knowledge of Chicken +Scheme. + +Once the Zilch libraries are installed (or inside a `nix-shell` inside the +Zilch repo), open an interpreter (or an editor). + +These examples assume you are running in an interpreter with +`(scheme base)`, `(zilch zexpr)`, and `(zilch magic)` in scope. + +== Creating and building derivations + +To start off, creating an arbitrary derivation can be done with one call: + +[,scheme,line-comment=;] +---- +(define normal-derivation + (store-path-for-drv + "hello" ; <1> + "x86_64-linux" ; <2> + '("/bin/sh" "-c" "echo hi > $out") ; <3> + '() ; <4> + '("out"))) ; <5> +;; (("out" . #)) +---- +<1> Name for the derivation (used in output store paths) +<2> Architecture that this derivation needs to be built on. +<3> Derivation builder and its arguments (unlike Nix, these are considered one list) +<4> Any environment variables to set whilst executing the builder +<5> The list of output names to use for this derivation. + +This procedure returns an alist of `` records, which refer to +derivation store paths. These derivations may or may not exist in the Nix +store, however, as Zilch only writes them to the store when necessary, or when +requested. To do this manually, call `store-path-materialize`: + +[,scheme] +---- +(store-path-materialize (cdar normal-derivation)) +---- + +After materialization, it's now possible to build this derivation, both +inside Zilch and outside it. + +[,console] +---- +$ nix-build '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out' +this derivation will be built: + /nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv +building '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv'... +/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello + +$ cat /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello +hi +---- + +[,scheme] +---- +(store-path-build (cdar normal-derivation)) +;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"] +;> [0/1 builds, 1 running] +;; 1 +---- + +These store paths are a type of ````; a format used to describe data that +depends on paths from the Nix store. + +== Zexpressions +While it's possible to manually build every store paths you need, and handle +them as plain strings, this is cumbersome and error-prone. It also doesn't work +when you have to deal with placeholders, which are used for content-addressed +derivations. So, to handle these, the `store-path` concept is extended to +generic Scheme expressions. These use `zexp` (or `++#~++`) and you can use +`zexp-unquote` (or `++#$++`) to read out the contents of one `zexp` inside another. +Finally, to read out a `zexp` outside another `zexp`, you can use `zexp-unwrap`: + +[,scheme] +---- +(define example-zexp + (zexp (zexp-unquote (cdar normal-derivation)))) +;; # + +(define unwrapped (zexp-unwrap example-zexp)) +;; # "out")), +;; srcs: ()> +---- + +A `zexp` represents an arbitrary S-expression. It's also possible to run code +when a `zexp` is used, similarly to `quasiquote`. Like `quasiquote`, this uses +`unquote` (or its syntax sugar). Doing this also retains the context of any +``zexp-unquote``d value: + +[,scheme] +---- +(define more-complex + (zexp ,(string-append + "The store path is at " + (zexp-unquote example-zexp)))) + +(zexp-unwrap more-complex) +;; # "out")), +;; srcs: ()> +---- + +These expressions can then be used in derivations (e.g. through +`store-path-for-drv`) too: + +[,scheme] +---- +(define another-derivation + (store-path-for-drv "hello2" "x86_64-linux" + '("/bin/sh" "-c" "echo \"$foo\" > $out") + (list (cons "foo" more-complex)) + '("out"))) +;; (("out" . #)) + +(store-path-materialize (cdar another-derivation)) + +(store-path-build (cdar another-derivation)) +;! /nix/store/nz1fv397b3cix74d58i8kh2nj8knvb72-hello2 contains +;! "The store path is at /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello" +---- + +As ``zexp``s aren't limited to strings, it's possible to write expressive +derivations relatively easy. + +[,scheme] +---- +(define another-derivation + (store-path-for-drv "hello2" "x86_64-linux" + #~("/bin/sh" "-c" ,(string-append "echo \"" #$more-complex "\" > $out")) + '() + '("out"))) +;; (("out" . #)) + +(store-path-materialize (cdar another-derivation)) +---- + +== Content-addressed derivations +Zilch implements xref:./derivations.adoc[its own content-addressed derivations], +which can be used almost as easily as input-addressed derivations. However, +they have an important caveat: Their output store paths are only known at build +time. To improve performance, the store paths render as a "placeholder", which +gets substituted at build time. + +[,scheme] +---- +(define content-addressed-derivation + (store-path-for-ca-drv "hello" "x86_64-linux" + '("/bin/sh" "-c" "echo hi > $out") + '() + '("out"))) +;; (("out" . #)) +---- + +Content addressed store paths can be determined by the output path not starting +with the Nix store, and the `ca~` marker in their representation. These +derivations also cannot be materialized; `store-path-materialize` will ignore +them. + +When these store paths are used in other content-addressed derivations, they +will work as intended. Their output, when ``zexp-unwrap``ed, however, will +still contain placeholders: + +[,scheme] +---- +(zexp-unwrap (cdar content-addressed-derivation)) +;; # "out")), +;; srcs: ()> +---- + +To resolve their placeholders, you need to use `store-path-realised`. This will +evaluate the `zexp` or `store-path`, and resolve placeholders: + +[,scheme] +---- +(store-path-realised (cdar content-addressed-derivation)) +;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello" +---- + +Alternatively, it's possible to use `store-path-devirtualise`. This does the +same thing, but returns a `zexp`, which can then be used in an input-addressed +derivation. diff --git a/docs/modules/ROOT/pages/go.adoc b/docs/modules/ROOT/pages/go.adoc deleted file mode 100644 index 213b01e..0000000 --- a/docs/modules/ROOT/pages/go.adoc +++ /dev/null @@ -1,66 +0,0 @@ -= Go - -Currently, the primary binary produced by the `zilch` repository is a tool to -generate content-addressed derivations to incrementally build any Go project, -called `zilch-cli-go`. - -It requires the `ca-derivations` and `impure-derivations` experimental features -to be enabled on the Nix daemon, and a Nix daemon to be available at the -default store path. It also requires the daemon to be able to run -`x86_64-linux` derivations, right now. - -Once run, it will use Zilch to build a series of derivations, and output a -`.drv` for each executable package in the module. These derivations will then -incrementally build the source code: if a package's changes do not impact its -compatibility with the packages that depend on it, it will quickly notice and -skip building dependents, thanks to Nix. - -== Example usage - -[source] ----- -$ git clone https://github.com/tailscale/tailscale # tailscale.com -$ git clone https://go.googlesource.com/net x-net # golang.org/x/net - -# build tailscale, but with the golang.org/x/net dependency replaced -# with the local checkout. this might take a few minutes, as it pulls in -# all dependencies one by one. -$ zilch-cli-go -m tailscale -r x-net tailscale.com/cmd/tailscaled - -# use `--option substitute false', as there are a lot of paths to check -# against the binary cache otherwise -$ nix-build --option substitute false /nix/store/qj0dh4pdgldxkb798pj1f3fs5n1nwnx3-tailscale.com_cmd_tailscaled.drv - -[.. change e.g. `GODEBUG` to `GODEBUG2` in `x-net/http2/http2.go` ..] -$ zilch-cli-go --module-dir tailscale --replace x-net tailscale.com/cmd/tailscaled -$ nix-build --option substitute false /nix/store/nm2ycs5zb7v2wdlwgy913apnd22gs4pn-tailscale.com_cmd_tailscaled.drv ----- - -Once `zilch-cli-go` returns, it returns a `.drv`, which, when built, will -incrementally build the changed modules. If a package has been changed without -impacting the exported functions too much, only it will be rebuilt, with all -packages that depend on it staying the same, and being skipped. - - -== Help page - -[source] ----- -Usage: zilch-cli-go [OPTION] [PACKAGE...] -Process the given module (or the current directory, if unspecified) and -output derivations for each package given on the command line (or all -executables in the module, if unspecified) - - -h, --help Print this help message. - -b, --build Build the store paths, rather than show their - derivations. - -L, --print-build-logs Print derivation logs as they come in. - -m, --module-dir DIR The directory to use as root module. - -r, --replace DIR Replace the module specified by the go.mod - with this source directory, rather than using - the upstream module. Can be specified more - than once. - - --debug Crash on the first error, rather than - continuing with the next package. ----- diff --git a/docs/modules/ROOT/pages/index.adoc b/docs/modules/ROOT/pages/index.adoc index aa3d958..07d1b0b 100644 --- a/docs/modules/ROOT/pages/index.adoc +++ b/docs/modules/ROOT/pages/index.adoc @@ -13,10 +13,6 @@ Features: * Batteries included * Intercompatible with Nixpkgs and other, arbitrary, Nix expressions. -== Current work -Current effort in Zilch is working on making "incremental", bitesize, -derivations to work. This is xref:go.adoc[currently being implemented for Go]. - == Contributing Come join [.line-through]#us# me at `#zilch` on https://libera.chat[libera.chat]!