puck/zilch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Document core concepts

Browse source 
Change-Id: I6a6a6964d6bded229cd640463eaac70fd52df233
This commit is contained in:
puck 2025-06-23 12:22:20 +00:00
parent e76c57a388
commit f0ce185d5c
7 changed files with 427 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/modules/ROOT/nav.adoc
View file

@ -1,6 +1,10 @@
* xref:index.adoc[] * xref:index.adoc[]
* xref:go.adoc[]
* xref:zexp.adoc[]
* Code * Core concepts
** xref:core/derivations.adoc[]
** xref:core/zexp.adoc[]
** xref:core/vfs.adoc[]
** xref:core/zilch-as-lib.adoc[]
* Code reference
include::generated:partial$nav.adoc[] include::generated:partial$nav.adoc[]

95
docs/modules/ROOT/pages/core/derivations.adoc Normal file
View file

@ -0,0 +1,95 @@
= Content-addressed derivations and the Zilch build scheduler
:page-pagination: next
Zilch implements its own content-addressed derivations, along with a subset of
the Nix build scheduler. This brings a few important features with it, such as
derivation fallbacks and post-build hooks. It also means that content-addressed
derivations can be experimented with inside Zilch without worrying about
changes to Nix breaking it, and without enabling any `experimental-features`.
== Content-addressed derivations
Currently, the content-addressed derivation format used by Zilch closely
matches that of Nix, with a few subtleties. content-addressed derivations are
never written to the Nix store, as that would require Nix support them.
If a Zilch CA derivation is built, it is translated to an input-addressed
derivation and built. Each output of this derivation is then queried and the
NAR hash is used to copy the output path to its final, content-addressed,
destination.footnote:[This means self-references aren't supported; though as
these are not correctly handled in Nix either this problem has been punted for
now.] These store paths are then used to substitute the placeholder in the
generated derivation for any derivation that depends on it.
The output of these content-addressed derivations has the Deriver set properly,
so it's possible to use the Nix CLI to quickly query the (input-addressed
equivalent) derivation used to build them, using a shell script:
[source,bash]
----
nix derivation show "$(nix path-info --json /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello | jq -r '.[].deriver')
----
== Build hooks
By implementing the content-addressed derivation logic itself, Zilch is able to
provide a few features Nix is unable to: Build hooks. When a content-addressed
derivation fails to build, it is possible to swap out the derivation, and
instead build a different derivation. This is done using xref:generated:zilch.magic.adoc#store-path-register-fallback[`store-path-register-fallback`],
which takes a thunk that is evaluated when the build fails, and returns a new
derivation to build.
[source,scheme]
----
(define fails-to-build
(cdar (store-path-for-ca-drv
"hello" "x86_64-linux"
'("/bin/sh" "-c" "echo i fail to build")
'() '("out")))
;; #<store path /0qln6zx68yai0x802a804hvwszvddjpkhv87phsdy0snvlnmr3q8
;; (ca~ /nix/store/7cp7gl3sdi6k9kvrg0b628fq4khlk5py-hello.drv!out)>
(define fallback
(cdar (store-path-for-ca-drv
"hello" "x86_64-linux"
'("/bin/sh" "-c" "echo hi > $out")
'() '("out")))
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
(store-path-register-fallback fails-to-build
(lambda () fallback))
;; #<store path /0qln6zx68yai0x802a804hvwszvddjpkhv87phsdy0snvlnmr3q8
;; (ca~ /nix/store/7cp7gl3sdi6k9kvrg0b628fq4khlk5py-hello.drv!out)>
(store-path-realised fails-to-build)
;> [..building "/nix/store/qmx86vqz7pl191vwpcc8zrpnh98rils8-hello.drv"]
;> [0/1 builds, 1 running]
;> i fail to build
;>
;> Error: daemon logger received error: (error "Error" 0 "builder for '\x1b[35;1m/nix/store/qmx86vqz7pl191vwpcc8zrpnh98r...")
;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"]
;> [0/1 builds, 1 running]
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
----
The post-build hook works similarly, but calls the callback with an `alist` of output name to output store path:
[source,scheme]
----
(define example
(cdar (store-path-for-ca-drv
"hello" "x86_64-linux"
'("/bin/sh" "-c" "echo hi > $out")
'() '("out")))
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
(store-path-register-post-build example
(lambda (vals) (write vals) (newline)))
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
(store-path-realised example)
;> (("out" . "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"))
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
----

54
docs/modules/ROOT/pages/core/vfs.adoc Normal file
View file

@ -0,0 +1,54 @@
= The Zilch Virtual Filesystem
:page-pagination:
In many places, it's necessary to build up a structure of files and
directories. These structures are a subset of e.g. a project's source code. To
make these, Zilch has a `vfs` record.
Core Zilch can create a VFS from two sources: an on-disk directory, as well as
a Nix store path:
[,scheme,line-comment=;]
----
(vfs-from-directory "/home/puck/example")
;; #<zilch.vfs#<vfs>>
(define example-vfs
(vfs-from-store
(zdir "zero" (zsymlink "/dev/zero")))
;; #<zilch.vfs#<vfs>>
----
These VFSes can be read out using `vfs-contents`:
[,scheme,line-comment=;]
----
(mapping->alist
(vfs-contents example-vfs))
;; ((("" . "zero")
;; . #<z-symlink -> "/dev/zero">))
----
VFS records store their contents as an SRFI 146 mapping, with a pair
`(dir . name)` as key, and the store path (or zexpr) to point at as its value.
Directories are indicated by the directory itself having a `'directory` symbol
as contents; these markers are used during serialization.
`(zilch vfs)` contains a series of helpers to make standard changes to the
filesystem; such as filtering, and reading subtrees.
== Serializing virtual filesystems to the store
When a virtual filesystem is written to the store using `vfs-to-store`, a
xref:generated:zilch.file.adoc[`(zilch file)`]-based structure is created,
which can then be ``zexp-unquote``d safely inside a `zexp`. An important caveat
to this, however, is that the created structure uses symlinks to every file in
the vfs, rather than copying. This is chosen to both limit the expense of
copying the contents of a large file to the store when many VFSes contain it,
and a logistical limitation in the way Zilch interacts with Nix. It's possible
this restriction will be lifted in the future.
== Language-specific
Zilch's Go support has a special handler that can create a VFS from a `go.sum`
line. See the xref:go/library.adoc#vfs[Go] documentation for this procedure.

78
docs/modules/ROOT/pages/core/zexp.adoc Normal file
View file

@ -0,0 +1,78 @@
= ``zexp``s, store paths, and context
:page-pagination:
In Nix, strings carry around a "context", corresponding to a list of
dependencies that string has. For example, the `outPath` of a derivation in Nix
has a context that contains its derivation, plus the output name. When strings
are concatenated, their contexts are merged together.
In Zilch, a similar concept exists; encapsulated in so-called "zexpressions".
A `zexpr` is a lazily evaluated thunk that, when evaluated, has access to other
``zexpr``s and closely related objects. When a `zexpr` is unwrapped, it returns
the resulting value, plus a list of derivations and outputs that are depended
on.
[source,scheme]
----
(define example (zexp foo))
;; #<zexp val: foo>
(zexp-unwrap example)
;; #<zexp-evaluation val: foo; drvs: (); srcs: ()>
----
The primary way a zexp that contains context is made, is by using
xref:generated:zilch.magic.adoc#store-path-for-drv[`store-path-for-drv`] and xref:generated:zilch.magic.adoc#store-path-for-ca-drv[`store-path-for-ca-drv`]. This returns a
`<store-path>`, which can be unwrapped like a `zexp`.
[source,scheme]
----
(define drv
(store-path-for-drv
"hello" "x86_64-linux"
'("/bin/sh" "-c" "echo hi > $out") '()
'("out"))
;; (("out"
;; . #<store path /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
;; (/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out)>))
(zexp-unwrap (cdar drv))
;; #<zexp-evaluation
;; val: "/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello";
;; drvs: ((#<derivation …> "out"));
;; srcs: ()>
----
A `zexp` behaves similarly to the `quasiquote` macro, in that it is possible to
use `unquote` (or its reader syntax) to unquote values. Unquoting a `zexp`,
however, requires a special mechanism, and is executed before `unquote`. This
allows for full expressiveness when generating, for example, arguments or
environments for derivations.
[source,scheme]
----
(define complex
(zexp
(unquote
(string-append
"The path for the derivation's output is "
(zexp-unquote (cdar #3)))))
(zexp-unwrap complex)
;; #<zexp-evaluation
;; val: "The path for the derivation's output is /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello";
;; drvs: ((#<derivation …> "out"));
;; srcs: ()>
----
A store path, after being created by `store-path-for-(ca-)drv`, is not
guaranteed to exist in the store. To ensure it exists, xref:generated:zilch.magic.adoc#store-path-materialize[`store-path-materialize`]
is used; this is transparently done when a `store-path` is unwrapped inside a
`zexp`.
== Reader syntax
For convenience, `+#~foo+` is used as reader syntax for `(zexp foo)`, while
`+#$foo+` and `+#$@foo+` are syntax for `(zexp-unwrap foo)` and `(zexp-unquote-splicing foo)`
respectively.

193
docs/modules/ROOT/pages/core/zilch-as-lib.adoc Normal file
View file

@ -0,0 +1,193 @@
= Using Zilch as a library
:page-pagination: prev
Zilch is able to be used as a library; provided you have knowledge of Chicken
Scheme.
Once the Zilch libraries are installed (or inside a `nix-shell` inside the
Zilch repo), open an interpreter (or an editor).
These examples assume you are running in an interpreter with
`(scheme base)`, `(zilch zexpr)`, and `(zilch magic)` in scope.
== Creating and building derivations
To start off, creating an arbitrary derivation can be done with one call:
[,scheme,line-comment=;]
----
(define normal-derivation
(store-path-for-drv
"hello" ; <1>
"x86_64-linux" ; <2>
'("/bin/sh" "-c" "echo hi > $out") ; <3>
'() ; <4>
'("out"))) ; <5>
;; (("out" . #<store path /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
;; (/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out)>))
----
<1> Name for the derivation (used in output store paths)
<2> Architecture that this derivation needs to be built on.
<3> Derivation builder and its arguments (unlike Nix, these are considered one list)
<4> Any environment variables to set whilst executing the builder
<5> The list of output names to use for this derivation.
This procedure returns an alist of `<store-path>` records, which refer to
derivation store paths. These derivations may or may not exist in the Nix
store, however, as Zilch only writes them to the store when necessary, or when
requested. To do this manually, call `store-path-materialize`:
[,scheme]
----
(store-path-materialize (cdar normal-derivation))
----
After materialization, it's now possible to build this derivation, both
inside Zilch and outside it.
[,console]
----
$ nix-build '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out'
this derivation will be built:
/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv
building '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv'...
/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
$ cat /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
hi
----
[,scheme]
----
(store-path-build (cdar normal-derivation))
;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"]
;> [0/1 builds, 1 running]
;; 1
----
These store paths are a type of ``<zexp>``; a format used to describe data that
depends on paths from the Nix store.
== Zexpressions
While it's possible to manually build every store paths you need, and handle
them as plain strings, this is cumbersome and error-prone. It also doesn't work
when you have to deal with placeholders, which are used for content-addressed
derivations. So, to handle these, the `store-path` concept is extended to
generic Scheme expressions. These use `zexp` (or `++#~++`) and you can use
`zexp-unquote` (or `++#$++`) to read out the contents of one `zexp` inside another.
Finally, to read out a `zexp` outside another `zexp`, you can use `zexp-unwrap`:
[,scheme]
----
(define example-zexp
(zexp (zexp-unquote (cdar normal-derivation))))
;; #<zexp val: (zexp-unquote (cdar normal-derivation))>
(define unwrapped (zexp-unwrap example-zexp))
;; #<zexp-evaluation val: "/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello",
;; drvs: ((#<derivation "hello" …> "out")),
;; srcs: ()>
----
A `zexp` represents an arbitrary S-expression. It's also possible to run code
when a `zexp` is used, similarly to `quasiquote`. Like `quasiquote`, this uses
`unquote` (or its syntax sugar). Doing this also retains the context of any
``zexp-unquote``d value:
[,scheme]
----
(define more-complex
(zexp ,(string-append
"The store path is at "
(zexp-unquote example-zexp))))
(zexp-unwrap more-complex)
;; #<zexp-evaluation
;; val: "The store path is at /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello",
;; drvs: ((#<derivation "hello" …> "out")),
;; srcs: ()>
----
These expressions can then be used in derivations (e.g. through
`store-path-for-drv`) too:
[,scheme]
----
(define another-derivation
(store-path-for-drv "hello2" "x86_64-linux"
'("/bin/sh" "-c" "echo \"$foo\" > $out")
(list (cons "foo" more-complex))
'("out")))
;; (("out" . #<store path /nix/store/nz1fv397b3cix74d58i8kh2nj8knvb72-hello2
;; (/nix/store/gvglnwmalgdnri3zwzmkscg61ll8nas1-hello2.drv!out)>))
(store-path-materialize (cdar another-derivation))
(store-path-build (cdar another-derivation))
;! /nix/store/nz1fv397b3cix74d58i8kh2nj8knvb72-hello2 contains
;! "The store path is at /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello"
----
As ``zexp``s aren't limited to strings, it's possible to write expressive
derivations relatively easy.
[,scheme]
----
(define another-derivation
(store-path-for-drv "hello2" "x86_64-linux"
#~("/bin/sh" "-c" ,(string-append "echo \"" #$more-complex "\" > $out"))
'()
'("out")))
;; (("out" . #<store path /nix/store/91v91bmyyjl04ccirp9x1bb1ync3c0f5-hello2
;; (/nix/store/a2315xkzjssicyxgf0ji0j8a3y085hkm-hello2.drv!out)>))
(store-path-materialize (cdar another-derivation))
----
== Content-addressed derivations
Zilch implements xref:./derivations.adoc[its own content-addressed derivations],
which can be used almost as easily as input-addressed derivations. However,
they have an important caveat: Their output store paths are only known at build
time. To improve performance, the store paths render as a "placeholder", which
gets substituted at build time.
[,scheme]
----
(define content-addressed-derivation
(store-path-for-ca-drv "hello" "x86_64-linux"
'("/bin/sh" "-c" "echo hi > $out")
'()
'("out")))
;; (("out" . #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>))
----
Content addressed store paths can be determined by the output path not starting
with the Nix store, and the `ca~` marker in their representation. These
derivations also cannot be materialized; `store-path-materialize` will ignore
them.
When these store paths are used in other content-addressed derivations, they
will work as intended. Their output, when ``zexp-unwrap``ed, however, will
still contain placeholders:
[,scheme]
----
(zexp-unwrap (cdar content-addressed-derivation))
;; #<zexp-evaluation val: "/106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv",
;; drvs: ((#<derivation "hello" …> "out")),
;; srcs: ()>
----
To resolve their placeholders, you need to use `store-path-realised`. This will
evaluate the `zexp` or `store-path`, and resolve placeholders:
[,scheme]
----
(store-path-realised (cdar content-addressed-derivation))
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
----
Alternatively, it's possible to use `store-path-devirtualise`. This does the
same thing, but returns a `zexp`, which can then be used in an input-addressed
derivation.

66
docs/modules/ROOT/pages/go.adoc
View file

@ -1,66 +0,0 @@
= Go
Currently, the primary binary produced by the `zilch` repository is a tool to
generate content-addressed derivations to incrementally build any Go project,
called `zilch-cli-go`.
It requires the `ca-derivations` and `impure-derivations` experimental features
to be enabled on the Nix daemon, and a Nix daemon to be available at the
default store path. It also requires the daemon to be able to run
`x86_64-linux` derivations, right now.
Once run, it will use Zilch to build a series of derivations, and output a
`.drv` for each executable package in the module. These derivations will then
incrementally build the source code: if a package's changes do not impact its
compatibility with the packages that depend on it, it will quickly notice and
skip building dependents, thanks to Nix.
== Example usage
[source]
----
$ git clone https://github.com/tailscale/tailscale # tailscale.com
$ git clone https://go.googlesource.com/net x-net # golang.org/x/net
# build tailscale, but with the golang.org/x/net dependency replaced
# with the local checkout. this might take a few minutes, as it pulls in
# all dependencies one by one.
$ zilch-cli-go -m tailscale -r x-net tailscale.com/cmd/tailscaled
# use `--option substitute false', as there are a lot of paths to check
# against the binary cache otherwise
$ nix-build --option substitute false /nix/store/qj0dh4pdgldxkb798pj1f3fs5n1nwnx3-tailscale.com_cmd_tailscaled.drv
[.. change e.g. `GODEBUG` to `GODEBUG2` in `x-net/http2/http2.go` ..]
$ zilch-cli-go --module-dir tailscale --replace x-net tailscale.com/cmd/tailscaled
$ nix-build --option substitute false /nix/store/nm2ycs5zb7v2wdlwgy913apnd22gs4pn-tailscale.com_cmd_tailscaled.drv
----
Once `zilch-cli-go` returns, it returns a `.drv`, which, when built, will
incrementally build the changed modules. If a package has been changed without
impacting the exported functions too much, only it will be rebuilt, with all
packages that depend on it staying the same, and being skipped.
== Help page
[source]
----
Usage: zilch-cli-go [OPTION] [PACKAGE...]
Process the given module (or the current directory, if unspecified) and
output derivations for each package given on the command line (or all
executables in the module, if unspecified)
-h, --help Print this help message.
-b, --build Build the store paths, rather than show their
derivations.
-L, --print-build-logs Print derivation logs as they come in.
-m, --module-dir DIR The directory to use as root module.
-r, --replace DIR Replace the module specified by the go.mod
with this source directory, rather than using
the upstream module. Can be specified more
than once.
--debug Crash on the first error, rather than
continuing with the next package.
----

4
docs/modules/ROOT/pages/index.adoc
View file

@ -13,10 +13,6 @@ Features:
* Batteries included * Batteries included
* Intercompatible with Nixpkgs and other, arbitrary, Nix expressions. * Intercompatible with Nixpkgs and other, arbitrary, Nix expressions.
== Current work
Current effort in Zilch is working on making "incremental", bitesize,
derivations to work. This is xref:go.adoc[currently being implemented for Go].
== Contributing == Contributing
Come join [.line-through]#us# me at `#zilch` on https://libera.chat[libera.chat]! Come join [.line-through]#us# me at `#zilch` on https://libera.chat[libera.chat]!