docs: Document core concepts
Change-Id: I6a6a6964d6bded229cd640463eaac70fd52df233
This commit is contained in:
parent
e76c57a388
commit
f0ce185d5c
7 changed files with 427 additions and 73 deletions
95
docs/modules/ROOT/pages/core/derivations.adoc
Normal file
95
docs/modules/ROOT/pages/core/derivations.adoc
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
= Content-addressed derivations and the Zilch build scheduler
|
||||
:page-pagination: next
|
||||
|
||||
Zilch implements its own content-addressed derivations, along with a subset of
|
||||
the Nix build scheduler. This brings a few important features with it, such as
|
||||
derivation fallbacks and post-build hooks. It also means that content-addressed
|
||||
derivations can be experimented with inside Zilch without worrying about
|
||||
changes to Nix breaking it, and without enabling any `experimental-features`.
|
||||
|
||||
== Content-addressed derivations
|
||||
Currently, the content-addressed derivation format used by Zilch closely
|
||||
matches that of Nix, with a few subtleties. content-addressed derivations are
|
||||
never written to the Nix store, as that would require Nix support them.
|
||||
|
||||
If a Zilch CA derivation is built, it is translated to an input-addressed
|
||||
derivation and built. Each output of this derivation is then queried and the
|
||||
NAR hash is used to copy the output path to its final, content-addressed,
|
||||
destination.footnote:[This means self-references aren't supported; though as
|
||||
these are not correctly handled in Nix either this problem has been punted for
|
||||
now.] These store paths are then used to substitute the placeholder in the
|
||||
generated derivation for any derivation that depends on it.
|
||||
|
||||
The output of these content-addressed derivations has the Deriver set properly,
|
||||
so it's possible to use the Nix CLI to quickly query the (input-addressed
|
||||
equivalent) derivation used to build them, using a shell script:
|
||||
|
||||
[source,bash]
|
||||
----
|
||||
nix derivation show "$(nix path-info --json /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello | jq -r '.[].deriver')
|
||||
----
|
||||
|
||||
|
||||
== Build hooks
|
||||
By implementing the content-addressed derivation logic itself, Zilch is able to
|
||||
provide a few features Nix is unable to: Build hooks. When a content-addressed
|
||||
derivation fails to build, it is possible to swap out the derivation, and
|
||||
instead build a different derivation. This is done using xref:generated:zilch.magic.adoc#store-path-register-fallback[`store-path-register-fallback`],
|
||||
which takes a thunk that is evaluated when the build fails, and returns a new
|
||||
derivation to build.
|
||||
|
||||
[source,scheme]
|
||||
----
|
||||
(define fails-to-build
|
||||
(cdar (store-path-for-ca-drv
|
||||
"hello" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo i fail to build")
|
||||
'() '("out")))
|
||||
;; #<store path /0qln6zx68yai0x802a804hvwszvddjpkhv87phsdy0snvlnmr3q8
|
||||
;; (ca~ /nix/store/7cp7gl3sdi6k9kvrg0b628fq4khlk5py-hello.drv!out)>
|
||||
|
||||
(define fallback
|
||||
(cdar (store-path-for-ca-drv
|
||||
"hello" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo hi > $out")
|
||||
'() '("out")))
|
||||
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
|
||||
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
|
||||
|
||||
(store-path-register-fallback fails-to-build
|
||||
(lambda () fallback))
|
||||
;; #<store path /0qln6zx68yai0x802a804hvwszvddjpkhv87phsdy0snvlnmr3q8
|
||||
;; (ca~ /nix/store/7cp7gl3sdi6k9kvrg0b628fq4khlk5py-hello.drv!out)>
|
||||
|
||||
(store-path-realised fails-to-build)
|
||||
;> [..building "/nix/store/qmx86vqz7pl191vwpcc8zrpnh98rils8-hello.drv"]
|
||||
;> [0/1 builds, 1 running]
|
||||
;> i fail to build
|
||||
;>
|
||||
;> Error: daemon logger received error: (error "Error" 0 "builder for '\x1b[35;1m/nix/store/qmx86vqz7pl191vwpcc8zrpnh98r...")
|
||||
;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"]
|
||||
;> [0/1 builds, 1 running]
|
||||
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
|
||||
----
|
||||
|
||||
The post-build hook works similarly, but calls the callback with an `alist` of output name to output store path:
|
||||
|
||||
[source,scheme]
|
||||
----
|
||||
(define example
|
||||
(cdar (store-path-for-ca-drv
|
||||
"hello" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo hi > $out")
|
||||
'() '("out")))
|
||||
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
|
||||
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
|
||||
|
||||
(store-path-register-post-build example
|
||||
(lambda (vals) (write vals) (newline)))
|
||||
;; #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
|
||||
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>
|
||||
|
||||
(store-path-realised example)
|
||||
;> (("out" . "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"))
|
||||
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
|
||||
----
|
||||
54
docs/modules/ROOT/pages/core/vfs.adoc
Normal file
54
docs/modules/ROOT/pages/core/vfs.adoc
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
= The Zilch Virtual Filesystem
|
||||
:page-pagination:
|
||||
|
||||
In many places, it's necessary to build up a structure of files and
|
||||
directories. These structures are a subset of e.g. a project's source code. To
|
||||
make these, Zilch has a `vfs` record.
|
||||
|
||||
Core Zilch can create a VFS from two sources: an on-disk directory, as well as
|
||||
a Nix store path:
|
||||
|
||||
[,scheme,line-comment=;]
|
||||
----
|
||||
(vfs-from-directory "/home/puck/example")
|
||||
;; #<zilch.vfs#<vfs>>
|
||||
|
||||
(define example-vfs
|
||||
(vfs-from-store
|
||||
(zdir "zero" (zsymlink "/dev/zero")))
|
||||
;; #<zilch.vfs#<vfs>>
|
||||
----
|
||||
|
||||
These VFSes can be read out using `vfs-contents`:
|
||||
|
||||
[,scheme,line-comment=;]
|
||||
----
|
||||
(mapping->alist
|
||||
(vfs-contents example-vfs))
|
||||
;; ((("" . "zero")
|
||||
;; . #<z-symlink -> "/dev/zero">))
|
||||
----
|
||||
|
||||
VFS records store their contents as an SRFI 146 mapping, with a pair
|
||||
`(dir . name)` as key, and the store path (or zexpr) to point at as its value.
|
||||
Directories are indicated by the directory itself having a `'directory` symbol
|
||||
as contents; these markers are used during serialization.
|
||||
|
||||
`(zilch vfs)` contains a series of helpers to make standard changes to the
|
||||
filesystem; such as filtering, and reading subtrees.
|
||||
|
||||
== Serializing virtual filesystems to the store
|
||||
|
||||
When a virtual filesystem is written to the store using `vfs-to-store`, a
|
||||
xref:generated:zilch.file.adoc[`(zilch file)`]-based structure is created,
|
||||
which can then be ``zexp-unquote``d safely inside a `zexp`. An important caveat
|
||||
to this, however, is that the created structure uses symlinks to every file in
|
||||
the vfs, rather than copying. This is chosen to both limit the expense of
|
||||
copying the contents of a large file to the store when many VFSes contain it,
|
||||
and a logistical limitation in the way Zilch interacts with Nix. It's possible
|
||||
this restriction will be lifted in the future.
|
||||
|
||||
== Language-specific
|
||||
|
||||
Zilch's Go support has a special handler that can create a VFS from a `go.sum`
|
||||
line. See the xref:go/library.adoc#vfs[Go] documentation for this procedure.
|
||||
78
docs/modules/ROOT/pages/core/zexp.adoc
Normal file
78
docs/modules/ROOT/pages/core/zexp.adoc
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
= ``zexp``s, store paths, and context
|
||||
:page-pagination:
|
||||
|
||||
In Nix, strings carry around a "context", corresponding to a list of
|
||||
dependencies that string has. For example, the `outPath` of a derivation in Nix
|
||||
has a context that contains its derivation, plus the output name. When strings
|
||||
are concatenated, their contexts are merged together.
|
||||
|
||||
In Zilch, a similar concept exists; encapsulated in so-called "zexpressions".
|
||||
A `zexpr` is a lazily evaluated thunk that, when evaluated, has access to other
|
||||
``zexpr``s and closely related objects. When a `zexpr` is unwrapped, it returns
|
||||
the resulting value, plus a list of derivations and outputs that are depended
|
||||
on.
|
||||
|
||||
[source,scheme]
|
||||
----
|
||||
(define example (zexp foo))
|
||||
;; #<zexp val: foo>
|
||||
|
||||
(zexp-unwrap example)
|
||||
;; #<zexp-evaluation val: foo; drvs: (); srcs: ()>
|
||||
----
|
||||
|
||||
The primary way a zexp that contains context is made, is by using
|
||||
xref:generated:zilch.magic.adoc#store-path-for-drv[`store-path-for-drv`] and xref:generated:zilch.magic.adoc#store-path-for-ca-drv[`store-path-for-ca-drv`]. This returns a
|
||||
`<store-path>`, which can be unwrapped like a `zexp`.
|
||||
|
||||
[source,scheme]
|
||||
----
|
||||
(define drv
|
||||
(store-path-for-drv
|
||||
"hello" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo hi > $out") '()
|
||||
'("out"))
|
||||
|
||||
;; (("out"
|
||||
;; . #<store path /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
|
||||
;; (/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out)>))
|
||||
|
||||
(zexp-unwrap (cdar drv))
|
||||
;; #<zexp-evaluation
|
||||
;; val: "/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello";
|
||||
;; drvs: ((#<derivation …> "out"));
|
||||
;; srcs: ()>
|
||||
----
|
||||
|
||||
A `zexp` behaves similarly to the `quasiquote` macro, in that it is possible to
|
||||
use `unquote` (or its reader syntax) to unquote values. Unquoting a `zexp`,
|
||||
however, requires a special mechanism, and is executed before `unquote`. This
|
||||
allows for full expressiveness when generating, for example, arguments or
|
||||
environments for derivations.
|
||||
|
||||
[source,scheme]
|
||||
----
|
||||
(define complex
|
||||
(zexp
|
||||
(unquote
|
||||
(string-append
|
||||
"The path for the derivation's output is "
|
||||
(zexp-unquote (cdar #3)))))
|
||||
|
||||
(zexp-unwrap complex)
|
||||
;; #<zexp-evaluation
|
||||
;; val: "The path for the derivation's output is /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello";
|
||||
;; drvs: ((#<derivation …> "out"));
|
||||
;; srcs: ()>
|
||||
----
|
||||
|
||||
A store path, after being created by `store-path-for-(ca-)drv`, is not
|
||||
guaranteed to exist in the store. To ensure it exists, xref:generated:zilch.magic.adoc#store-path-materialize[`store-path-materialize`]
|
||||
is used; this is transparently done when a `store-path` is unwrapped inside a
|
||||
`zexp`.
|
||||
|
||||
== Reader syntax
|
||||
For convenience, `+#~foo+` is used as reader syntax for `(zexp foo)`, while
|
||||
`+#$foo+` and `+#$@foo+` are syntax for `(zexp-unwrap foo)` and `(zexp-unquote-splicing foo)`
|
||||
respectively.
|
||||
|
||||
193
docs/modules/ROOT/pages/core/zilch-as-lib.adoc
Normal file
193
docs/modules/ROOT/pages/core/zilch-as-lib.adoc
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
= Using Zilch as a library
|
||||
:page-pagination: prev
|
||||
|
||||
Zilch is able to be used as a library; provided you have knowledge of Chicken
|
||||
Scheme.
|
||||
|
||||
Once the Zilch libraries are installed (or inside a `nix-shell` inside the
|
||||
Zilch repo), open an interpreter (or an editor).
|
||||
|
||||
These examples assume you are running in an interpreter with
|
||||
`(scheme base)`, `(zilch zexpr)`, and `(zilch magic)` in scope.
|
||||
|
||||
== Creating and building derivations
|
||||
|
||||
To start off, creating an arbitrary derivation can be done with one call:
|
||||
|
||||
[,scheme,line-comment=;]
|
||||
----
|
||||
(define normal-derivation
|
||||
(store-path-for-drv
|
||||
"hello" ; <1>
|
||||
"x86_64-linux" ; <2>
|
||||
'("/bin/sh" "-c" "echo hi > $out") ; <3>
|
||||
'() ; <4>
|
||||
'("out"))) ; <5>
|
||||
;; (("out" . #<store path /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
|
||||
;; (/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out)>))
|
||||
----
|
||||
<1> Name for the derivation (used in output store paths)
|
||||
<2> Architecture that this derivation needs to be built on.
|
||||
<3> Derivation builder and its arguments (unlike Nix, these are considered one list)
|
||||
<4> Any environment variables to set whilst executing the builder
|
||||
<5> The list of output names to use for this derivation.
|
||||
|
||||
This procedure returns an alist of `<store-path>` records, which refer to
|
||||
derivation store paths. These derivations may or may not exist in the Nix
|
||||
store, however, as Zilch only writes them to the store when necessary, or when
|
||||
requested. To do this manually, call `store-path-materialize`:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(store-path-materialize (cdar normal-derivation))
|
||||
----
|
||||
|
||||
After materialization, it's now possible to build this derivation, both
|
||||
inside Zilch and outside it.
|
||||
|
||||
[,console]
|
||||
----
|
||||
$ nix-build '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv!out'
|
||||
this derivation will be built:
|
||||
/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv
|
||||
building '/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv'...
|
||||
/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
|
||||
|
||||
$ cat /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello
|
||||
hi
|
||||
----
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(store-path-build (cdar normal-derivation))
|
||||
;> [..building "/nix/store/76w21n1f03fs5kw8fnffphx7qrqffw6r-hello.drv"]
|
||||
;> [0/1 builds, 1 running]
|
||||
;; 1
|
||||
----
|
||||
|
||||
These store paths are a type of ``<zexp>``; a format used to describe data that
|
||||
depends on paths from the Nix store.
|
||||
|
||||
== Zexpressions
|
||||
While it's possible to manually build every store paths you need, and handle
|
||||
them as plain strings, this is cumbersome and error-prone. It also doesn't work
|
||||
when you have to deal with placeholders, which are used for content-addressed
|
||||
derivations. So, to handle these, the `store-path` concept is extended to
|
||||
generic Scheme expressions. These use `zexp` (or `++#~++`) and you can use
|
||||
`zexp-unquote` (or `++#$++`) to read out the contents of one `zexp` inside another.
|
||||
Finally, to read out a `zexp` outside another `zexp`, you can use `zexp-unwrap`:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(define example-zexp
|
||||
(zexp (zexp-unquote (cdar normal-derivation))))
|
||||
;; #<zexp val: (zexp-unquote (cdar normal-derivation))>
|
||||
|
||||
(define unwrapped (zexp-unwrap example-zexp))
|
||||
;; #<zexp-evaluation val: "/nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello",
|
||||
;; drvs: ((#<derivation "hello" …> "out")),
|
||||
;; srcs: ()>
|
||||
----
|
||||
|
||||
A `zexp` represents an arbitrary S-expression. It's also possible to run code
|
||||
when a `zexp` is used, similarly to `quasiquote`. Like `quasiquote`, this uses
|
||||
`unquote` (or its syntax sugar). Doing this also retains the context of any
|
||||
``zexp-unquote``d value:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(define more-complex
|
||||
(zexp ,(string-append
|
||||
"The store path is at "
|
||||
(zexp-unquote example-zexp))))
|
||||
|
||||
(zexp-unwrap more-complex)
|
||||
;; #<zexp-evaluation
|
||||
;; val: "The store path is at /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello",
|
||||
;; drvs: ((#<derivation "hello" …> "out")),
|
||||
;; srcs: ()>
|
||||
----
|
||||
|
||||
These expressions can then be used in derivations (e.g. through
|
||||
`store-path-for-drv`) too:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(define another-derivation
|
||||
(store-path-for-drv "hello2" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo \"$foo\" > $out")
|
||||
(list (cons "foo" more-complex))
|
||||
'("out")))
|
||||
;; (("out" . #<store path /nix/store/nz1fv397b3cix74d58i8kh2nj8knvb72-hello2
|
||||
;; (/nix/store/gvglnwmalgdnri3zwzmkscg61ll8nas1-hello2.drv!out)>))
|
||||
|
||||
(store-path-materialize (cdar another-derivation))
|
||||
|
||||
(store-path-build (cdar another-derivation))
|
||||
;! /nix/store/nz1fv397b3cix74d58i8kh2nj8knvb72-hello2 contains
|
||||
;! "The store path is at /nix/store/mjs27ix6ig2bkbi3s3sm470vrv4lf7ic-hello"
|
||||
----
|
||||
|
||||
As ``zexp``s aren't limited to strings, it's possible to write expressive
|
||||
derivations relatively easy.
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(define another-derivation
|
||||
(store-path-for-drv "hello2" "x86_64-linux"
|
||||
#~("/bin/sh" "-c" ,(string-append "echo \"" #$more-complex "\" > $out"))
|
||||
'()
|
||||
'("out")))
|
||||
;; (("out" . #<store path /nix/store/91v91bmyyjl04ccirp9x1bb1ync3c0f5-hello2
|
||||
;; (/nix/store/a2315xkzjssicyxgf0ji0j8a3y085hkm-hello2.drv!out)>))
|
||||
|
||||
(store-path-materialize (cdar another-derivation))
|
||||
----
|
||||
|
||||
== Content-addressed derivations
|
||||
Zilch implements xref:./derivations.adoc[its own content-addressed derivations],
|
||||
which can be used almost as easily as input-addressed derivations. However,
|
||||
they have an important caveat: Their output store paths are only known at build
|
||||
time. To improve performance, the store paths render as a "placeholder", which
|
||||
gets substituted at build time.
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(define content-addressed-derivation
|
||||
(store-path-for-ca-drv "hello" "x86_64-linux"
|
||||
'("/bin/sh" "-c" "echo hi > $out")
|
||||
'()
|
||||
'("out")))
|
||||
;; (("out" . #<store path /106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv
|
||||
;; (ca~ /nix/store/bl3mp0i3kd2rssjgynga84gwzx5cj653-hello.drv!out)>))
|
||||
----
|
||||
|
||||
Content addressed store paths can be determined by the output path not starting
|
||||
with the Nix store, and the `ca~` marker in their representation. These
|
||||
derivations also cannot be materialized; `store-path-materialize` will ignore
|
||||
them.
|
||||
|
||||
When these store paths are used in other content-addressed derivations, they
|
||||
will work as intended. Their output, when ``zexp-unwrap``ed, however, will
|
||||
still contain placeholders:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(zexp-unwrap (cdar content-addressed-derivation))
|
||||
;; #<zexp-evaluation val: "/106ghhc6jy33ycylgj3ndzwb1l6sdkm75likgd7fm5pr7fjfx5cv",
|
||||
;; drvs: ((#<derivation "hello" …> "out")),
|
||||
;; srcs: ()>
|
||||
----
|
||||
|
||||
To resolve their placeholders, you need to use `store-path-realised`. This will
|
||||
evaluate the `zexp` or `store-path`, and resolve placeholders:
|
||||
|
||||
[,scheme]
|
||||
----
|
||||
(store-path-realised (cdar content-addressed-derivation))
|
||||
;; "/nix/store/a5izqk5bgpxcrrnrm1m8n1fvyq2jlc52-hello"
|
||||
----
|
||||
|
||||
Alternatively, it's possible to use `store-path-devirtualise`. This does the
|
||||
same thing, but returns a `zexp`, which can then be used in an input-addressed
|
||||
derivation.
|
||||
Loading…
Add table
Add a link
Reference in a new issue