puck/zilch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

(zilch lang go): document

Browse source 
Change-Id: I6a6a6964558b4fe2f96d78120b2e899f91d48c22
This commit is contained in:
puck 2025-06-23 12:22:20 +00:00
parent f0ce185d5c
commit 18f2887eba
13 changed files with 457 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

210
docs/modules/ROOT/pages/go/library.adoc Normal file
View file

@ -0,0 +1,210 @@
= Writing Go packages directly in Zilch
:page-pagination:
It's possible to use Zilch as generic build system for Go. This page
serves to document that, as well as the internals of the Go build
library in Zilch.
== Go packages in Zilch
Each Go package is represented in Zilch by a xref:generated:zilch.lang.go.adoc#++_go-package_++[`<go-package>`],
which represents the compiled API and ABI as separate store paths, plus the
name of the package, and its dependencies.
The simplest possible Go package consists of a single file:
[,scheme,line-comment=;]
----
(define example
(go-package-compile
"example.com/package" ; <1>
'() ; <2>
`(("example.go" . ,(zfile "package example"))))) ; <3>
;; #<go-package example.com/package
;; api: #<store path /1ahdindygxf34gjhdj9l6h43i6cwsky0v02ai75g35bp5n1i80gp
;; (ca~ /nix/store/f4bj92n5yfhp5f29jmacydghsvkaj7fq-example.com_package-src.drv!api)>
;; code: #<store path /0ln85mblval4j4cpjgw65f4phrbiagzh1hd5pryjlnnknbq75agj
;; (ca~ /nix/store/8dqr20s72i4w4llp73imm9bkcwmx9via-go-example.com_package-code.drv!code)>
;; deps: ()>
----
<1> Name of the package
<2> List of dependency packages (in this case, none)
<3> Mapping of filename to in-store files
To create an executable, though, a bit more work is necessary, which is
abstracted by the Go build system by default. The "main" package, which
contains the `func main()`, always has the package name "main". However, this
is undesirable for the name in panics. For this case, the extended
`go-package-compile` procedure allows setting both:
[,scheme,line-comment=;]
----
(define binary
(go-package-compile
"main" ; <1>
"example.com/binary" ; <2>
(list example) ; <3>
`(("main.go" . ,(zfile "package main\nfunc main() { }"))) ; <4>
'() '() ; <5>
'() '())) ; <6>
;; #<go-package
;; main (example.com/binary)
;; api: #<store path /0p33flbfjc0s9ykgmcyzqbr3dhsq0344zswhl6xjvqdbin9ysqch
;; (ca~ /nix/store/hg5j28f63b4czfxm4cg76rjbh4bx6ivy-main-src.drv!api)>
;; code: #<store path /15is1garmlclnb06qj9ymxdi8xpsgakrx97qdb2jmx0562wn7kgx
;; (ca~ /nix/store/z9w0rccki2rhjrgxr0y063qijz64m6p8-go-example.com_binary-code.drv!code)>
;; deps: ("example.com/package")>
----
<1> The package name, as provided to the compiler (must be `main` for binaries)
<2> The package name as shown in stacktraces
<3> Direct dependencies for this package
<4> The source file for this package
<5> Assembly-related files
<6> ``go:embed``-related files
To use this binary, it still needs to be linked. This can be done with
`go-package-link`.
[IMPORTANT]
====
If you're linking a Go binary, make sure to add the `runtime` package to your
dependencies, either directly or indirectly. Otherwise, you will end up with an
error like so:
[,scheme,line-comment=;]
----
(store-path-realised (go-package-link binary))
;> [..building "/nix/store/jr17baky22c0zzpjfi97cw16k5wmqqc3-example.com_binary.drv"]
;> [0/1 builds, 1 running]
;> loadinternal: cannot find runtime
;> panic: could not look up runtime.mapinitnoop
----
This is caused by Zilch not adding any standard library code by default, while
the Go compiler depends on structures in the standard library to provide runtime
support for the garbage collector and goroutines.
====
== Using the Go standard library
The Go standard library, unlike most other languages, is shipped as source code.
When you compile Go code, the compiler will also compile the parts of the
standard library you use. Zilch does the same.
`(zilch lang go stdlib)` contains a single procedure, `go-stdlib-ref`, which
handles this for you:
[,scheme,line-comment=;]
----
(go-stdlib-ref "fmt")
;; #<go-package fmt
;; api: #<store path /1al016fvsnknhpzdzbcf7kjxyzbf3fj8sbm40p8h32x3kr0aswbv
;; (ca~ /nix/store/vpvpmx2d43ix6m142jxxg0z1kqddypzl-fmt-src.drv!api)>
;; code: #<store path /0sssb21sd2d398fv4vwlykf0shnlyr31kf372k65djqrgxrz5674
;; (ca~ /nix/store/pi9nraymjdn12hg7r18h13js56w0s1b6-go-fmt-code.drv!code)>
;; deps: ("errors" "internal/fmtsort" "io" "math" "os" "reflect"
;; "slices" "strconv" "sync" "unicode/utf8")>
(define hello-world
(go-package-compile
"main"
"example.com/binary"
(list example (go-stdlib-ref "fmt"))
`(("main.go"
. ,(zfile "package main
import \"fmt\"
func main() {
fmt.Printf(\"Hello, world!\\n\")
}")))
'() '()
'() '()))
(define linked (go-package-link hello-world))
;; #<store path /1csyxx5m27a4819g5m8bnm88gfgcspxisg19rv2kg6b7cv7wn066
;; (ca~ /nix/store/jjqhwl7cq2crhdjdsrkby4ng584n3pc1-example.com_binary.drv!out)>
(store-path-realised linked)
;> [..building "/nix/store/5bgrfjgnvzr8wqksl5j23xr5wm7rahnw-zilchfile.drv"]
;> …
;> [0/2 builds, 0 running]
;> [..building "/nix/store/jdk0wbnrz16sbn41r0y6f82hxl93q87r-example.com_binary.drv"]
;; "/nix/store/p71p43d3cv48rgi4yh9dvlzssgfkb5vf-example.com_binary"
----
The binary that is output by this code is incrementally built, one package at a
time, including the standard library.
[#vfs]
== Creating virtual filesystems for Go modules
Go's `go.sum` files contain enough information to recover the full file
structure of the module, through a hash format called `dirhash`. Zilch
implements this, and uses it to guide its VFS generating. Once you have a
`go.sum` line, e.g. through `parse-go-sum-line`, passing it to
`vfs-from-dirhash` will have it generate a VFS from a dirhash, plus the Go
module proxy to fetch the zip:
[,scheme,line-comment=;]
----
(define sum-line (parse-go-sum-line "golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw="))
;; #<go-sum golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=>
(define x-net-vfs (vfs-from-dirhash sum-line))
;> (pre)fetching "module.zip" "https://proxy.golang.org/golang.org/x/net/@v/v0.41.0.zip"
;> …
;; #<zilch.vfs#<vfs>>
(store-path-realised
(vfs-to-store x-net-vfs))
;> [..building "/nix/store/vvs2xsgjpab4gwm530dvzcfzfcs5ak4m-zilchfile.drv"]
;> …
;; "/nix/store/r1p1y6bnmddmcrl2avsygp659d8iiih1-zilchfile/-"
----
== Dynamically generating ``go-package``s from packages
Zilch supports generating `go-package` records dynamically, from `go.mod` files
and `go.sum` files recursively. To do this, get a VFS for the primary module
you want to work with, and any module you want to resolve automatically (Zilch
will automatically crawl `go.sum` files to resolve dependencies, where needed.)
[,scheme,line-comment=;]
----
(define-values (root-module-path requirements)
(collect-requirements-for-module x-net-vfs '()))
;> Collecting required modules
;> [..building "/nix/store/s9fmg08x56px39hjkwlba2pra5g1abgr-go.mod.json.drv"]
;> - found "golang.org/x/net" (requires 4 modules)
;> - found "golang.org/x/crypto" (requires 4 modules)
;> …
;; root-module-path -> "golang.org/x/net"
;; requirements
;; -> #<mapping
;; "golang.org/x/net" -> (#f . #<zilch.vfs#<vfs>>)
;; "github.com/google/go-cmp" -> ("v0.6.0" . #<zilch.vfs#<vfs>>)
;; "golang.org/x/crypto" -> ("v0.39.0" . #<zilch.vfs#<vfs>>)
;; …
;; "golang.org/x/mod" -> ("v0.25.0" . #<zilch.vfs#<vfs>>)>
----
The resulting values are a comprehensive set of modules mapped to their final
versions and the VFS of that module. This information can then be used to
create a procedure that returns the `go-package` for any package in this set of
modules, similar to how `go-stdlib-ref` works:
[,scheme,line-comment=;]
----
(define x-net-ref
(collect-packages-from-requires requirements))
(x-net-ref "golang.org/x/net/idna")
;; #<go-package idna (golang.org/x/net/idna)
;; api: #<store path /1w32lfajzbs83ibvgnn2cx4zk33nmg3xb20dhra1b89zyg3xin4i
;; (ca~ /nix/store/az0g6zjc33mg3kfacy7gnfhkbkps8m2y-golang.org_x_net_idna-src.drv!api)>
;; code: #<store path /1r40gj99ycm404ak3f2rkhl4qhy19hx1dkhjpmp0rbbsgy2f4wj7
;; (ca~ /nix/store/b93f07hkv2yxf80pdwbra2r760yxkh3r-go-golang.org_x_net_idna-code.drv!code)>
;; deps: ("fmt" "bidirule" "bidi" "norm" "math" "strings" "unicode/utf8")>
(x-net-ref "golang.org/x/text/runes")
;; #<go-package runes (golang.org/x/text/runes) …> <1>
----
<1> Even packages defined in a dependency can be looked up using this method.

58
docs/modules/ROOT/pages/go/man.adoc Normal file
View file

@ -0,0 +1,58 @@
= zilch-cli-go(1)
Puck Meerburg
v0.0.1
:doctype: manpage
:manmanual: ZILCH-CLI-GO
:mansource: ZILCH
:page-pagination: prev
== Name
zilch-cli-go - builds a Go module using Zilch
== Synopsis
*zilch-cli-go* [_OPTION_]... _PACKAGE_...
== Description
This command uses Zilch to build one or more Go packages entirely
inside Nix, using content-addressed derivations.
A Go module is recognized by its `go.mod`, which contains information
about the dependencies of any Go package. This versioning info is used
identically to a normal `go build` call.
== Options
*-h*::
*--help*::
Print a help message
*-j* _COUNT_::
*--max-jobs* _COUNT_::
The maximum amount of builds to run. Defaults to the amount of cores.
*-v*::
*--verbose*::
Increase the verbosity configured in the Nix daemon. Can be specified
multiple times.
*-L*::
*--print-build-logs*::
Print derivation logs as they come in.
*-m* _DIR_::
*--module-dir* _DIR_::
The directory to use as root module. All packages to be built must
come from this module.
*-r* _DIR_::
*--replace* _DIR_::
Replace a module from the `go.mod` of the root module with
this directory, based on the name of the replaced module's `go.mod`.
Can be specified multiple times.
*--debug*::
Crash on the first error, rather than continuing to build the next
package.

49
docs/modules/ROOT/pages/go/usage.adoc Normal file
View file

@ -0,0 +1,49 @@
= Usage
:page-pagination: next
Zilch supports compiling any Go code incrementally using `zilch-cli-go`, with
no special configuration needed on the part of the Go module being built:
[,console]
----
$ git clone https://github.com/tailscale/tailscale
$ zilch-cli-go --module-dir tailscale/ <1>
tailscale.com/client/tailscale/example/servetls /nix/store/727ci6sssga0pbi4aqb08vnhfvmh9nl0-tailscale.com_client_tailscale_example_servetls
tailscale.com/cmd/addlicense /nix/store/ckc5bx4s0c29sancd05asvhz0dy43xj7-tailscale.com_cmd_addlicense
…
tailscale.com/cmd/xdpderper /nix/store/8wn9w91mjipw1dzks4p3kcmkwcn18jp7-tailscale.com_cmd_xdpderper
$ zilch-cli-go --module-dir tailscale/ \
> tailscale.com/cmd/tailscaled <2>
tailscale.com/cmd/tailscaled /nix/store/hqrm0f8sd8sx54am921na25w8za67p3m-tailscale.com_cmd_tailscaled
----
<1> Building all binary packages in a module
<2> Building a specifically targeted package
Right now, the daemon has to be able to build `x86_64-linux` derivations,
and `zilch-cli-go` will only output statically linked amd64 binaries as well.
While running, all processing of the Go module, its dependencies, etc, are done
inside of Nix. Once all dependencies have been resolved, a series of Nix
derivations will be used to then build the requested packages. If any source
file's changes do not affect the way dependent packages use it, those packages
will not need rebuilding; only the final result will have to be re-linked.
== Replacing dependencies
As part of Zilch, it's also possible to quickly build a module with one of its
(transitive) dependencies replaced. This keeps the same guarantees as before:
Any changes made that do not involve the output changing will only need
relinking of the resulting binary. This is also very simple:
[,console]
----
$ git clone https://go.googlesource.com/net x-net # golang.org/x/net
$ zilch-cli-go --module-dir tailscale --replace x-net/ tailscale.com/cmd/tailscaled
…
tailscale.com/cmd/tailscaled /nix/store/hqrm0f8sd8sx54am921na25w8za67p3m-tailscale.com_cmd_tailscaled
----
After editing a file any of the replaced dependencies, if this package is used
in the final build, Zilch will apply early-cutoff where possible.