zilch/docs/modules/ROOT/pages/go/library.adoc

= Writing Go packages directly in Zilch
:page-pagination:

It's possible to use Zilch as generic build system for Go. This page
serves to document that, as well as the internals of the Go build
library in Zilch.

== Go packages in Zilch

Each Go package is represented in Zilch by a xref:generated:zilch.lang.go.adoc#++_go-package_++[`<go-package>`],
which represents the compiled API and ABI as separate store paths, plus the
name of the package, and its dependencies.

The simplest possible Go package consists of a single file:

[,scheme,line-comment=;]
----
(define example
 (go-package-compile
  "example.com/package" ; <1>
  '() ; <2>
  `(("example.go" . ,(zfile "package example"))))) ; <3>
;; #<go-package example.com/package
;;   api: #<store path /1ahdindygxf34gjhdj9l6h43i6cwsky0v02ai75g35bp5n1i80gp
;;          (ca~ /nix/store/f4bj92n5yfhp5f29jmacydghsvkaj7fq-example.com_package-src.drv!api)>
;;   code: #<store path /0ln85mblval4j4cpjgw65f4phrbiagzh1hd5pryjlnnknbq75agj
;;          (ca~ /nix/store/8dqr20s72i4w4llp73imm9bkcwmx9via-go-example.com_package-code.drv!code)>
;;   deps: ()>
----
<1> Name of the package
<2> List of dependency packages (in this case, none)
<3> Mapping of filename to in-store files

To create an executable, though, a bit more work is necessary, which is
abstracted by the Go build system by default. The "main" package, which
contains the `func main()`, always has the package name "main". However, this
is undesirable for the name in panics. For this case, the extended
`go-package-compile` procedure allows setting both:

[,scheme,line-comment=;]
----
(define binary
 (go-package-compile
  "main" ; <1>
  "example.com/binary" ; <2>
  (list example) ; <3>
  `(("main.go" . ,(zfile "package main\nfunc main() { }"))) ; <4>
  '() '() ; <5>
  '() '())) ; <6>
;; #<go-package
;;   main (example.com/binary)
;;   api: #<store path /0p33flbfjc0s9ykgmcyzqbr3dhsq0344zswhl6xjvqdbin9ysqch
;;          (ca~ /nix/store/hg5j28f63b4czfxm4cg76rjbh4bx6ivy-main-src.drv!api)>
;;   code: #<store path /15is1garmlclnb06qj9ymxdi8xpsgakrx97qdb2jmx0562wn7kgx
;;          (ca~ /nix/store/z9w0rccki2rhjrgxr0y063qijz64m6p8-go-example.com_binary-code.drv!code)>
;;   deps: ("example.com/package")>
----
<1> The package name, as provided to the compiler (must be `main` for binaries)
<2> The package name as shown in stacktraces
<3> Direct dependencies for this package
<4> The source file for this package
<5> Assembly-related files
<6> ``go:embed``-related files

To use this binary, it still needs to be linked. This can be done with
`go-package-link`.

[IMPORTANT]
====
If you're linking a Go binary, make sure to add the `runtime` package to your
dependencies, either directly or indirectly. Otherwise, you will end up with an
error like so:

[,scheme,line-comment=;]
----
(store-path-realised (go-package-link binary))
;> [..building "/nix/store/jr17baky22c0zzpjfi97cw16k5wmqqc3-example.com_binary.drv"]
;> [0/1 builds, 1 running]
;> loadinternal: cannot find runtime
;> panic: could not look up runtime.mapinitnoop
----

This is caused by Zilch not adding any standard library code by default, while
the Go compiler depends on structures in the standard library to provide runtime
support for the garbage collector and goroutines.
====

== Using the Go standard library
The Go standard library, unlike most other languages, is shipped as source code.
When you compile Go code, the compiler will also compile the parts of the
standard library you use. Zilch does the same.

`(zilch lang go stdlib)` contains a single procedure, `go-stdlib-ref`, which
handles this for you:

[,scheme,line-comment=;]
----
(go-stdlib-ref "fmt")
;; #<go-package fmt
;;   api: #<store path /1al016fvsnknhpzdzbcf7kjxyzbf3fj8sbm40p8h32x3kr0aswbv
;;          (ca~ /nix/store/vpvpmx2d43ix6m142jxxg0z1kqddypzl-fmt-src.drv!api)>
;;   code: #<store path /0sssb21sd2d398fv4vwlykf0shnlyr31kf372k65djqrgxrz5674
;;           (ca~ /nix/store/pi9nraymjdn12hg7r18h13js56w0s1b6-go-fmt-code.drv!code)>
;;   deps: ("errors" "internal/fmtsort" "io" "math" "os" "reflect"
;;          "slices" "strconv" "sync" "unicode/utf8")>

(define hello-world
 (go-package-compile
  "main"
  "example.com/binary"
  (list example (go-stdlib-ref "fmt"))
  `(("main.go"
      . ,(zfile "package main
                 import \"fmt\"
                 func main() {
                   fmt.Printf(\"Hello, world!\\n\")
                 }")))
  '() '()
  '() '()))

(define linked (go-package-link hello-world))
;; #<store path /1csyxx5m27a4819g5m8bnm88gfgcspxisg19rv2kg6b7cv7wn066
;;   (ca~ /nix/store/jjqhwl7cq2crhdjdsrkby4ng584n3pc1-example.com_binary.drv!out)>

(store-path-realised linked)
;> [..building "/nix/store/5bgrfjgnvzr8wqksl5j23xr5wm7rahnw-zilchfile.drv"]
;> …
;> [0/2 builds, 0 running]
;> [..building "/nix/store/jdk0wbnrz16sbn41r0y6f82hxl93q87r-example.com_binary.drv"]
;; "/nix/store/p71p43d3cv48rgi4yh9dvlzssgfkb5vf-example.com_binary"
----

The binary that is output by this code is incrementally built, one package at a
time, including the standard library.

[#vfs]
== Creating virtual filesystems for Go modules
Go's `go.sum` files contain enough information to recover the full file
structure of the module, through a hash format called `dirhash`. Zilch
implements this, and uses it to guide its VFS generating. Once you have a
`go.sum` line, e.g. through `parse-go-sum-line`, passing it to
`vfs-from-dirhash` will have it generate a VFS from a dirhash, plus the Go
module proxy to fetch the zip:

[,scheme,line-comment=;]
----
(define sum-line (parse-go-sum-line "golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw="))
;; #<go-sum golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=>

(define x-net-vfs (vfs-from-dirhash sum-line))
;> (pre)fetching "module.zip" "https://proxy.golang.org/golang.org/x/net/@v/v0.41.0.zip"
;> …
;; #<zilch.vfs#<vfs>>

(store-path-realised
 (vfs-to-store x-net-vfs))
;> [..building "/nix/store/vvs2xsgjpab4gwm530dvzcfzfcs5ak4m-zilchfile.drv"]
;> …
;; "/nix/store/r1p1y6bnmddmcrl2avsygp659d8iiih1-zilchfile/-"
----

== Dynamically generating ``go-package``s from packages

Zilch supports generating `go-package` records dynamically, from `go.mod` files
and `go.sum` files recursively. To do this, get a VFS for the primary module
you want to work with, and any module you want to resolve automatically (Zilch
will automatically crawl `go.sum` files to resolve dependencies, where needed.)

[,scheme,line-comment=;]
----
(define-values (root-module-path requirements)
 (collect-requirements-for-module x-net-vfs '()))

;> Collecting required modules
;> [..building "/nix/store/s9fmg08x56px39hjkwlba2pra5g1abgr-go.mod.json.drv"]
;> - found "golang.org/x/net" (requires 4 modules)
;> - found "golang.org/x/crypto" (requires 4 modules)
;> …
;; root-module-path -> "golang.org/x/net"
;; requirements
;;  -> #<mapping
;;       "golang.org/x/net" -> (#f . #<zilch.vfs#<vfs>>)
;;       "github.com/google/go-cmp" -> ("v0.6.0" . #<zilch.vfs#<vfs>>)
;;       "golang.org/x/crypto" -> ("v0.39.0" . #<zilch.vfs#<vfs>>)
;;        …
;;       "golang.org/x/mod" -> ("v0.25.0" . #<zilch.vfs#<vfs>>)>
----

The resulting values are a comprehensive set of modules mapped to their final
versions and the VFS of that module. This information can then be used to
create a procedure that returns the `go-package` for any package in this set of
modules, similar to how `go-stdlib-ref` works:

[,scheme,line-comment=;]
----
(define x-net-ref
 (collect-packages-from-requires requirements))

(x-net-ref "golang.org/x/net/idna")
;; #<go-package idna (golang.org/x/net/idna)
;;   api: #<store path /1w32lfajzbs83ibvgnn2cx4zk33nmg3xb20dhra1b89zyg3xin4i
;;          (ca~ /nix/store/az0g6zjc33mg3kfacy7gnfhkbkps8m2y-golang.org_x_net_idna-src.drv!api)>
;;   code: #<store path /1r40gj99ycm404ak3f2rkhl4qhy19hx1dkhjpmp0rbbsgy2f4wj7
;;           (ca~ /nix/store/b93f07hkv2yxf80pdwbra2r760yxkh3r-go-golang.org_x_net_idna-code.drv!code)>
;;   deps: ("fmt" "bidirule" "bidi" "norm" "math" "strings" "unicode/utf8")>

(x-net-ref "golang.org/x/text/runes")
;; #<go-package runes (golang.org/x/text/runes) …> <1>
----
<1> Even packages defined in a dependency can be looked up using this method.