Introduction

What is Want?

Want is a build system (WBS).

Software projects use build systems to reliably and reproducibly derive artifacts from source code. WBS manages setting up and tearing down isolated environments to build artifacts like static sites, assets, and executables for release or testing.

Why would I use Want?

If you've ever tried to get up and running with a new software project, you will have dealt with the problem of downloading all the tools that the project uses.

Even for simple tasks like building the docs, or producing shippable executables, you may have to install several tools. This requires developers to know about and understand tools which they will never work with themselves, to merely benefit from the output of a computation that others have set up.

If you or someone else put in the time to get something to work, it should also just work for you going forward. You shouldn't have to replay previous steps like: picking the right version, learning command line arguments, order of operations, etc.

With Want, every build target you add will be computed in exactly the same way on your laptop tomorrow, on your co-worker's development machine, and in CI. Want forces build steps to be fully and exactly specified, so they can be reproduced by anyone at anytime, in any environment.

Why is it called Want?

You write down what you want, and Want builds it.

More seriously: .want and .wants were good extensions for the files where you specify build targets, and naming the entire build system after them came later.

Using Want

This section will get you up and running with Want.

Most of the time you should run want build. This command is suitable to use as a pre-merge check. It will build all the targets in the current module. It does not and cannot perform publishing or other side effects.

Creating a Want Module

Want operates on Modules. A Module is just a filesystem tree with a file named WANT at its root. The WANT file holds module level configuration, and its presence is what tells Want that a directory is meant to be built.

To start a new module, run want init in your project's root. It writes a WANT file with the default module config — the want standard library as a dependency — and a default .wantignore:

.deps = [
	{"want", \(_: Ctx) FS LATEST},
]

See Module Files for what the WANT file can contain, and how the sibling .wantignore file controls which paths are imported.

Creating a Build Target

Want calls the things that you, the user, want, but don't yet have, and need to build, Targets.

A target is declared by a file ending in .want. Create a file named hello.txt.want at the root of your module, and fill it with the following:

build = (#import "want:build")
paths = (#import "want:build/paths")

.input = [
    (paths.unit "./input.txt")
]
.do = \(ctx: build.Ctx) build.FS {
    (build.pick ctx.jobs ctx.input "input.txt")
}

The file declares two things: the module paths the build reads (.input), and the computation that produces the output (.do). Here do simply picks the input file input.txt out of the target's input tree and returns it.

Create input.txt next to it containing Hello World!, then run

$ want cat hello.txt
Hello World!

The unit file hello.txt.want produces the output path hello.txt (its file name minus the .want extension), and want cat reads that file out of the build output.

Inspecting the Build Output

Normally cat hello.txt would print the source file, but want cat reads from the build output, while the regular cat reads from the regular filesystem, which is the build input.

Want organizes build targets into an output filesystem. This filesystem only exists virtually, within Want, but it can be inspected using UNIX-like tools (ls and cat).

want ls <path> will list the contents of a tree in the build output. Compare this to the output of regular ls.

want serve-http <addr> builds the output and serves it as a read-only HTTP file tree (for example want serve-http 127.0.0.1:8080): directories render as an index of links, and files are written with a content type guessed from their extension. Stop it with Ctrl-C.

This simple translation of inputs to outputs is what makes Want so easy to use and reason about. You will never find yourself in a situation where you don't know when/how/where a Target is being computed.

Armed with just want ls, want cat, and want build, you can get pretty far. To learn more about what you can build with Want, read the Build Configuration section.

Publishing and Secrets

A unit file may export an optional .sink proc: the side effect to run after the build is green. Sinks do not run in a normal build; want build --publish runs them. A sink belongs to its target, consumes the target's output as ctx.input, and reaches the outside world through the job port and the capabilities in build.SinkCtx.

Secrets reach a sink through ctx.secrets, a list of ports in the order the WANT's .secrets grants name them. Supply them on the command line:

# one raw secret
want build --publish --secret mySecret:/path/to/my_secret.txt
# or one spore map carrying several named values
want build --publish --secrets cloudflare:~/.config/want/cloudflare.sp
want build --publish --secrets github:~/.config/want/github.sp

Only the targets the build query names directly are published; a target pulled in to satisfy a dependency is built but its sink does not run. A publish build fails if a directly queried target (or the sink that will run for it) is granted a secret that was not supplied. See Module Files for the .secrets grant syntax.

Concepts

This section provides a conceptual overview of WBS. If something doesn't make sense or you want to learn more about WBS's architecture, this is a good section to read.

TLDR

Want represents all build inputs and outputs as values in a content-addressed data model called Mycelium. Every value has a cryptographic hash, and equal values have equal hashes, so Want can easily tell whether two trees, files, or computations are identical.

A build is a transformation from an input filesystem (a Module, plus the outputs of its dependencies) to an output filesystem. Everything a computation reads is named explicitly, and the only way to reach outside the module is through the small set of executors, so a computation can always be cached and replayed.

The pieces you will read about next:

  • Data Model: how Mycelium values and the Want filesystem (wantfs) are shaped.
  • Tasks & Jobs: how units of work are named, cached, and run.
  • Executors: responsible for executing Tasks, in the context of a Job.
  • Spore: the language that want files are written in.

Data Model

Mycelium

Want stores all data in the Mycelium format. Mycelium is a serialization format like JSON or Protocol Buffers. In addition to the functionality of those two technologies, Mycelium allows functions to be sent over the wire and stored.

Mycelium uses cryptographic hashes as references. So all data-structures stored in Mycelium are cryptographic commitments to the full transitive closure of a node in the data structure.

Mycelium allows Want to cache anything in a meaningful representation. This covers Bits, Bytes, Strings, Programs, and Procedures, as well as Lists, Arrays, Products, and Sums of all those. Instead of converting those to another less accurate format like JSON, which also throws away any concept of reference, Mycelium allows a faithful representation.

The Want Filesystem (WantFS)

WBS stores all file data in the WantFS format. WantFS is built on top of Mycelium, and inherits the property of being content-addressed, and copy-on-write.

WantFS is used to represent file data whenever it has to be cached, or presented as the input to a build step. The output of want files is also stored in WantFS.

The first step in a build is to import the module from the local filesystem into WantFS.

Tasks & Jobs

Tasks

A Task is a well-defined unit of work. Well-defined means that all the information that could be required to complete the Task is explicitly contained in the Task.

A Task has 2 parts: an operation name and an input (an AnyValue):

Task = (#Struct {
  .op_name = String
  .input   = AnyValue
})

The operation names one of a small number of built-in operators, for example:

  • want.pick
  • want.place
  • download.tree
  • linux.cspawn
  • wasm.run
  • mvm.apply

The input is the operator's argument value. Both parts are part of the hash which becomes the Task's identity, so a Task with the same operation and input is the same piece of work.

Want caches the output of Tasks when they complete, and checks this cache before computing a Task. An immediate rerun of want build without any changes therefore produces all the same Tasks, and every one of them can be skipped.

Jobs

Jobs are used to track running computations in Want.

Jobs are created to compute Tasks. A Job carries active state, like whether it has completed and when it started and finished: it is spawned when created, queued when admitted, running once a worker picks it up, and done (with an ok value or an error) once it has an outcome.

Jobs are organized into a hierarchical tree. All the major operations in Want create a new root Job; that Job may spawn additional child Jobs.

Child Jobs give a computation access to additional compute, since children run concurrently, and they are each an opportunity for caching.

The Job System Port

A build computation is not given access to the host; instead it is handed a jobs port, a capability it uses to spawn and await child Jobs. In Spore this port has the type build.JobCtx, and a Task's do proc receives it on ctx.jobs.

Through the port a computation can:

  • spawn jc limits op input: start a child Job for a Task and get its JobID immediately, under a resource ceiling. A Job that runs past its max_duration ceiling is cancelled automatically.
  • limits jc: the current Job's resource ceiling. Spawning with it passes the ceiling down unchanged.
  • await jc id: block until that child finishes and return its result; a child that failed faults the parent.
  • inspect jc id: check a child's state without blocking.
  • awaitAny jc ids: block until any one of a set of children finishes.

A Job cannot spawn root or sibling Jobs; it can only spawn its immediate children. This keeps runaway computations from escaping their parent, and gives every Job an audit trail back to the user who asked for the result.

Executors

WBS dispatches Tasks to Executors. An Executor knows how to do particular types of Tasks, and can provide any additional context.

Executors are the small set of built-in operations that can touch the outside world or run programs. They are the only place where a computation can do something non-deterministic (like use the network), and each is designed so that its successful outputs are fully determined by its inputs.

Want Executor

Tasks directly related to planning, building, or manipulating files are executed by the Want Executor.

Here are a few examples:

  • want.pick extracts a subpath from a WantFS instance and returns a new instance containing just the subpath shifted to the root.
  • want.place is the opposite of pick.
  • want.build builds a want module provided as a WantFS instance.
  • want.plan is the planning portion of the build, separated so that it can be cached separately.

I/O Executor

This executor performs all input and output tasks. If something is incoming or outgoing to/from the build system, then it goes through this executor.

Linux Executor

This executor spawns Linux containers to execute tasks. Think of this as "Linux as a Service".

Wasm Executor

This executor runs WebAssembly programs.

MVM Executor

This executor performs computations defined in the Mycelium format. Mycelium Procs can be applied to data with mvm.apply. Mycelium Lazys can be evaluated with mvm.eval.

All of the want files are evaluated using this executor.

Spore Executor

This executor calls into the Spore compiler. It is used to compile want files, module configuration files, and Spore libraries into Mycelium programs, which can be executed by the MVM executor.

Build Configuration

Want configuration is done using 3 kinds of files, all written in Spore:

When want build runs, it:

  1. Checks that the current directory has a WANT file which is a valid module config; if not, it exits with a "not a module" error.
  2. Finds every file ending in .want and .wants, and evaluates each into one or more Targets.
  3. Validates that the Targets together are a valid build plan.
  4. Runs the build according to the plan.

Each .want/.wants file is evaluated independently as a single-file Spore body, and must export the fields described for its kind.

The build departs from the module's source files only through the paths a target declares in its .input. Which paths are imported into the module in the first place is controlled by the root .wantignore file, described in Module Files.

Module Files

Want operates on units called Modules. A Module is a filesystem tree with a file named WANT at its root.

The WANT file provides module-level configuration and doubles as a sanity check that a directory is supposed to be built by Want. It is a Spore body that evaluates to a module config:

.deps = [
    {"want", \(_: Ctx) FS LATEST},
]

.deps

.deps lists the module's dependencies. Each dep is a tuple of a name and a proc that produces the dependency's filesystem from a build context ((#Product String (#Proc build.Ctx build.FS))). At plan time every dep's proc runs and its returned filesystem is registered as a Spore source under <name>:; planning files import its packages as <name>:pkg.

The WANT file is compiled with a preamble binding Ctx (build.Ctx), FS (build.FS, a wantfs.Segment) and LATEST (the compiled-in want standard library as an FS), so the default dep above gives the module the standard library under want: without any imports. The spore standard library (core:) is always implicit; a module that does not declare a want dep has no want stdlib. want init writes the default config above.

.secrets

.secrets grants named secrets to the module's planning files. It is a list of {paths, names} tuples, where paths is a paths.Set matched against a planning file's module-root-relative path, and names is the list of secret names that file may access:

paths = (#import "want:build/paths")

.secrets = [
    {{.all = {}}, ["cloudflare"]},                       // the whole module
    {(paths.prefix "site/"), ["S3_KEY"]},                // files under site/
    {(paths.unit "build/doc.want"), ["DOCS_TOKEN"]},     // one file
]

Only want build --publish uses secrets. Each matching target's build and sink contexts receive the granted names as a (#List (#Port Unit Unit)) in grant order (first occurrence wins), so a sink reads the first granted secret with (#listslot ctx.secrets 0). The names must be supplied on the command line:

  • --secret NAME:PATH — a single secret: the file's bytes.

  • --secrets NAME:PATH — a spore module whose exports are the map entries:

    .CLOUDFLARE_API_TOKEN = "..."
.CLOUDFLARE_ACCOUNT_ID = "..."

--secrets cloudflare:~/.config/want/cloudflare.sp supplies the map under the name cloudflare, matching the grant above. A GitHub release upload takes a file exporting .GITHUB_TOKEN in the same way: --secrets github:~/.config/want/github.sp. A name supplied twice is an error; names supplied but unused are ignored. A publish build fails before running any target if a file it will publish is granted a name that was not supplied. Secret values are loaded into memory (never the content store), and only port tokens travel through tasks. Keep secret files outside the module tree (or dot-ignored, as .* in the default .wantignore) so the workspace import does not capture them as ordinary sources.

.wantignore

Which paths are imported into the module is configured separately, by a root .wantignore file. It uses gitignore-style patterns, one per line: any path matching a pattern is left out of the imported module tree.

By default a project that uses Git should ignore its VCS metadata and build outputs. For example:

.*
zig-out/
zig-cache/

.* ignores every dotfile (including .git), which keeps the import fast; the other entries ignore build output directories.

.want is always ignored, and files starting with .want cannot be imported.

Unit Files

Unit files end in .want. Each unit file declares a single Target.

A unit file is a Spore body with two exports:

build = (#import "want:build")
paths = (#import "want:build/paths")

.input = [
    (paths.unit "./input.txt")
]
.do = \(ctx: build.Ctx) build.FS {
    (build.pick ctx.jobs ctx.input "input.txt")
}
  • .input is a list of PathSets the target reads; the list is implicitly OR'd together. An empty list means the target reads nothing.
  • .do is a proc that is called to produce the build. It is handed a build.Ctx (its job port on ctx.jobs, and its filtered input tree on ctx.input) and returns an FS.

The output path

A unit file does not declare its output; it is derived from the file's name. A file hello.txt.want produces the output path hello.txt, and a file build/app.want produces build/app.

The FS returned by do is shifted into place under that path. A flat single-file FS (the result of picking a file) lands exactly at the output path; a tree becomes the subtree beneath it.

The build fails during planning if the derived output path already exists in the source tree, so a target can never silently overwrite a source file.

Path rules

Paths in .input are module-root-relative, with two exceptions:

  • A path beginning with ./ is relative to the directory of the unit file. In a/build_my_output.want, the input ./input.txt names a/input.txt, while input.txt names the file at the module root.
  • Paths may not begin with / (absolute paths are rejected), may not be empty, and may not contain a .. segment.

Inputs implicitly exclude the target's own output path.

Example: download a rootfs

fetch.want downloads a pinned Alpine minirootfs and returns it:

build   = (#import "want:build")
alpine  = (#import "want:linux/dist/alpine")

.input = []
.do = \(ctx: build.Ctx) build.FS {
    (alpine.rootfs ctx.jobs)
}

The result lands at fetch/ in the build output. want ls fetch lists it, and want build computes the whole module.

Multi Files

Multi files end in .wants. Unlike a unit file, a multi file declares a list of Targets, and each Target must declare the paths it is allowed to write.

A multi file is a Spore body with a single export, .targets:

build = (#import "want:build")
paths = (#import "want:build/paths")

.targets = [
  {
    .input = [
      (paths.unit "./input.txt")
    ]
    .output = [
      (paths.suffix ".out.txt")
    ]
    .do = \(ctx: build.Ctx) build.FS {
        // generate a bunch of files ending in ".out.txt"
    }
  }
]
  • .input is a list of PathSets the target reads, exactly as for a unit file.
  • .output is a list of PathSets the target may write to. The FS returned from do is filtered through this set: writing to an undeclared path fails the build.
  • .do is the proc that produces the build, the same as for a unit file.

Rooting

Every target in a multi file is rooted at the parent directory of the .wants file. The FS returned by do is positioned there, and .output is interpreted there too.

A protobuf.wants file at the root of a module produces files at the module root. If it were at a/b/c/protobuf.wants, and its do returned a filesystem containing messages.pb.go and api.pb.go, the resulting output would be:

a/b/c/messages.pb.go
a/b/c/api.pb.go

The .output declaration does not need to repeat the prefix. Either of these would accept the files above:

.output = [
  (paths.unit "messages.pb.go")
  (paths.unit "api.pb.go")
]

or, more flexibly, allowing the names to change as long as they keep the suffix:

.output = [
  (paths.suffix ".pb.go")
]

Why declare outputs?

A unit file's output path is fixed by its name, so conflicts with other targets are impossible. A multi file can write anywhere under its directory, so declaring .output lets Want check, before anything runs, that no two targets claim overlapping outputs, and that a target only writes where it said it would.

Standard Library

Want ships a standard library of Spore packages. Planning files can import these packages. An import names a package (a directory of .sp files) or a single file. Standard-library packages are namespaced: the want standard library is want: and the spore standard library is core:. An import with no colon resolves in the same source as the importing file — a module-relative path in your module, or a sibling package within a standard library:

ns    = (#import "core:ns")
build = (#import "want:build")
linux = (#import "want:linux")
paths = (#import "want:build/paths")

Callers are responsible for spawning and awaiting Jobs through their job port; the standard library functions take that port (build.JobCtx) as their first argument, by convention. A tool package may set up a package-specific context instead and take it first: the zig and go commands take a zig.Ctx or go.Ctx (built with new from the job port, limits, platform, and version).

build

The core of the build library (stdlib/build).

Types

  • FS: a wantfs tree. This is the value a do proc returns and the type almost everything operates on.
  • Ctx: the build context handed to a do proc: 
    Ctx = (#Struct { .jobs = JobCtx .input = FS .secrets = (#List (#Port Unit Unit)) })
    secrets is empty unless the build publishes; see SinkCtx below.
  • SinkCtx: the context handed to a unit file's .sink proc by want build --publish: (#Struct { .jobs = JobCtx .input = FS .internet = (#Port Unit Unit) .secrets = (#List (#Port Unit Unit)) }). input is the target's do result; secrets is the file's granted secret ports, in the order the WANT's .secrets names them (supplied with --secret NAME:PATH or --secrets NAME:PATH). Sinks run only for targets the build query names directly, and a publish build fails if a directly queried target (or its running sink) is granted a name that was not supplied.
  • JobCtx: the jobs port used to spawn and await child Jobs.
  • Task, JobID, JobState, JobResult: the job system's wire types.
  • Limits: the resource ceiling a spawn requests for its child ((#Struct { .memory = B64 .cpu = B64 .max_duration = B64 })). memory is bytes, cpu is microseconds per second, and max_duration is seconds (0 means no ceiling); a child running longer than max_duration is cancelled.

Jobs

  • spawn jc limits op input -> JobID: start a child job for a Task under limits.
  • limits jc -> Limits: the current job's spawn-time ceiling. Passing it to spawn propagates the ceiling down the tree unchanged.
  • await jc id -> AnyValue: wait for a child; a failed child faults the parent.
  • inspect jc id -> JobState: check a child without blocking.
  • awaitAny jc ids -> JobID: wait for any of several children.

Filesystem

  • pick jc root path -> FS: re-root the tree at path; picking a file yields a flat single-file FS.
  • pickRoot jc root path -> FS: the same operation, named for re-rooting a directory.
  • fileAt jc root path -> FS: pick a single file (a synonym for pick).
  • place jc ents -> FS: mount named single-file FSes at their names: 
    (build.place jc [
    (build.ent "bin/hello" exe)
])
  • ls jc root path -> (#List String): the immediate children of a directory.
  • filterPaths jc root set -> FS: keep only the entries of root whose path the PathSet set contains; entry paths are unchanged, so the filtered tree is not re-rooted.
  • emptyDir jc -> FS: a tree with no entries.

Downloads

  • downloadFile jc task -> FS: fetch a file from one of task.urls, verified against task.hash; a non-none task.compression decompresses the verified payload into the returned file.
  • downloadTree jc task -> FS: fetch an archive (tar, zip, or deb) and expand it into a tree. task.compression names the archive payload's compression (gzip, xz, zstd, or none); decompression is explicit, never sniffed.

OCI

  • downloadOciImage jc task -> FS: pull an OCI/Docker image into an OCI layout.
  • pushOciImage jc task -> PushedOCIImage: push an OCI layout to a registry repository and attach the task's tags. The push is idempotent: blobs, manifests, and tags the registry already holds are skipped. The registry credentials come from task.secrets, a --secrets map whose exports include .OCI_USERNAME and .OCI_PASSWORD.
  • OCIImageTask (url, repo, algo, hash), PushOCIImageTask (url, repo, tags, image, secrets), and PushedOCIImage (digest, tags): the OCI wire types.

Importing build also gives you its download, job, and oci modules' exported functions, since a package is compiled from every immediate child .sp file.

build/paths

The PathSet constructors used in .input and .output:

  • Set: the PathSet tagged union itself ((#TaggedUnion {.unit = String .prefix = String .suffix = String})), the parameter type of build.filterPaths.
  • unit p -> Set: the single path p.
  • prefix p -> Set: every path beginning with p.
  • suffix s -> Set: every path ending in s.

linux

The Linux container executor (stdlib/linux).

  • run jc cspec -> CSpawnResult: run a container described by cspec, and return its exit code plus the tree captured from its output mount.
  • CSpec: the container description: init, args, env, wd, mounts, network, stdin, stderr, stdout.
  • initMemfd fs: run an executable from a flat single-file FS (copied into an anonymous memfd).
  • initPath p: run the executable already at path p inside the container.
  • Mount sources: snapshot fs (read-only), output, emptyDir, tmpfs n, drive n, overlay rlayer wlayer (lower may be a snapshot/drive; upper must be a writable drive/tmpfs), devNull, procfs, sysfs, cgroup2fs, devpts.
  • mount dst src -> Mount: pin a source at a container path.
  • net name backend, loopback, private, internet cap: network interfaces.
  • stdio: stdinNull, stdin ctx, stdout ctx, stderr ctx, outputLog, outputClosed, stdinClosed.
  • fileAtPath jc root path -> FS: pull a file out of a container tree.

linux/dist/alpine

A pinned Alpine minirootfs: a small musl userland with the busybox shell.

  • rootfs jc -> FS: expand the pinned minirootfs tarball into a tree.
  • fileAt jc path -> FS: pull a single file out of it.
  • version, arch: the pinned release.

linux/dist/debian

A pinned rblaine95/debian base image (a GHCR mirror of Docker Hub's official image). Like Alpine, but a full glibc userland.

  • image jc -> FS: pull the pinned image as an OCI layout.
  • rootfs jc -> FS: unpack the image's root filesystem.
  • fileAt jc path -> FS: pull a single file out of it.

linux/dist/arch

Arch Linux packaging on top of a pinned official Arch container image (with the bash, pacman, and bsdtar makepkg itself uses). package takes already-built artifacts as a build.FS tree and produces the .pkg.tar.zst pacman installs; nothing is compiled.

  • rootfs jc -> FS: the pinned image's root filesystem.
  • fileAt jc path -> FS: pull a single file out of it.
  • version: the pinned image build.
  • Spec: the required metadata (.name, .version, .arch, .desc, .builddate, .files). version is the full pkgver including the pkgrel (and optional epoch), and builddate is a decimal Unix timestamp that fixes every file's mtime, so equal inputs produce equal packages.
  • Meta: the full metadata record; meta spec -> Meta fills the optional fields with defaults.
  • withUrl, withPackager, withLicense, withDepends, withOptdepends, withProvides, withConflicts, withReplaces, withGroups, withBackup, withInstall: rebind one optional field.
  • filename meta -> String: the output file's name, name-version-arch.pkg.tar.zst.
  • pkginfo meta -> String: the rendered .PKGINFO.
  • package jc meta -> FS: assemble the package and return the tree captured from the output mount (one file, filename meta); use build.fileAt for the package as a flat single-file tree. The target faults if the packager fails.
  • checkExit code -> Unit: fault unless a container exited zero.

The payload is a tree rooted at /, built with build.place or picked from a build output:

files = (build.place jc [
    (build.ent "usr/bin/hello" exe),
])
m = (arch.meta {.name = "hello" .version = "1.0-1" .arch = "x86_64"
                .desc = "..." .builddate = "1700000000" .files = files})
m = (arch.withLicense m ["MIT"])
pkg = (arch.package jc m)

docker

  • pull jc task -> FS: pull an OCI image.
  • umoci jc -> FS: the pinned umoci binary.
  • rootfs jc img -> FS: unpack a pulled image's root filesystem.

git

A pinned static git build (darkvertex/static-git), used to give a container a git binary with no package manager or network.

  • tool jc -> FS: the expanded static git tree (git at its root). Mount it on PATH (e.g. at /git) and set GIT_TEMPLATE_DIR to an existing directory.

github

Upload release assets to an existing GitHub release (stdlib/github). Credentials come from a --secrets map whose only required export is .GITHUB_TOKEN, a token with write access to the repository's releases (contents: write). The preferred setup is a spore file:

.GITHUB_TOKEN = "..."

want build --publish --secrets github:~/.config/want/github.sp
  • Ctx: the GitHub context ((#Struct {.jobs = JobCtx .secrets = (#Port Unit Unit)})); build it once with new.
  • new jc secrets -> Ctx: bind the job port and the secrets port granted by the WANT's .secrets.
  • UploadOpts: (#Struct {.owner = String .repo = String .tag = String}), naming the existing release to attach assets to.
  • Asset: one uploaded asset (.id, .name, .size, .content_type, .browser_download_url).
  • ReleaseUpload: the op's result: the release and the assets it now carries.
  • uploadRelease ctx opts assets -> ReleaseUpload: upload every top-level file in assets as an asset of the named release; an existing asset with the same name is replaced.

go

The pinned Go toolchain (stdlib/go).

  • Ctx: the go environment ((#Struct {.jc = JobCtx .limits = Limits .platform = Platform .version = Version .modcache = FS .env = (#List String)})). Every command takes it as its first argument.
  • new jc limits platform version -> Ctx: set up the environment once, then hand it to every command. platform/version select the toolchain distribution to fetch; jc and limits are the job resources commands run under. The module cache starts empty.
  • withModcache ctx modcache -> Ctx: rebind the module cache to the tree returned by pkg.
  • withEnv ctx env -> Ctx: rebind the command-wide environment. Entries are prepended to the fixed environment and the first entry for a variable wins, so they override it (for example GOFLAGS, GOPROXY, CGO_ENABLED).
  • withTarget ctx platform -> Ctx: prepend GOOS/GOARCH for platform, cross-compiling pure-Go builds with the host toolchain.
  • LATEST, v1_27_1: the pinned Version values; linux_amd64 is the pinned Platform; GOOS_*/GOARCH_* name every target.
  • dist ctx -> FS: download the pinned distribution and re-root it as a container root.
  • pkg ctx src hash -> FS: run go mod download for the module tree src (under the context's environment) in a network-enabled container and return the resulting module cache, audited against hash.
  • cmd ctx cwd args -> CSpawnResult: run the pinned go tool over the module tree cwd (mounted at /src) in a container with no network. args is the full argv, e.g. ["go", "build", "-o", "/out/app", "./cmd/foo"].
  • build ctx modfs entrypoint -> FS: compile the main package at entrypoint into a static executable.
  • test ctx modfs -> CSpawnResult: run go test ./...; its stdout and stderr are captured as /stdout and /stderr in the output tree.
  • testBinaries ctx modfs -> CSpawnResult: compile every package's test binaries into the output tree without running them.
  • checkExit code -> Unit: fault the build unless code is zero.

go/dist

The Go release pins and the mirror URLs used to fetch them (stdlib/go/dist.sp, part of the go package): Version, LATEST, v1_27_1, Platform, linux_amd64, tarballName, topDir, dist, and mkurls.

python

The pinned Python toolchains: CPython (the interpreter, from python-build-standalone) and uv (the package manager), each fetched from its own release artifacts (stdlib/python).

  • Ctx: the python environment ((#Struct {.jc = JobCtx .limits = Limits .platform = Platform .uv = UvVersion .python = PythonVersion .venv = FS .env = (#List String)})). Every command takes it as its first argument.
  • new jc limits platform uv python -> Ctx: set up the environment once, then hand it to every command. platform selects both distributions; uv and python pin the versions to fetch (see below). jc and limits are the job resources commands run under. The project environment starts empty.
  • withVenv ctx venv -> Ctx: rebind the project environment to the tree returned by pkg; commands mount it at /venv and prefer its bin/ on PATH.
  • withEnv ctx env -> Ctx: rebind the command-wide environment. Entries are prepended to the fixed environment and the first entry for a variable wins, so they override it (for example UV_NO_DEV).
  • UV_LATEST, PYTHON_LATEST, PYTHON_3_13: the pinned UvVersion and PythonVersion values; x86_64_linux is the pinned Platform.
  • uvDist ctx -> FS: the pinned uv distribution (uv and uvx), to mount at /uv.
  • pythonDist ctx -> FS: the pinned CPython distribution, to mount at /python.
  • pkg ctx src hash -> FS: run uv sync --frozen for the project tree src (which must contain pyproject.toml and uv.lock) in a network-enabled container with the pinned toolchains, and return the resulting venv, audited against hash. Mount the result on the context with withVenv before running commands.
  • cmd ctx cwd argv -> CSpawnResult: run any command over the tree cwd in the hermetic command container; argv is resolved on PATH through the venv, the pinned interpreter, and uv, and each element is passed verbatim.
  • python ctx cwd args, uv ctx cwd args, sh ctx cwd script: cmd with the pinned interpreter, uv, or the Debian shell (for pipelines and redirects) as the program.
  • checkExit code -> Unit: fault the build unless code is zero.

python/dist

The uv and CPython release pins and the asset URLs used to fetch them (stdlib/python/dist.sp, part of the python package): UvVersion, PythonVersion, UV_LATEST, PYTHON_LATEST, PYTHON_3_13, Platform, x86_64_linux, uvUrl, pythonUrl, uvDist, and pythonDist.

zig

The pinned Zig toolchain.

  • Ctx: the zig environment ((#Struct {.jc = JobCtx .limits = Limits .platform = Platform .version = Version .pkgfs = FS .env = (#List String)})). Commands take it as their first argument.
  • new jc limits platform version -> Ctx: set up the environment once, then hand it to every command. platform/version select the toolchain; jc and limits are the job resources commands run under. The package directory starts empty.
  • withPkgfs ctx pkgfs -> Ctx: rebind the package directory to the tree returned by pkg; commands mount it at /src/zig-pkg.
  • withEnv ctx env -> Ctx: rebind the command-wide environment. Entries are prepended to the fixed environment and the first entry for a variable wins, so they override it.
  • LATEST, STABLE, v0_16_0: the pinned Version values; x86_64_linux is the pinned Platform.
  • dist ctx -> FS: download the pinned distribution and re-root it as a container root.
  • pkg ctx src hash -> FS: fetch the build.zig.zon dependencies of src through zig build --fetch, audited against hash (the zig-pkg fingerprint).
  • cmd ctx cwd args -> CSpawnResult: run the pinned zig tool over the source tree cwd (mounted at /src); args is the full argv, e.g. ["/zig/zig", "build", "test"].
  • buildArgs ctx src args -> CSpawnResult: cmd plus the fixed --build-file, --prefix, and --cache-dir flags, for arbitrary zig build steps; returns the exit code plus the captured output tree.
  • build ctx src target -> FS: run the binaries step for the Zig target triple target and return its output tree.
  • checkExit code -> Unit: fault the build unless code is zero.
  • run ctx args -> AnyValue: run the compiler with stdio wired to the host.
  • watch ctx stdin stdout stderr -> B8: run zig build --watch on the build context's working directory, with stdio wired to the given ports; returns the container's exit code.

zig/dist

The Zig release pins and the mirror URLs used to fetch them (stdlib/zig/dist.sp, part of the zig package): Version, LATEST, STABLE, v0_16_0, Platform, x86_64_linux, tarballName, topDir, dist, and mkurls.

mdbook

Renders a markdown book to static HTML, using the pinned static-musl mdbook release inside an Alpine container. The environment (Alpine rootfs and mdbook) is mounted from constant snapshots, so only the book source changes between builds.

  • tool jc -> FS: the pinned mdbook distribution, re-rooted.
  • build jc src book_toml summary -> FS: render the markdown in src, with book_toml as its book.toml and summary as src/SUMMARY.md, and return the generated site. The build faults if mdbook fails or produces no index.html.

sys

Types for the surrounding system context used by entrypoints (not by build planning files): Ctx, ctxPort, and the NetBackend choices.

ns

Helpers for building namespaces (NS, get, mkEntry), used when a Spore entrypoint is handed its context as a namespace.

bits

Bit-width aliases (B8…B256) and the bitwise primitives: AND, OR, and XOR on Bit, plus b32AND and b32OR on B32.

strings

Byte-level string helpers mirroring a subset of Go's strings (a String is a List of bytes):

  • len s -> B32.
  • hasPrefix s p -> Bit, hasSuffix s p -> Bit, contains s p -> Bit.
  • index s p -> (Maybe B32): the first index of p in s, or Nothing.
  • count s p -> B32: non-overlapping instances of p; an empty p counts len(s) + 1 insertion points.
  • split s sep -> (#List String): an empty sep splits into single bytes.
  • join parts sep -> String, repeat s count -> String.
  • trimPrefix s p -> String, trimSuffix s p -> String, trimSpace s -> String (ASCII whitespace).

The Spore Programming Language

Spore is a small programming language which compiles to Mycelium. WBS uses Spore as the configuration language.

Spore is designed for simplicity; it is a smaller language than Python or even Go. This article gives a complete tour of Spore. It resembles what an engineer would write on a whiteboard when outlining a program.

Spore is a compiled, typed language with immutable values.

Symbols & Variables

Spore has immutable variables, which you can bind with =. Variables must be bound to a symbol. Symbols must start with a letter and can contain alphanumerics and _.

Assign a number to myVar.

myVar = 1234

Literals

Bits

A bit is the unit of information. There are two possible values 0 and 1.

firstBit = 0
secondBit = 1

The natural type of the numerals 0 and 1 is Bit, but they can be coerced to other types as well.

Numbers

We have already seen numbers used in the previous examples. Spore's numbers are coerced to (#Array Bit _) where _ is some value of type (#Array Bit 32) aka B32. Numbers must start with - or a numeral.

Underscores are allowed for readability

aMillion = 1_000_000

Hexadecimal can be specified with 0x

sixteen = 0x10

Binary can be specified with 0b

eight = 0b1000

The preamble to the compiler (included before all code sent for compilation) includes a few basic integer types, to which these numeral literals can be coerced.

B8
B16
B32
B64
B128
B256

You can define an integer type of any bitwidth

(#Array Bit 4096)

The natural type of a numeral is the smallest bit array that can represent it. For example 31 would have type (#Array #Bit 5)

Strings

Strings are literal values of type List[Byte].

Double-quoted strings interpret escape sequences:

EscapeByte
\nnewline
\ttab
\rcarriage return
\0zero byte
\\backslash
\"double quote
\xHHthe byte with hex value HH
aString = "this is a string"
aTabbed = "col1\tcol2"

Backtick strings take everything between the backticks literally, newlines, backslashes, and double quotes included. They cannot contain a backtick.

aRawString = `C:\path\to\file`
aMultiline = `set -e
echo hello
`

Arrays

Spore Arrays are ordered, finite sequences of elements, which are all the same type. They can be coerced to Mycelium Arrays or Lists.

Lists are delimited with [ and ]. The elements are separated with a comma. A trailing comma is tolerated and not required.

aListOfStrings = ["a", "b", "c"]

Tuples

A Tuple is an ordered, finite sequence of elements, each of an independent and not necessarily distinct type. Tuples compile to Mycelium Products.

Tuples are delimited with { and }. The elements are separated with a comma. A trailing comma is tolerated and not required.

aTuple = {"a", 1, "b", 2}

Lookup Table

Lookup Tables (LUT) are the primitive unit of computation in Spore. They look like match/switch statements in other languages. A LUT is a mapping from one set of values to another.

LUTs are delimited by { and }. Inside are comma separated cases. A case is a pair joined by =>.

{
  0 => "zero",
  1 => "one",
}

The natural type of a LUT is a Proc type, which takes the type of all the things to the left of the =>, and returns a value of the same type as the things to the right. If a common type cannot be inferred, then a type must be specified.

((#Proc B2 String) {
  0 => "a",
  1 => "b",
  2 => "c",
  3 => "d",
})

All LUT cases must be mutually exclusive.

_ can be used once per LUT as a catch-all.

myLookTableProc = ((#Proc String B2) {
  "a" => 1,
  "b" => 2,
  "c" => 3,
  
  _ => 0,
})

Lambdas

Lambdas represent procedures. Lambdas start with \, which is followed by a list of arguments, a return type, and then an expression.

This lambda takes a String and returns a String which repeats it 3 times.

repeatThrice = \(a: String) String {
  (#flatten [a, a, a])
}

Lazies

Lazy values are delayed computations which require no additional input. They are specified like Lambdas, but without the arguments.

thisWillBeA4 = \B32 {
  two = (B32 2)
  (#add two two)
}

Content-ID (CID)

A CID is the hash of some data. CIDs are often long, and copy-pasted rather than written by humans. They start with an @ and contain hex characters after that. CIDs take up 256 bits.

ref = ((#Ref (#Run Byte)) @deadbeef)

As a special case, 160-bit values are padded with zeros to 256 bits. This is to accommodate SHA1 hashes.

S-Expressions

In Spore, computation is expressed by enclosing a function and an input to that function together in parenthesis. The whole thing, from ( to the corresponding ), is called an S-Expression, or SExpr for short.

Apply a Procedure

twice = \(x: B32) B32 (#add x x)
(twice 50)

evaluates to 100 :: B32

The first element of the SExpr is evaluated first. If it is a procedure, then that procedure is applied to a product created from the rest of the elements of the SExpr.

Eval a Lazy

An SExpr with a single element will evaluate a Lazy value.

lazy = \B32 (#add 1 2)
(lazy)

evaluates to 3 :: B32

Type an Expression

You can force any expression to be a certain type by having a type in the first position of the SExpr.

tenAsB4Naturally = 10
tenAsB32 = ((#Array Bit 32) 10)

Both of these are fine

3array = ((#Array String 3) ["a", "b", "c"])
list = ((#List String) ["a", "b", "c"])

Access a Field

A Tuple is like a function from indices to values.

aTuple = {"a", 2, \B32 4}

(#check (aTuple 0) "a")
(#check (aTuple 1) 2)
(#check ((aTuple 2)) 4)

Primitives

Primitives are operations exposed by the compiler. They are defined in the compiler's source, not anywhere in the Spore source. A Primitive always starts with #, followed by alphanumerics and symbols.

Importing

  • mypkg = (#import "a/b/c") imports a directory of Spore files at a/b/c.
  • mypkg = (#import "dir/file.sp") imports a single Spore file at dir/file.sp.

Math

  • (#add 1000 1000) performs modular arithmetic on integers of the same type. 1000 has type (#Array Bit 10) naturally.

Types

  • #Bit the bit type.
  • (#Array ElementType len) evaluates to a new Array type.
  • (#List ElementType) evaluates to a new List type.
  • (#Product TypeA TypeB TypeC)
  • (#Sum TypeA TypeB TypeC)
  • (#Proc Input Output)
  • (#Lazy Output)
  • (#Ref ElementType)
  • (#Distinct BaseType anyValue)
  • #AnyType a type capable of holding instances of any type.
  • #AnyValue a type capable of holding instances of any value (that's everything).
  • #AnyProg a type capable of holding any program.

Errors

  • (#panic value) faults the MVM.
  • result = (#brace thingThatCouldPanic arg1 arg2). Returns a (#Sum Output #AnyValue) if thingThatCouldPanic has return type Output.
  • (#check expr) panics if expr is not equal to 1.
  • (#check expr1 expr2) panics if expr1 and expr2 are not equal.

Store

  • ref = (#post value) turns any T into a (#Ref T).
  • value = (#load ref) turns any (#Ref T) into a T.

Miscellaneous

  • (#=> data proc) is the same as (proc data)
  • 'expr quotes the next value: it evaluates to an #AST value instead of compiling the expression.
  • #AST is the type of a quoted AST node.
  • (#compileAST ast) compiles a compile-time #AST at the call site, so it resolves names in the call site's scope.
  • (#TypeOfEval ast) returns the AnyType that (#compileAST ast) would produce.
  • (#TypeOf 123) returns the type of 123, which is B7, without evaluating the expression.
  • (#TypeOf (#post "abc")) => (#Ref String) and does not evaluate the expression. "abc" will not be posted to the store.
  • (#anyValue "a string") creates an AnyValue wrapper containing "a string"
  • (#anyType #Bit) creates an AnyType wrapper containing (#Bit)
  • (#unwrapAnyValue Type anyValue) panics if anyValue does not contain a value of Type and evaluates to an AnyValue
  • (#unwrapAnyType Type2 anyType) panics if anyType does not contain a value (it will also be a Type) of Type2

Macros

  • X = (#fnr (params) body) defines a find-and-replace macro: (X a b) substitutes the argument forms into body.
  • X = (#macro fn) defines a function macro: fn takes #AST arguments and returns #AST, and (X a b) evaluates it at compile time and compiles the AST it returns.

Namespaces

Symbols in Spore are used to identify values in a namespace. The current local namespace's symbols can always be used directly, as we have seen previously. To access symbols in other namespaces, like an imported namespace, use the . operator.

Symbol within Import

bits = (#import "abc")

(#check (bits.xor 0 1) 1)
(#check (bits.xor 1 1) 0)

Exporting Symbols

Symbols can be exported from the current namespace, for import into another namespace, by prefixing them with ..

localOnly = "abc123"
.exported = "exported"

Namespaced Types

The #Struct and #TaggedUnion types:

myStructType = (#Struct {
  .fieldA = B32
  .fieldB = String
})

myStructValue = (myStructType {
  .fieldA = 1234

  tmp1 = "foo"
  tmp2 = "bar"
  .fieldB = (#flatten [tmp1 tmp2])
})

myTaggedUnionType = (#TaggedUnion {
  .fieldA = B32
  .fieldB = String
})
myTaggedUnionValue1 = (myTaggedUnionType {
  .fieldB = "foobar"
  // also providing .fieldA is not allowed
})
myTaggedUnionValue2 = (myTaggedUnionType {
  .fieldA = 12345
  // also providing .fieldB is not allowed
})

Core Library

Spore has a core library of packages. It is importable using the core: prefix with #import.

This is not the same as Want's own standard library, accessible with the want: prefix.

The bits package provides functions for bitwise operations.

bits = (#import "core:bits")

The ns package provides helpers for building namespaces (NS, get, mkEntry), used when a Spore entrypoint is handed its context as a namespace.

ns = (#import "core:ns")

Mycelium

Mycelium is a typed data serialization format. Mycelium values are made up of fundamental values: Bits, Refs, and Ports, and all other types composed from this fundamental set.

Mycelium values are immutable; any change to a Value is necessarily a new Value.

0. Opaque

Opaque is a type that gives no information; it cannot be instantiated. It exists mostly to make the 0 value for TypeCodes an obvious error.

1. Bit

The Bit type requires no additional information; it has two Values 0 and 1.

2. Array

An Array is a finite sequence of elements, all the same type, where the length of the sequence and the type of an element are known at compile time. Arrays are parameterized by 1 type (the element type) and 1 value (the length).

3. Run

A Run is a contiguous segment of elements, all the same type, where the length is unknown. Runs cannot be used directly; they must always exist within a Ref.

4. Ref

A Ref is a reference to another value, kept in the store. Refs are all the same size, and are redeemable for a Value. A Ref can be converted to a Value by reading from the store. Refs are parameterized by 1 type and 0 values.

5. Sum

A Sum has several possible integer-identified fields. Each field has its own independent type. Only one field may be set at a time.

6. Product

A Product is a collection of fields, each of which has an independent Value of an independent type. All fields must be set. Products are parameterized by N types and 0 values.

7. List

A List is an ordered collection of elements, all the same type, with a length known at runtime. A List is represented as a Ref[Run] of known length, where the length is tracked in the list. Lists are parameterized by 1 type and 0 values.

8. Lazy

Lazy is a program which requires no additional input, producing an output of a known type. Lazys are parameterized by 1 type and 0 values.

9. Procedure (Proc)

A Procedure is a program which requires additional input, producing an output of a known type. Procs are parameterized by 2 types (input and output) and 0 values.

10. Port

A Port is a token for communication outside the Mycelium virtual machine.

11. Distinct

A Distinct type is marked with a value. This causes the type to become distinct from its base type. A Distinct type is parameterized by 1 type (the base) and 1 value (the mark).

12. Any

Any is a family of formats that can hold a wide variety of values with different types. These values must be converted to another type in order to be used.

AnyValue

AnyValue is a type capable of holding any value.

AnyType

AnyType is a type capable of holding the type of any value.

AnyProg

AnyProg is a type capable of holding any program. Programs are Runs of B32 instructions.

255. Type2

The type of a Type is a Type2. Type2s always take up 32 bits in Mycelium.