Tasks & Jobs

Tasks

A Task is a well-defined unit of work. Well-defined means that all the information that could be required to complete the Task is explicitly contained in the Task.

A Task has 2 parts: an operation name and an input (an AnyValue):

Task = (#Struct {
  .op_name = String
  .input   = AnyValue
})

The operation names one of a small number of built-in operators, for example:

  • want.pick
  • want.place
  • download.tree
  • linux.cspawn
  • wasm.run
  • mvm.apply

The input is the operator's argument value. Both parts are part of the hash which becomes the Task's identity, so a Task with the same operation and input is the same piece of work.

Want caches the output of Tasks when they complete, and checks this cache before computing a Task. An immediate rerun of want build without any changes therefore produces all the same Tasks, and every one of them can be skipped.

Jobs

Jobs are used to track running computations in Want.

Jobs are created to compute Tasks. A Job carries active state, like whether it has completed and when it started and finished: it is spawned when created, queued when admitted, running once a worker picks it up, and done (with an ok value or an error) once it has an outcome.

Jobs are organized into a hierarchical tree. All the major operations in Want create a new root Job; that Job may spawn additional child Jobs.

Child Jobs give a computation access to additional compute, since children run concurrently, and they are each an opportunity for caching.

The Job System Port

A build computation is not given access to the host; instead it is handed a jobs port, a capability it uses to spawn and await child Jobs. In Spore this port has the type build.JobCtx, and a Task's do proc receives it on ctx.jobs.

Through the port a computation can:

  • spawn jc limits op input: start a child Job for a Task and get its JobID immediately, under a resource ceiling. A Job that runs past its max_duration ceiling is cancelled automatically.
  • limits jc: the current Job's resource ceiling. Spawning with it passes the ceiling down unchanged.
  • await jc id: block until that child finishes and return its result; a child that failed faults the parent.
  • inspect jc id: check a child's state without blocking.
  • awaitAny jc ids: block until any one of a set of children finishes.

A Job cannot spawn root or sibling Jobs; it can only spawn its immediate children. This keeps runaway computations from escaping their parent, and gives every Job an audit trail back to the user who asked for the result.