Skip to content
quarry Documentation
Login

Documentation

Summary

Setting up CI for pikchr-studio, the second repository on zeitforge to get it, took two JSON files to say one thing, timeouts in nanoseconds, a retry block whose zero means "run once", a worker group typed by hand, task names that had to start with the repository's name (learned from a 403), and a dedicated runner with a production token in a root-only file on the box, guarded by a flag written that afternoon. Every one of those is a fact about the engine that the author had to learn.

Codeberg's Woodpecker, GitHub Actions, GitLab and sourcehut all converge on the same shape, and this RFC adopts it:

Motivation

The deploy question showed the gap. "Deploy pikchr-studio after a landing" needed a new pipeline key (quarry_when: default-branch), a new runner flag (--branch trunk), a new system user, and a second worker token, and it still only works for trunk. The user asked the right question: why only trunk, and not wherever the user says?

The answer every other forge gives has two halves, and quarry had only been building one:

  1. The file says where a step wants to run. It is read at the check-in under test, so a branch can rewrite it. It is a convenience.
  2. Settings say where a secret may go. A branch cannot edit settings. This is the safeguard.

GitHub puts a deploy's branch policy on the environment, in settings, not in the workflow; GitLab exposes protected variables to protected branches only; Woodpecker limits each secret by event and keeps it from pull requests by default. quarry already has the right primitive for the second half: a protected branch (RFC 0007) is one a check-in reaches only through a landing somebody accepted. "Secrets reach protected branches only" is therefore "secrets reach reviewed code only", configured where the owner already configures who may land.

The file

ci/<name>.yml, one workflow per file, as RFC 0009 already named it. The vocabulary is Woodpecker's where Woodpecker has a word, because Codeberg users will copy from it and the words are plain: steps, commands, depends_on, when, environment, from_secret. Nothing promises that a .woodpecker.yaml runs unchanged; see What this is not.

pikchr-studio's two workflows, as they would be written:

# ci/test.yml: every push and every landing
steps:
  - name: test
    commands:
      - npm test
    environment:
      PIKCHR_STUDIO_DIAG: "1"
    timeout: 30m
  - name: worker
    commands:
      - npx --yes [email protected] deploy --dry-run --outdir dist-dry-run
    depends_on: []

# ci/deploy.yml: publish after a landing
when:
  event: push
  branch: trunk
steps:
  - name: test
    commands: [npm test]
  - name: deploy
    depends_on: [test]
    commands:
      - npx --yes [email protected] deploy
    environment:
      CLOUDFLARE_API_TOKEN:
        from_secret: cloudflare

What the author no longer writes, and where it went:

Today Now
ci/runner.json naming each task's argv the step's commands, run as one set -e script
task type pikchr-studio.test derived: quarry names the task from the repository and step
"worker_group": "pikchr-studio" derived: always the repository's own group (RFC 0009 already refuses any other)
"timeout": 1800000000000 timeout: 30m; a default applies when absent
"retry": {"max_attempts": 0} the default is once; retries: 2 when wanted
"quarry_when": "landing" / "default-branch" when:
"_comment" YAML comments

Steps run in the order written unless one has depends_on, which turns the workflow into a graph (Woodpecker's rule, so the common case needs no graph at all). A workflow may depends_on another workflow by file name.

when:

Allowed at the top of a file (the whole workflow) and on a step. Every key is a filter; all given keys must match; a list means any of.

Key Matches Notes
event push (a check-in on a branch), landing (a landing's tested merge, RFC 0009 §premerge), manual (Re-run, ci.trigger), cron absent means push and landing, today's behaviour
branch the check-in's branch, or for landing the target branch; globs, include/exclude same doublestar matching Woodpecker uses
path a changed file; globs, include/exclude for landing, every file the landing changes
cron a schedule's name, when event: cron schedules come from dagnats' existing on.schedule

Deliberately absent: Woodpecker's evaluate (an expression language) and GitHub's if:. RFC 0009 rule 1 stands: filters decide where, tasks decide logic. A workflow that needs a real condition runs a step that exits early.

quarry_when in a JSON pipeline keeps working and is documented as the engine-format spelling of when: {event: landing} and of when: {event: push, branch: <default>}.

Errors

Unknown keys are errors with a line and a column; so are a depends_on naming no step, a cycle, a from_secret naming no secret the repository has, and a when.event quarry does not emit. They show on the CI page where the runs would be and on the landing's checks panel, per RFC 0009 rule 2. A misspelt key never silently runs everywhere.

Secrets

A Secrets section in repository settings, gated by a (RFC 0008), holds named values. Each secret has a policy:

Values are write-only in the UI: set, replace, delete, never shown. The settings trail (as for every setting) records who changed which secret and when, never the value.

Delivery never goes through the engine. dagnats persists a run's input, so a secret in it would sit in NATS storage. Instead the runner, when it claims a task whose step asks for secrets, asks the forge's bridge for them with its worker token. The forge answers only if the task's check-in qualifies, decided the way --branch decides today: it reads the check-in's branch from its own index, not from the task input, and refuses a pair outright. The runner puts the values in the step's environment and masks them in the log it streams (every occurrence of a value is replaced before it leaves the machine).

This retires the deploy runner. A runner holds a worker token and nothing else; quarry runner --branch stays as an extra lock for owners who want a machine that runs one branch, but it is no longer how a secret is protected.

Where each piece lives

RFC 0009's three layers, unchanged in shape:

Layer Does
dagnats CI module parses and validates the YAML (steps, commands, depends_on, timeouts, retries, environment, from_secret names), compiles it to a WorkflowDef, and returns each workflow's and step's when as structured data it does not interpret
quarry trigger evaluates when against quarry's events (it is the only one that knows what a landing or a protected branch is), launches the matching workflows with the matching steps
quarry runner runs a step's commands from the task input; fetches and masks secrets; still honours ci/runner.json for JSON pipelines
quarry forge the Secrets page, storage, the bridge endpoint that decides delivery

when stays out of dagnats on purpose: its GitHub front-end has its own events, and a forge-agnostic compiler that knew "landing" would not be.

Build order

Rung Work Waits on
1 This RFC; the three open questions answered the owner
2 dagnats: spec v2 in the CI module: steps with commands, depends_on, duration strings, environment with from_secret (names only), when passed through as data; line/column errors; the current checks: spec keeps compiling a dagnats issue (to file)
3 quarry runner: a shell task type that runs commands from the task input; masks a list of values in its log sink nothing
4 quarry trigger: read ci/*.yml through the compiler, evaluate when per workflow and per step, map quarry_when onto it; errors on the CI page and checks panel rung 2
5 Secrets: settings section, storage, trail, the bridge endpoint with the delivery rule, runner fetch and masking rung 3
6 Starter files on the empty CI page (test; test plus deploy); docs/running.md and a how-to rung 4
7 Move quarry's own CI and pikchr-studio to YAML; pikchr-studio's deploy uses a cloudflare secret; stop the deploy runner and delete its token rungs 4, 5

Rungs 2 and 3 are independent and can go in parallel; rung 5 can start beside rung 4. The interim trunk-only deploy (quarry_when: default-branch, --branch trunk) is what pikchr-studio uses until rung 7.

Rung 5's storage

Secrets are encrypted at rest with a key the forge reads from its environment (not from the database it encrypts), so a copied database or backup does not carry usable secrets. A backup restored without the key restores the names and the policies, and every value must be set again; the Secrets section says so when it happens.

What this is not

Open questions

  1. Format: YAML or TOML. Recommended: YAML. No mainstream CI uses TOML; the four above all use YAML, people copy CI files between projects, and the nested when reads worse in TOML. YAML's traps (no read as false, anchors) are answered by strict parsing, which the dagnats compiler already does.
  2. Vocabulary: Woodpecker's (steps, commands, when), GitHub's (on, jobs, needs, run), or dagnats' current (checks, deploy). Recommended: Woodpecker's. It is the smallest, Codeberg's users know it, and it has one word for "where" (when) instead of GitHub's two (on and if).
  3. Where the files live: ci/ or a dot-directory (.quarry/, as .woodpecker/ and .github/workflows/). Recommended: stay in ci/. RFC 0009 and every repository that has CI today already use it, and zit has no reason to hide the directory.

Testing