Skip to content
quarry Artifact [0621e4b0a6]
Login

Artifact [0621e4b0a6]

Check-ins Using Download Text

Artifact 0621e4b0a654fe03fb6b012fff9cdd04fec773e3f2be40c57fb75506228829a2:

Summary

quarry names the things that happen in a repository, once, in a vocabulary that pipelines, webhooks and notifications all subscribe to. An event is a noun and an action — ticket.closed, landing.opened, branch.created — carried on the bus that already exists, with a payload shape declared per noun.

A pipeline runs because something happened, because the clock said so, or because a person asked. dagnats already implements all three — cron, NATS-subject, webhook and durable-HTTP triggers, with debounce — so quarry builds no scheduler and no subscription plumbing. What quarry owes is well-named subjects (§2.3), and the vocabulary is the subject hierarchy. Creating a trigger is still a forge act, gated on fossil's a capability so a repository's own admin owns its automation (§2.4).

Today there are two and a half paths and no vocabulary. Webhooks derive an event's kind by slicing a notification's idempotency string. CI does not use events at all: it polls the index for new check-ins behind a per-repository watermark. Notifications have their own workflow names. Adding one more trigger — a landing opening, say — means either a third path or generalising anyway, so this RFC generalises first.

The vocabulary is the load-bearing part, and the reason this is an RFC rather than a commit: a pipeline definition names events in its on: clause, so an event name is a published interface. Renaming one breaks every pipeline in every repository that used it, in a system where the pipeline definition is itself a versioned artifact in someone else's store.

Motivation

What exists

ci.Trigger runs after every fold. It asks the index for check-ins newer than a watermark, reads ci/*.json at each one, registers the definition at that check-in, and starts a run. pollCI later polls unfinished runs and, on a terminal status, mints a ci-pass / ci-fail tag on the check-in.

That is a complete, working pipeline trigger for exactly one event, with the event's identity left implicit. Four things in it are check-in-shaped:

today needed
trigger poll NewCheckins(watermark) subscribe to a named event
payload {repo, checkin, branch, date, user, comment, cloneUrl, pipeline} per-noun, common envelope
watermark one check-in per repository per subscription
verdict sink WriteTag(verdict, checkin, runID) per-noun; a landing wants a reply

Every one of those blocks the landing trigger (quarry ticket 49323c33), and none of them is made easier by doing the landing trigger first.

What the field does

Four vocabularies were read before proposing a fifth.

Fossil hooks (www/hooks.md) are artifact-level and carry no semantics at all. Three types — after-receive, before-commit, commit-msg — and the after-receive hook's stdin is a list of artifact hashes with human-readable descriptions. There is no "on branch created" or "on merge". The doc states its purpose plainly: to "start up a background testing or CI process".

Fossil alerts (src/alerts.c) carry fossil's only semantic vocabulary, as subscription letters on the subscriber.ssub column:

a Announcements          c Check-ins            f Forum posts
n New forum threads      r Replies to my posts  t Ticket changes
u User permission changes  w Wiki changes       x Edits to forum posts

Eight types. It splits the forum four ways by verb, and has no event for a branch, a tag, a technote or a merge.

GitHub publishes 75 webhook events, most of which are not about version control: roughly six VCS core, fifteen collaboration, eight CI, ten security scanning, fifteen governance, and twenty platform and commerce events (marketplace, sponsorships, apps, Pages, Projects). quarry has a concept for about twenty of them. Gitea publishes about 27, GitLab 15.

The shape decision

The three forges disagree about the axis, and the disagreement is the design question:

quarry takes GitHub's. Under Gitea's rule every new verb is a new permanent event name, and the name is a published interface. Under GitHub's, a subscriber can ask for ticket and get every change, or ticket.closed and get one, and adding a verb later costs nothing. Fossil's split shows the cost of the other choice inside one product: f and n and x are three names for three verbs on one noun, and a subscriber who wants "any forum activity" must name all three and remember to add the fourth when it appears.

Guide-level explanation

An event is noun.action, published on a subject that carries the repository:

zit.evt.<repo>.<noun>.<action>

A workflow subscribes with a dagnats subject trigger, so the selecting is NATS' and quarry adds no syntax of its own:

zit.evt.quarry.landing.opened     one repository, one action
zit.evt.*.landing.opened          that action anywhere
zit.evt.quarry.ticket.*           every ticket action in one repository
zit.evt.quarry.>                  everything in one repository

A pipeline with no trigger of its own still runs on checkin.created, which is today's behaviour, so every pipeline already in a repository keeps working without being edited.

Every event carries the same envelope, and a payload keyed by its noun:

{
  "event":  "landing.opened",
  "repo":   "quarry",
  "at":     "2026-08-24T09:48:57.123",
  "actor":  "dmestas",
  "id":     "activity:landing:9802c157…",
  "payload": { "landing": "9802c157…", "branch": "wiki-struct", "target": "trunk" }
}

id is the idempotency key. It is stable for one occurrence of one event, and it is how a consumer that sees an event twice can tell that it has.

Reference-level explanation

§1 The vocabulary

Nouns and their actions. An action marked † is not yet raised by any code path and is reserved so that adding it later is not a vocabulary change.

noun actions payload
checkin created hash, branch, parents, user, comment
branch created, moved, closed† name, tip, previous tip
tag created, removed† name, target, value
ticket opened, changed, closed, reopened id, title, fields changed
wiki edited, created, deleted title, hash
technote created, edited event id, hash
forum posted, replied, edited†, moderated thread root, post hash
landing opened, commented, accepted, rejected id, branch, target
ci started, completed run id, checkin or landing, verdict
mirror pulled, failed url, artifacts received
repo created, removed, settings name
clone cloned, fetched actor or anonymous, artifacts sent
account added, caps, removed account, repository, caps

Fourteen nouns, thirty-odd actions. For comparison: fossil's alerts name eight things, GitHub's 75 events name about twenty that quarry has a concept for.

Clone counts. clone.* is observable because every sync request goes through one place: Fronter proxies POST /r/{repo}/xfer and already runs a ReadGate per request, so the choke point exists. Counting fetches is free at that point. Distinguishing a clone — a first fetch by a peer holding nothing — from an incremental pull requires reading the request body, since that distinction lives in the sync protocol's announcement and nowhere else. Both counts are operational per §2.1 and so are recorded in OLTP as they happen; the bus carries notice of a row that already exists.

GitHub reports clones and unique cloners separately. Uniqueness needs an identity to count by, and a sync request may be anonymous, so quarry counts unique by account where there is one and reports anonymous clones as a separate figure rather than guessing at a person behind an address.

Three of these exist in no other forge, and are here because quarry has the concept and an operator needs the hook: technote (fossil has the artifact but no alert for it), mirror (a forge that tracks 25 upstreams needs to fire on a failed pull), and landing.

§2 What the bus is, verified

Read from internal/events/events.go rather than assumed, because §2's promises depend on it:

desired := &nats.StreamConfig{
    Retention: nats.LimitsPolicy,
    Discard:   nats.DiscardOld,
    MaxBytes:  1 << 30,   // 1 GiB ceiling across all repos
    MaxAge:    time.Hour, // live window; older needs a reindex
}

The stream is a one-hour live window, not a log of record. It is a transport between the store and the index, and the durable truth is the pair either side of it: the store (authority) and the publisher cursor in the KV bucket, which advances only after a fold succeeds. A consumer that falls outside the window does not catch up by replaying; it rebuilds from the store.

Two consequences this RFC must respect rather than wish away:

  1. A consumer down for an hour has missed events permanently. CI's current watermark-and-poll survives that, because it reads the index, which is durable. A naive move to bus subscriptions would trade a robust trigger for a fragile one — §2.1 is how that is avoided.
  2. Per-node durable consumers each read the whole stream (limits retention was chosen over work-queue for exactly this: every node must fold every event). So three consumers subscribing independently is the shape the bus was already built for.

§2.0 A gap is detectable, but only if you look

The documentation does not say what a consumer sees when MaxAge drops a message it has not read, so it was measured against this forge's own embedded NATS (internal/events/gap_test.go). Three messages aged out unread:

stream   first_seq=7  last_seq=7  msgs=1
consumer ack_floor=3  delivered=3 pending=1

Two facts, and the design turns on the second:

  1. The gap is visible: first_seq - ack_floor - 1 = 3. A consumer can know exactly how many messages it lost.
  2. Nothing tells it. The next Fetch succeeds and returns the message after the gap. pending counts only what survives. A consumer that does not compare sequences carries on with a hole in its view and no error.

So the window is not silent data loss if every consumer checks. That turns "a consumer down for an hour has missed events" into something recoverable:

Every consumer MUST compare the stream's first_seq against its own ack_floor on each fetch. On a gap, a consumer of derived events rebuilds from the index (§2.1); a consumer of operational events reads the OLTP rows it missed. Neither is expensive, and the alternative is a forge that is quietly wrong after any hour of downtime.

§2.1 Two classes of event, and why the distinction is load-bearing

The one-hour window is survivable for some events and not others, and the line is whether the event is derivable from the DAG:

Derived events — checkin, branch, tag, ticket, wiki, technote, forum, landing. Every one is a reading of artifacts the store holds. A consumer that missed one can re-derive it by walking the index, exactly as the ingest pipeline rebuilds after a gap. A missed event costs latency, not truth.

Operational events — ci, mirror, repo, account, clone. None of these is in the DAG. mirror.failed is a fact about a network attempt; account.caps is OLTP state; a clone is a request that happened. If the delivery is missed, the fact is gone.

So operational events MUST be recorded in OLTP as they are raised, and the bus carries a notification of a record that already exists. Derived events need no such backing — the store is their backing.

This is not symmetry for its own sake: it is the difference between a mirror.failed an operator can page on and one that silently did not happen.

§2.2 Delivery semantics, which the medium decides

Fossil's hook documentation states the constraint, and quarry inherits it rather than choosing it:

A push might not deliver all of the artifacts for a checkin. If Fossil knows that a /xfer HTTP request is incomplete, it will defer running the after-receive push for 60 seconds, or until a complete /xfer request is received. This helps to prevent after-receive hooks from running when incomplete checkins exist in the repository, but it does not provide hard guarantees, as there is no way to do that in a distributed system.

So:

  1. At-least-once. A consumer may see an event twice. id is the idempotency key and a consumer that acts on an event MUST be idempotent under it. quarry's own notification plane already works this way.
  2. Possibly batched. Fossil warns that two pushes' artifacts "might be combined into a single after-receive callback". quarry's fold has the same property; a single fold can produce many events, and a consumer must not assume one event per delivery.
  3. Possibly partial. An event may describe state that is still arriving. A consumer that needs the whole picture reads the index at handling time rather than trusting the payload to be complete.
  4. Ordered within a repository, not across. The bus preserves order per repository because the fold does; nothing orders two repositories against each other, and nothing should be written that needs it.

§2.3 Three trigger sources, of which events are one

A pipeline runs because something happened, because the clock said so, or because a person asked. Only the first is an event, and conflating the three is how "on:" ends up meaning two different things.

source on: what it carries
event checkin.created, landing.opened the event envelope of §1
schedule schedule: "0 * * * *" the fire time and the ref it runs against
dispatch dispatch: true the actor and any inputs they supplied

This is GitHub's split (schedule, workflow_dispatch) and it is the right one: a cron run has no actor and no artifact, and a dispatched run has an actor but nothing happened. Forcing either into the event envelope would mean a payload full of fields that are empty by construction, which is how a schema stops meaning anything.

Schedules and dispatches are OLTP state, not artifacts. A schedule is a forge configuration — which repository, which pipeline, which cron expression, enabled or not — and it is exactly the shape of the existing webhook table. Both are per-repository, and both survive a reindex because neither is derived from the DAG.

dagnats already has all four sources, and more. Read from internal/trigger/types.go rather than probed over HTTP — the probe was the wrong lens, because triggers are NATS- and KV-native and CLI-created, so nothing about them answers on a URL:

type TriggerDef struct {
    Cron     *CronConfig            // {Expression}
    Subject  *SubjectConfig         // {Subject} — a NATS subject, wildcards included
    Webhook  *WebhookConfig         // {Path, Secret} — POST /hooks/{path}
    HTTP     *HTTPConfig            // durable endpoints, /api/{path}, with a respond step
    External *ExternalTriggerConfig // trigger kinds contributed by workers
    Debounce *DebounceConfig        // {Period, Timeout, Key}
}

and its envelope's own comment states the property §2.3 was about to ask for: "the standard workflow input produced by all trigger types. Workflows always know how they were triggered."

So quarry builds no scheduler, no dispatch plumbing, and no subscription mechanism. What quarry owes the engine is well-named subjects, and the vocabulary of §1 is exactly that.

§2.3.1 The subject namespace is the interface

Events publish to:

zit.evt.<repo>.<noun>.<action>

and NATS wildcards do the selecting, so the on: clause this RFC first proposed is unnecessary — a workflow's own triggers array already says it:

subscription subject filter
every landing opened, anywhere zit.evt.*.landing.opened
everything in one repository zit.evt.quarry.>
every ticket action zit.evt.*.ticket.*
one repository's CI zit.evt.quarry.ci.*

This is why noun-and-action was the right shape for a second reason the motivation did not know: the vocabulary is a subject hierarchy, and Gitea's issue_assign naming would have flattened it into strings a wildcard cannot take apart.

The practical consequence for §1: the noun and action MUST be single tokens with no dots, because a dot is the subject separator. Every name in the vocabulary already obeys this; it is now a rule rather than an accident.

§2.3.2 Debounce, and one resolved question

DebounceConfig{Period, Timeout, Key} delays a run until events stop arriving, resetting on each new one, with a hard upper bound.

That answers the RFC's own first unresolved question — whether branch.moved fires per fold or per check-in — and answers it better than either option: fire per check-in and let the subscriber debounce. A ten-commit push then runs one build for a pipeline that debounces and ten for one that does not, and the choice belongs to the pipeline rather than to quarry. Coalescing at the source would have taken it away from both.

§2.3.3 One bus, or two

For a dagnats subject trigger to see a quarry event, both must be on the same NATS. Today a quarry deployment runs two: its own embedded server for ingest, and dagnats' for workflow execution.

The seam already exists — events.Connect(url) joins an external cluster and is what the multi-node flag uses, with events.Start(dir) embedding one otherwise. Pointing quarry at dagnats' NATS makes the two into one bus and the subject triggers work with no bridge.

Two constraints on doing it, both real:

§2.4 Who may schedule and who may dispatch

dagnats stores the trigger; quarry decides who may ask for one. Creating a schedule or dispatching a run is repository configuration and spends the engine's time, so both follow RFC 0004's rule: they take a — fossil's Admin capability, the same letter that already gates repo.update, repo.reindex, ticket.config(write) and the mirror verbs.

That the record lives in dagnats' KV rather than quarry's OLTP does not move the gate: the forge is what knows who holds a on a repository, and an engine that trusted its callers would let any of them schedule work on anyone's.

That answers "repo admins should have these" in fossil's own vocabulary rather than by inventing a role: an account holding a on a repository can create, edit, disable and delete its schedules, and can dispatch a run. An account without it can see that a schedule exists — a pipeline that runs nightly is not a secret from the people reading the repository — and cannot change it.

Deliberately NOT a new capability letter. RFC 0004 §6's rule is that a letter which exists in fossil carries its fossil meaning and a surface which exists only in quarry names an existing letter rather than minting one; a forge that grows private letters stops being a fossil forge by increments.

One consequence worth stating: a excludes s, so a repository admin may schedule work on their own repository and not on anyone else's, and a fleet-wide schedule is a site-admin action. That falls out of the cap model rather than needing its own rule.

§3 Handlers do not run inside the fold

Fossil again:

Fossil holds a write transaction on the repository while the after-receive hook is running.

A hook that writes to the repository it was fired by therefore deadlocks. The same is true here for a different reason: quarry mints artifacts through the lease holder, and the fold is not it.

An event handler that mints an artifact runs outside the fold, through the lease holder, asynchronously. The CI verdict already obeys this by accident of structure — pollCI runs on a timer, not in the ingest path. This RFC makes it a rule rather than an accident, because the landing verdict (700b22b6) would otherwise be written from exactly the wrong place.

§4 One stream, three consumers

consumer today after
webhooks Raise() from fireNotify, kind sliced out of a TxnID subscribes by name
CI polls NewCheckins behind a watermark subscribes by name
notifications own workflow names subscribes by name

The watermark does not disappear: it becomes per-subscription rather than per-repository, because two consumers may be at different positions and one falling behind must not advance the other.

§5 Verdict sinks

A CI run's verdict is an artifact, because 0001 §6's no-hostage rule says the result must clone with the repository. Which artifact depends on the noun that triggered it:

The rule generalises: the verdict takes the shape of the thing it judges.

§6 What is deliberately absent

Named so the absence reads as a decision rather than an oversight:

The last one is the one most likely to be asked for later. It should be its own RFC, and it should not reuse these names.

Drawbacks

A published vocabulary is hard to change. That is the point, and also the cost: a name chosen badly here is one every pipeline in every store repeats. The mitigation is the reserved actions in §1 and the noun-plus-action shape, which absorbs new verbs without new names.

One stream is a single point of failure where today CI's polling is independently robust: if the bus stops, CI stops. The watermark survives, so nothing is lost, but a stalled bus becomes a stalled forge in a way a stalled poll loop is not. §2's at-least-once rule and per-subscription watermarks are what make recovery a replay rather than a repair.

Schedules and dispatches put quarry in the business of running things nobody asked for at that moment. A cron pipeline consumes engine time on a repository whose owner may have forgotten it exists, and unlike an event-driven run there is no push to point at. The a requirement in §2.4 bounds who can create one; nothing here bounds how many, and a per-repository ceiling is probably wanted before this ships.

Fourteen nouns is more than fossil has and fewer than GitHub. Both comparisons will be made, and neither is the argument: the set is what quarry can honestly raise today plus what it will raise next.

Rationale and alternatives

Do the landing trigger without a vocabulary. Rejected: all four generalisations in the motivation table are needed for it anyway, and the result is a third trigger path. Ticket 49323c33 says so in its own words — "rides the existing event plane … rather than growing a second trigger path".

Adopt fossil's after-receive shape: hand the handler a list of artifacts. Rejected. It is honest and it is what fossil does, but it pushes the whole interpretation problem to every consumer: each pipeline would re-derive "is this a new branch" from artifact hashes, and they would derive it differently. quarry already folds artifacts into meaning; the event stream should carry the meaning it already has.

Adopt Gitea's noun_verb naming. Rejected in §Motivation: every new verb becomes a permanent name.

Use GitHub's names verbatim where they overlap. Tempting for recognisability, and rejected in two places: issues is called ticket here because that is what fossil calls it and fossil compatibility is the north star, and pull_request is called landing because it is not one — nothing is transported and nobody requests a pull (zit#127).

Prior art

Unresolved questions

  1. ~~Does branch.moved fire per check-in or per fold?~~ Resolved in §2.3.2: per check-in, because dagnats' debounce lets each pipeline decide whether to coalesce. Choosing at the source would take the choice from both.
  2. Should ci.completed be subscribable? It would let a pipeline trigger on another pipeline, which is either composition or a foot-gun. GitHub allows it (workflow_run) and guards against recursion. Deferred until something needs it.
  3. How does a subscription express "my repository only" versus fleet-wide? The webhook plane is per-repository today; CI is per-repository; a fleet-wide subscriber (an operator watching mirror.failed across 25 stores) has no way to say so.
  4. ~~JetStream delivery guarantees have not been traced.~~ Resolved and folded into §2: the stream is a one-hour window with limits retention and per-node durable consumers, and the durable truth is the store plus the publisher cursor. The consequence — that operational events need OLTP backing and derived ones do not — is §2.1.

Future possibilities