Timeline

Same mistake as before: extras was printed and then the commit ran without the add. Caught by checking the check-in rather than the worktree, which is the only check that works. closed check-in: 63ad319ef2 user: dmestas tags: ops-budget

500ms was a literal in a template and a constant nowhere. The panel coloured a p95 over it and nothing else knew, so the forge could show a route in red and tell no one.

obs.Budget is now the single definition. The panel colours against it — sent on the wire rather than hardcoded, so the page and the threshold cannot disagree — and crossing it triggers a notification.

First crossing only, on errlog's rule and for errlog's reason: the same page is slow on every request until somebody fixes it, and an alert channel that repeats itself gets muted. A muted channel is worse than none, because it reads as healthy. The decision is made under the lock, so two slow requests racing on one route cannot both report it.

This is the difference between an operator watching the forge and the forge watching itself: the ops page has to be looked at; a notification arrives. check-in: fc8ef2165b user: dmestas tags: ops-budget

The rings die with the process, and this fleet restarts constantly — a 4.24s render and a 546 load average both left no trace. obs.Sink had one implementation and it did nothing; this is the durable one.

DuckDB because the workload is columnar: append-only, time-ordered, aggregate-queried, never point-looked-up by a page. RFC 0001 defers the lake to 'when hydration and analytics — the things columnar is for — exist', and reserves the operational surface for SQLite. This is that analytics workload, not a second operational store. Wide flat tables, no keys and no indexes, which is also what a Parquet COPY TO wants when the lake arrives.

Isolated behind a build constraint, in its own package, imported only by the composition root. Both halves of the reason were measured rather than assumed: CGO_ENABLED=0 fails on go-duckdb's bindings, and GOOS=js reports 'build constraints exclude all Go files'. Leaving the import off the request path is not enough — any reachable import breaks a wasm build — so cmd/quarry has two files, one per side of the tag. Verified: GOOS=js GOARCH=wasm builds cmd/quarry clean at 76MB, without DuckDB.

Buffered, never a write per request, and flushed on a timer so a quiet forge still lands rows rather than holding them in memory for days.

Failure is not fatal. The first run failed on 'at' being reserved in DuckDB; the server started anyway, the panels worked, and the log said what was wrong. Analytics is derived data about a forge that works without it.

Costs, measured: +462MB module cache, binary 57MB -> 104MB, cold build +10s, and CGO_ENABLED=0 no longer builds the server. closed check-in: 564f272d3c user: dmestas tags: ops-history

Seven test files were written this session, reported as landed, and never committed: `zit commit` with no paths captures TRACKED files, and new files need `zit add` first. make check passed every time because they were on disk in the worktree, which proves nothing about what is in the check-in.

What was missing:

- search kinds, held in agreement across four places (the index list, two ops validators, the scope chips, hitOf). This is the guard that caught `doc` and again `file` the moment each was added; without it the validator silently refused a kind the chips offered. - technote edits collapsing to one timeline row, folded in three different orders, because a clone delivers artifacts in wire order. - wiki and technote search matching the CURRENT revision only, the "terrified" case. - Contributors(0) meaning every contributor rather than a silent 100. - the ops phase states, the stalled-store trap, and an empty fleet.

Verified by removing them and running the suite: it stays green, which is precisely the problem — trunk has been passing on their absence. closed check-in: 24fdaa723a user: dmestas tags: search-guards

Check-in 2cacce9b01aa added obs.Default.Spawned, RecordSubprocess, RecordRender and Snapshot calls across zitcli, web and api, but the package itself was never added — so trunk did not compile at all.

The files are the ones from the authoring session's tree, unmodified, tests included. Nothing here is my design; this only makes the import resolve so the tree builds again. check-in: 5229b05522 user: dmestas tags: trunk

The panels were steady-state numbers and said nothing while a fleet came up — which is the window an operator most wants to watch and the one where a forge most looks hung. A multi-repo re-ingest on this fleet was mistaken for a hang precisely because nothing said '23 of 25 stores ready' while it worked.

A phase banner leads the page: starting, indexing, syncing or rest, with stores ready over stores served, artifacts this session, the slowest recent round, and the names of what is rebuilding — which, not merely that something is.

Derived rather than tracked. Every store already reports Indexing, LastOK, LastCount, Total and Stalled; a phase field somewhere would be one more thing to keep true. The test ordering is the reader's priority: rebuilding outranks syncing, and a store that has never finished a round outranks both.

A stalled store is DONE waiting — it needs a person, not time — so it does not hold the fleet in 'starting' forever and does not sit in the bar as pending. Ready and stalled are drawn as two segments of one track, or the bar could never fill. closed check-in: 78fbef9d30 user: dmestas tags: ops-phase

quarry could not answer two questions about its own behaviour. A render time was printed in the page footer and discarded, so a 4.24s files page was visible only to whoever loaded it and read the footer. Subprocess load was visible nowhere: a warm loop once ran 134 concurrent `zit ls --long` against the same hash, load average 546 on 14 cores, and /fleet said "all healthy" throughout.

/fleet/ops, site-admin only. Slowest routes by p95 — a mean hides the tail, and the tail is the thing worth finding. Store subprocesses rolled up per verb, with the number that actually mattered: how many run at once, and the peak, which outlives the incident. Recent renders with ops time and the zit time inside it.

Instrumented at the two seams that cover everything: the ops registry, which every page's data crosses, and the outermost middleware, which is the only place the template's own cost is known — the footer's breakdown is assembled before the template runs, so a page can never report the time it spent rendering itself.

Datastar with no no-script path, deliberately: an internal dashboard that cannot move while an operator watches a fleet index is worth much less than one that can, and no consumer depends on it. The op behind it is registered and versioned like any other.

The rings are small and in memory. This is the snapshot; history belongs in the analytics store behind obs.Sink, which is an interface with a no-op default so the persistent implementation can be cgo without the request path importing it. closed check-in: 022966bc05 user: dmestas tags: fleet-ops

zit ls --long adds size, permission and text/binary, and the last of those reads every listed blob. On a large store the annotated forms cost ~18ms per file where the plain two-field listing costs ~0.09ms: measured on tcl, 0.20s plain against 38s for --size and 41s for --long. The cost scales with the STORE, not the listing (filed as zit#148).

Four callers read only .Name and threw the rest away: the About marker scan, a root check-in's file set, the doc index, and CI's file list. They use LsNames now, which runs the plain listing and caches it beside the annotated one.

The tree page and its warm still need the columns they display, so they are unchanged; that cost is the upstream issue's to fix. check-in: 78db482057 user: dmestas tags: trunk

The poll loop ran 'go cli.Ls(ctx, tip)' on EVERY tick as a bare goroutine whose result was discarded. On a store where that call takes seconds the next tick started another before the last finished, and across a fleet they compounded.

Measured: 134 concurrent 'zit ls --long', all against the SAME hash, under one server, load average 546 on a 14-core machine.

The listing is immutable per version, so it needs computing once per version. Now it skips when the tip has not moved and when a warm is already in flight. check-in: a3475428ac user: dmestas tags: trunk

warmReads and indexTree each spawn several zit subprocesses — ls, ls-under, a cat per file, an archive — and on a store tcl's size each takes seconds. They were launched as bare goroutines from every round that moved the tip, so the next round started another set before the last had finished and they stacked.

Measured on the demo fleet: 43 zit children under one server, three of them the same ls against the same hash, load average 546 on 14 cores.

A warm pass already running IS the warm pass. CompareAndSwap claims the slot and the winner clears it on the way out; a round that finds one in flight skips rather than piling on. check-in: 0325ba4a2c user: dmestas tags: trunk

Accepting an already-accepted landing wrote another state reply, and four stacked up on one record before it was noticed.

Two causes. The guard read state from the INDEX, which lags the store by an ingest round, so a second press inside that window saw 'open' and passed. And zit does not refuse the write either — an already-contained branch still gets a reply saying 'accepted', with 'nothing to merge' in the body.

So: fold first, re-read, then decide. A settle is deliberate and rare and can afford one ingest round to know what it is settling. And already-settled now succeeds writing nothing, because that is the end state the caller asked for; an error made a double press look broken when the landing was exactly as they wanted.

Proven against a running server: three accepts in a row leave one 'accepted this' reply and no 'nothing to merge'. check-in: f102c4c7ec user: dmestas tags: trunk

A border-right per item is the single-row idiom, and it breaks the moment the strip wraps — the item ending a row keeps a divider that lands against the container edge, the last item overall has none, and nothing rules between rows. Measured live at 390px the activity strip is three rows, so wrapping is the normal case.

Letting the container's colour show through a 1px gap draws every divider the layout actually needs, vertical and horizontal, with no rule needing to know where a row ends. closed check-in: 76dc2b8d96 user: dmestas tags: pager-shape

The glyph was always centred; the boxes were not. The current page is a span and took 0.5rem of padding with no border, every other page is an anchor and took 0.6rem plus a 1px border — 48px against 53.2px for the same digit.

So the highlight changed size as the pointer moved along the row, and a number sitting still inside a box that grew around it read as though it had slid off centre. Padding and border are stated once for both now.

Measured: every cell 53.2x32.7 at desktop, 37.2 wide at 390px, glyph centred in all of them. check-in: 2ae6328950 user: dmestas tags: pager-shape

The count is a statement about the list, not a control in it, so on a narrow screen it drops to its own line. The controls are trimmed to keep theirs: measured at 390px the row wanted 398px of a 358px container, and most of the excess was the number cells' 2rem min-width plus padding.

Before, whatever spilled — usually 'older' — landed centred under the numbers beside the count, because justify-content centres each wrapped line independently and it read as though it belonged to them. closed check-in: d42e59bdef user: dmestas tags: events-public

It was a flex container with no flex-wrap, so every bare text run between the elements — ' · started ', ' · last reply ' — was its own anonymous flex item. On a narrow screen the default nowrap shrank each item until its text wrapped INSIDE it, stacking the words in columns instead of wrapping the line once.

Normal flow wraps it the way prose wraps. The name with its mark, and each date, are held together with nowrap so the break lands between them rather than through them. check-in: 968a59a2b0 user: dmestas tags: events-public

Every other repository page leaves visibility to the ops layer. A gateAt here made this the one page a public repository would not show an anonymous reader, while its timeline, forum and landings all did.

Safe because of what is on it. The derived events restate artifacts the timeline already shows. The operational ones carry no identity at all — Counted is func(repo string, clone bool), with no user and no address anywhere in the path — so a clone event says how many, never who. check-in: 318949277b user: dmestas tags: events-public

The forge could not answer for minutes after starting. Register runs before ListenAndServe and probes each store with 'zit submanifest show', and the note there said it costs ~250ms — measured on small stores. On tcl it is 116 SECONDS, on sqlite 16. A fleet with either in it held the whole site down.

The probe is bounded at 2s now. Past the deadline it is simply an unanswered question, which the existing false already handled: the fossil clone URL is withheld rather than offered wrongly.

Registration is also timed and logged, because each repository's cost is the whole forge's downtime and a slow one should be nameable rather than suspected. closed check-in: 65e953863a user: dmestas tags: startup-latency

A wiki page and a technote are EDITED: every revision is its own artifact under one stable name. Keyed by artifact hash they piled up in search_doc, and Search's group-by then returned whichever revision matched BEST — so a query found text deleted years ago and linked to a page that no longer contains it. On the live fleet "terrified" matched spatialite_gui's wiki home, which has not said it since 2016; libfossil/home had 117 revisions all searchable.

One row per page now, holding the current revision, newest wins on date so a clone may deliver revisions in any order. Every revision stays readable in the page's own history, which is where the question 'what did this used to say' belongs.

Existing indexes are rebuilt from wiki_edit and technote at open rather than left to the fold: the fold only fires on artifacts that arrive after this code, so deleting the stale rows alone would empty wiki search until somebody happened to reindex. closed check-in: e262fc246c user: dmestas tags: search-current

"use after free" on tk returned three rows identical on screen — same repository, same title, same snippet — because the same fix landed on three branches and the row carried nothing that differed. The only way to find out which one you wanted was to open all three.

Rows now carry the hash, date, author and branch for a check-in; the status and date for a ticket; the date and author for wiki, forum and technote. A doc or a file is identified by its path and gets nothing, rather than a date nobody asked for.

Decorated by primary key, for the page actually shown: the FTS query groups over a match set that can be thousands of rows, so joining there would decorate all of them to render fifty. On the fleet, decorated AFTER the merge trims — every repository is asked for the full limit so the merge can pick freely, and decorating before it meant 25 indexes times 50 rows to render 50.

One definition of the line, shared by both search pages. They are separate templates, which is why the first version of this reached only the per-repo one. closed check-in: 5cc9306b59 user: dmestas tags: hit-meta

Read straight off the vocabulary stream — no table, no fold, nothing to keep in sync, so it cannot disagree with what the forge published.

The window is the stream's window, 24 hours, and the page says so: a log that let a reader assume history would be lying about a live tail.

Asks the consumer how much is pending before fetching. Without that a quiet repository waits out the fetch timeout, which would make 'nothing has happened yet' the slowest page on the site — 2s became 9ms.

Linked from the webhooks settings, not given a tab: that is where the question gets asked. closed check-in: a5c5fe771e user: dmestas tags: praise-ui

quarry rendered both as first-class surfaces (the docs tab, the file browser) and could find neither. Fossil searches documentation but not source; this does both, from one pass.

One tarball, not a read per file. cat per path costs a subprocess and a store open each: 2,199 files on sqlite is ~31s, against 2.2s for one archive — the tarball is the whole tree in one pass, which is the shape of the question.

Tip-only. Indexing every revision of every file would dwarf the rest of the index many times over to answer a question nobody asks; what a file used to say is a question its own history answers.

ref carries the version, not just the path: a link has to name the check-in the row was indexed at, and resolving a branch name at click time would be a second source of truth about which tree the row describes.

Binaries and files over 1MB are skipped and counted — a search that silently covers less than the tree beside it is the failure this change is about. closed check-in: 8df971b482 user: dmestas tags: tree-search

Reset drops every derived table, doc rows included, and the doc pass only fired on a tip change or once at boot — so a reindex left documentation silently unsearchable until somebody happened to commit, which on a mirror of a quiet project is never. Observed live: fossil's 173 documents went to zero after a rebuild and stayed there.

ReplaceDocs swaps the whole set, so a pass that found documents and could not read them would overwrite a good index with nothing. It now keeps the previous set and says so. A tree with genuinely no documents still writes an empty set, which is true. closed check-in: 24586a4771 user: dmestas tags: doc-reindex

Restores the backend the trunk merge dropped, and adds the two surfaces that get pressed.

The button sits in the stats strip beside the clone count, drawn only when there is an account behind it. Pressed state rides on aria-pressed as well as colour.

/praised is where praise is revisited and withdrawn. It exists apart from /people/{user} because that page lists CONTRIBUTORS derived from the DAG, and a praiser need never have authored anything. check-in: 0cba46d4ed user: dmestas tags: praise-ui

quarry rendered tree documents as a first-class surface — the docs tab, the reading furniture — and could not find them. A reader could open www/quickstart.wiki and get nothing searching for it. Fossil searches c d t w e f; quarry had everything but d.

Indexed at the default branch tip only, asynchronously, on the trigger warmReads already uses: a round that moved the tip, plus once at boot. Not through dagnats — dagnats is optional, a forge without one "simply has no CI rather than a broken one", and search must not inherit that. This is local derivation from the store, the same class of work as the fold, which Reindex already routes around the bus on purpose.

isDocPath is exported so the indexer and the docs listing share one definition of what a document is; two would drift and the drift would be invisible.

Adding the kind exposed how many places the list is written down: the index, two ops validators, the scope chips, hitOf. The validator was missed and refused every doc search before it reached the query. Two tests now hold the four in agreement. closed check-in: 57a2f8e3d2 user: dmestas tags: doc-search

They fell through hitOf's switch with no URL, so the title rendered as — which a browser resolves to the current page. Clicking a forum result reloaded the search. The index found them, the facet chips offered them, and the one step that turns a hit into somewhere to go was missing.

Their refs are not artifact hashes: searchDoc files a forum post under its thread root and a technote under its event id, which is what a reader wants to land on.

The test walks index.SearchKinds rather than a hand-written list, so a kind added to the index without a link here fails instead of shipping a dead result. closed check-in: 69f5a86ccd user: dmestas tags: search-links

The capabilities table sized to an <input size=28> it could not shrink, so the whole page was wider than the viewport and every field ran off the right edge. It scrolls in its own box now.

Webhooks were a three-column table with an input and three buttons in one cell — the same bug, added by me. Rows stack instead.

Help text cut to one line; the noun chips and the field's datalist say the rest. closed check-in: 0a60aac30c user: dmestas tags: settings-mobile

A clone reads and writes nothing, so it is the one kind of forge activity that leaves no trace in the DAG at all.

The sync edge classifies the protocol's cards: clone SEQNO 1 opens a new clone, later rounds do not, so a big store cloned in forty rounds still counts once. Two sinks: the vocabulary event for anything reacting, and a JetStream counter for how many, which converges across nodes instead of losing increments to a read-modify-write. check-in: dc058fa551 user: dmestas tags: clone-counts

Four commits: the fan-out onto the vocabulary, per-subscription event selection, the settings surface, and the RFC correction that preceded them. check-in: 69fc3f06d1 user: dmestas tags: trunk

Subscriptions were CLI-only. The settings page now lists them, takes an endpoint plus its event patterns, pauses, resumes and deletes — gated on 'a' like everything else there. Patterns are checked before saving.

The section is gated on the store being attached, not on the list being non-empty, or the first subscription could never be created. closed check-in: 1c5d758256 user: dmestas tags: hooks-vocabulary

A webhook took everything or nothing. It now names vocabulary patterns: '*', a bare noun for every action on it, or an exact kind. Checked at the moment it is set, because a webhook that silently delivers nothing reads as a broken forge. check-in: e9547dcc49 user: dmestas tags: hooks-vocabulary

A ticket closed by push notified nobody, because only ops queued webhooks. One consumer on quarry.evt.> turns every raised event into deliveries, so arrival path stops mattering.

Deletes kindOf: the delivery's kind was sliced off an idempotency key. check-in: f2ea5f5c1a user: dmestas tags: hooks-vocabulary