Timeline

The live panels read in-memory rings, so a forge whose store failed to open looked exactly like a healthy one. Sink.Err() existed and nothing asked it. The page now distinguishes three states: off, broken, and recording to a named path.

Retention was 'append for ever', which is a default nobody was asked about. --analytics-keep sets the window, 14d by default, 0 keeps everything. Pruned at startup and daily. check-in: e2b74749b4 user: dmestas tags: fleet-census

Marks were stored second-granular like index post dates, so opening a thread in the same second its newest post landed left the thread the reader was looking at unread. Marks now carry a fractional part, which makes the two orderable: a post from the view's own second was in the page and reads as seen; any later second re-brightens.

Adds a harness test that walks the whole loop, including the part worth pinning — forum.get does not mark read, only forum.read does. check-in: c7bf7c08f0 user: dmestas tags: trunk

The fleet listing called Counts, the per-section census the repository nav needs: eight aggregates per store, of which a fleet row displays two. A fold invalidates the memo, so during indexing every row went cold at once.

Cold, on tcl (668MB, 42k check-ins): 230ms of census per repository against 80ms for the two numbers. Contributors now reads the maintained contributor table rather than a DISTINCT scan over every event — verified identical across all 25 demo indexes. check-in: c5d8786875 user: dmestas tags: fleet-census

Dimming keyed on a 30-day window meant a quiet forum rendered almost entirely grey. Read-state replaces it: forum_read stores when an account last viewed a thread, and a thread is read only while its newest post is no newer than that view — so a reply re-brightens it with no flag to clear. Anonymous readers have no read-state and see every row at full ink.

Marking read is its own op, not a side effect of forum.get, so crawlers and API clients cannot spend a reader's unread state. Titles now take the accent colour on hover (gated on hover: hover) and on :active, so touch keeps the affordance. check-in: c8a93fda64 user: dmestas tags: trunk

dev.local.mk said nothing about what it held, so the only way to learn what `make dev` needed was to read someone's copy — and there was no copy to read, because it is not committed.

.env.local now holds one named key per setting instead of a single 1,697-character DEV_ARGS line, and the Makefile composes the flags. A key left empty is omitted from the command line rather than passed as an empty string, which the server would read as a deliberate value. Corpus repositories are named once each and their paths derived, rather than --repo name=path repeated twenty-three times.

.env.example is committed and documents every key, including which two are secrets and which of --data and --oltp survives a rebuild.

dev.local.mk is still included, so an existing checkout keeps working, and both names are ignored. Verified equivalent: the old file and the new one produce the same 8 flags and the same 23 repositories.

`make dev-args` prints the composed flags without starting anything. check-in: 6be29eb1b4 user: dmestas tags: trunk

The check-ins default lived only in the page handler, so a reader on the default timeline had an empty query string — and the live stream opened with it asked for no filter at all. The first tick after a landing replaced their check-ins with everything.

Both the page and the stream now resolve through timelineQuery. closed check-in: 550f9601a2 user: dmestas tags: tl-filter

Three files written in the same minute on 15 Aug and committed by accident. g1.json is a wiki API response for a page titled 'evil/traversal', g2.html the rendered edit page for it, g3.json a failed call that returned 'no such endpoint'. curl -o output, named g1 through g3, left in the repository root where they read as though they belong to the project.

Nothing references them: no Go file, no Makefile, no Dockerfile. They shipped with every clone and told a reader nothing except that someone was probing path traversal in wiki page names that evening. check-in: 2946cb4429 user: dmestas tags: trunk

Repo was read with req.PathValue in the outermost middleware, where the router has not set it — path values live on the request the mux passes DOWN, a copy that middleware never sees. Same shape as the route-pattern bug, and it went unnoticed longer because an empty string looks like 'no repository' rather than like a failure.

Every render recorded until now carried an empty repo: 61 of 61 rows in the analytics store, on a fleet where repository size is the dominant cost of almost everything. The drilldown's by-repository table showed a single '(none)' row, which is what surfaced it.

Read from the path instead, unescaping as the router would. closed check-in: 059fe0bdde user: dmestas tags: ops-drill

/fleet/ops linked OUT to /fleet seven times and nothing anywhere linked IN. The page was live, gated, and reachable only by typing the URL.

Second surface to ship that way after the notes feed, so it gets a test rather than a promise to remember — both directions, so an admin is never stranded on either.

The link goes in the summary strip that already carries the fleet's counts and the error badge, which is where somebody reading health already is. /fleet answers 'are my repositories healthy'; /fleet/ops answers 'is the forge healthy'. check-in: 3314daeff6 user: dmestas tags: ops-drill

The panel finds a slow route; this says why. Two things the roll-up cannot show.

WHICH REPOSITORY. A route pattern folds pikchr and tcl into one line, and repository size is the dominant cost on this fleet — GET /r/{repo} at 420ms could be either, and the two differ by orders of magnitude.

WHICH CALLS. Each recent request is listed with the store calls it made, in the order it made them, and what they cost together. A request with no calls listed spent its time somewhere else, which is itself the answer when the store is not to blame.

Reads the same rings the panels read, so a request that has aged out simply is not there. That is the honest limit of a snapshot: this answers what a slow request just did, not what happened yesterday. check-in: bb6ec6b896 user: dmestas tags: ops-drill

--data is derived and disposable; the OLTP is truth and is not. That is only worth stating if something checks it.

Wiping the derived directory must keep accounts and capability grants, and the converse test keeps the old interleaved layout honest: without the flag the same wipe destroys them, which is exactly what the split exists to escape.

Confirmed against the real binary too: wiping a 4.3M data directory leaves the forge serving 200 with its file listing intact and the index rebuilt from the stores, and the account still there. check-in: 6c2ed45467 user: dmestas tags: trunk

The panel could find a slow route and never say why: the render list and the subprocess list were two independent tables, so 'GET /search took 1132ms' and 'zit artifact ran 499 times' could not be connected even when one caused the other.

Every request is now tagged at the outermost middleware, and every store call records the tag. Work with no request behind it — ingest rounds, warm passes, the doc indexer — records the empty string, which is itself the useful distinction: store load with nothing to blame is background, and telling the two apart is most of what the subprocess panel is for.

Also fixes a silent hole of the same shape as the route-pattern bug: Repo was read with req.PathValue in the outermost middleware, where the router has not set it — the values live on the request the mux passes DOWN. Every render recorded until now carried an empty repo, 61 of 61 rows in the analytics store, which made the numbers unattributable on a fleet where repository size is the dominant cost. Read from the path instead. check-in: bc35cc0c45 user: dmestas tags: ops-drill

The fleet recorded GET /r/{repo}/timeline.sse at a p95 of 435 SECONDS. That is not a slow render, it is a connection somebody kept open for seven minutes — chat, the timeline and the ops panel all hold one for as long as the reader keeps the tab.

Two consequences, the second worse than the first. The slowest-routes panel was headed by four streams, burying GET /{$} at 1198ms against a 13ms median — the one real finding on the page. And every stream crosses the render budget by design, so the first-crossing alert fired on connections that were working perfectly: the muted-channel failure that rule exists to prevent, defeated from inside the same feature.

Detected from the response, not the path. A handler that streams says so in its Content-Type; a rule matching '.sse' would miss the next streaming route added under another name. Prefix-matched, so a charset does not defeat it.

Found by the panel on its first look at real traffic, which is the argument for having built it. closed check-in: cf9a277e13 user: dmestas tags: ops-streams

--data holds two things that behave nothing alike. Per-repository indexes, built archives and the event store are DERIVED — rebuildable from the stores (0001 5), and therefore private to one node. oltp.db is not: accounts, capability grants, webhooks, praise, mirrors, saved views and CI runs are forge-local truth (0001 8), derivable from nothing.

Interleaved in one directory they could not be deployed differently, so two nodes could not have their own fast local index without also having two account databases that disagree. 256KB of the 4.5GB was the whole obstacle to running more than one node.

--oltp defaults to <data>/oltp.db, so a single-node deployment is unchanged. Every command that opens the OLTP takes it.

Verified: node A creates an account, node B with its own --data and the shared --oltp sees it; node B without the flag reports no accounts, which is the divergence this removes. Both nodes serve at once, each with its own index. check-in: 308751929f user: dmestas tags: trunk

Same mistake as before: extras was printed and then the commit ran without the add. Caught by checking the check-in rather than the worktree, which is the only check that works. closed check-in: 63ad319ef2 user: dmestas tags: ops-budget

500ms was a literal in a template and a constant nowhere. The panel coloured a p95 over it and nothing else knew, so the forge could show a route in red and tell no one.

obs.Budget is now the single definition. The panel colours against it — sent on the wire rather than hardcoded, so the page and the threshold cannot disagree — and crossing it triggers a notification.

First crossing only, on errlog's rule and for errlog's reason: the same page is slow on every request until somebody fixes it, and an alert channel that repeats itself gets muted. A muted channel is worse than none, because it reads as healthy. The decision is made under the lock, so two slow requests racing on one route cannot both report it.

This is the difference between an operator watching the forge and the forge watching itself: the ops page has to be looked at; a notification arrives. check-in: fc8ef2165b user: dmestas tags: ops-budget

The rings die with the process, and this fleet restarts constantly — a 4.24s render and a 546 load average both left no trace. obs.Sink had one implementation and it did nothing; this is the durable one.

DuckDB because the workload is columnar: append-only, time-ordered, aggregate-queried, never point-looked-up by a page. RFC 0001 defers the lake to 'when hydration and analytics — the things columnar is for — exist', and reserves the operational surface for SQLite. This is that analytics workload, not a second operational store. Wide flat tables, no keys and no indexes, which is also what a Parquet COPY TO wants when the lake arrives.

Isolated behind a build constraint, in its own package, imported only by the composition root. Both halves of the reason were measured rather than assumed: CGO_ENABLED=0 fails on go-duckdb's bindings, and GOOS=js reports 'build constraints exclude all Go files'. Leaving the import off the request path is not enough — any reachable import breaks a wasm build — so cmd/quarry has two files, one per side of the tag. Verified: GOOS=js GOARCH=wasm builds cmd/quarry clean at 76MB, without DuckDB.

Buffered, never a write per request, and flushed on a timer so a quiet forge still lands rows rather than holding them in memory for days.

Failure is not fatal. The first run failed on 'at' being reserved in DuckDB; the server started anyway, the panels worked, and the log said what was wrong. Analytics is derived data about a forge that works without it.

Costs, measured: +462MB module cache, binary 57MB -> 104MB, cold build +10s, and CGO_ENABLED=0 no longer builds the server. closed check-in: 564f272d3c user: dmestas tags: ops-history

Seven test files were written this session, reported as landed, and never committed: `zit commit` with no paths captures TRACKED files, and new files need `zit add` first. make check passed every time because they were on disk in the worktree, which proves nothing about what is in the check-in.

What was missing:

- search kinds, held in agreement across four places (the index list, two ops validators, the scope chips, hitOf). This is the guard that caught `doc` and again `file` the moment each was added; without it the validator silently refused a kind the chips offered. - technote edits collapsing to one timeline row, folded in three different orders, because a clone delivers artifacts in wire order. - wiki and technote search matching the CURRENT revision only, the "terrified" case. - Contributors(0) meaning every contributor rather than a silent 100. - the ops phase states, the stalled-store trap, and an empty fleet.

Verified by removing them and running the suite: it stays green, which is precisely the problem — trunk has been passing on their absence. closed check-in: 24fdaa723a user: dmestas tags: search-guards

Check-in 2cacce9b01aa added obs.Default.Spawned, RecordSubprocess, RecordRender and Snapshot calls across zitcli, web and api, but the package itself was never added — so trunk did not compile at all.

The files are the ones from the authoring session's tree, unmodified, tests included. Nothing here is my design; this only makes the import resolve so the tree builds again. check-in: 5229b05522 user: dmestas tags: trunk

The panels were steady-state numbers and said nothing while a fleet came up — which is the window an operator most wants to watch and the one where a forge most looks hung. A multi-repo re-ingest on this fleet was mistaken for a hang precisely because nothing said '23 of 25 stores ready' while it worked.

A phase banner leads the page: starting, indexing, syncing or rest, with stores ready over stores served, artifacts this session, the slowest recent round, and the names of what is rebuilding — which, not merely that something is.

Derived rather than tracked. Every store already reports Indexing, LastOK, LastCount, Total and Stalled; a phase field somewhere would be one more thing to keep true. The test ordering is the reader's priority: rebuilding outranks syncing, and a store that has never finished a round outranks both.

A stalled store is DONE waiting — it needs a person, not time — so it does not hold the fleet in 'starting' forever and does not sit in the bar as pending. Ready and stalled are drawn as two segments of one track, or the bar could never fill. closed check-in: 78fbef9d30 user: dmestas tags: ops-phase

quarry could not answer two questions about its own behaviour. A render time was printed in the page footer and discarded, so a 4.24s files page was visible only to whoever loaded it and read the footer. Subprocess load was visible nowhere: a warm loop once ran 134 concurrent `zit ls --long` against the same hash, load average 546 on 14 cores, and /fleet said "all healthy" throughout.

/fleet/ops, site-admin only. Slowest routes by p95 — a mean hides the tail, and the tail is the thing worth finding. Store subprocesses rolled up per verb, with the number that actually mattered: how many run at once, and the peak, which outlives the incident. Recent renders with ops time and the zit time inside it.

Instrumented at the two seams that cover everything: the ops registry, which every page's data crosses, and the outermost middleware, which is the only place the template's own cost is known — the footer's breakdown is assembled before the template runs, so a page can never report the time it spent rendering itself.

Datastar with no no-script path, deliberately: an internal dashboard that cannot move while an operator watches a fleet index is worth much less than one that can, and no consumer depends on it. The op behind it is registered and versioned like any other.

The rings are small and in memory. This is the snapshot; history belongs in the analytics store behind obs.Sink, which is an interface with a no-op default so the persistent implementation can be cgo without the request path importing it. closed check-in: 022966bc05 user: dmestas tags: fleet-ops

zit ls --long adds size, permission and text/binary, and the last of those reads every listed blob. On a large store the annotated forms cost ~18ms per file where the plain two-field listing costs ~0.09ms: measured on tcl, 0.20s plain against 38s for --size and 41s for --long. The cost scales with the STORE, not the listing (filed as zit#148).

Four callers read only .Name and threw the rest away: the About marker scan, a root check-in's file set, the doc index, and CI's file list. They use LsNames now, which runs the plain listing and caches it beside the annotated one.

The tree page and its warm still need the columns they display, so they are unchanged; that cost is the upstream issue's to fix. check-in: 78db482057 user: dmestas tags: trunk

The poll loop ran 'go cli.Ls(ctx, tip)' on EVERY tick as a bare goroutine whose result was discarded. On a store where that call takes seconds the next tick started another before the last finished, and across a fleet they compounded.

Measured: 134 concurrent 'zit ls --long', all against the SAME hash, under one server, load average 546 on a 14-core machine.

The listing is immutable per version, so it needs computing once per version. Now it skips when the tip has not moved and when a warm is already in flight. check-in: a3475428ac user: dmestas tags: trunk

warmReads and indexTree each spawn several zit subprocesses — ls, ls-under, a cat per file, an archive — and on a store tcl's size each takes seconds. They were launched as bare goroutines from every round that moved the tip, so the next round started another set before the last had finished and they stacked.

Measured on the demo fleet: 43 zit children under one server, three of them the same ls against the same hash, load average 546 on 14 cores.

A warm pass already running IS the warm pass. CompareAndSwap claims the slot and the winner clears it on the way out; a round that finds one in flight skips rather than piling on. check-in: 0325ba4a2c user: dmestas tags: trunk

Accepting an already-accepted landing wrote another state reply, and four stacked up on one record before it was noticed.

Two causes. The guard read state from the INDEX, which lags the store by an ingest round, so a second press inside that window saw 'open' and passed. And zit does not refuse the write either — an already-contained branch still gets a reply saying 'accepted', with 'nothing to merge' in the body.

So: fold first, re-read, then decide. A settle is deliberate and rare and can afford one ingest round to know what it is settling. And already-settled now succeeds writing nothing, because that is the end state the caller asked for; an error made a double press look broken when the landing was exactly as they wanted.

Proven against a running server: three accepts in a row leave one 'accepted this' reply and no 'nothing to merge'. check-in: f102c4c7ec user: dmestas tags: trunk

A border-right per item is the single-row idiom, and it breaks the moment the strip wraps — the item ending a row keeps a divider that lands against the container edge, the last item overall has none, and nothing rules between rows. Measured live at 390px the activity strip is three rows, so wrapping is the normal case.

Letting the container's colour show through a 1px gap draws every divider the layout actually needs, vertical and horizontal, with no rule needing to know where a row ends. closed check-in: 76dc2b8d96 user: dmestas tags: pager-shape

The glyph was always centred; the boxes were not. The current page is a span and took 0.5rem of padding with no border, every other page is an anchor and took 0.6rem plus a 1px border — 48px against 53.2px for the same digit.

So the highlight changed size as the pointer moved along the row, and a number sitting still inside a box that grew around it read as though it had slid off centre. Padding and border are stated once for both now.

Measured: every cell 53.2x32.7 at desktop, 37.2 wide at 390px, glyph centred in all of them. check-in: 2ae6328950 user: dmestas tags: pager-shape

The count is a statement about the list, not a control in it, so on a narrow screen it drops to its own line. The controls are trimmed to keep theirs: measured at 390px the row wanted 398px of a 358px container, and most of the excess was the number cells' 2rem min-width plus padding.

Before, whatever spilled — usually 'older' — landed centred under the numbers beside the count, because justify-content centres each wrapped line independently and it read as though it belonged to them. closed check-in: d42e59bdef user: dmestas tags: events-public

It was a flex container with no flex-wrap, so every bare text run between the elements — ' · started ', ' · last reply ' — was its own anonymous flex item. On a narrow screen the default nowrap shrank each item until its text wrapped INSIDE it, stacking the words in columns instead of wrapping the line once.

Normal flow wraps it the way prose wraps. The name with its mark, and each date, are held together with nowrap so the break lands between them rather than through them. check-in: 968a59a2b0 user: dmestas tags: events-public

Every other repository page leaves visibility to the ops layer. A gateAt here made this the one page a public repository would not show an anonymous reader, while its timeline, forum and landings all did.

Safe because of what is on it. The derived events restate artifacts the timeline already shows. The operational ones carry no identity at all — Counted is func(repo string, clone bool), with no user and no address anywhere in the path — so a clone event says how many, never who. check-in: 318949277b user: dmestas tags: events-public

The forge could not answer for minutes after starting. Register runs before ListenAndServe and probes each store with 'zit submanifest show', and the note there said it costs ~250ms — measured on small stores. On tcl it is 116 SECONDS, on sqlite 16. A fleet with either in it held the whole site down.

The probe is bounded at 2s now. Past the deadline it is simply an unanswered question, which the existing false already handled: the fossil clone URL is withheld rather than offered wrongly.

Registration is also timed and logged, because each repository's cost is the whole forge's downtime and a slow one should be nameable rather than suspected. closed check-in: 65e953863a user: dmestas tags: startup-latency

A wiki page and a technote are EDITED: every revision is its own artifact under one stable name. Keyed by artifact hash they piled up in search_doc, and Search's group-by then returned whichever revision matched BEST — so a query found text deleted years ago and linked to a page that no longer contains it. On the live fleet "terrified" matched spatialite_gui's wiki home, which has not said it since 2016; libfossil/home had 117 revisions all searchable.

One row per page now, holding the current revision, newest wins on date so a clone may deliver revisions in any order. Every revision stays readable in the page's own history, which is where the question 'what did this used to say' belongs.

Existing indexes are rebuilt from wiki_edit and technote at open rather than left to the fold: the fold only fires on artifacts that arrive after this code, so deleting the stale rows alone would empty wiki search until somebody happened to reindex. closed check-in: e262fc246c user: dmestas tags: search-current

"use after free" on tk returned three rows identical on screen — same repository, same title, same snippet — because the same fix landed on three branches and the row carried nothing that differed. The only way to find out which one you wanted was to open all three.

Rows now carry the hash, date, author and branch for a check-in; the status and date for a ticket; the date and author for wiki, forum and technote. A doc or a file is identified by its path and gets nothing, rather than a date nobody asked for.

Decorated by primary key, for the page actually shown: the FTS query groups over a match set that can be thousands of rows, so joining there would decorate all of them to render fifty. On the fleet, decorated AFTER the merge trims — every repository is asked for the full limit so the merge can pick freely, and decorating before it meant 25 indexes times 50 rows to render 50.

One definition of the line, shared by both search pages. They are separate templates, which is why the first version of this reached only the per-repo one. closed check-in: 5cc9306b59 user: dmestas tags: hit-meta

quarry rendered both as first-class surfaces (the docs tab, the file browser) and could find neither. Fossil searches documentation but not source; this does both, from one pass.

One tarball, not a read per file. cat per path costs a subprocess and a store open each: 2,199 files on sqlite is ~31s, against 2.2s for one archive — the tarball is the whole tree in one pass, which is the shape of the question.

Tip-only. Indexing every revision of every file would dwarf the rest of the index many times over to answer a question nobody asks; what a file used to say is a question its own history answers.

ref carries the version, not just the path: a link has to name the check-in the row was indexed at, and resolving a branch name at click time would be a second source of truth about which tree the row describes.

Binaries and files over 1MB are skipped and counted — a search that silently covers less than the tree beside it is the failure this change is about. closed check-in: 8df971b482 user: dmestas tags: tree-search