Timeline

Repo was read with req.PathValue in the outermost middleware, where the router has not set it — path values live on the request the mux passes DOWN, a copy that middleware never sees. Same shape as the route-pattern bug, and it went unnoticed longer because an empty string looks like 'no repository' rather than like a failure.

Every render recorded until now carried an empty repo: 61 of 61 rows in the analytics store, on a fleet where repository size is the dominant cost of almost everything. The drilldown's by-repository table showed a single '(none)' row, which is what surfaced it.

Read from the path instead, unescaping as the router would. closed check-in: 059fe0bdde user: dmestas tags: ops-drill

/fleet/ops linked OUT to /fleet seven times and nothing anywhere linked IN. The page was live, gated, and reachable only by typing the URL.

Second surface to ship that way after the notes feed, so it gets a test rather than a promise to remember — both directions, so an admin is never stranded on either.

The link goes in the summary strip that already carries the fleet's counts and the error badge, which is where somebody reading health already is. /fleet answers 'are my repositories healthy'; /fleet/ops answers 'is the forge healthy'. check-in: 3314daeff6 user: dmestas tags: ops-drill

The panel finds a slow route; this says why. Two things the roll-up cannot show.

WHICH REPOSITORY. A route pattern folds pikchr and tcl into one line, and repository size is the dominant cost on this fleet — GET /r/{repo} at 420ms could be either, and the two differ by orders of magnitude.

WHICH CALLS. Each recent request is listed with the store calls it made, in the order it made them, and what they cost together. A request with no calls listed spent its time somewhere else, which is itself the answer when the store is not to blame.

Reads the same rings the panels read, so a request that has aged out simply is not there. That is the honest limit of a snapshot: this answers what a slow request just did, not what happened yesterday. check-in: bb6ec6b896 user: dmestas tags: ops-drill

--data is derived and disposable; the OLTP is truth and is not. That is only worth stating if something checks it.

Wiping the derived directory must keep accounts and capability grants, and the converse test keeps the old interleaved layout honest: without the flag the same wipe destroys them, which is exactly what the split exists to escape.

Confirmed against the real binary too: wiping a 4.3M data directory leaves the forge serving 200 with its file listing intact and the index rebuilt from the stores, and the account still there. check-in: 6c2ed45467 user: dmestas tags: trunk

The panel could find a slow route and never say why: the render list and the subprocess list were two independent tables, so 'GET /search took 1132ms' and 'zit artifact ran 499 times' could not be connected even when one caused the other.

Every request is now tagged at the outermost middleware, and every store call records the tag. Work with no request behind it — ingest rounds, warm passes, the doc indexer — records the empty string, which is itself the useful distinction: store load with nothing to blame is background, and telling the two apart is most of what the subprocess panel is for.

Also fixes a silent hole of the same shape as the route-pattern bug: Repo was read with req.PathValue in the outermost middleware, where the router has not set it — the values live on the request the mux passes DOWN. Every render recorded until now carried an empty repo, 61 of 61 rows in the analytics store, which made the numbers unattributable on a fleet where repository size is the dominant cost. Read from the path instead. check-in: bc35cc0c45 user: dmestas tags: ops-drill

The fleet recorded GET /r/{repo}/timeline.sse at a p95 of 435 SECONDS. That is not a slow render, it is a connection somebody kept open for seven minutes — chat, the timeline and the ops panel all hold one for as long as the reader keeps the tab.

Two consequences, the second worse than the first. The slowest-routes panel was headed by four streams, burying GET /{$} at 1198ms against a 13ms median — the one real finding on the page. And every stream crosses the render budget by design, so the first-crossing alert fired on connections that were working perfectly: the muted-channel failure that rule exists to prevent, defeated from inside the same feature.

Detected from the response, not the path. A handler that streams says so in its Content-Type; a rule matching '.sse' would miss the next streaming route added under another name. Prefix-matched, so a charset does not defeat it.

Found by the panel on its first look at real traffic, which is the argument for having built it. closed check-in: cf9a277e13 user: dmestas tags: ops-streams

--data holds two things that behave nothing alike. Per-repository indexes, built archives and the event store are DERIVED — rebuildable from the stores (0001 5), and therefore private to one node. oltp.db is not: accounts, capability grants, webhooks, praise, mirrors, saved views and CI runs are forge-local truth (0001 8), derivable from nothing.

Interleaved in one directory they could not be deployed differently, so two nodes could not have their own fast local index without also having two account databases that disagree. 256KB of the 4.5GB was the whole obstacle to running more than one node.

--oltp defaults to <data>/oltp.db, so a single-node deployment is unchanged. Every command that opens the OLTP takes it.

Verified: node A creates an account, node B with its own --data and the shared --oltp sees it; node B without the flag reports no accounts, which is the divergence this removes. Both nodes serve at once, each with its own index. check-in: 308751929f user: dmestas tags: trunk