C Concurrency\sreview:\swhat\scame\sof\sit
D 2026-09-28T02:23:12.876
E 2026-09-28T02:23:12.876 31b4011e381f7b435553069456a02d993a1bec0f
N text/x-markdown
U zeitgeist
W 3394
In September @danshearer reviewed the forge layer for concurrency problems, at trunk 4910c69016e5. Every finding became a ticket, and the full list with priorities lives on the wiki: [Concurrency review 2026-09](/r/dmestas/quarry/wiki/Concurrency%20review%202026-09). This note is the short version of what came of it.

## The one-node forge

Fourteen of the fifteen single-node findings are fixed and deployed, and the fifteenth is fixed and in review. The ones that mattered most:

- **A crash that took the whole server down.** Opening or closing a chat tab during a presence change could kill the process with a concurrent map error, for every visitor at once (54dc6a4148).
- **Leaks that grew with every closed tab.** Live-update streams left a goroutine behind per closed tab (508fd44ab0), and a failed reindex left its census goroutine and a `zit frames` process behind on every retry (86a22fabef).
- **No timeouts.** The server had no HTTP timeouts and no limit on anonymous chat streams (61cd995161).
- **Losing the index to a small error.** Any error reading the ingest cursor started a full reindex that emptied the index first (c21e27c2d6), and a node that fell behind the event stream's one-hour window was never noticed (b880cb13c8).
- **Writes that could race.** SQLite settings reached only one pooled connection, so concurrent writes failed busy (7ae62edbcc); several checks ran outside the transaction that acted on them (3d2980cf29); some store writes skipped the lease (8e6ecbb15c); two accepts of one landing could both proceed (e60889c0d7).
- **Smaller ones:** the login rate limiter trusted a header a client can set (fe5069ef76), a CI result could be lost (2af687e64f), CI webhook events shared ids (4a9f45bab5), package-level state was written per request (83b111e004), and shutdown was not graceful (128ffbf478).

## Four more, found while landing the work

Fixing the list turned up four follow-ups, and all are fixed: a race-detector failure in the web tests (3a2d821f59), a failed rebuild that never resumed the maintenance it paused (55d25c1639), a stale reset that could clear a fresh index (955c5cee5a), and every store rebuilding at once after long downtime (21e312b2f9).

Separately, a push carrying a file with a line like `login TEXT NOT NULL,` was refused, because the sync gate read protocol cards out of the pushed file's contents. The gate now frames payloads card by card.

## What it looks like now

A restart of zeitforge.app is serving again in about 25 to 30 seconds, and the most recent ones rebuilt no stores and logged no errors beyond clients dropping during the restart. Earlier the same day, before the event-window and shed fixes landed, one restart spent about nine minutes and over twelve hundred log lines rebuilding stores that did not need it.

## Still to do: more than one node

The multi-node findings are designed in RFC 0001 and matter only once a second node runs. Node identity is done (a598314e3f). The private-repository list's cache is in review (8af7bf74ca). Two are open: the lease is checked once before a long write (935f72e648), and background jobs would run on every node (1d09827992). Two are deferred as design work: the shared OLTP file assumes one host (18ca45f20e), and write routing to the lease holder is not built (4116d2f4a2).

Thanks to Dan Shearer for a review that was specific enough to act on line by line.

Z cde762019f12130ab590324f06d0325f