zit frames streams a WHOLE store: 507 MB peak on sqlite, more on tcl. quarry ran one ingest goroutine per repository with nothing between them, so a cold start — or a mirror round landing artifacts in several big stores at once — ran that many censuses simultaneously and the memory added. Observed as zit spiking to gigabytes after pulling sqlite and tcl together.

--ingest-limit (default 3) puts them in a work lane. The queue is generous and ingest is never shed: dropping a round would leave an index behind its store until the next poll, trading a correctness surprise for a saving the limit already delivers. check-in: e6fd32e82e user: dmestas tags: trunk

fail() mapped only ErrNotFound, so every other kind rendered as 500 — and a 500 tells a client the request failed, not to try again. Two retryable paths lost that distinction: a single-writer lease refusal (0001 §3) and a shed from a saturated work lane.

ErrInvalid now renders 400 and ErrUnavailable 503 with Retry-After, matching what the API router has always done.

Verified with the archive endpoint, live for the first time now that zit 10a1236a7a fixes tarball -R: twenty concurrent requests give ten 200s and ten 503s with Retry-After, RSS bounded at 247 MB. check-in: e9d441a552 user: dmestas tags: trunk

Nothing in quarry limited concurrency — every zit subprocess ran in the request goroutine with the arrival rate as the only bound. work.Lane is a concurrency limit plus a queue cap that SHEDS past it: a request accepted into a queue it cannot be served from consumes the capacity that would have served someone else.

Wired to archive first, which buffers 256 MiB per request on an anonymous route. That is latent rather than live — archive 503s on zit#97 — which is why the bound belongs there now, so the endpoint is born bounded rather than acquiring a limit after someone finds it.

Footer precision: '<1ms' was true and useless once most pages went sub-millisecond, collapsing the interesting range into one label. timing.Format keeps ~3 significant figures across five orders of magnitude, so 900µs and 90µs no longer read the same. Fossil has the mirror-image bug: three decimals of seconds prints everything faster than a millisecond as 0.000s. check-in: 45bb75ade2 user: dmestas tags: trunk

zit cat costs ~147ms even for a small file, against a 3ms process floor, and every file view forked one. Under load that capped the forge at 32 req/s, degrading to 24 at twenty concurrent readers.

hash-versioned file page: 393ms -> 18ms 50 concurrent: 201 req/s, still climbing

Bounded by BYTES, not entries: values are file contents, and an entry-counted cache holding a 30MB blob is a memory leak with a polite name. Per repository, so a busy store cannot evict a quiet one's entries.

resolveVersion only expands hash PREFIXES. An earlier draft fell back to the index's branch table for names, which served tk a README from 2020: tk renamed its trunk to main, so a branch literally called trunk still exists and is six years stale, while zit resolves the word correctly. 0001 §2 makes zit the authority on names and that is not negotiable — so a symbolic version stays uncached, and the way to fix that is to make links carry the resolved hash. check-in: 0cf9a57e88 user: dmestas tags: trunk

The ticket list derived itself per request: GROUP BY over every change row (tk: 35,903), two correlated subqueries per ticket to find the newest title and status — 11,618, each with its own temp b-tree — and a HAVING that filtered after grouping, so paging saved nothing.

tk 200ms -> 1ms sqlite 1ms fossil <1ms

Written from refreshTicketDoc, which already folds exactly this state for the search doc, so the summary cannot disagree with what search sees. Backfilled at open for indexes written before the table existed.

Second instance of the same bug the timeline had, so plan_test now covers this query too. check-in: 4c973ff323 user: dmestas tags: trunk

The footer said 'rendered in 344ms'; it now says 'rendered in 344ms (zit 270ms)'. One number says a page is slow, not which layer to look at, and quarry's layers differ by three orders of magnitude — an index read is microseconds, zit ls --long on tk is 2.6 seconds.

The accumulator rides the request context, which is what makes it correct under load: package-level counters would blend concurrent requests together, and concurrency is what this exists to measure.

zit read calls take a context now and use exec.CommandContext, so an abandoned request kills its subprocess instead of leaving it to finish a 2.6-second listing nobody will read.

SQL is not instrumented yet; the index does not take a context. check-in: 85b9909d1c user: dmestas tags: trunk

The timeline assembled itself at read time from a five-way UNION ALL, ending in USE TEMP B-TREE FOR ORDER BY — every request sorted every event in the repository to return fifty. tk cost 536ms; fossil serves the same page in 3-4ms whatever the repository size.

This is fossil's shape: one event table with a kind column and an index on date, so the read is an index walk that stops at the limit. Triggers maintain it, like search_fts, so it cannot drift from its sources. isleaf and the ticket label are resolved for the returned page only.

tk 536ms -> 4ms sqlite 210ms -> 5ms fossil 163ms -> 3ms pikchr 22ms -> 1ms home 399ms -> 3ms (it asked each repo for its newest event)

plan_test asserts the plan, not the types: nothing about the old query was ill-typed, and the defect was only ever visible in EXPLAIN.

Also fixes limit being ignored on every wildcard route: the router's second decode pass re-applied defaults to fields the path did not name, so ?limit=200 became 50. OverlayValues sets only what it names. check-in: 450a8ad0f3 user: dmestas tags: trunk

Fossil publishes its own ('This page was generated in about 0.003s') and that is what made our timeline regression measurable against it. Same number, same place, every page.

semdiff gains data-volatile so the disposability harness compares state rather than the stopwatch — without it a wipe-and-reingest fails on '<1ms' vs '4ms', which is true about the clock and false about the index. check-in: cb17edbefa user: dmestas tags: trunk

Was ~52px per row across three columns for one short filename; fits 38 entries where 25 fit before. Drops the History column — thirty identical links, and the file page already links its own history — adds dir/file marks, right-aligns sizes with tabular figures, and folds four lines of chrome into one.

Names are no longer underlined: every name in the listing is a link, so marking them all marked nothing.

The space the History column freed is where the last-change column goes once the index carries per-file data. check-in: 574cf5b97e user: dmestas tags: trunk

A bare README was rendered as markdown 'by convention'. libspatialite's is an autotools plain-text file whose commands carry the root prompt #, so markdown read '# sudo make install' as a heading and turned the build instructions into a dozen h2s.

Extension decides now, and anything unrecognized falls through to escaped

 with its line breaks intact — fossil's rule in
wiki_render_by_mimetype. Also recognizes README.wiki.

Counts tell a reader where a store's content is — pikchr is 245 forum threads and one ticket, tk is 5,809 tickets and one post. The index memoizes them per fold; on tk they cost ~110ms cold and 3ms after.

Also: the current section is marked, empty sections are dimmed but stay linked, the repository outranks the forge name in the header, and timestamps render relative with the exact value in the title. check-in: 3a330c9560 user: dmestas tags: trunk

Adds mirror.list/set/delete/run ops (admin), a mirror table in OLTP, and a scheduler that pulls each configured store on its own interval. Pulls run under the store's single-writer lease and re-ingest only when artifacts actually arrived.

zitcli.Pull is the one call that cannot pass -R: zit pull refuses the flag and works by working directory. pullChanged reads zit's result positively and errors on an unrecognized line rather than defaulting to 'nothing changed', which would leave a mirror green while it silently stopped tracking. check-in: 9c7b4cff5b user: dmestas tags: trunk

Adds fleet.status and fleet.sync ops (admin) and a /fleet page: last round, artifacts, parse fallbacks, and consecutive failures per store. With eleven repos a wedged store is invisible on a page that shows only counts — its counts are whatever they were when it broke.

gate gains gateAt so the fleet pages can ask for Admin at an empty repo rather than Token at a repo; forbid now words the refusal to match. check-in: 5c9f9f41b1 user: dmestas tags: trunk

FramesEach hands each artifact to a callback as it is read; peak RSS on sqlite's 13.2GB census drops to 0.48GB. Frames is kept as a collecting wrapper for tests. A callback error kills the child rather than draining the stream, and the cursor is not saved on a failed round. check-in: 05d01cbb93 user: dmestas tags: trunk

The harness writes .agent-config/trace/<session>.jsonl into the working tree and it was tracked: 2MB, carried across ten check-ins, growing with every tool call, so each commit stored another revision of a log nobody working on quarry writes or reads. .claude/scheduled_tasks.lock is a lock file and equally not source.

Both untracked with 'zit rm', which keeps the files on disk, and already ignored via .zitignore so they do not return as extras. History keeps what was recorded; this stops the accumulation. check-in: 4d8f962778 user: dmestas tags: trunk

The harness writes .agent-config/trace/<session>.jsonl into the working tree and it is tracked: 2MB, carried across ten check-ins, growing with every tool call, so each commit stored another revision of a log nobody working on quarry writes or reads. .claude/scheduled_tasks.lock is a lock file and equally not source.

Both untracked with 'zit rm' (the files stay on disk — zit keeps them unless asked otherwise) and added to .zitignore so they do not come back as extras. History keeps the revisions already recorded; this only stops the accumulation.

Also lands gofmt churn in web_test.go from an earlier formatting pass: import order and one alignment block, no logic. check-in: bcd8d75db7 user: dmestas tags: trunk

The Dockerfile and compose were committed unverified because no container runtime was installed. With one available they were wrong in three ways, none visible by reading:

- The Zig download 404s. Naming is zig-<arch>-linux-<version>; I had the older zig-linux-<arch>-<version>. - The zit clone fails: the GitHub mirror is private, so an anonymous clone asks for a username. zit's source now arrives as a named build context, which needs no credentials and pins the exact checkout being shipped. A public fossil mirror exists at fossil.craftdesign.group/zit and would make a clone step viable later. - /data was root-owned. VOLUME creates a missing directory as root while the process runs as uid 10001, so quarry could not open its own database — 'permission denied' on the first line of the log.

Now verified end to end: a 90.1MB image, /etc/quarry-build recording both versions, and a container serving libfossil's 3,835 check-ins from a read-only mounted store with zit reading it from inside. check-in: a8cbc48878 user: dmestas tags: trunk

fixtures/out is 599MB of built stores and it is what a 'zit addremove' once swept into this repository, along with a 45MB binary — 559MB of bloat that took a shun and a reclaim to undo.

Ignoring the BUILT output rather than the whole directory: the scripts and README beside it are source.

.zitignore rather than the ignore-glob setting, because zit says plainly that a root .zitignore overrides the setting, and because a tracked file travels with a clone while a local setting does not. check-in: 52a8ca8b77 user: dmestas tags: trunk

quarry had no packaging at all while notify already had a working Dockerfile and compose. The image ships zit WITH quarry, deliberately: 0001 §2 makes the zit CLI the entire store interface, so zit's behaviour IS quarry's, and today alone three zit builds changed what quarry does — frames got faster, a card fix changed which artifacts parse, stat -R enabled bare-store serving. ZIT_REF is a build argument and /etc/quarry-build records the pair, so a container can answer which zit it carries.

compose follows notify's shape, including the constraint its own file documents: dagnats binds loopback in standalone mode, so everything shares its network namespace. quarry joins it too — its embedded NATS uses DontListen and binds nothing, so there is no port to collide with.

docs/running.md is the by-hand path, which is what this was actually developed against: the launchd-signed dagnats, notify's bootstrap and workflows, the topic-vs-subscriber addressing, and the two credentials that close the write and push gates.

Unverified, and said so in the doc: no container runtime is installed here, so the images have never been built. The two build recipes inside them ARE verified natively — zit compiles from a clean clone with zig build --release=fast, quarry builds fully static with CGO_ENABLED=0.

Also lands the publish-throughput measurement as an opt-in test. It is what disproved the hypothesis that synchronous publish was the ingest bottleneck (62k msg/s sync), which redirected the search to zit frames and quarry's folding. check-in: bbe731e042 user: dmestas tags: trunk

zit#95 landed 'zit tarball', so quarry can offer what every forge does: this check-in as a file. An op, a download route with the same posture as attachments (always a download, never an inline render), and a link beside the clone box.

It does not work yet, and the reason is upstream: zit tarball ignores -R and demands a checkout in the working directory, which a server process never has (zit#97). ls honours -R from the same directory against the same store; tarball does not. The route is correct and stays — it starts answering the moment the flag is honoured — and the failure is reported as unavailable rather than not-found, because the repository is there and the archive is what cannot be produced.

Not worked around: 0001 §2 makes the CLI the whole store interface, and a checkout materialised per download to dodge the bug would defeat both that and the reproducibility the feature exists for. check-in: c81b757987 user: dmestas tags: trunk

Three separate faults, worst first.

Branches() used a correlated subquery — one full scan of checkin per row. On tcl (42,074 check-ins, 1,855 branches) that took 434 SECONDS, which is what made the page look hung rather than slow. One grouped scan with SQLite's bare-column-with-max rule, plus an index on (branch, date): 0.042s.

'ls --long' costs ~4ms per file because it reads each blob for size and the binary flag, so tcl's 2,198-path tip took 8.5s. Fossil never pays this — browse.c builds a directory listing from its own indexed tables and shows no size at all — but quarry reads through the CLI by design, so the answer is to compute once: a check-in's file set is IMMUTABLE, so listings are memoised per version, the tip is warmed after each ingest, and the tree op resolves 'trunk' to its hash so the page asks for the key that was warmed.

Only a hash may be cached. Caching 'trunk' would serve a stale tree forever after the next commit — a correctness bug dressed as a performance fix — so the memo refuses anything that can move, with a test pinning it.

The branch selector rendered every branch: 1,855 <option> elements, 90% of a 239KB page, and unusable as a control. Now the 25 most recently active, plus the branch being viewed even when it is older.

tcl's repo home: 8.5s -> 0.38s, 239KB -> 29KB. sqlite: 5.0s -> 0.32s. check-in: c754084dc7 user: dmestas tags: trunk

A present-but-empty U card — a bare U, or one followed by an empty token — is specified as the literal string "anonymous", and quarry left the author empty. The two are different claims: one says an anonymous person did this, the other says nobody did.

Absence of the card entirely stays a separate matter, governed by the per-type required-card rule of §5.2.

Noticed while tracing the §4.2 empty-token defect (zit#86), which is the same shape: a MUST a reader can skip without anything complaining. check-in: adf5f7c322 user: dmestas tags: trunk

Two follow-ups to the CI slice.

A check-in now carries its verdict wherever it appears — a pass/fail badge on timeline cards and the check-in page, and a ci field on the API. It is read from the store's tag artifacts, not from dagnats, so it answers with the engine down and travels with a clone. A check-in with no verdict shows nothing rather than a neutral badge that would read as a result.

The CI page follows runs live. dagnats' /runs/{id}/events is a JSON snapshot rather than a stream, so this polls and sends only when the rendered fragment actually differs, reusing the SSE shape the timeline and fleet already use. data-init sits on a wrapper OUTSIDE the patched element — the mistake that silently killed both other streams this morning and only showed up in a browser.

Verified in a browser rather than by asserting the server is right: a commit to synthetic made the table go from one row to two with no reload. check-in: f50a282ab1 user: dmestas tags: trunk

Completes the loop 0001 §6 describes. The trigger runs after each ingest fold — the moment a check-in becomes visible whatever brought it in, a local commit, a push or a sync. A poller refreshes runs quarry still thinks are going, and a terminal run becomes a tag artifact on the check-in, so CI history clones with the repository instead of living in the forge.

The verdict tag is --raw, and that is not a detail: without it the name enters the symbolic namespace where it resolves to a check-in like a release name, and the SECOND check-in to pass would make 'ci-pass' ambiguous for both. zit#80 landed exactly this last night.

It is written through the lease holder, like every artifact quarry mints, so two nodes cannot both record a verdict.

Proven end to end against the live dagnats: a commit to synthetic launched ci/build.json read AT that check-in, the run was recorded, cancelling it drove the poller to write ci-fail with the run id as the tag's value, and the tag came back through ingest into the index. check-in: e326831901 user: dmestas tags: trunk

0001 §6 splits this narrowly: dagnats runs pipelines, quarry notices a check-in and shows the result. quarry defines nothing, schedules nothing, retries nothing.

A pipeline is TRACKED CONTENT — ci/*.json in the repository, dagnats' native format — read at the check-in under test, so the pipeline for a historical check-in is the pipeline that ran and there is no forge-side store to drift. Only files directly under ci/ count; a nested path is somebody's fixture.

Two behaviours worth naming. First sight of a repository takes the tip as a watermark and launches NOTHING: running a pipeline per historical check-in would be a denial of service against the engine and would say nothing about code nobody just changed — on an imported corpus that is tens of thousands of runs. And an unrecognised run status counts as still running, because claiming a run finished when it has not is how a verdict gets written against a build still going.

Verified against the live dagnats for the wire shapes; the trigger itself is covered by tests. check-in: 0502a93834 user: dmestas tags: trunk

/r/{repo} was a 404. A repository's landing page was the timeline, which is fossil's instinct rather than a forge's, and there was no way to browse files at all: /file/{hash}/{path} served content only if you already knew the exact path, and nothing linked to it.

0002 §1 lists this as a v1 row that never got built. tree.list turns zit ls's flat path set into one directory level — a store has no directory objects, so a directory is a shared prefix and a count of what is under it — with the directory's README rendered beneath, breadcrumbs, and files linking into the existing highlighted view.

The branch selector carries each tip hash, not just the name: zit permits two branches to share a name where forgejo assumes uniqueness, so a name-keyed control would silently pick one.

The clone box emits the reader's own host, since a hardcoded base is wrong for every deployment but one. check-in: 7058721205 user: dmestas tags: trunk

The page dumped every folded field into one table, which read as noise and lost most of the ticket. Fossil's model has three parts and quarry was collapsing all of them into one:

- Scalar state (status, priority, resolution…) folds last-write-wins. - A leading + is the APPEND form: 'J +comment more' grows the field rather than replacing it. quarry stored a separate field literally named '+comment' — sqlite's tickets use it 255 times. - icomment belongs to the CHANGE, never to the ticket. Folding it left one comment standing where a conversation was: the busiest tcl ticket showed 1 of its 84.

Per-change plumbing (icomment, mimetype, login, username) no longer appears as ticket state. The page now leads with the title, shows fossil's header order, renders the description and every comment through their own recorded mimetype, and attributes each — including fossil's 'claiming to be' when the typed name differs from the login, which on an open tracker it may.

Verified on tcl 408840ff: 11 clean fields, 84 comments, 5 attachments. check-in: 9ac012f7f2 user: dmestas tags: trunk

Running the real companion found two faults no stub would have.

NotifyEvent.To is documented as "a user, or a repo:<name> topic", but quarry put that string in subscriberId. notify addresses the two through DIFFERENT fields, so an activity event did not fan out — it invented a subscriber literally named repo:synthetic and every watcher heard nothing. Verified against notify: the message existed, addressed to nobody real.

notify also refuses a topic key containing a colon (lowercase, digits, hyphen, underscore only) and rejects the whole trigger, so quarry's own convention could never have been a valid key. repoTopicKey translates at the boundary; quarry keeps its internal vocabulary. Names differing only in punctuation or case collapse to one key, which is documented rather than hashed away — an operator has to be able to read the key they subscribe someone to.

End to end on a real notify + dagnats: a wiki save reaches every member of repo-synthetic, an @mention reaches the person, and both render in quarry's inbox. check-in: 526e25a5d6 user: dmestas tags: trunk

Two things, one of them a live bug. notify's /v1 routes sit behind an API-key middleware and quarry's client sent no Authorization header, so every trigger it fired would have been answered 401. It went unnoticed because --notify was never configured and a trigger failure is logged rather than raised. --notify-key (or QUARRY_NOTIFY_KEY) supplies it, and serve says so at startup when it is missing rather than leaving the answer to be discovered in the log.

The inbox is notify's, embedded (0001 §5): GET /v1/messages?channel=in_app for the signed-in account, rendered with unread emphasised. quarry models no notification state — it displays what notify owns.

The wire field names are notify's Go field names: its domain.Message carries db tags but no json tags, so the encoder falls back to the identifiers. A test pins that, since decoding lowercase would silently produce empty subjects.

An outage is not emptiness: a failed fetch says notifications are unavailable instead of showing an empty list, which would be a lie about whether anyone had written to you. check-in: e33f1c4e8c user: dmestas tags: trunk

Basic auth works and stays for the API, but a browser has no way to log out of it and the credential lives in the browser's keeping. A gated page now redirects to /login instead of raising a Basic prompt, the header names who is signed in and offers sign-out, and Basic remains for scripts.

Session tokens are stored as SHA-256 digests, so the table cannot be read back into working cookies. Cookies are HttpOnly, Path=/, SameSite=Lax — Lax being what stops another origin POSTing quarry's write forms on a reader's behalf — and Secure under TLS.

A password change or an account deletion ends that account's sessions; otherwise 'change your password' would leave the browser already holding one still signed in. Authorization for a cookie runs the same RoleOf lookup the password path uses, so a cookie is never worth more than a password.

safeNext keeps the post-login redirect on this site: a reader is at their most trusting on the hop straight after authenticating, which is exactly when an open redirect pays. check-in: 7bdf7a189b user: dmestas tags: trunk