50 descendants of 88ae8ba61972eeca
|
2026-08-13
| ||
| 16:09 | export-git: mirror canonical history into git check-in: ae7c26a1a3 user: dmestas tags: sub-manifest | |
| 16:09 | bench: qualify incremental git export scaling — memo parse bound pinned; althttpd/wapp export clean (fsck 0), phantom-parent lineage refuses naming both hashes check-in: 75fb61025d user: dmestas tags: sub-manifest | |
| 15:18 | e2e: verify git export commit by commit check-in: 08ca21c63c user: dmestas tags: sub-manifest | |
| 15:14 | cli: export git mirrors incrementally check-in: 5b2c6139d5 user: dmestas tags: sub-manifest | |
| 15:03 | export-git: publish mirrors without partial state check-in: 41dfef63b2 user: dmestas tags: sub-manifest | |
| 14:32 | export-git: translate check-ins refs and failures precisely check-in: 74c5c8aba0 user: dmestas tags: sub-manifest | |
| 13:52 | export-git: plan topologically and enumerate through the seam check-in: 491cbc50c5 user: dmestas tags: sub-manifest | |
| 13:35 | export-git: persist a disposable local mark map check-in: 17c02d8950 user: dmestas tags: sub-manifest | |
| 13:23 | export-git: serialize bounded fast-import streams (the writer files the previous commit's message described) check-in: bb1cc4a01b user: dmestas tags: sub-manifest | |
| 13:23 | export-git: serialize bounded fast-import streams check-in: 2450e4f142 user: dmestas tags: sub-manifest | |
| 13:08 | git: gather every interop root into src/git/ — cli.zig joins the zit module (nine importers rerouted), moved roots take parent deps through the module; ~986 duplicate test executions eliminated check-in: 49c08bb191 user: dmestas tags: sub-manifest | |
| 11:41 | submanifest: replay history to pick k (#41) — analyze replays last 200 commits through the real builder+delta encoder (both cost axes, bootstrap split, baseline-charged materialization); enable pins the replayed argmin, snapshot objective as fallback. Advisory only. check-in: b9ceda433f user: dmestas tags: sub-manifest | |
| 03:06 | clone: stop inflating served/received delta containers for their header — prefix-inflate 16B, base frame read header-only; server CPU per 1M-artifact clone 300s+ -> 22s, clone verifies 0 faults check-in: f5eafc6a2f user: dmestas tags: sub-manifest | |
| 01:25 | serve: notice a --force shun/pack generation swap — stat the path beside the held handle, reopen between rounds; shunned bytes stop serving with no restart. Closes #17 check-in: faa8befc3b user: dmestas tags: sub-manifest | |
| 00:16 | serve: scale the write deadline with the reply — 4MB clone batches at the flat 5s cut healthy slow peers mid-write (#22 shape); slow-drain e2e proves it check-in: fa230db9e8 user: dmestas tags: sub-manifest | |
|
2026-08-12
| ||
| 23:21 | timeline: order tied runs by generation, not per-pair ancestry walks — 305k-commit CVS import renders in 5s vs CPU-hours; B1 descendant-first preserved exactly check-in: d823a2811d user: dmestas tags: sub-manifest | |
| 21:47 | clone: byte-budgeted batches (4MB) — 159 rounds vs ~1000 for a 1M-artifact store; warm clones plant the checkpoint (first command 0.06s); print transfer/derive phases. Closes #53 check-in: c32414008b user: dmestas tags: sub-manifest | |
| 19:56 | checkpoint: accept delta manifests; anchor at newest check-in (mainline fold collapsed on real repos); sync: index the pending-delta queue (O(1) hasTarget, O(deps) completeFor) — zit now clones 2GB pkgsrc-2012 from real fossil in 10min, resumes opens in 9s vs 4min derive check-in: c70f48f825 user: dmestas tags: sub-manifest | |
| 09:04 |
checkpoint: retire the tree shape — sidecar only, structurally
The tree shape minted permanent SUBMANIFEST artifacts for benefits already had elsewhere: the rebuild win lives in the prefix record, verify's §6.2 oracle runs buildTree in memory, and the subtree-sharing bytes re-materialize at the first real S-emitting commit at identical cost. Minting early only created unreferenced permanent artifacts and the allow-tree/demotion/gate apparatus needed to keep automation from doing it by accident — all deleted with it. build no longer opens the log for writing or reads the emission settings at all; sidecar layout v3 (old sidecars miss and rebuild in seconds). Proven at scale: SHA-256 of pkgsrc's 593MB store is byte-identical across a full build; cold 4:12, rebuild 3.4s. check-in: 2a8fdcff60 user: dmestas tags: sub-manifest | |
|
2026-08-11
| ||
| 22:45 |
checkpoint: one-pass build, incremental advance, opportunistic refresh
Build paid a bypass derive, a whole-prefix delta scan, and a capture re-derive; a view-free captureRecords (byte-identical to the deriving walk, pinned at every boundary) now does capture and delta detection in one pass, and an existing sidecar's body plus its delta-free attestation reduce a rebuild to the unscanned tail. pkgsrc: cold build 17:58 -> 4:21; rebuild or anchor advance -> 5.1s. Repository.open's healing paths keep an existing checkpoint fresh: witness shape forced (the automatic path can never mint an artifact, whatever §8.1 says — a tree pin advances by demotion and is re-pinned deliberately), gated on 4MB growth AND anchor progress so blob-only growth never churns the sidecar. No checkpoint, no auto-creation: building once is the opt-in. check-in: adae602908 user: dmestas tags: sub-manifest | |
| 20:04 |
repostate: carry the C-card comment in the crosslink record
An unlimited timeline re-read and re-parsed every manifest just to print its comment first-line — 163k inflate+parse cycles on pkgsrc. The comment now rides the crosslink record into the derived state (commentOf; image backing falls back to the manifest read), so the render loop is a lookup per row: full timeline 12.3s -> 1.06s. Record-format consumers bumped so old bodies miss instead of misparsing (repocache v3, checkpoint v2 — one re-derive per repo, then warm). zit log's pinned B-log-1 walk untouched. check-in: 57d8de4120 user: dmestas tags: sub-manifest | |
| 19:07 |
repocache: persist the content log's scan index
The lazy content channel re-walked every frame header on each open — O(store) before the first byte of any comment, cat, or annotate. The scan's product (entries, hash index, shun rows, config cursor, tail key) now persists as .zit-cache/index, self-validating and LogKey-keyed: an exact tail adopts whole, a grown log resumes the walk over the tail alone, and anything else falls back to the authoritative scan (--no-cache never consults it). pkgsrc: info 1.7s -> 0.44s, timeline -n 5 4.3s -> 0.96s. check-in: f2bfd36e74 user: dmestas tags: sub-manifest | |
| 18:17 |
logstore: O(1) LRU eviction in PlaintextCache
Replace the pick-the-minimum-stamp linear scan with an intrusive doubly linked recency list; get/put/evict are all constant-time. The scan was invisible at small occupancy and dominant at full: a pkgsrc clone kept the cache pinned at max_entries and spent over half its client CPU sweeping the map once per stored artifact. Measured: clone 12:38 -> 8:39 wall, client CPU 694s -> 463s; cloned store verifies with 0 faults. check-in: 0df83f350a user: dmestas tags: sub-manifest | |
|
2026-08-10
| ||
| 23:48 |
Move the git-interop implementation into src/git/
Seven files relocate with unchanged basenames: git_fast_export(+test), git_import, git_import_impl, git_import_cli, git_import_verify, git_import_benchmark_support. The six standalone module roots (test, refs_test, cli_test, e2e, benchmark, pipe_helper) stay flat because a Zig module cannot import above its root file's directory — the same rule that keeps main/e2e/dst/smoke at src/. Future export-git files follow the same split: roots flat, implementation in src/git/. Path-string edits only; build.zig unchanged. check-in: ce3c87e25c user: dmestas tags: sub-manifest | |
| 23:12 |
Add zit checkpoint: durable prefix resume for hybrid history
.zit-cache/checkpoint pairs a validated log-prefix key with the prefix's crosslink record body; Repository.open resumes past the flat segment on a cache miss instead of re-parsing it. Witness shape under a closed emission gate creates zero artifacts; tree shape pins the anchor's file set as the canonical S-tree (batch-oracle byte-identical, one interior root node). verify re-derives every claim from real history; --no-cache bypasses both sidecars. New verb: zit checkpoint [show|build|verify|drop]. pkgsrc (163k commits, 60k files): cold open 8.4s vs 352s full derive; verify passes; 366/367 tree nodes deduplicated against the imported tree. check-in: e4026acd7b user: dmestas tags: sub-manifest | |
| 20:17 |
Delete the legacy reserved-name tier (consolidation Task 7)
The policy is three names and a prefix: .zit, .zit-checkout, .zit-cache, .zit-tmp-*. Removed: the absorbed sibling reservations (.zit-managed/-undo/ -bisect), the pre-directory cache/vstat spellings and their read tolerance, the retired scratch spellings, and the import-provenance names (refs, .zit-import-map.jsonl). Behavior shift: in never-re-opened old checkouts these leftovers are now visible to extras and deletable by clean. The container's byte-1 fold stays, so a legacy authoritative sibling still migrates losslessly on first contact (e2e: every-leftover checkout converges to the clean root with membership intact). check-in: c0a116d458 user: dmestas tags: sub-manifest | |
| 19:31 |
Fold managed/undo/bisect into the .zit-checkout container v2 (consolidation Task 6)
Sectioned container (ZITCKOT, tagged table, whole-file CRC); payloads are the existing formats unchanged. Byte-1 dispatch migrates legacy checkouts on first write; siblings dead once a container exists. Undo/redo restore and the commit/merge record+membership writes each collapse to one atomic splice. DST gains a legacy-seeded migration sweep (full-legacy or full-container at every crash cut). Registry: the three names move to the legacy tier. Perf on pkgsrc: adjacent update 6.50s (was 6.8), no-op 5.14s (was 7.69); migration of a 66k-file checkout invisible inside one update. check-in: efa29d234f user: dmestas tags: sub-manifest | |
| 17:25 |
control files: registry, provenance opt-in, one scratch prefix, advisory dir
Tasks 1-5 of the control-file consolidation plan. The checkout root goes from twelve zit-owned names to six live ones; the authoritative-tier container (task 6) and legacy removal (task 7) are deliberately not in this change. Registry: control_paths now carries every reserved root name with its tier -- authoritative, advisory, scratch, legacy -- and is the single authority the case table, the extras exclusions and the ignore-glob coverage pin all derive from or test against. A name added without its glob line or case coverage is a test failure, proven by mutation. The pin caught two live gaps on arrival: a crashed update's transaction directory had no ignore-glob line, and a crashed import's staging directory was never reserved at all. import-git --provenance: the per-object map and the source ref manifest are now instrumentation, published only on request. The default destination is the repository and its checkout record -- fossil's own shape. Both files are still written and verified in staging every run; without the flag they leave with the raw generation, and the staging manifest check enforces whichever set was chosen. Scratch: one prefix. .zit-tmp-txn- (update transactions), .zit-tmp-import- (import staging), .zit-tmp-pack (pack scratch; name only, the truncate-reuse and rewrite-exclusion lock protocol are untouched). One registry prefix row and one glob line cover the tier including names that do not exist yet. Advisory: .zit-cache is a directory -- repo (derived-state cache) and vstat (materialization plan), with the tree-checkpoint's member reserved. A legacy cache FILE is served through a read fallback and migrated by the first write via rename through a scratch name: the derivation survives with no bytes copied and no re-derive. Verified on a 66k-file pkgsrc checkout: identical SHA-256 across migration of a 35.8MB cache, fallback reads at parity, no-op update 7.69s vs 7.74s. Legacy spellings of every renamed or relocated name stay reserved and ignore-globbed until the removal release. check-in: f1ac625093 user: dmestas tags: sub-manifest | |
| 13:41 |
update: cost tracks the diff, not the width
Three fixes, each found by measuring the previous one's residue. markUnchanged: a path byte-identical between baseline and target -- same digest, same mode, with requireClean having already proven the on-disk file matches -- is skipped by every mutation phase: staging both directions, the undo journal, deletion and rewrite. An update used to delete and rewrite the entire tree regardless of how little changed. .zit-vstat: the stat sidecar written at materialization is now the BASELINE PLAN -- version-bound, CRC'd, carrying mode and plan order -- so requireClean rebuilds the baseline with zero tree flattens when it describes the current check-in, and proves most files clean by inode/size/mtime instead of reconstructing their content out of the log. Advisory throughout: any mismatch, staleness or corruption falls back to the flatten and the content compare, which stay authoritative. ManagedSet.fromSortedPaths: building the 66k-entry managed set through insert was quadratic -- an O(n) sorted-position memmove plus the LIVE sortedUnique assert per call -- and measured at 23.9 SECONDS of a pkgsrc update, larger than every real phase combined. The plan's own order is already sorted and unique, so the set is now built in one pass, order enforced per adjacent pair, invariant asserted once. Measured, same store and targets as #52's gate: sqlite (2.2k files): 117.8s flat -> 0.6s adjacent / 1.2s distant pkgsrc (66k files): 42.2s flat -> 6.8s adjacent / 8.9s distant pkgsrc user CPU: ~25s -> 1.3s Residue is stat I/O (two ~66k-file sweeps), not computation. The lockstep flatten prune #52 proposed was never built: the flatten measured 178ms at 66k files once the real costs fell. Tests: dirty AND delete of a skipped-diff file both refuse with the tree untouched; sidecar intact/absent/rebound-to-wrong-check-in all produce identical trees; an unchanged file provably keeps its inode; an updated tree equals a from-scratch materialization. Closes #52. check-in: 8797f53546 user: dmestas tags: sub-manifest | |
| 01:23 |
repocache: extend across appends; import warms at completion
The derived-state cache was all-or-nothing: any log append was a total miss, and the next read re-derived the whole store. On a real 1M-artifact pkgsrc store that was 352s per cold read, re-paid after every commit. Repository.open's miss path now distinguishes growth from rewrite. On growth it replays the cached record body (parse-free), folds only the tail frames through the same crosslink fold a cold derive uses, and rewrites the cache: every append healed by the next read at O(tail). Measured on pkgsrc: 352.31s cold, 2.40s after a commit, output byte-identical to a forced full re-derive. Sound because extension refuses whenever the fold could be wrong rather than stale: validatePrefix proves the old tail frame still verifies at its offset (a rewritten log cannot extend), and a shun fingerprint in the cache header forces the full derive on any shun-table change, since the body was distilled under the old mask. Cache format v2; a v1 cache misses once and rebuilds. import-git now warms the cache at completion, after the atomic rename, against the published store -- clone has done this since warmCacheFromLog existed, so an import-built store no longer defers the whole derive onto its first reader. Tests: extension equals cold derive with a fork and control tag in the tail; a forged-cache canary proving the fold starts from the stored body; shun change takes the full derive with the mask applied. Also backs out 1a4b74f2d5 (parallel staging): measured ~3% on a corpus where staging is ~5s of 149s -- threading complexity the number never justified. Closes #51. check-in: 84d3fe8f32 user: dmestas tags: sub-manifest | |
|
2026-08-09
| ||
| 16:32 |
update: stage plan entries in parallel
Staging reconstructs and writes every file in a plan -- one decompress, delta-chain application and hash verification each -- and on a wide repository that is tens of thousands of files for a single open or update. The work is per-file independent, and the store's read path is offset-addressed pread over an index immutable for the duration, so it parallelises without coordination. The one thing that cannot be written concurrently is last_stage_fault, a module global. Workers keep their own and the winner is chosen by ENTRY INDEX, which is exactly the fault sequential order would have raised, so this is observationally identical to the loop it replaces including which failure gets reported. Below 64 entries it stays sequential, and a thread that fails to spawn is not an error -- its share drains off the same cursor. Measured 2:33.5 to 2:28.8 on a 299 MB sqlite store, about 3%, with CPU never leaving 101%. So staging is NOT the dominant term there: the flatten's per-node reconstruction is, which this does not touch. Committed so it can be measured at pkgsrc scale, where the file count is 27x higher and the size distribution is different. If it does not move that number either, revert it -- it buys complexity for 3%. check-in: 1a4b74f2d5 user: dmestas tags: sub-manifest | |
| 04:46 |
import-git: sub-manifest mode drops on the pair proof, not on reproduction
A POLICY CHANGE, not a bug fix. Gates 5 to 7 prove the mirror's manifest is byte-reproducible from the check-in this import builds. Flat mode claims that; sub-manifest mode restructures the file set behind S cards, so its check-in is a different artifact by construction and claims nothing of the kind. Refusing to release the metadata because a mode failed to reproduce something it never promised is a category error. What makes the drop safe is gate 3 -- the bytes hash to the name a sibling manifest.uuid records, which a project file merely called 'manifest' cannot satisfy. That gate is untouched and still catches the look-alike. Gate 8 still guards the tags member. The B-card and S-card shape checks move with gates 5 to 7: a delta manifest is a real check-in, and refusing it only ever mattered because gate 5 needs a complete F run to compare against. On a sound source the win is small: bench measures 180 retained on sqlite, and this takes that to about 1. The win is on sources holding shunned or phantom content, where the origin's manifests name bytes the origin no longer has and no export can reproduce them -- permanent stranding today. libfossil goes from 293 of 3819 commits validated to 3817, and its store from 35.0 MB to 30.2 MB. Costs, stated rather than discovered later: this retires the only signal that a source repository is incomplete, and it leaves mirrorTreeMatches's tree arm unreachable from classification, annotated in place with the condition for deleting it. check-in: 063a66e4c5 user: dmestas tags: sub-manifest | |
|
2026-08-08
| ||
| 20:53 |
import-git: mint the D card from a proven mirror manifest
Fossil's git export derives the git commit time by truncating a floating-point Julian Day, so a whole-second origin timestamp lands a second early about half the time. Minting from commit.committer meant zit built those check-ins at a provably different instant than the origin, their manifests were never redundant, and their mirror metadata was kept forever: 4533 of 36383 commits on a real sqlite mirror. Where gate 3 has proved a mirror manifest authentic and gate 4 parsed it, the D card is now that manifest's own. Per commit, only where the proof succeeded -- an unproven mirror is never trusted. On the sqlite corpus timestamp_mismatch goes 4533 to 0 and the sha1-named era drops 18373 of 18392. Not under an explicit --attribution author: that flag asks for the author's instants, and adopting the origin's would leave it doing nothing on most commits while still appearing set. Those imports drop nothing, stated rather than silent. Gate 7 also compared against the committer unconditionally while minting the attributed identity, so under --attribution author it could drop a manifest on a proof about a check-in that was never built. It now compares against the instant actually minted. check-in: 7495b2e141 user: dmestas tags: sub-manifest | |
| 18:46 |
import-git: report blobs whose bytes are check-ins; pin sterilizer position
A tracked file can be a complete check-in manifest -- a vendored fossil source tarball is the ordinary route, since fossil stopped sterilizing archive exports in 2.18. The import stores it verbatim rather than mutating it the way fossil's own git import does: the file the user committed should be the file we store, and open already refuses to materialize a check-in whose content is absent. But the operator should hear it at import, not at the first open, so such blobs are counted and reported. Two bytes reject ordinary content before anything is parsed. The --manifest e2e now pins the marker's POSITION, not just its presence: it follows the final card and is not line 1. A refactor moving it to line 1 would still sterilize and would still pass every previous assertion, while silently diverging from fossil and breaking the mirror strip. Errata for artifact-format 4.1.2 filed separately; the draft states the wrong position, the wrong rejection gate, and a literal no fossil code path emits. check-in: 12ce319d18 user: dmestas tags: sub-manifest | |
| 18:07 |
import-git: strip the sterilizer where fossil actually writes it
sterilize_manifest() appends its marker after the last card. On an ordinary manifest that is the end of the blob, but on a PGP-clearsigned one the cards are followed by the signature block and the marker lands immediately before it. The strip only handled the trailing case, so a signed check-in's mirror bytes never hashed to the origin's recorded name, the pair proof refused it, and its manifest/manifest.uuid/ manifest.tags were retained permanently. On a real sqlite mirror that is 1076 check-ins, all 2009-2011. With the splice every one of the corpus's 36383 commits proves: 35307 by the trailing strip, 1076 only by this. Deliberately narrow -- only the two positions fossil writes. A marker anywhere else is still left alone, so a blob that merely contains the line is not reshaped into one that proves. check-in: a161ee719a user: dmestas tags: sub-manifest | |
| 17:40 |
import-git: name artifacts SHA3-256, not SHA-1
The algorithm was a hardcoded .sha1 that mis-named every import from a SHA3-256 Fossil, in both output modes. Sub-manifest mode reproduces nothing by construction, so it has nothing to inherit; flat mode is meant to inherit its source's algorithm, but that is unimplemented and sqlite shows it cannot be one choice anyway -- its history spans the promotion ratchet, so a single manifest there names both 40- and 64-hex blobs. SHA3-256 is the correct answer for a source whose algorithm cannot be derived, and the only defensible one for new addresses after SHATTERED. Now one constant, with the store's default policy and the submanifest k-analysis derived from it rather than guessed. Four name-verification sites in the storage layer used the same constant and rejected every SHA3 name; they now use artifact.verify, which reads the algorithm off the name's own width and cannot go stale. Two test repairs the change exposed: the ISSUE #24 e2e compared parent and child check-in names that both alias one readCheckoutState buffer, passing on the length difference alone, and that fixture no longer produces a mixed-width store -- noted inline, it needs its own fixture. Hashing is ~2.9x slower than SHA-1 on Apple Silicon, which lengthens hash-bound work. check-in: 51afeef206 user: dmestas tags: sub-manifest | |
| 13:42 |
e2e: pin the foreign-check-in refusal end to end
Two CLI-level fixtures for the class behind #42, differing only in their T cards, both dated 2099 so mtime cannot explain the difference: - self-tagged: carries trunk's tag pair, wins the tip, and open must refuse with exit 1, name the class, and leave the tree untouched. - untagged: a leaf and visible in 'zit leaves', but never resolvable, so open succeeds on the real tip. Also repairs assertions the earlier trunk-relabel and open-diagnostic work left stale in the two opt-in e2e steps, which 'zig build test' does not run: branch topology now maps the source's main line to trunk, and open's untracked-file diagnostic includes the resolved check-in line. Closes #44. check-in: 33d9286c19 user: dmestas tags: sub-manifest | |
| 13:02 |
import-git: skip refs outside refs/ instead of aborting the import
git fast-export --all emits a 'reset worktrees/NAME/HEAD' pseudo-ref for every detached-HEAD linked worktree. canonicalRef correctly refuses it, but that refusal unwound the whole import, so one worktree in the source aborted a complete history after all the work was done. Skip it, counted as skipped_foreign_refs. The skip is on the refs/ prefix alone; canonicalRef's traversal refusal still aborts. Fixes #45. check-in: 31346e784b user: dmestas tags: sub-manifest | |
| 00:22 |
import-git: persist a HashPolicy, not a HashAlgo ordinal
The header byte and the config field are both HashPolicy values. The import wrote @intFromEnum(HashAlgo.sha1) into each. The enums overlap numerically and mean different things -- HashAlgo is {sha1=0, sha3_256=1}, HashPolicy is {sha1=0, auto=1, sha3=2, ...} -- so every imported repository advertised policy sha1 by coincidence of ordinal, and the obvious 'switch it to sha3_256' repair would have written 1 = auto, silently arming 7.4's promotion ratchet. Now derived through artifact.defaultPolicy rather than written as a literal. The import mints SHA-1 names throughout, so 7.2's rule gives auto. When the naming algorithm learns to follow its source the predicate changes and the derivation does not. Header byte on a fresh import: 00 to 01. check-in: 4efd2e36d2 user: dmestas tags: sub-manifest | |
| 00:14 |
import-git: a child's parent-proof needs identity, not its parent's other gates
mirrorParentProven required the parent's validated_uuid, set only when a commit FULLY dropped. So proving commit N's parent meant commit N-1 had to clear every gate for itself -- tree shape, timestamp, manifest shape -- none of which bear on whether its UUID is correct. One retain anywhere permanently poisoned every descendant. On a real 36,432-commit sqlite mirror that produced zero drops and 36,252 parent_unproven retentions: its 2000-05-29 genesis commit fails the timestamp gate, and essentially all of history descends from it. Gate 3 already proves identity per commit and cryptographically. That proof is now recorded as attested_uuid whether or not the commit drops, and the parent proof accepts either. Drop eligibility is unchanged -- attestation governs only what a CHILD may rely on. The existing test asserted the defect as intent, comment and all; it now asserts the corrected behaviour. (#43) check-in: e7fd62ec4d user: dmestas tags: sub-manifest | |
|
2026-08-07
| ||
| 21:50 |
update: pin the fail-closed refusal of a foreign check-in
Regression tests for issue 42. A blob whose bytes are a valid check-in manifest IS a check-in -- structure types artifacts, and repo-state 4.4 exempts only what a parser declines to classify -- so a repository that tracks such a file holds check-ins of another repository. That derivation is required; what must not regress is the refusal. Asserts materialization fails with FileContentMissing, names the file, and leaves nothing written -- the fail-closed property stated as a test so it cannot be quietly traded later. Also pins that the diagnostic explains the class and never alleges corruption, since the store is intact and verify says so. Records the T-card grammar that makes such a manifest resolvable: 'T *branch * trunk' self-targets, while 'T *branch trunk' puts a branch name in the UUID field and the manifest is rejected entirely. (#42) check-in: 0369e62f19 user: dmestas tags: sub-manifest | |
| 21:42 |
open/update: refuse a tip whose content this repository never held
The format types artifacts by structure, and repo-state 4.4 exempts only what a parser declines to classify -- so a stored blob whose bytes are a valid check-in manifest IS a check-in, joins the DAG, and may carry its own branch tags. A repository tracking such a file (a Fossil-to-Git mirror's 'manifest' is the common source) therefore holds check-ins of another repository, whose blobs it never had. That derivation is conformant; Fossil does the same. What was not acceptable was spending minutes materializing before finding out, then reporting it as a missing file with no way to see why. open now presence-checks the resolved tip's file set before invoking the engine and refuses naming the class; update prints the same explanation. Advisory only: a files() fault falls through, and stagePlan remains the sole authority, so this cannot let anything through -- it can only refuse earlier. Verified against a foreign self-tagged manifest that genuinely hijacks trunk: exit 1, working tree untouched. (#42) check-in: a81e9a8745 user: dmestas tags: sub-manifest | |
| 21:10 |
update: cover the legacy trailing-separator plan path
parseForImport normalizes a fossil-origin F card into a fresh buffer and the manifest borrows that copy, so every F-card hash the plan reads points into it rather than into the stored bytes. Nothing exercised it: every synthetic corpus here is git-native and never carries the separator, so normalized is null throughout the suite. The path is correct -- the test passes -- but it was covered only by repositories nobody runs the tests against. (#42) check-in: 15db533ea7 user: dmestas tags: sub-manifest | |
| 20:57 |
open: name the check-in it resolved when materialization fails
open picks its own target -- the tip of the configured main branch -- rather than being handed one, so a failure that omits the check-in cannot distinguish materializing the expected one going wrong from resolving a different one than the operator believes is the tip. There was no other way to see which. (#42) check-in: da081a3eca user: dmestas tags: sub-manifest | |
| 18:59 |
update: copy the staging fault's path instead of borrowing freed memory
StageFault held a slice into PlanEntry.path, documented as borrowing a plan that outlives the call. It does not: applyCheckin frees the plan from its own defer on the way out, so the caller rendered freed heap -- ten bytes of garbage where a filename belonged, in the very diagnostic added to make this failure legible. Path is copied now, as the artifact name already was. Adds the end-to-end test that would have caught it: the earlier test hand-built a fault and so never exercised the lifetime that was wrong. check-in: 41dc7067d8 user: dmestas tags: sub-manifest | |
| 18:20 |
update: name the file staging cannot materialize
Three separate sites printed the identical 'cannot read repository: repository is corrupt' -- deriving the view, reopening the log, and applyCheckin. So a materialization failure was indistinguishable from a failure to read the repository, which is a different claim about a different subsystem. stagePlan's three refusals now return FileContentMissing carrying the file path and the artifact name: a file the check-in references has no artifact here, or has none stored, or the stored bytes do not match the name. verify cannot surface any of these -- it checks that stored artifacts are valid, never that referenced ones exist. Being in the closed ApplyError set, the compiler required all four verb mappers to handle it. (#42) check-in: 0b7b4af7fe user: dmestas tags: sub-manifest | |
| 17:49 |
repoview: name the artifact a failed derive rejected
'repository is corrupt' says a repository cannot be read, not what in it is unreadable -- and the walk that discovers it visits every artifact in the store, minutes on a large one. Whoever holds the broken repository was left bisecting it or attaching a debugger. The derive now records which artifact it rejected and why; openRepo reports it, so every verb's open gets the name. Plain data, no allocation, so it survives the errdefer that tears down a half-built view. Repository.open clears it per attempt because callers declare a Repository undefined. Also splits open's two identical diagnostics: deriving the view and reopening the log for materialization both printed the same line, which made a failure in one indistinguishable from the other. (#42) check-in: 37f746e59f user: dmestas tags: sub-manifest | |
| 17:17 |
pack cache: derive the entry cap from the byte budget
A real 53,773-commit import evicted 99,097 times, every one forced by the 4096-entry cap and not one by the 64 MiB byte cap, with the resident set at 23,551 KiB. The cache was held to a third of its own stated size by a second, unrelated number, and the 51 percent hit rate that produced looked like a working-set problem rather than an arithmetic accident. The entry cap now divides the byte budget by a planning entry size, so the two cannot disagree about which one is bounding: bytes bind for normal data, entries only for many tiny items, which is what an entry cap is for. (#40) check-in: ac89bad5fd user: dmestas tags: sub-manifest | |
| 15:29 |
import-git: name the main line trunk, whatever git called it
A git default branch is master or main; this format's is trunk, and so is every place that must name the main line without being told -- a fresh checkout, zit open resolving a tip, a clone initialising its own state. A repository whose main line was called master satisfied none of them: it cloned and verified clean, then could not be opened, because the branch the destination looked for held no commits. labelBranches now takes the name to MATCH on and the name to APPLY separately: the head is still found by its source name, the segment is labelled trunk. Side branches keep their source names, and the git-side refs mapping still records sym-heads/master, so nothing about round-trip identity moves. If a side branch already publishes trunk, the source name is kept rather than colliding two segments under one label. (#42) check-in: 5caf866d86 user: dmestas tags: sub-manifest | |
| 13:03 |
storage: a block cache for FileStorage reads, opt-in and not yet wired
Profiling a real pkgsrc-scale import put readv at over half of all CPU samples. A block cache in front of pread is the shape that addresses it: the store is append-only, so a block is retained only when the read that filled it came back FULL -- which proves those bytes already existed, so appends cannot alter them and need no invalidation. Only truncate does. Deliberately not attached to any caller. Wired into import-git it reached a 99 percent hit rate over 392,421 reads and did NOT make a small corpus faster: at that size the page cache already served those reads, so the syscalls were cheap and the lookup was pure overhead. The prior from the real profile is strong but unproven at scale, and shipping it on 'should help' is how the reverted top-level cache in issue 32 happened. useReadCache is the one-line opt-in for profiling it. (#39) check-in: f3becbedd1 user: dmestas tags: sub-manifest | |
| 11:34 |
pack cache: name WHICH budget forced each eviction
A real 53,773-commit run reported 99,097 evictions against 69,485 lookups -- evicting more than it was asked about. But two caps can force that, max_bytes and max_entries, and they bind under different data, so the count alone said only 'raise something'. Evictions are now split by cause and the resident set is reported, so a run names the budget to raise instead of requiring a blind sweep. (#40) check-in: 88ae8ba619 user: dmestas tags: sub-manifest | |