| Comment: | perf: cache prepared statements in db.Querier to stop per-call SQL re-parse (#171)
The hot per-artifact queries in the phantom-fill crosslink cascade — the content/blob-by-rid lookup and the per-node orphan/delta-child lookups in afterDephantomize — re-tokenized, re-parsed, and re-compiled their SQL text on every call because db.Querier delegated straight to the raw driver each time with no statement reuse. On a large repository (67k+ artifacts) the SQLite parser/compiler dominated CPU during the slow clone stretch. Add a statement cache to both Querier implementations, keyed by exact query text, with lifetimes matched to each backing type: - *DB caches *sql.Stmt prepared from its long-lived *sql.DB pool (safe for concurrent use, manages its own connections). Statements are prepared lazily on first use, reused thereafter, and closed in DB.Close. - *Tx caches *sql.Stmt scoped to a single *sql.Tx; the cache is torn down (all statements closed) when the transaction commits or rolls back, so no statement outlives its transaction or leaks across a clone/sync session. Exec/QueryRow/Query transparently get-or-prepare the cached statement — no signature or caller-visible behavior change. QueryRow preserves database/sql's deferred-error contract by falling back to the raw path on prepare failure so the error still surfaces from Scan. Multi-statement strings (DDL setup batches) bypass the cache and run through the raw driver path, since a prepared statement compiles only one statement. Benchmark (Apple M4 Pro, modernc driver): the blob-by-rid lookup drops from 4779 ns/op to 2289 ns/op (~2.1x). Cache reuse, transaction-boundary cleanup, and multi-statement bypass are covered by new tests in db/stmtcache_test.go. |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA3-256: |
218ebaec74e8d4bef460008b03795e9d |
| User & Date: | [email protected] on 2026-07-26 00:03:00.000 |
| Other Links: | manifest | tags |
|
2026-07-26
| ||
| 13:15 |
ci: pin canonical fossil to 2.28 instead of the distro package (#173)
Both test jobs installed the canonical fossil binary with an unpinned `apt-get install -y fossil`. On the Ubuntu runner that resolves to 2.23, while all local real-fossil parity work is validated against 2.28 -- so CI was checking our output against a different reference implementation than development does. The distro package is not stable even at a fixed version string: it tracks different upstream check-ins over time, so two runs at nominally the same version can behave differently. Install the official upstream release tarball instead, pinned by version and by SHA-256 so it is a fixed artifact rather than a moving target. The version lives in one workflow-level env value both jobs reference, and a verification step asserts the installed binary actually reports it, so an upstream change fails loudly instead of drifting silently. This also explains the intermittent `table global_config already exists` failure recorded in #162. `global_config` is not part of the repository schema at all -- it is the only table in `~/.fossil`, fossil's per-user global config database, created lazily on first use. Under concurrent fossil invocations against a HOME that has no `~/.fossil` yet (a fresh CI runner, never a dev machine) 2.23 races on that CREATE TABLE; 2.28 does not. Reproduced at 24-way concurrency on a fresh HOME: 2.23 hit the error in 30/30 trials, 2.28 in 0/30, and 0/60 at 48-way. Our schema creation is not implicated. 1500 iterations of TestCreateRepoSchema_FossilValidation against 2.28 (500 in-process plus 200 separate processes per driver variant) and 100 full `./db/...` runs passed with zero failures, and 40 independently created schema artifacts across both drivers produced byte-identical DDL. Closes #162 check-in: 9ff6e07d74 user: [email protected] tags: trunk | |
| 00:03 |
perf: cache prepared statements in db.Querier to stop per-call SQL re-parse (#171)
The hot per-artifact queries in the phantom-fill crosslink cascade — the content/blob-by-rid lookup and the per-node orphan/delta-child lookups in afterDephantomize — re-tokenized, re-parsed, and re-compiled their SQL text on every call because db.Querier delegated straight to the raw driver each time with no statement reuse. On a large repository (67k+ artifacts) the SQLite parser/compiler dominated CPU during the slow clone stretch. Add a statement cache to both Querier implementations, keyed by exact query text, with lifetimes matched to each backing type: - *DB caches *sql.Stmt prepared from its long-lived *sql.DB pool (safe for concurrent use, manages its own connections). Statements are prepared lazily on first use, reused thereafter, and closed in DB.Close. - *Tx caches *sql.Stmt scoped to a single *sql.Tx; the cache is torn down (all statements closed) when the transaction commits or rolls back, so no statement outlives its transaction or leaks across a clone/sync session. Exec/QueryRow/Query transparently get-or-prepare the cached statement — no signature or caller-visible behavior change. QueryRow preserves database/sql's deferred-error contract by falling back to the raw path on prepare failure so the error still surfaces from Scan. Multi-statement strings (DDL setup batches) bypass the cache and run through the raw driver path, since a prepared statement compiles only one statement. Benchmark (Apple M4 Pro, modernc driver): the blob-by-rid lookup drops from 4779 ns/op to 2289 ns/op (~2.1x). Cache reuse, transaction-boundary cleanup, and multi-statement bypass are covered by new tests in db/stmtcache_test.go. check-in: 218ebaec74 user: [email protected] tags: trunk | |
|
2026-07-25
| ||
| 15:31 |
fix: thread sync context into the general push/pull phantom-fill cascade (#167) (#169)
storeReceivedFile's mid-round AfterDephantomize crosslink cascade became context-aware in #166, and the clone and server-push receive paths already pass their real deadline context into it. The third caller -- the general push/pull path -- did not: (s *session) handleFileCard hardcoded context.Background(), so an ordinary Sync could not interrupt a long-running crosslink cascade via its caller's deadline the way a Clone now can. Thread the sync loop's own context the one remaining hop: - processResponse gains a ctx parameter and passes it to handleFileCard. - handleFileCard gains a ctx parameter and hands it to storeReceivedFile instead of context.Background(). - Sync's round loop passes its ctx into processResponse. Existing processResponse callers in tests pass context.Background(), the convention already used elsewhere. A new regression test drives the general sync path (processResponse) with a context cancelled mid-cascade against a fanout large enough to cross crosslinkCancelCheckStride and asserts the cascade stops early rather than crosslinking every child. check-in: 5ef8deb561 user: [email protected] tags: trunk | |