Artifact 448b0cac2e2e1badcf51358d5fd6f785e563e96aed1012abc062f5d5e7cd536b:
- File
docs/site/content/docs/guides/testing.md
— part of check-in
[b451efa274]
at
2026-10-31 16:23:44
on branch main
— docs: the libfossil_noslog tag, and how the publisher engine gets smaller still
The TinyGo section of the install guide and examples/tinygo described libfossil_nohttp but not libfossil_noslog, which only the changelog mentioned. Both now say what the tag leaves out and what changes.
The install guide gains a "Smaller still" section pointing at the html-artifact-publisher engine's build: no scheduler, a reactor export, explicit panic unwinding and a patched standard library, with the behavior the patch changes. It says plainly that these steps are the publisher's and that this repository does not build that way.
CI never compiled the code behind either tag, so a change could break a tagged embedder unnoticed until the publisher's CI ran against a release. A new build-size-tags job builds the library under both tags, vets the repository under libfossil_noslog, and runs the check-in linker's tests under it. (user: [email protected], size: 14277) [annotate] [blame] [check-ins using] [more...]
--- title: Testing weight: 50 aliases: ["/docs/testing/"] --- This guide describes libfossil's test strategy for contributors adding tests or investigating failures. For the module layout referenced throughout, see [Architecture](/docs/concepts/architecture/). ## Overview libfossil's test suite has these layers: 1. **Unit tests** — standard `go test ./...` coverage for individual packages under the root module and `internal/...`. 2. **Conformance tests** — `./conformance`, which checks the library against the Fossil Internet-Drafts in `conformance/rfcs/`, requirement by requirement, and against real `fossil` and `zit` binaries where they are present. See [Conformance](/docs/reference/conformance/). 3. **DST simulation tests** — deterministic simulation scenarios under `./dst/...` that drive multiple repos through a virtual clock with fault injection and invariant checks. 4. **Fuzz targets** — Go native fuzz tests for the parsers and codecs. 5. **Dual-driver matrix** — the whole suite runs against two SQLite drivers (`modernc` and `ncruces`) to catch driver-specific behavior. ## Running tests The `Makefile` wraps the common targets: - `make test` — runs `go test ./... -count=1 -timeout=120s` with the default `modernc` driver. - `make test-drivers` — runs the full suite twice: `go test ./... -count=1 -timeout=120s` followed by `go test -tags test_ncruces ./... -count=1 -timeout=120s`. - `make test-otel` — runs the out-of-workspace OTel submodule with `cd observer/otel && GOWORK=off go test ./... -count=1`. - `make test-all` — chains `test-drivers`, `test-otel`, and `go build ./cmd/libfossil/`. - `make ci` — mirrors `.github/workflows/test.yml`. It uses `GOWORK=off`, runs the packages other than `dst` with a 120-second timeout, runs `./dst/...` with a 300-second timeout, runs the driver submodule tests, `go vet` and the CLI build, then repeats the tests under `-tags test_ncruces` and runs the OTel submodule. Run it before opening a PR. `make release VERSION=vX.Y.Z` runs it before it tags. `make setup-hooks` points `core.hooksPath` at `.githooks/`. The installed pre-commit hook runs both drivers with `-short`, `go vet`, the OTel submodule, the CLI build, and a check that the generated SDK reference pages are in sync, in roughly 45 seconds. Skip with `git commit --no-verify` only in emergencies. Tests that compare against real binaries skip when the binary is missing. `REQUIRE_FOSSIL_BIN=1` turns a missing `fossil` into a failure, and CI sets it. `fossil` keeps a global config in `$HOME`, so run those tests with a scratch `HOME` if you do not want your own touched. ## Fuzzing There are seven fuzz targets. Each is also a regular test over its seed corpus, so `go test` runs the seeds without `-fuzz`. | Target | Package | Exercises | |---|---|---| | `FuzzUnified` | `internal/diff` | Unified diff output. | | `FuzzTableSyncPayload` | `internal/sync` | Extension-table sync payload parsing. | | `FuzzParseXDelete` | `internal/xfer` | The `xdelete` card parser. | | `FuzzDecode` | `internal/xfer` | The xfer message decoder and its size limits. | | `FuzzArtifactRoundTrip` | `conformance` | Artifact parse and re-encode. | | `FuzzDeltaApply` | `conformance` | Applying arbitrary deltas. | | `FuzzDeltaRoundTrip` | `conformance` | Delta create then apply. | Run one with, for example: ``` go test ./conformance -run '^$' -fuzz FuzzDeltaApply -fuzztime 60s ``` CI does not run `-fuzz`, and no Makefile target or script runs the fuzz targets or a seed sweep. They are run by hand. ## Corpus parity gate `TestCloneCorpusParity` (`apitest/` package, `apitest/clone_corpus_parity_test.go`) clones one repository two ways — with the canonical `fossil` binary and with `libfossil.Clone` — and compares what each derived. It is opt-in: ``` LIBFOSSIL_CORPUS_PARITY=1 go test -run TestCloneCorpusParity -v . ``` Without the variable it skips, so `make test` is unchanged in both runtime and outcome. It also needs a fossil binary and skips without one (or fails, under `REQUIRE_FOSSIL_BIN=1`). Budget roughly a second and under a megabyte of temp space; everything lands in `t.TempDir()` and is removed on exit. It exists because the per-package parity tests compare our derivation against fossil's for fixtures built in-test, which only covers the shapes someone thought to build. Issue #184 — a silent 100% loss of the ticket-change artifact shape — passed that suite and was found by an external benchmark instead. The gate asserts two things: - **Per-`event`-type counts match**, not just the total. This distinction is the point: on the real Fossil repository the totals read 23,396 vs 27,099, a 14% shortfall easy to wave off, while the per-type breakdown put ticket events at 96 vs 3,808. - **Row totals match for `blob`, `event`, `mlink`, `plink`, `tagxref` and `leaf`**, reported in both directions. A surplus matters as much as a shortfall — our `tagxref` ran +17,119 over fossil's on the real repository, which is what writing a tag without finishing the work it marks looks like. Failures name the table or type and the delta, e.g. `event type "t": got 0, want 10 (-10)`. The corpus is built by the canonical binary at test time rather than downloaded: seconds and kilobytes instead of the ~10 minutes and ~1 GB the real Fossil repository costs, no network dependency, and contents that are a property of the test rather than of whatever a remote server holds today. It covers check-ins (including a branch and a merge), ticket creations and updates, a wiki page, a technote, an attachment, and a tag control artifact. `requireCorpusShapes` classifies every blob by its card set and fails if any shape falls below its expected floor. Without that check a corpus that quietly stopped producing ticket artifacts would let the gate pass while covering nothing — the exact blind spot it exists to remove. Not covered: forum posts (`f` events), because fossil exposes no CLI to create them; the scale-dependent behaviour of a large repository; and per-row content comparison — this gate counts rows, and `internal/manifest` diffs them. ## Driver matrix libfossil abstracts SQLite behind `db/` and two driver submodules. Both must pass identical tests: - **`modernc`** (default) — pure-Go port of SQLite. Used by every build that doesn't set a tag. Registered via `internal/testdriver/modernc.go` under `//go:build !test_ncruces && !test_mattn`. - **`ncruces`** (wasm-capable) — selected with `-tags test_ncruces`, suitable for WASM targets. Registered via `internal/testdriver/ncruces.go` under `//go:build test_ncruces`. GitHub Actions (`.github/workflows/test.yml`) runs `test`, `test-ncruces`, `build-wasm`, `build-tinygo`, `build-size-tags`, `test-otel` and `test-docs` as parallel jobs. The first two install the pinned fossil 2.28, build every package, then run the unit tests, DST and the driver submodule tests under modernc and ncruces respectively. `build-wasm` compiles the packages for `js/wasm` with the ncruces tag, `build-tinygo` builds `examples/tinygo` with TinyGo 0.43, `build-size-tags` builds the library under `libfossil_nohttp` and `libfossil_noslog` and tests the check-in linker under `libfossil_noslog`, and `test-docs` builds this site with Hugo. For the canonical local invocation, use `make test-drivers` or `make ci` rather than hand-rolling tag flags. ## DST (Deterministic Simulation Testing) DST is inspired by FoundationDB and TigerBeetle: run many repos in a single process under a simulated clock, inject faults, and replay a failing seed deterministically. The harness lives in `dst/simulator.go` and `dst/invariants.go`. ### Seeds and reproducibility `dst.SimConfig.Seed` drives every random source used by the simulator: the event queue, network fault decisions, and a `SeededBuggify` PRNG that gates fault-injection sites. Any failure on seed N reproduces exactly on seed N. `dst/scenario_test.go` exposes three flags driving `TestDST`: ``` go test ./dst -run 'TestDST$' -seed=42 -level=hostile -steps=10000 ``` `TestDST` skips unless at least one flag is set. It builds a master repo with 50 artifacts, 5 unversioned files and 3 private blobs, starts a simulator with 3 leaves, and checks safety invariants as it runs. - `-seed=<int64>` — seed passed to `SimConfig.Seed`. In `TestDST`, 0 means seed 1; other scenarios use their own fixed seeds. - `-level={normal,adversarial,hostile}` — picks a `severity` struct setting `DropRate` and `Buggify`: `normal` is no drops and no BUGGIFY, `adversarial` is 10% drops with BUGGIFY, `hostile` is 20% drops with BUGGIFY. An empty or unknown level is `normal`. - `-steps=<int>` — caps the number of events processed (default 10000 in `TestDST`). Seed-parameterized scenarios (e.g. `TestCloneDSTSeedSweep` in `dst/clone_test.go`, which runs seeds 0 to 19) loop over seeds via `t.Run(fmt.Sprintf("seed_%d", seed), ...)`. To reproduce a single sub-seed: `go test ./dst -run TestCloneDSTSeedSweep/seed_7 -v`. ### Invariants `dst/invariants.go` defines per-node and cross-node checks. `Simulator.CheckSafety` runs the per-node set after every `SafetyCheckInterval` steps; convergence checks run once the sim has quiesced: - **Safety (anytime):** `CheckBlobIntegrity` (UUID matches hash of expanded content), `CheckDeltaChains` (`delta.srcid` resolves to a blob), `CheckNoOrphanPhantoms`, `CheckUVIntegrity`, `CheckTableSyncIntegrity`. `CheckTagxrefIntegrity` is a separate check that scenarios call directly. - **Convergence (after fault-free sync):** `CheckConvergence` (leaves match master blob-for-blob), `CheckSubsetOf`, `CheckUVConvergence`, `CheckTableSyncConvergence`, `CheckTombstoneConvergence`. Failures return a `*InvariantError{Invariant, NodeID, Detail}` naming the failing invariant and the offending node. The scenarios add the seed and level to the failure message, so a log line is enough to replay the run. The checks run with BUGGIFY suspended, so they see what is persisted and not faults injected into their own reads. ## BUGGIFY BUGGIFY marks code points where faults can be injected — shrunk send budgets, dropped or rejected cards, bad nonces, byte-flipped content. About twenty sites exist in `internal/sync`, `internal/blob` and `internal/content`. In production the check is a single boolean read; under DST the seeded PRNG decides whether to fire. Two styles appear in the tree: 1. **Process-global, used outside the sync path.** `simio.Buggify(probability)` consults a process-global state toggled by `simio.EnableBuggify(seed)` / `DisableBuggify()` (see `simio/buggify.go`). Example from `internal/content/content.go`: ```go // BUGGIFY: flip a byte in expanded content to exercise the hash check // below under DST. if check && simio.Buggify(0.01) && len(content) > 0 { corrupted := make([]byte, len(content)) copy(corrupted, content) corrupted[0] ^= 0xFF content = corrupted verified = false } ``` 2. **Scoped, passed through session options.** `BuggifyChecker` is an interface (`Check(site string, probability float64) bool`) defined in `internal/sync/session.go` and re-exported from the root as `libfossil.BuggifyChecker`. Sync code threads a checker through so each site has a stable name: ```go // BUGGIFY: drop the last file card to simulate partial send. if s.opts.Buggify != nil && s.opts.Buggify.Check("sync.buildFileCards.skip", 0.05) && len(cards) > 0 { cards = cards[:len(cards)-1] } ``` `dst.SeededBuggify` in `dst/simulator.go` implements `BuggifyChecker` with a seeded `*rand.Rand` (it ignores the site name). `Simulator.New` enables both styles when `SimConfig.Buggify` is true, wiring independent PRNG streams into the client, the server-side `MockFossil`, and the global `simio` state. ## Interop with fossil 2.28 and zit Two other implementations check libfossil's output. **Fossil.** CI downloads the official fossil 2.28 release tarball (`FOSSIL_VERSION` in `.github/workflows/test.yml`) and verifies its SHA-256 and its reported version, so every run compares against the same build. Tests in `conformance/`, `cli/`, `apitest/` and the root package create repositories with stock `fossil`, have it open and verify what libfossil wrote, and compare the rows libfossil derives with the rows fossil derives. Locally they use the `fossil` on `PATH`; `fossil version` should report 2.28. **zit.** zit is a Zig reimplementation of Fossil. The `TestZit*` tests in `conformance/*zit*_test.go` (20 of them) cover the newer drafts: sub-manifest trees (`zit_interop_test.go`, `zit_writer_test.go`), sparse cones and sparse stores, and horizons (serve, structural seeds, fallback, push check, cursor, nodes, history URL and merge base). Each runs both ways where it can: go-libfossil clones from and pushes to `zit serve`, and zit clones from and pushes to go-libfossil's handler. They need a `zit` on `PATH` (or `ZIT_BIN`) at a recent enough commit and skip otherwise. `REQUIRE_ZIT_BIN=1` makes a missing or older zit a failure. CI has no zit, so there the bundle tests (`TestZitTreeBundle`, `TestZitWriterBundle`) read recorded copies of what zit wrote from `conformance/testdata/`. `ZIT_REGENERATE=1` rewrites those copies from a live zit. Signed check-ins are covered the same way: `TestSignedCheckin` has fossil and zit clone and read libfossil's signed check-ins, and checks that a check-in zit signs verifies in libfossil. ## Writing new tests - Prefer table-driven tests in unit code; include success and failure rows. - If you touch sync, merge, clone, or checkin, add a DST scenario under `dst/` and run it across a seed range (see `TestCloneDSTSeedSweep` for the pattern). - Mark any new fault-sensitive code point with BUGGIFY. Use the scoped `BuggifyChecker` style when the site lives inside the sync session; reserve `simio.Buggify` for code that runs outside an explicit session scope. - Assert invariants. If the new behavior warrants one, add it to `dst/invariants.go` and wire it into `CheckSafety` or a convergence helper. - Confirm tests pass under both drivers (`make test-drivers`) before requesting review.