Artifact 21e012af968aa776c4405e247db7aff9b6eaae9906a3c937dc6e8db235a8a994:
- File
internal/sync/projectcode_test.go
— part of check-in
[18bfbc166c]
at
2026-07-27 23:31:22
on branch trunk
— fix(sync): derive the project code, and write artifact visibility inside the storing transaction (#204)
* fix(sync): derive the project code from the repo instead of panicking (#199)
Supplying User/Password made SyncOpts.ProjectCode mandatory in practice: buildLoginCard passed the caller's raw value straight to computeLogin, which asserts it is non-empty, so an authenticated Sync on a repo whose project code the caller had not repeated took the process down. Nothing in the public API signalled that a password made the field required, and a server embedding the library died with it.
buildRequest already fell back to the repo's stored project-code for the push/pull cards; that fallback was local to one function, which is exactly how the login path drifted out of step with it. Resolve the code once when the session is built — caller value first, the repo's own config second — and let buildRequest and buildLoginCard both read it. A code that genuinely cannot be determined now returns a descriptive error from the site that needs it rather than panicking.
Sync's deferred result-not-nil assertion fired during panic unwinding, where err and result are both nil, replacing the original cause with its own message. The invariant only holds on a normal return, so it no longer asserts while a panic is in flight.
* fix(sync): write received artifacts' visibility inside the storing transaction (#200)
Receiving an artifact cost three write transactions, not one. The blob was stored in a transaction, its dephantomize cascade committed another, and then applyPrivateStatus re-derived the rid on the pool and issued a bare DELETE FROM private -- an implicit transaction of its own, with its own BEGIN IMMEDIATE and fsync, for a row that a transaction microseconds earlier had guaranteed did not exist. Every igot card announcing an artifact already held cost one more. Measured against a real fossil peer: 1509 statements executed outside a transaction to receive 1504 artifacts.
That is minutes of pure commit overhead on a large repository -- roughly 200,000 write transactions for a 67k-artifact pull -- and it is fragile as well as slow. Each of those writes contests the database's single write lock independently, so N artifacts means N chances to lose that contest and fail outright rather than one chance to wait at a single BEGIN.
Visibility now travels with the content it describes: storeReceivedFile takes the visibility a private card announced and records it inside the transaction that stores the blob, using the rid that transaction already has. The status is now atomic with the content rather than a separate write that could be interrupted between the two. The igot path buffers its updates and writes them once per round. The same three-transaction shape on the server's receive path goes with it.
The same real-fossil pull now executes 2 statements outside a transaction for those 1504 artifacts, down from 1509. db.DB.PoolExecs makes that budget assertable, which is how the regression test pins O(rounds) rather than O(artifacts) without needing a large corpus.
On the failure originally reported: the "database is locked" error came from a harness that copied a live SQLite database out from under a running server without its WAL, so that specific lock holder is not a product defect and this does not claim to fix it. The transaction storm above is a real performance and robustness defect on its own evidence, and is what this changes.
* test(sync): pin the transaction budget, not just writes outside a transaction
The budget tests measured PoolExecs, which counts statements issued outside a transaction. That pins one half of the defect and misses the other: moving a per-artifact write from the pool into its own WithTx leaves PoolExecs unchanged while every artifact still pays a BEGIN IMMEDIATE and an fsync, and still contests the single write lock on its own. That is the cost issue #200 actually measured -- roughly three write transactions per received file, about 203,000 for one clone-sized pull.
Verified by mutation: reverting the batched visibility flush to one transaction per update left both budget tests green. They now count transactions too, via a new db.DB.WriteTxns counter, and the same mutation fails with "began 51 write transactions for 50 igot cards over 1 rounds; want <= 4".
Also stop the contention test from starving the pull it is meant to observe. The rival connection re-acquired the write lock immediately on release, so on a loaded machine it could hold the pull's single BEGIN past busy_timeout and fail the test even when the batching under test is perfect -- which is what happened on CI. It now yields between bursts, modelling another process doing occasional writes rather than testing SQLite's fairness. (user: [email protected], size: 4358) [annotate] [blame] [check-ins using]
package sync import ( "context" "path/filepath" "strings" "testing" "github.com/danmestas/go-libfossil/internal/repo" _ "github.com/danmestas/go-libfossil/internal/testdriver" "github.com/danmestas/go-libfossil/internal/xfer" "github.com/danmestas/go-libfossil/simio" ) // newProjectCodeRepo creates a repo and returns it alongside its stored // project-code. func newProjectCodeRepo(t *testing.T) (*repo.Repo, string) { t.Helper() r, err := repo.Create(filepath.Join(t.TempDir(), "repo.fossil"), "testuser", simio.CryptoRand{}, "") if err != nil { t.Fatalf("repo.Create: %v", err) } t.Cleanup(func() { r.Close() }) code, err := r.Config("project-code") if err != nil { t.Fatalf("repo.Config(project-code): %v", err) } if code == "" { t.Fatal("repo.Create left project-code empty") } return r, code } // captureLogin runs one converged sync round and returns the login card the // client put on the wire (nil when it sent none). func captureLogin(t *testing.T, r *repo.Repo, opts SyncOpts) (*xfer.LoginCard, *SyncResult, error) { t.Helper() var login *xfer.LoginCard mt := &MockTransport{Handler: func(req *xfer.Message) *xfer.Message { for _, c := range req.Cards { if lc, ok := c.(*xfer.LoginCard); ok { login = lc } } return &xfer.Message{} }} res, err := Sync(context.Background(), r, mt, opts) return login, res, err } // A password does not make ProjectCode the caller's problem: the repo being // synced already stores its own project-code, so the login card derives from // it (issue #199). func TestSyncDerivesProjectCodeForLoginCard(t *testing.T) { r, code := newProjectCodeRepo(t) login, res, err := captureLogin(t, r, SyncOpts{Pull: true, User: "syncer", Password: "pw123"}) if err != nil { t.Fatalf("Sync: %v", err) } if res == nil { t.Fatal("Sync returned nil result on success") } if login == nil { t.Fatal("no login card sent") } want := sha1hex(login.Nonce + sha1hex(code+"/syncer/pw123")) if login.Signature != want { t.Fatalf("Signature = %q, want %q (derived from repo project-code %s)", login.Signature, want, code) } } // An explicitly supplied project code still wins over the derived one. func TestSyncExplicitProjectCodeWins(t *testing.T) { r, code := newProjectCodeRepo(t) const explicit = "0123456789abcdef0123456789abcdef01234567" if explicit == code { t.Fatal("test constant collides with the generated project-code") } login, _, err := captureLogin(t, r, SyncOpts{ Pull: true, User: "syncer", Password: "pw123", ProjectCode: explicit, }) if err != nil { t.Fatalf("Sync: %v", err) } if login == nil { t.Fatal("no login card sent") } want := sha1hex(login.Nonce + sha1hex(explicit+"/syncer/pw123")) if login.Signature != want { t.Fatalf("Signature = %q, want %q (explicit project code)", login.Signature, want) } } // A project code that genuinely cannot be determined is a caller error, not a // crash. func TestSyncUnderivableProjectCodeReturnsError(t *testing.T) { r, _ := newProjectCodeRepo(t) if _, err := r.DB().Exec("DELETE FROM config WHERE name='project-code'"); err != nil { t.Fatalf("delete project-code: %v", err) } defer func() { if p := recover(); p != nil { t.Fatalf("Sync panicked instead of returning an error: %v", p) } }() _, _, err := captureLogin(t, r, SyncOpts{Pull: true, User: "syncer", Password: "pw123"}) if err == nil { t.Fatal("Sync succeeded without a project code") } if !strings.Contains(err.Error(), "project-code") { t.Fatalf("err = %v, want it to name project-code", err) } } // panicTransport panics on Exchange, standing in for any panic raised inside // the round loop. type panicTransport struct{ msg string } func (p *panicTransport) Exchange(context.Context, *xfer.Message) (*xfer.Message, error) { panic(p.msg) } // Sync's result-not-nil invariant only holds on a normal return; asserting it // while a panic unwinds replaces the real cause (issue #199). func TestSyncPanicSurfacesOriginalCause(t *testing.T) { r, _ := newProjectCodeRepo(t) defer func() { p := recover() if p == nil { t.Fatal("expected the transport panic to propagate") } if got, _ := p.(string); got != "boom" { t.Fatalf("recovered %v, want the original cause %q", p, "boom") } }() _, _ = Sync(context.Background(), r, &panicTransport{msg: "boom"}, SyncOpts{Pull: true}) }