Artifact d5272c3bdf8c7516209af97b9709cab0e6e469ae256dba7ef93daaa336fddca4:
- File
docs/site/content/docs/reference/sdk/db/api.md
— part of check-in
[1fd12b3ed3]
at
2026-10-28 21:53:09
on branch feat/chunked-blobs
— Store blobs over a hosted value limit across blobpart rows
A hosted SQLite that caps value size (a Durable Object's 2 MB) cannot hold a large file's stored blob in one blob.content value. With the new db.DriverConfig.MaxValueBytes set, a stored value over the limit is written in 1,000,000-byte parts to blobpart(rid, seq, content), created on first use, leaving blob.content NULL and blob.size the expanded length, so phantoms (size < 0) stay unambiguous. Chunking sits below compression: hashes, deltas and the wire format are unchanged.
internal/blob owns the seam (ReadStored, WriteStored, InsertStored, StoredHead, DropStored, SweepParts and two scan expressions), and every direct reader or writer of blob.content now goes through it: delta rewrite and undelta, receive, clone send and walk, compaction, sparse demotion, seed drop, shun purge, horizon cost and cone preparation.
Zero, the default, keeps every value inline and creates no table, so local stores stay byte-identical. A store with blobpart rows is not readable by stock fossil; enable it only for hosted backends.
Closes #459 (user: [email protected], size: 22880) [annotate] [blame] [check-ins using]
--- title: db --- <!-- Code generated by gomarkdoc. DO NOT EDIT --> # db ```go import "github.com/danmestas/go-libfossil/db" ``` Package db provides a SQLite database layer with pluggable drivers. Two SQLite drivers ship with libfossil — modernc \(pure Go, default\) and ncruces \(WASM\-capable\). Select one at build time by importing its driver package: ``` import _ "github.com/danmestas/go-libfossil/db/driver/modernc" ``` [Open](<#Open>) and [OpenWith](<#OpenWith>) handle DSN construction, WAL/pragma setup, and WASM\-specific workarounds. Use [DB.WithTx](<#DB.WithTx>) for transaction scoping with automatic rollback on error. The [Querier](<#Querier>) interface \(Exec, QueryRow, Query\) is satisfied by both [DB](<#DB>) and [Tx](<#Tx>), allowing callers to write transaction\-agnostic code. ## Index - [Constants](<#constants>) - [func CreateRepoSchema\(d \*DB\) error](<#CreateRepoSchema>) - [func DefaultPragmas\(\) map\[string\]string](<#DefaultPragmas>) - [func OpenSQL\(path string, cfg OpenConfig, params map\[string\]string\) \(\*sql.DB, error\)](<#OpenSQL>) - [func ReceiptID\(q Querier\) \(int64, error\)](<#ReceiptID>) - [func Register\(cfg DriverConfig\)](<#Register>) - [func SQLDriverAndDSN\(path string, cfg OpenConfig, params map\[string\]string\) \(string, string\)](<#SQLDriverAndDSN>) - [func ScanInt\(v any\) \(int, bool\)](<#ScanInt>) - [func ScanJulianDay\(v any\) \(float64, bool\)](<#ScanJulianDay>) - [func ScanTextJulianDay\(v any\) \(float64, bool\)](<#ScanTextJulianDay>) - [func ScanTime\(v any\) \(time.Time, bool\)](<#ScanTime>) - [func SeedConfig\(d \*DB, rng simio.Rand, projectCode string\) error](<#SeedConfig>) - [func SeedDefaultUsers\(d \*DB, rng simio.Rand\) error](<#SeedDefaultUsers>) - [func SeedNobody\(d \*DB, caps string\) error](<#SeedNobody>) - [func SeedUser\(d \*DB, login string\) error](<#SeedUser>) - [type CheckpointMode](<#CheckpointMode>) - [func \(m CheckpointMode\) String\(\) string](<#CheckpointMode.String>) - [type DB](<#DB>) - [func Open\(path string\) \(\*DB, error\)](<#Open>) - [func OpenWith\(path string, cfg OpenConfig\) \(\*DB, error\)](<#OpenWith>) - [func \(d \*DB\) ApplicationID\(\) \(int32, error\)](<#DB.ApplicationID>) - [func \(d \*DB\) Bound\(\) bool](<#DB.Bound>) - [func \(d \*DB\) Checkpoint\(mode CheckpointMode\) error](<#DB.Checkpoint>) - [func \(d \*DB\) Close\(\) error](<#DB.Close>) - [func \(d \*DB\) Driver\(\) string](<#DB.Driver>) - [func \(d \*DB\) Exec\(query string, args ...any\) \(sql.Result, error\)](<#DB.Exec>) - [func \(d \*DB\) Path\(\) string](<#DB.Path>) - [func \(d \*DB\) PoolExecs\(\) int64](<#DB.PoolExecs>) - [func \(d \*DB\) Query\(query string, args ...any\) \(\*sql.Rows, error\)](<#DB.Query>) - [func \(d \*DB\) QueryRow\(query string, args ...any\) \*sql.Row](<#DB.QueryRow>) - [func \(d \*DB\) SetApplicationID\(id int32\) error](<#DB.SetApplicationID>) - [func \(d \*DB\) SqlDB\(\) \*sql.DB](<#DB.SqlDB>) - [func \(d \*DB\) UseReceipt\(rc \*Receipt\) \*Receipt](<#DB.UseReceipt>) - [func \(d \*DB\) WithBoundReadTx\(fn func\(bound \*DB\) error\) error](<#DB.WithBoundReadTx>) - [func \(d \*DB\) WithBoundTx\(fn func\(bound \*DB\) error\) error](<#DB.WithBoundTx>) - [func \(d \*DB\) WithReadTx\(fn func\(tx \*Tx\) error\) error](<#DB.WithReadTx>) - [func \(d \*DB\) WithTx\(fn func\(tx \*Tx\) error\) error](<#DB.WithTx>) - [func \(d \*DB\) WriteTxns\(\) int64](<#DB.WriteTxns>) - [func \(d \*DB\) Writes\(\) int64](<#DB.Writes>) - [type DriverConfig](<#DriverConfig>) - [func RegisteredDriver\(\) \*DriverConfig](<#RegisteredDriver>) - [type OpenConfig](<#OpenConfig>) - [type Querier](<#Querier>) - [type Receipt](<#Receipt>) - [func \(rc \*Receipt\) ID\(q Querier\) \(int64, error\)](<#Receipt.ID>) - [type Tx](<#Tx>) - [func \(t \*Tx\) Exec\(query string, args ...any\) \(sql.Result, error\)](<#Tx.Exec>) - [func \(t \*Tx\) Query\(query string, args ...any\) \(\*sql.Rows, error\)](<#Tx.Query>) - [func \(t \*Tx\) QueryRow\(query string, args ...any\) \*sql.Row](<#Tx.QueryRow>) - [func \(t \*Tx\) UseReceipt\(rc \*Receipt\) \*Receipt](<#Tx.UseReceipt>) - [func \(t \*Tx\) WithSavepoint\(fn func\(tx \*Tx\) error\) error](<#Tx.WithSavepoint>) ## Constants <a name="HashPolicySHA1"></a>Fossil's hash\-policy config values, in Fossil's own order \(see \`fossil help hash\-policy\`\). The row holds the integer as text. Naming follows from the value: SHA1 below HashPolicySHA3, SHA3 at or above it, except HashPolicyAuto, which names with SHA1 until a SHA3 artifact enters the repo. From HashPolicySHA3Only up, an artifact already stored under a legacy SHA1 name is not reused. ```go const ( HashPolicySHA1 = 0 HashPolicyAuto = 1 HashPolicySHA3 = 2 HashPolicySHA3Only = 3 HashPolicyShunSHA1 = 4 ) ``` <a name="CreateRepoSchema"></a> ## func CreateRepoSchema ```go func CreateRepoSchema(d *DB) error ``` <a name="DefaultPragmas"></a> ## func DefaultPragmas ```go func DefaultPragmas() map[string]string ``` DefaultPragmas returns the default pragma settings. <a name="OpenSQL"></a> ## func OpenSQL ```go func OpenSQL(path string, cfg OpenConfig, params map[string]string) (*sql.DB, error) ``` OpenSQL opens a raw \*sql.DB using the registered driver and the same DSN construction as OpenWith. params are appended as URI query parameters after driver pragmas, so callers can request options such as mode=ro without constructing driver\-specific DSNs. <a name="ReceiptID"></a> ## func ReceiptID ```go func ReceiptID(q Querier) (int64, error) ``` ReceiptID returns the rcvid content stored through q is recorded under: the receipt of q's transaction, which is the one given by UseReceipt or, when none was, one made for that transaction alone. Content stored outside a transaction is a receipt of its own. <a name="Register"></a> ## func Register ```go func Register(cfg DriverConfig) ``` Register registers a SQLite driver for use by Open/OpenWith. Must be called exactly once \(typically from a driver package's init\(\)\). Panics if called more than once. <a name="SQLDriverAndDSN"></a> ## func SQLDriverAndDSN ```go func SQLDriverAndDSN(path string, cfg OpenConfig, params map[string]string) (string, string) ``` SQLDriverAndDSN returns the driver name and DSN OpenWith would use, with optional URI query params appended after driver\-specific options. <a name="ScanInt"></a> ## func ScanInt ```go func ScanInt(v any) (int, bool) ``` ScanInt converts a scanned value to int. SQLite drivers differ on BOOLEAN columns: modernc returns int64, ncruces returns bool. Scan the column into an \`any\` variable, then pass it here. <a name="ScanJulianDay"></a> ## func ScanJulianDay ```go func ScanJulianDay(v any) (float64, bool) ``` ScanJulianDay converts a scanned mtime value to a float64 Julian Day Number. SQLite drivers return mtime differently: modernc returns float64, ncruces returns time.Time for DATETIME/TIMESTAMP/DATE columns. Scan the column into an \`any\` variable, then pass it here. The float64 branch returns the scanned value verbatim, and that exactness is load\-bearing rather than incidental. A float64 read off the row IS the stored value, so any "canonicalizing" transform can only move it away from what the row holds. Two consumers compare a scanned mtime against other stored mtimes and would silently return a wrong answer \(never an error\) if the value were perturbed even by one ulp: - Timeline's pagination cursor, whose predicate \`e.mtime \< ? OR \(e.mtime = ? AND b.rid \< ?\)\` needs the cursor to compare exactly equal to the row it came from. Raise it by an ulp and that row repeats as the next page's first row; lower it and rows are skipped. - tag propagation's \`tagxref.mtime \< ?\`, where SQLite compares the bound value against raw stored floats. Neither is hypothetical for values this library did not write: a value another writer computed differently, or at finer than millisecond resolution, is not on the grid TimeToJulian writes, and snapping it to that grid changes it. The time.Time branch is the one case where a transform is required rather than harmful — see timeToJulian. <a name="ScanTextJulianDay"></a> ## func ScanTextJulianDay ```go func ScanTextJulianDay(v any) (float64, bool) ``` ScanTextJulianDay is ScanJulianDay for a column fossil writes as SQL text: event, plink, attachment and forumpost times. Those hold the 16\-digit form \(fsltype.TextJulianDayFromMillis\), so a time.Time a driver hands back is converted to that form rather than to the exact julianday\(\) ScanJulianDay gives for tagxref.mtime. A float64 passes through unchanged either way. <a name="ScanTime"></a> ## func ScanTime ```go func ScanTime(v any) (time.Time, bool) ``` ScanTime converts a scanned mtime value to time.Time, at the millisecond SQLite's date functions give it: the time a check\-in's D card records. Same driver\-compatibility rationale as ScanJulianDay. <a name="SeedConfig"></a> ## func SeedConfig ```go func SeedConfig(d *DB, rng simio.Rand, projectCode string) error ``` SeedConfig seeds the config table with project\-code, server\-code, the schema markers, and hash\-policy. If projectCode is empty, a fresh 40\-char lowercase\-hex project\-code is generated from rng. If non\-empty, it must match ^\[0\-9a\-f\]\{40\}$; otherwise an error is returned and no rows are written. Server\-code is always generated. hash\-policy is seeded to sha3, which is what \`fossil init\` writes and so what a repo created here has to say to name its artifacts the same way. A clone is the exception: it adopts its source's policy, not this one, so sync.Clone drops the row the same way it drops project\-code. <a name="SeedDefaultUsers"></a> ## func SeedDefaultUsers ```go func SeedDefaultUsers(d *DB, rng simio.Rand) error ``` SeedDefaultUsers inserts the four built\-in users \`fossil new\` creates, with fossil's capabilities \(src/db.c db\_create\_default\_users\). nobody's set is what every unauthenticated sync request is granted, so it is clone, read and zip only: check\-in rights must be granted explicitly. developer and reader are templates other users inherit through v and u. anonymous gets a random password, as fossil gives it; the others have none, so no login can reach them \(VerifyLogin requires a stored password\). <a name="SeedNobody"></a> ## func SeedNobody ```go func SeedNobody(d *DB, caps string) error ``` SeedNobody inserts a "nobody" user with the given capabilities. This controls anonymous access policy for the repo. A new repository gets fossil's built\-in users from SeedDefaultUsers instead. <a name="SeedUser"></a> ## func SeedUser ```go func SeedUser(d *DB, login string) error ``` <a name="CheckpointMode"></a> ## type CheckpointMode CheckpointMode mirrors SQLite's PRAGMA wal\_checkpoint\(\<mode\>\) argument. ```go type CheckpointMode int ``` <a name="CheckpointPassive"></a> ```go const ( // CheckpointPassive checkpoints frames not held by readers; never blocks. // Appropriate for periodic background checkpoints. CheckpointPassive CheckpointMode = iota // CheckpointFull blocks new writers until all frames are checkpointed. CheckpointFull // CheckpointRestart is FULL plus restarts the WAL file. CheckpointRestart // CheckpointTruncate is RESTART plus truncates the WAL file to zero bytes. // Produces the most compact on-disk file and is what Close uses. CheckpointTruncate ) ``` <a name="CheckpointMode.String"></a> ### func \(CheckpointMode\) String ```go func (m CheckpointMode) String() string ``` <a name="DB"></a> ## type DB DB wraps a SQLite database connection. ```go type DB struct { // contains filtered or unexported fields } ``` <a name="Open"></a> ### func Open ```go func Open(path string) (*DB, error) ``` Open opens a SQLite database with the registered driver and default pragmas. <a name="OpenWith"></a> ### func OpenWith ```go func OpenWith(path string, cfg OpenConfig) (*DB, error) ``` OpenWith opens a SQLite database with explicit configuration. <a name="DB.ApplicationID"></a> ### func \(\*DB\) ApplicationID ```go func (d *DB) ApplicationID() (int32, error) ``` <a name="DB.Bound"></a> ### func \(\*DB\) Bound ```go func (d *DB) Bound() bool ``` Bound reports whether d is a DB WithBoundTx handed its callback, whose statements run in a transaction that d does not own. <a name="DB.Checkpoint"></a> ### func \(\*DB\) Checkpoint ```go func (d *DB) Checkpoint(mode CheckpointMode) error ``` Checkpoint runs PRAGMA wal\_checkpoint\(\<mode\>\) against the database. Safe to call on a live database. PASSIVE never blocks; TRUNCATE produces the most compact on\-disk layout. On a non\-WAL database the underlying PRAGMA is a no\-op and returns nil. <a name="DB.Close"></a> ### func \(\*DB\) Close ```go func (d *DB) Close() error ``` Close runs PRAGMA wal\_checkpoint\(TRUNCATE\) before closing the underlying connection so the on\-disk file is readable by external SQLite/fossil tooling. The WASM/WASI build path does not use WAL, so the checkpoint is skipped there. Checkpoint errors are joined with the close error so the connection is always closed regardless of checkpoint outcome. <a name="DB.Driver"></a> ### func \(\*DB\) Driver ```go func (d *DB) Driver() string ``` <a name="DB.Exec"></a> ### func \(\*DB\) Exec ```go func (d *DB) Exec(query string, args ...any) (sql.Result, error) ``` <a name="DB.Path"></a> ### func \(\*DB\) Path ```go func (d *DB) Path() string ``` <a name="DB.PoolExecs"></a> ### func \(\*DB\) PoolExecs ```go func (d *DB) PoolExecs() int64 ``` PoolExecs returns how many statements have been executed on the pool rather than inside a transaction. Each one runs in its own implicit transaction, so it costs a BEGIN IMMEDIATE and an fsync of its own and contests the database's single write lock independently of every other. A path that issues one per artifact therefore degrades from slow to failing the moment anything else writes, which is what issue \#200 cost a large pull. Tests pin the per\-round budget against this counter. <a name="DB.Query"></a> ### func \(\*DB\) Query ```go func (d *DB) Query(query string, args ...any) (*sql.Rows, error) ``` <a name="DB.QueryRow"></a> ### func \(\*DB\) QueryRow ```go func (d *DB) QueryRow(query string, args ...any) *sql.Row ``` <a name="DB.SetApplicationID"></a> ### func \(\*DB\) SetApplicationID ```go func (d *DB) SetApplicationID(id int32) error ``` <a name="DB.SqlDB"></a> ### func \(\*DB\) SqlDB ```go func (d *DB) SqlDB() *sql.DB ``` SqlDB returns the underlying \*sql.DB connection. <a name="DB.UseReceipt"></a> ### func \(\*DB\) UseReceipt ```go func (d *DB) UseReceipt(rc *Receipt) *Receipt ``` UseReceipt is Tx.UseReceipt for the transaction a bound DB runs in. Only a bound DB has a transaction to give it to. <a name="DB.WithBoundReadTx"></a> ### func \(\*DB\) WithBoundReadTx ```go func (d *DB) WithBoundReadTx(fn func(bound *DB) error) error ``` WithBoundReadTx is WithBoundTx over a read transaction \(see WithReadTx\): a DB bound to one snapshot that takes no write lock. fn must not write. <a name="DB.WithBoundTx"></a> ### func \(\*DB\) WithBoundTx ```go func (d *DB) WithBoundTx(fn func(bound *DB) error) error ``` WithBoundTx runs fn in one write transaction and hands it a DB bound to that transaction. Every statement on the bound DB runs in the transaction, and every WithTx or WithBoundTx on it is a savepoint there, so code written against a \*DB, including code that opens its own transactions, joins this one instead of contending for the write lock from a second connection. The transaction commits when fn returns nil and rolls back everything fn did otherwise. The bound DB is valid only until fn returns, and it cannot be closed or checkpointed, nor hand out its \*sql.DB. On a DB that is itself bound, WithBoundTx is a savepoint of that transaction, as WithTx is. <a name="DB.WithReadTx"></a> ### func \(\*DB\) WithReadTx ```go func (d *DB) WithReadTx(fn func(tx *Tx) error) error ``` WithReadTx runs fn in a read transaction: one snapshot of the database, taken without the write lock, so it neither waits for a writer nor holds one up. fn must not write; the drivers refuse or contest a write in it. The transaction ends when fn returns, with nothing to commit. On a bound DB it runs fn in the bound transaction, as WithTx does. <a name="DB.WithTx"></a> ### func \(\*DB\) WithTx ```go func (d *DB) WithTx(fn func(tx *Tx) error) error ``` WithTx runs fn in a write transaction, which commits when fn returns nil and rolls back otherwise. On a DB that WithBoundTx bound, it runs fn in a savepoint of the bound transaction instead, so it neither waits for a second connection nor commits on its own: fn's error undoes only fn's writes, and the bound transaction decides whether the rest commits. <a name="DB.WriteTxns"></a> ### func \(\*DB\) WriteTxns ```go func (d *DB) WriteTxns() int64 ``` WriteTxns counts transactions begun through WithTx. Each is its own BEGIN IMMEDIATE and fsync, and each takes the single write lock independently, so a path that opens one per artifact is both slow and fragile under contention \-\- the cost issue \#200 measured at roughly three per received file. Tests pin the per\-round transaction budget against this, which PoolExecs cannot see: moving a per\-artifact write from the pool into its own transaction leaves PoolExecs unchanged while the real cost stays. <a name="DB.Writes"></a> ### func \(\*DB\) Writes ```go func (d *DB) Writes() int64 ``` Writes counts writes of this DB that have finished: pool Exec calls that returned and WithTx transactions that ended, committed or not. A reader that keeps what it read across calls compares two counts to learn whether a write may have landed in between; counting at the end, not the start, is what makes an unchanged count mean that. A write through another handle on the same file, in this process or another, is not counted. <a name="DriverConfig"></a> ## type DriverConfig DriverConfig defines a SQLite driver's name and DSN builder. ```go type DriverConfig struct { Name string BuildDSN func(path string, pragmas map[string]string) string // MaxValueBytes is the largest single value the backend accepts, for a // hosted SQLite (such as a Durable Object's) that caps value size. A // stored blob larger than this is written across rows of a blobpart // table instead of into blob.content; hashes, deltas and the wire // protocol are unchanged. Zero, the default, disables this: every value // is stored inline and no blobpart table is created, so a local store // stays byte-identical to stock fossil's. // // A store holding blobpart rows cannot be read by stock fossil, which // sees those blobs as NULL content. Set it only for hosted backends. // A nonzero value must be at least 1,000,000, the part size. MaxValueBytes int } ``` <a name="RegisteredDriver"></a> ### func RegisteredDriver ```go func RegisteredDriver() *DriverConfig ``` RegisteredDriver returns a copy of the currently registered driver config, or nil if none. <a name="OpenConfig"></a> ## type OpenConfig OpenConfig allows callers to customize driver selection and pragmas. ```go type OpenConfig struct { Driver string // override driver name (empty = use registered driver) Pragmas map[string]string // additional/override pragmas (merged with defaults) } ``` <a name="Querier"></a> ## type Querier Querier is the common interface satisfied by both \*DB and \*Tx. Functions that need to work inside transactions accept Querier instead of \*DB. ```go type Querier interface { Exec(query string, args ...any) (sql.Result, error) QueryRow(query string, args ...any) *sql.Row Query(query string, args ...any) (*sql.Rows, error) } ``` <a name="Receipt"></a> ## type Receipt Receipt is one arrival of content in the repository: a commit, a sync or clone, a push a server accepts. Fossil records each as a row of rcvfrom, made the first time that arrival stores anything \(content\_rcvid\_init in src/content.c\), and every blob or unversioned file it stores names the row in its rcvid. A receipt that stores nothing leaves no row. The fields are what the row will hold, read when it is made: UID is the user.uid the content came from, Nonce the nonce of the login card that authenticated it, and IPAddr the peer's address or, for local content, a description of its source. An empty Nonce or IPAddr is stored as NULL, as fossil stores an absent one. The row's mtime is the Julian day of the moment it is made, by Clock \(nil means real time\). A Receipt is not safe for concurrent use. ```go type Receipt struct { UID int64 Nonce string IPAddr string Clock simio.Clock // contains filtered or unexported fields } ``` <a name="Receipt.ID"></a> ### func \(\*Receipt\) ID ```go func (rc *Receipt) ID(q Querier) (int64, error) ``` ID returns the rcvid of the receipt's row, making the row through q if the receipt has none yet. Fossil holds that id for its process in g.rcvid and never looks at the row again \(src/content.c, content\_rcvid\_init\); a fossil whose transaction rolls back ends with it. A Receipt outlives transactions, so the transaction or savepoint that makes its row also forgets it if it rolls back \(Tx.forgetFrom\), and the next use makes a fresh row rather than naming one some other insert may since have taken. A kept id costs no query. A row made through a Querier that is neither a Tx nor a DB is taken as kept; a row made in one transaction should not be used from another connection before that transaction commits. <a name="Tx"></a> ## type Tx ```go type Tx struct { // contains filtered or unexported fields } ``` <a name="Tx.Exec"></a> ### func \(\*Tx\) Exec ```go func (t *Tx) Exec(query string, args ...any) (sql.Result, error) ``` <a name="Tx.Query"></a> ### func \(\*Tx\) Query ```go func (t *Tx) Query(query string, args ...any) (*sql.Rows, error) ``` <a name="Tx.QueryRow"></a> ### func \(\*Tx\) QueryRow ```go func (t *Tx) QueryRow(query string, args ...any) *sql.Row ``` <a name="Tx.UseReceipt"></a> ### func \(\*Tx\) UseReceipt ```go func (t *Tx) UseReceipt(rc *Receipt) *Receipt ``` UseReceipt makes rc the receipt content stored in t is recorded under, and returns the one it replaces so a caller can put it back. <a name="Tx.WithSavepoint"></a> ### func \(\*Tx\) WithSavepoint ```go func (t *Tx) WithSavepoint(fn func(tx *Tx) error) error ``` WithSavepoint runs fn in a savepoint of t, as WithTx does on a DB bound to a transaction: fn's error undoes fn's writes alone and is returned, leaving t open with the writes before it. Generated by [gomarkdoc](<https://github.com/princjef/gomarkdoc>)