Artifact [82cd2b1bf1]

Artifact 82cd2b1bf19bbcfa0337c5ea4d7348d26cb98131e0900f3aa7578b5e5e96bc15:

• File internal/content/cone_reference_prefix.go — part of check-in [b28c06506b] at 2026-10-22 02:57:50 on branch feat/horizon-cones — fix(content): classify sparse stores from crosslink tables, not the whole store

  Sparse classification read, inflated and delta-expanded every held blob, file content included, under a whole-store byte budget. On a clone of the Fossil SCM repository (67,894 blobs, 8.4 GB expanded) LoadAbsences failed at once with "byte budget exceeded", so no path-cone or content-horizon store of a real repository could classify its phantoms, and pull, verify and compaction failed with it.

  Reference facts now come from the tables wherever they provably stand in for raw cards. A blob the crosslinker recorded as no artifact and mlink names as a file version is never read. A completely linked check-in (event row, no orphan row, a primary parent edge) whose primary parent and baseline trees are themselves reproduced contributes its mlink rows as F cards and its plink, cherrypick and tagxref rows as references; by induction over the primary chain those rows give the same (name, file) pairs as its raw cards. Everything else is still read raw: unlinked and orphaned check-ins, roots, check-ins whose parent never arrived, other artifacts, and every check-in once the store has seen a sub-manifest, since trees are not crosslinked. Budgets are charged per round for the selected blobs and their delta ancestors only, and a base's bytes are released once its last delta is applied.

  A content horizon now reads only its held set's manifests and the baselines and nodes they reach, on demand, with one visited set and a byte budget scaled to the held set.

  On the Fossil SCM clone, cone {src} classifies in about 2 s at a 65 MiB peak heap, reading 8,655 of 67,894 blobs; the inside/outside sets equal the all-raw result, and Outside/Required agree on every artifact. (user: [email protected], size: 1446) [annotate] [blame] [check-ins using] [more...]

package content

import (
	"context"
	"encoding/binary"
	"fmt"

	"github.com/danmestas/go-libfossil/db"
)

// The metadata-only pass must prove total stored bytes first: substr(BLOB) can
// materialize the entire value inside SQLite even though it returns six bytes.
// Charge those reads separately from later full driver-payload reads.
func referencePrefixes(
	ctx context.Context, q db.Querier, blobs []referenceBlob,
	budget ReferenceBudget, work *ReferenceWork, inflated *int64,
) error {
	for i := range blobs {
		if err := ctx.Err(); err != nil {
			return err
		}
		b := &blobs[i]
		if b.unreadable {
			continue
		}
		if b.stored > budget.StoredBytes-work.PrefixBytes {
			return fmt.Errorf("sparse preparation: prefix byte budget exceeded")
		}
		work.PrefixReads++
		work.PrefixBytes += b.stored
		var prefix []byte
		err := q.QueryRow(`SELECT substr(content,1,6) FROM blob WHERE rid=?
			AND typeof(content)='blob' AND length(content)=?`, b.rid, b.stored).Scan(&prefix)
		if err != nil {
			return err
		}
		b.unreadable = len(prefix) != 6
		if !b.unreadable {
			b.unreadable = prefix[4]&15 != 8 || prefix[4]>>4 > 7 ||
				binary.BigEndian.Uint16(prefix[4:])%31 != 0
		}
		if b.unreadable {
			continue
		}
		b.inflated = int64(binary.BigEndian.Uint32(prefix[:4]))
		if b.inflated > budget.InflatedBytes-*inflated {
			return fmt.Errorf("sparse preparation: inflation budget exceeded")
		}
		*inflated += b.inflated
	}
	return nil
}