Skip to content
go-libfossil Documentation
Login

Documentation

title: Sparse cones weight: 60

aliases: ["/docs/sparse-and-horizons/"]

A repository does not have to hold everything. A cone limits it by path. A horizon limits it by age, and is carried as a member of a cone. Both leave the history intact, and the blobs they skip can be fetched later.

Cones follow draft-fossil-sparse-cone-00. They are implemented in step with zit.

Cones

A cone is a set of repository paths. Each member is a directory subtree or a single file, and each is either an include or an exclude. Members are literal prefixes, not patterns (§3.4).

The longest member that covers a path decides it. An exclude carves a part out of an include, and a deeper include re-includes part of an exclude (§3.3):

include src
exclude src/vendor
include src/vendor/keep

src/a.c is covered. src/vendor/v.c is not. src/vendor/keep/k.c is covered again.

A cone must have at least one include, at most 255 members, and no exclude that no include covers. Order does not matter; members are sorted and reduced to the smallest set with the same coverage (§3.2).

Checkout cones and store cones

A checkout cone limits the working directory (§4). The repository still holds every file. A sparse checkout writes, scans and tracks only the covered files. A commit carries the uncovered files from its parent unchanged, so it is the same check-in a dense checkout would make. Adding, removing, renaming or reverting an uncovered path fails with ErrOutsideCone.

A store cone limits the repository itself (§5). Blobs outside the cone are not fetched, and can be dropped by compaction. A checkout opens over a sparse store only if its own cone stays inside the store's.

Manifests, sub-manifest nodes and control artifacts are always held, so the whole history stays readable. Only file content is elided.

Elided is not damaged

An artifact the store lacks is classified one of three ways (§5.2):

Only elided is by design. Sync never requests an elided artifact, whether the store is the client or the server, and verify reports a store that is complete within its cone as OK. Elision is computed from the cone and never stored, so changing the cone reclassifies. Losing the cone, or one this version cannot read, turns every elided artifact back into a transit one and the store heals toward dense.

Library usage

Build a cone

cone, err := libfossil.NewCone(
    []string{"src", "docs"},       // include
    []string{"src/vendor"},        // exclude
)
if err != nil {
    log.Fatal(err)
}
fmt.Println(cone.Covers("src/main.go"))           // true
covered, member := cone.Decide("src/vendor/x.c")  // false, "src/vendor"

A Cone is immutable. With, Without and WithHorizon return new ones. Include, Exclude, Digest, String and Horizon read it.

Sparse checkout

repo, err := libfossil.Open("repo.fossil")
if err != nil {
    log.Fatal(err)
}
defer repo.Close()

co, err := repo.CreateCheckout("work", libfossil.CheckoutCreateOpts{Cone: cone})
if err != nil {
    log.Fatal(err)
}
defer co.Close()

// Widen, narrow, or pass nil to make the checkout dense.
change, err := co.SetCone(wider)
if err != nil {
    log.Fatal(err)
}
fmt.Println(change.Materialized, change.Dematerialized)

SetCone writes the files the new cone adds and removes the ones it drops from the working directory. The repository keeps them. It refuses, changing nothing, when a file leaving the cone has local changes or a file entering it would overwrite an untracked one. A cone that carries a horizon fails with ErrHorizonCheckout.

Over a sparse store, widening needs content the store may not hold. SetConeContext takes a RequiredContentOpts{Peer: transport} and fetches it. With no peer the widening is refused and nothing changes.

Clone into a sparse store

t := libfossil.NewHTTPTransport("https://example.com/repo")
repo, res, err := libfossil.Clone(ctx, "repo.fossil", t, libfossil.CloneOpts{
    StoreCone: cone,
})
if err != nil {
    log.Fatal(err)
}
defer repo.Close()
fmt.Println(res.Elided, res.ConeServed)

The result counts:

Both are zero for a dense clone. SyncResult has the same fields. A server that will not serve a sparse clone refuses it. A cone with a horizon clones a horizon store; see Horizons.

Set, widen and drop on an existing store

// Set or replace, without a peer. Narrowing deletes nothing.
err = repo.SetStoreCone(next)

// Read it.
cur, err := repo.StoreCone()

// Widen, and fetch what that newly covers.
sr, err := repo.SetStoreConeContext(ctx, wider, libfossil.RequiredContentOpts{
    Peer: t,
})

// Drop the cone. Fetch everything it elided.
sr, err = repo.SetStoreConeContext(ctx, nil, libfossil.RequiredContentOpts{Peer: t})

// Drop held blobs the cone elides. See Limits.
n, err := repo.CompactStore(ctx)

SetStoreCone moves no content. Widening makes the newly covered elided artifacts wanted again, and the next pull fetches them. SetStoreConeContext does the pull itself through an ordinary cone-scoped sync. With artifacts to fetch and no Peer, it refuses, names the count, and leaves the cone unchanged. If the transfer fails after the cone is replaced, the wider policy stays and the next pull heals it. While a store cone is set, a commit carries no R card. CompactStore on a dense store fails with ErrNoStoreCone.

Command line

Open a sparse checkout

libfossil -R repo.fossil repo open [DIR] --cone src --cone docs --cone-not src/vendor

Edit the checkout cone

All of these take -d, --dir for the checkout directory (default .).

libfossil repo cone show
libfossil repo cone add src/extra
libfossil repo cone add --not src/gen
libfossil repo cone remove src/extra
libfossil repo cone set src docs --not src/vendor
libfossil repo cone dense
libfossil repo cone test src/a.c src/vendor/v.c

add, remove, set and dense can need content the store lacks. They take --remote (default: the saved last-sync-url), --sync-user, --sync-pass (never saved), --sync-private and --timeout (default 10m; 0 disables it).

Clone a sparse store

libfossil repo clone URL repo.fossil --cone src --cone-not src/vendor

--horizon, --leaves, --structural and --seed-key combine with these; see Horizons.

Change the store cone

libfossil -R repo.fossil repo cone store                          # show it
libfossil -R repo.fossil repo cone store src docs --not src/vendor
libfossil -R repo.fossil repo cone store --dense
libfossil -R repo.fossil repo cone compact

store takes the same --remote, --sync-user, --sync-pass, --sync-private and --timeout flags as above. Its --horizon, --leaves, --structural and --no-horizon flags are described under Horizons.

Interoperability

zit. The cone format matches zit's byte for byte, so a cone's digest is the same in both. Sparse sync works in both directions against zit's horizon-rfc revision. An implementation that predates horizons refuses a cone that carries one.

Stock fossil 2.28. A sparse store must not be served as if it were dense, because the clone would be silently incomplete. A store held sparsely is served only when its partial-mirror setting is on (repo config set partial-mirror on, SettingPartialMirror); otherwise it refuses. The refusal puts the push card ahead of the error so fossil 2.28 aborts instead of carrying on. Fossil cannot read sub-manifest check-ins either, so a repository that uses them is for libfossil and zit peers only.

Limits