Two releases ago a no-op pull — both peers already holding everything — took seventeen minutes. One release ago it took seven seconds. This one looks at what those seven seconds were made of, and finds that most of the bytes on the wire were a peer saying, one line at a time, "I have this" about nine hundred thousand things the other side already knew it had.
That is what igot does. A pull opens with the client announcing every
artifact it holds, so the server can suppress what it need not send and can
ask for what it lacks. It is the right protocol for a repository of a
thousand artifacts. For pkgsrc it is 904,898 lines, about sixty-three
megabytes, per direction, per round — and every round of every pull, whether
or not anything changed. It fits under zit's reply ceiling today. At a few
million artifacts it would not, and the pull would fail outright, which is
the kind of limit that arrives as an outage rather than a slowdown.
Fossil solved this in 2007 with the cluster artifact. A cluster is the
simplest artifact in the format: a sorted list of hashes and a checksum,
nothing else. It is igot in bulk. Once a server folds an artifact into a
cluster it stops announcing that artifact individually — the cluster's M
cards are the announcement — and because a cluster is itself an artifact, a
later cluster names it, and the whole repository becomes reachable from the
newest one by following the chain down. A peer that already holds the chain
announces almost nothing; a peer that lacks it asks for one cluster and
learns about eight hundred artifacts at once.
zit could read clusters. It had never used them. On receipt it folded their members into its request set correctly, but on send it announced everything, and it never minted a cluster of its own. A repository born in zit had no clusters at all, and a repository imported from fossil carried fossil's clusters as dead weight — 2,439 of them in pkgsrc, naming nearly every artifact, consulted by nothing.
This release consults them. A sync now keeps a small index — the names of every held cluster, kept warm by the persist path as it writes and rebuilt by one walk if lost — and resolves it at open into two facts: which artifacts some cluster already names, and which names a held cluster promises that this repository never collected. The first is what not to announce. The second is the part fossil gets for free from its database and zit has to derive: fossil records a phantom row for every promised-but-absent artifact and asks for it on every sync until it arrives; zit writes no phantom to disk, so it plants them again at each open from the clusters it holds. Without that, a pull interrupted halfway would leave holes that no later pull could see, because the bounded announcement is precisely the announcement that no longer mentions them.
Then it announces held-and-unclustered, on both sides, and mints. When a pull reply finds a hundred or more held artifacts that no cluster names, it folds them into clusters of eight hundred — sorted, checksummed, stored like any other artifact — exactly where fossil does it and at fossil's thresholds. A repository born in zit clusters itself the first time anyone pulls from it.
Measured on the fossil-imported pkgsrc store, a no-op pull that sent
sixty-three megabytes now sends 4,748 bytes. Most of those are not
announcements at all: they are sixty-three gimme cards, for sixty-three
artifacts that pkgsrc's clusters promise and that no peer holds — private or
shunned on the fossil side, years ago. Fossil asks for those every round and
says nothing. zit's first version of this change did something worse: it
counted them as artifacts the peer had failed to deliver and reported every
pull as a failure, forever. The fix was already written down in the code, in
the comment explaining why a shunned phantom is not counted: an expectation
the peer set and did not meet is a failed pull; a request the run made on its
own initiative is not. Cluster promises are now a one-line warning, and the
pull says already in sync.
On a repository born in zit, one hundred and sixty-one artifacts, the first pull after this release minted one cluster on the server and delivered it; the next announced one artifact instead of a hundred and sixty-one — two hundred and one bytes.
The index costs one walk of the store the first time a sync opens it — three and a half minutes on pkgsrc, once, then a hundred-kilobyte file adopted in a few milliseconds. And where the index is absent or stale, every path degrades to the old behaviour: the full announcement, correct and expensive. That was the design constraint throughout. Bounded announcement trades bytes for bookkeeping, and the bookkeeping is allowed to be wrong only in the direction that costs bytes.