edgesync: Agent & Bridge Deployment

Leaf Agent Architecture

The leaf agent is a Go daemon in its own module (leaf/) that opens a Fossil repo via libfossil, connects to NATS, and syncs artifacts on a poll loop. It acts as both sync client and server.

Lifecycle: New(Config) -> Start() -> Stop(). A single poll goroutine is the sole executor of doSync(). SyncNow() is a non-blocking signal (buffered channel, size 1) that coalesces if a sync is already running. This avoids concurrent SQLite access.

Signals: SIGUSR1 triggers SyncNow(). SIGINT/SIGTERM trigger graceful Stop() (cancel context, drain NATS, close repo).

Config Field	Default	CLI Flag	Env Var
`RepoPath`	(required)	`--repo`	`LEAF_REPO`
`NATSRole`	`"peer"`	`--nats-role`
`NATSUpstream`	`""` (embedded only)	`--nats`	`LEAF_NATS_URL`
`NATSClientPort`	`0` (random)	`--nats-client-port`	`LEAF_NATS_CLIENT_PORT`
`PollInterval`	`5s`	`--poll`
`User`	`""` (anonymous)	`--user`
`Password`	`""`	`--password`
`Push`	`true`	`--push` / `--no-push`
`Pull`	`true`	`--pull` / `--no-pull`
`ServeHTTPAddr`	`""` (disabled)	`--serve-http`
`ServeNATSEnabled`	`false`	`--serve-nats`
`UV`	`false`	`--uv`
`SubjectPrefix`	`"fossil"`	`--prefix`

Embedded NATS: Every agent runs an in-process NATS server. The agent's NATS client connects to localhost. If NATSUpstream is set, the embedded server joins external NATS as a leaf node. If iroh peers are configured, the embedded server connects to them via tunnels over QUIC.

NATSRole determines mesh topology:

Role	Accepts leaf connections	Solicits leaf connections	Use case
`peer` (default)	Yes	Yes (lower EndpointId solicits)	Native P2P mesh
`hub`	Yes	No	Dedicated server, cluster node
`leaf`	No	Yes (always solicits outward)	WASM browser, lightweight client

Hub Package

The hub package (hub/, in the root module) is an in-process host for an EdgeSync hub: fossil repo bootstrap, embedded NATS server with JetStream, auto-port allocation, and the libfossil HTTP server. Consumers (notably bones) use it to host a hub without rolling their own libfossil + nats-server bootstrap; everything is libfossil-free at the public API boundary. Leaf agents on the same host (or another host) sync to it via Hub.HTTPAddr() (HTTP-xfer) or Hub.LeafUpstream() (NATS leaf node). See hub-package.md.

Bridge Architecture

The bridge is a NATS-to-HTTP translator in its own module (bridge/). It subscribes to <prefix>.<project-code>.sync, decodes xfer messages, proxies them to a Fossil HTTP server via sync.HTTPTransport, and replies. One project per instance.

Error handling: Decode or HTTP failures produce an empty response (never leave the leaf hanging, never crash).

Config Field	Default	CLI Flag	Env Var
`FossilURL`	(required)	`--fossil`	`BRIDGE_FOSSIL_URL`
`ProjectCode`	(required)	`--project`	`BRIDGE_PROJECT_CODE`
`NATSUrl`	`nats://localhost:4222`	`--nats`	`BRIDGE_NATS_URL`
`SubjectPrefix`	`"fossil"`	`--prefix`	`BRIDGE_PREFIX`

NATS Transport

NATSTransport implements sync.Transport over NATS request/reply. Subject scheme: <prefix>.<project-code>.sync. Exchange flow: encode xfer message -> zlib compress -> NATS request -> decode reply.

Constraint: Leaf and bridge must agree on the subject prefix for a given project.

JetStream upgrade path: Replace Request with JetStream publish/consume. The Exchange signature stays the same.

Iroh P2P Transport

Third transport option alongside NATS and HTTP. Enables direct peer-to-peer sync without central infrastructure, using QUIC holepunching + relay fallback for NAT traversal.

Architecture: Rust sidecar binary (iroh-sidecar) communicates with the Go agent via HTTP on a Unix socket. The agent spawns the sidecar as a child process. This avoids CGo while leveraging iroh's Rust networking stack.

Identity: Ed25519 keypair. Each agent has a stable EndpointId derived from its key. Peers are configured by EndpointId, not address.

Config Field	Default	CLI Flag
`IrohEnabled`	`false`	`--iroh`
`IrohPeers`	`[]`	`--iroh-peer` (repeatable)
`IrohKeyPath`	`~/.config/edgesync/iroh.key`	`--iroh-key`

Agent lifecycle:

Start(): spawn sidecar (--socket, --key-path, --callback, --alpn /edgesync/xfer/1), wait for /status 200, register each peer as a syncTarget with IrohTransport
Poll loop: iterates all syncTargets uniformly (NATS, iroh, HTTP)
Incoming: sidecar forwards peer requests to agent's callback listener → repo.XferHandler()
Stop(): POST /shutdown to sidecar, wait 5s, SIGTERM, wait 2s, SIGKILL

Socket path: /tmp/iroh-<sha256(repoPath)[:12]>.sock — deterministic, collision-free for multiple agents.

IrohTransport: RoundTrip(ctx, payload) POSTs to http://iroh-sidecar/exchange/<endpointID> over Unix socket. Same Transport interface as NATS and HTTP — sync layer is unaware of the transport.

Discovery: Phase 1 is manual config only (--iroh-peer). DNS discovery and NATS-assisted peer exchange are future work.

HTTP Serving

serve_http.go in leaf/agent/ composes an HTTP mux with /healthz (200 OK, JSON status) and sync.XferHandler() for Fossil's /xfer endpoint. Operational endpoints live in the agent, not in libfossil (which is transport-agnostic).

Stock fossil clone and fossil sync work against the leaf's HTTP endpoint.

Docker Deployment

Stack lives in deploy/. Two containers: NATS + leaf agent.

Dockerfile: Multi-stage build. Build stage: golang:1.26-alpine with CGO (modernc SQLite). Run stage: alpine:3.20 (~20-30MB). Uses GOWORK=off, copies only libfossil/ and leaf/.

Compose services:

Service	Ports	Notes
`nats`	`100.64.0.1:4222` (Tailscale only)	JetStream enabled, no auth (Tailscale boundary)
`leaf`	`0.0.0.0:9000` (public HTTP)	Health check via `/healthz`, auto-restart

Volumes: deploy/data/*.fossil (host mount), nats-data (named volume).

Endpoints:

Endpoint	URL	Access
Public HTTPS	`https://sync.example.com`	Cloudflare Tunnel
Tailscale HTTP	`http://100.64.0.1:9000`	Tailnet only
Tailscale NATS	`nats://100.64.0.1:4222`	Tailnet only

Excluded from v1: TLS termination (use reverse proxy), NATS auth, multi-repo, non-root user, OTel in container, backup cron.

WASM Targets

Two targets, same agent API (New -> Start -> Stop):

Target	GOOS	SQLite	Network	HTTP Serve	Entry Point
WASI	`wasip1`	ncruces (as-is)	Host networking	Full	`leaf/cmd/leaf/`
Browser	`js`	ncruces + OPFS VFS	NATS via WebSocket	No-op (can't listen)	`leaf/cmd/wasm/`

Key decisions:

simio.Storage interface added (following Clock/Rand pattern): OSStorage (prod/WASI), MemStorage (tests), OPFSStorage (browser, //go:build js).
Browser NATS uses nats.SetCustomDialer() with a syscall/js WebSocket dialer.
Build-tagged no-ops: serve_http_js.go, signals_js.go replace platform features in browser.
Browser exposes JS globals (edgesync.newAgent, .start, .stop, .syncNow, .clone) and runs in a Web Worker.
stash/ and undo/ excluded from WASM (//go:build !js) -- checkout features, not needed for sync.

Observability

Observer Pattern

sync.Observer interface in libfossil -- zero OTel dependency. Lifecycle hooks injected via SyncOpts.Observer and CloneOpts.Observer. Nil observer uses nopObserver (zero-cost, ~2ns indirect call).

type Observer interface {
    Started(ctx context.Context, info SessionStart) context.Context
    RoundStarted(ctx context.Context, round int) context.Context
    RoundCompleted(ctx context.Context, round int, stats RoundStats)
    Completed(ctx context.Context, info SessionEnd, err error)
    Error(ctx context.Context, err error)
    HandleStarted(ctx context.Context, info HandleStart) context.Context
    HandleCompleted(ctx context.Context, info HandleEnd)
}

Design constraint: No per-card child spans. Card-level detail aggregated into RoundStats (files, gimmes, igots, bytes). HandleStarted/HandleCompleted cover server-side request lifecycle. Bridge uses Observer for tracing without an OTel dependency.

OTel Integration

All OTel code lives in leaf/telemetry/ (build-tagged !wasip1 && !js). WASM gets no-op stubs. libfossil and bridge modules have zero OTel dependencies.

Setup: telemetry.Setup() initializes TracerProvider + MeterProvider with OTLP exporters. Falls back to standard OTEL_* env vars. No endpoint = no-op (zero overhead).

Shutdown order: agent.Stop() first (drains in-flight sync), then telemetry.Shutdown() (flushes spans/metrics).

Metric	Type
`sync.sessions.total`	Counter
`sync.errors.total`	Counter
`sync.duration.seconds`	Histogram
`sync.rounds`	Histogram
`sync.files.sent`	Histogram
`sync.files.received`	Histogram
`sync.uv.files.sent`	Histogram
`sync.uv.files.received`	Histogram

slog integration: log.Printf replaced with slog.InfoContext/slog.ErrorContext across agent, bridge, and serve_http. Context carries trace/span IDs via otelslog bridge.

Backend swappable via env vars only (Honeycomb, Grafana Cloud, Jaeger, Axiom). OTel secrets managed by Doppler -- no .env files.

CLI Flag	Env Var
`--otel-endpoint`	`OTEL_EXPORTER_OTLP_ENDPOINT`
`--otel-headers`	`OTEL_EXPORTER_OTLP_HEADERS`
`--otel-service-name`	`OTEL_SERVICE_NAME`

Module Layout

Four modules in EdgeSync's go.work:

Module	OTel Deps	NATS Deps	Purpose
`.` (root)	No	No	CLI, sim tests, soak runner
`leaf/`	Yes	Yes	Agent daemon, telemetry
`bridge/`	No	Yes	NATS-HTTP translator
`dst/`	No	No	Deterministic simulation

libfossil is an external dependency at github.com/danmestas/libfossil (public, standalone repo). For local development, copy go.work.example to go.work and clone libfossil at ../libfossil.