D 2026-03-15T20:27:27.551 L NATS\sServer\s—\sSetup\s&\sArchitecture N text/x-markdown P ca9cc569520239ac7f73ae44224891038f2095cb0ac5d987dbc78e0d6814ddd1 U dmestas W 6495 # NATS Server — Setup & Architecture NATS 2.10 JetStream server for the **Fossil NATS Sync** project. Provides durable pub/sub messaging between Fossil repositories and their leaf-node agents. Public endpoint: `tls://sync.craftdesign.group:4222` --- ## Architecture ``` pikchr right # External clients C1: box "Leaf Node" color white fill 0x3a5a3a width 1.2 height 0.4 rad 4px move 0.1 arrow "TLS :4222" above color 0x555555 thick move 0.1 NS: box "nats-server" color white fill 0x2a4a8a width 1.4 height 0.4 rad 4px move 0.1 arrow color 0x555555 move 0.1 AC: box "Account JWT" fill 0xddeeff width 1.4 height 0.4 rad 4px move 0.1 arrow color 0x555555 move 0.1 JS: box "JetStream" color white fill 0x2a6b8a width 1.3 height 0.4 rad 4px # Second client below C2: box "Admin CLI" color white fill 0x3a5a3a width 1.2 height 0.4 rad 4px \ at C1.s + (0, -0.7) arrow from C2.e right 0.1 then to NS.sw color 0x555555 # TLS cert path below nats-server down TLS: box "acme.sh" fill 0xd0f0d0 width 1.0 height 0.35 rad 4px at NS.s + (0, -0.7) arrow from TLS.n to NS.s color 0x888888 <- "cert" rjust small CF: box "Cloudflare DNS" fill 0xd0f0d0 width 1.4 height 0.35 rad 4px at TLS.e + (0.6, 0) arrow <-> from TLS.e to CF.w color 0x888888 "DNS-01" above small # Streams below JetStream ES: box "EDGESYNC" fill 0xddeeff width 1.3 height 0.35 rad 4px at JS.s + (0, -0.6) GL: box "GO_LIBFOSSIL" fill 0xddeeff width 1.3 height 0.35 rad 4px at JS.s + (0, -1.1) arrow from JS.s to ES.n color 0x888888 arrow from JS.s down 0.05 then to GL.n color 0x888888 ``` --- ## What Is Installed | Component | Version | Path | |-----------|---------|------| | nats-server | 2.10.22 | `/usr/local/bin/nats-server` | | nats CLI | 0.1.4 | `/usr/local/bin/nats` | | nsc | 2.9.0 | `~/.local/bin/nsc` (dmestas) | | acme.sh | latest | `/root/.acme.sh/acme.sh` | | TLS cert | Let's Encrypt ECC | `/var/nats/tls/sync.craftdesign.group.{crt,key}` | ### Key Paths | Path | Purpose | |------|---------| | `/usr/local/bin/nats-server` | Server binary | | `/etc/nats/nats-server.conf` | Live rendered config | | `/etc/nats/nats-server.conf.template` | Config template (envsubst vars) | | `/etc/nats/operator.env` | Operator JWT (root-owned, 0600) | | `/etc/nats/cf-token.env` | Cloudflare API token (root-owned, 0600) | | `/var/nats/jetstream/` | JetStream message storage | | `/var/nats/jwt/` | Account JWT resolver directory | | `/var/nats/tls/` | TLS cert + key | | `~/Projects/infra/nats/creds/` | `.creds` files for each user | | `~/.local/share/nats/nsc/` | nsc operator/account/key store (dmestas) | | `/etc/systemd/system/nats.service` | Systemd unit | --- ## Auth Hierarchy ``` pikchr down OP: box "Operator: fossil-sync" fill 0xfff0cc width 2.4 height 0.45 rad 5px move 0.3 # Three accounts side by side SYS: box "SYS" fill 0xd0eeff width 0.9 height 0.4 rad 4px at OP.s + (-1.5, -0.5) ES: box "edgesync" fill 0xd0f0d0 width 1.1 height 0.4 rad 4px at OP.s + (0, -0.5) GL: box "go-libfossil" fill 0xd0f0d0 width 1.3 height 0.4 rad 4px at OP.s + (1.6, -0.5) line from OP.s to SYS.n color 0x888888 line from OP.s to ES.n color 0x888888 line from OP.s to GL.n color 0x888888 # Users under edgesync ESA: box "admin" fill 0xeeffee width 0.9 height 0.35 rad 4px at ES.s + (-0.55, -0.55) ESB: box "bridge" fill 0xeeffee width 0.9 height 0.35 rad 4px at ES.s + (0.55, -0.55) line from ES.s to ESA.n color 0x888888 line from ES.s to ESB.n color 0x888888 # Users under go-libfossil GLA: box "admin" fill 0xeeffee width 0.9 height 0.35 rad 4px at GL.s + (-0.55, -0.55) GLB: box "bridge" fill 0xeeffee width 0.9 height 0.35 rad 4px at GL.s + (0.55, -0.55) line from GL.s to GLA.n color 0x888888 line from GL.s to GLB.n color 0x888888 # Annotations text "pub/sub: >" small at ESA.s + (0, -0.2) text "fossil.edgesync.>" small at ESB.s + (0, -0.2) text "pub/sub: >" small at GLA.s + (0, -0.2) text "fossil.go-libfossil.>" small at GLB.s + (0, -0.2) ``` - **admin** users: full `>` pub/sub (local provisioning only) - **bridge** users: scoped to `fossil..>` (bridge daemon) - **SYS** account: NATS system events (server-internal use) --- ## JetStream Streams | Stream | Subject filter | Retention | Max size | |--------|---------------|-----------|---------| | `EDGESYNC` | `fossil.edgesync.>` | 7 days | 1 GB | | `GO_LIBFOSSIL` | `fossil.go-libfossil.>` | 7 days | 1 GB | Storage: file (`/var/nats/jetstream/`) Replicas: 1 (single-node) Discard policy: old (drop oldest when full) Dupe window: 2 minutes --- ## Install Scripts (4 Phases) All scripts live in `nats/` in the `infra` Fossil repo. ``` pikchr down P1: box "Phase 1: install.sh (root)" fill 0xfff0cc width 3.2 height 0.45 rad 5px text "binary · nats user · dirs · TLS cert · systemd unit" small at P1.s + (0, -0.18) arrow from P1.s down 0.28 color 0x888888 P2: box "Phase 2: bootstrap-creds.sh (dmestas)" fill 0xd0f0d0 width 3.2 height 0.45 rad 5px text "nsc operator · accounts · users · .creds files · operator.env" small at P2.s + (0, -0.18) arrow from P2.s down 0.28 color 0x888888 P3: box "Phase 3: start.sh (root)" fill 0xd0eeff width 3.2 height 0.45 rad 5px text "render config · start service · copy account JWTs" small at P3.s + (0, -0.18) arrow from P3.s down 0.28 color 0x888888 P4: box "Phase 4: provision-streams.sh (dmestas)" fill 0xd0f0ff width 3.2 height 0.45 rad 5px text "create EDGESYNC and GO_LIBFOSSIL streams" small at P4.s + (0, -0.18) ``` ### Running a fresh install ```sh # On the server, as dmestas: cd ~/Projects/infra # Phase 1 (root) — idempotent export NATS_VERSION=2.10.22 sudo --preserve-env=NATS_VERSION bash nats/install.sh # Phase 2 (dmestas) — idempotent bash nats/bootstrap-creds.sh # Phase 3 (root) — idempotent sudo bash nats/start.sh # Phase 4 (dmestas) bash nats/provision-streams.sh ``` **Prerequisites for Phase 1:** - `/etc/nats/cf-token.env` must exist with `CF_Token=""` and mode 0600 **Prerequisites for Phase 2:** - nsc installed (`~/.local/bin/nsc`) - Phase 1 complete --- ## Firewall UFW is used (not Hetzner Cloud firewall). ``` sudo ufw allow 4222/tcp # NATS TLS (already open) sudo ufw status ``` Port 4222 is the only NATS-related port that needs to be open externally. Z d2cb1ed55dd293358df6c8502956281a