D 2026-03-15T20:21:57.854 L NATS\sServer\s—\sSetup\s&\sArchitecture N text/x-markdown U dmestas W 7076 # NATS Server — Setup & Architecture NATS 2.10 JetStream server for the **Fossil NATS Sync** project. Provides durable pub/sub messaging between Fossil repositories and their leaf-node agents. Public endpoint: `tls://sync.craftdesign.group:4222` --- ## Architecture ```pikchr scale = 0.85 # External clients (left column) box "External Leaf Node" "(e.g. dev laptop)" fit rad 6px at (0, 2.4) box "External Leaf Node" "(e.g. CI runner)" fit rad 6px at (0, 1.2) box "Local Admin" "(nats CLI)" fit rad 6px at (0, 0.0) # Arrow labels arrow from 1st box.e right 0.3 then right 0.9; "TLS :4222" above arrow from 2nd box.e right 0.3 then right 0.9; "TLS :4222" above arrow from 3rd box.e right 0.3 then right 0.9; "TLS :4222" above # NATS server box NX = 2.8 NY = 1.2 box wid 2.2 ht 2.8 rad 6px at (NX, NY) "nats-server 2.10" "(vps-nbg1)" big bold fill lightyellow # JetStream box "JetStream" fit rad 4px at (NX, 1.8) fill lightblue arrow from (NX, 1.55) down 0.25 box "/var/nats/jetstream" fit rad 4px at (NX, 1.2) fill lightyellow # JWT resolver arrow from (NX, 0.95) down 0.25 box "/var/nats/jwt" fit rad 4px at (NX, 0.6) fill lightyellow # Accounts (right column) RX = 5.4 box "Account: edgesync" fit rad 6px at (RX, 2.2) fill lightgreen box " users: admin, bridge" fit rad 4px at (RX, 1.8) fill honeydew box "Account: go-libfossil" fit rad 6px at (RX, 1.1) fill lightgreen box " users: admin, bridge" fit rad 4px at (RX, 0.7) fill honeydew box "Account: SYS" fit rad 6px at (RX, 0.1) fill lightyellow arrow from (NX+1.1, 2.0) right 0.3; "" above arrow from (NX+1.1, 1.0) right 0.3; "" above # TLS / cert renewal (bottom) box "acme.sh" fit rad 6px at (1.4, -0.8) fill lightcyan box "Cloudflare DNS" fit rad 6px at (3.4, -0.8) fill lightcyan arrow from (1.4, -0.55) up 0.2 then to (NX-0.4, -0.1) "cert" above arrow <-> from (1.4+0.7, -0.8) right 0.6; "DNS-01" above # systemd (top) box "systemd nats.service" fit rad 6px at (NX, 3.2) fill lavender arrow from (NX, 3.0) down 0.15 ``` --- ## What Is Installed | Component | Version | Path | |-----------|---------|------| | nats-server | 2.10.22 | `/usr/local/bin/nats-server` | | nats CLI | 0.1.4 | `/usr/local/bin/nats` | | nsc | 2.9.0 | `~/.local/bin/nsc` (dmestas) | | acme.sh | latest | `/root/.acme.sh/acme.sh` | | TLS cert | Let's Encrypt ECC | `/var/nats/tls/sync.craftdesign.group.{crt,key}` | ### Key Paths | Path | Purpose | |------|---------| | `/usr/local/bin/nats-server` | Server binary | | `/etc/nats/nats-server.conf` | Live rendered config | | `/etc/nats/nats-server.conf.template` | Config template (envsubst vars) | | `/etc/nats/operator.env` | Operator JWT (root-owned, 0600) | | `/etc/nats/cf-token.env` | Cloudflare API token (root-owned, 0600) | | `/var/nats/jetstream/` | JetStream message storage | | `/var/nats/jwt/` | Account JWT resolver directory | | `/var/nats/tls/` | TLS cert + key | | `~/Projects/infra/nats/creds/` | `.creds` files for each user | | `~/.local/share/nats/nsc/` | nsc operator/account/key store (dmestas) | | `/etc/systemd/system/nats.service` | Systemd unit | --- ## Auth Hierarchy ```pikchr scale = 0.8 # Operator box "Operator: fossil-sync" wid 3.2 ht 0.5 rad 6px at (2.5, 3.0) fill lightyellow bold # Accounts row box "SYS" wid 1.2 ht 0.5 rad 6px at (0.8, 2.0) fill lightcyan box "edgesync" wid 1.4 ht 0.5 rad 6px at (2.5, 2.0) fill lightgreen box "go-libfossil" wid 1.6 ht 0.5 rad 6px at (4.4, 2.0) fill lightgreen line from (2.5, 2.75) to (0.8, 2.25) line from (2.5, 2.75) to (2.5, 2.25) line from (2.5, 2.75) to (4.4, 2.25) # Users under edgesync box "admin" wid 1.0 ht 0.4 rad 4px at (1.9, 1.0) fill honeydew box "bridge" wid 1.0 ht 0.4 rad 4px at (3.1, 1.0) fill honeydew line from (2.5, 1.75) to (1.9, 1.2) line from (2.5, 1.75) to (3.1, 1.2) # Users under go-libfossil box "admin" wid 1.0 ht 0.4 rad 4px at (3.8, 1.0) fill honeydew box "bridge" wid 1.0 ht 0.4 rad 4px at (5.0, 1.0) fill honeydew line from (4.4, 1.75) to (3.8, 1.2) line from (4.4, 1.75) to (5.0, 1.2) # Labels text "pub/sub: >" at (2.3, 0.55) small text "pub/sub: fossil.edgesync.>" at (2.5, 0.55) small text "pub/sub: >" at (4.2, 0.55) small text "pub/sub: fossil.go-libfossil.>" at (4.4, 0.55) small ``` - **admin** users: full `>` pub/sub (local provisioning only) - **bridge** users: scoped to `fossil..>` (bridge daemon) - **SYS** account: NATS system events (server-internal use) --- ## JetStream Streams | Stream | Subject filter | Retention | Max size | |--------|---------------|-----------|---------| | `EDGESYNC` | `fossil.edgesync.>` | 7 days | 1 GB | | `GO_LIBFOSSIL` | `fossil.go-libfossil.>` | 7 days | 1 GB | Storage: file (`/var/nats/jetstream/`) Replicas: 1 (single-node) Discard policy: old (drop oldest when full) Dupe window: 2 minutes --- ## Install Scripts (4 Phases) All scripts live in `nats/` in the `infra` Fossil repo. ```pikchr scale = 0.85 X = 0.0 W = 3.0 H = 0.7 G = 0.3 box "Phase 1: install.sh" wid W ht H rad 6px at (X, 3*H+3*G) fill lightyellow text "run as root" small at last box.s + (0,0.1) box "Phase 2: bootstrap-creds.sh" wid W ht H rad 6px at (X, 2*H+2*G) fill lightgreen text "run as dmestas" small at last box.s + (0,0.1) box "Phase 3: start.sh" wid W ht H rad 6px at (X, 1*H+1*G) fill lightblue text "run as root" small at last box.s + (0,0.1) box "Phase 4: provision-streams.sh" wid W ht H rad 6px at (X, 0) fill lightcyan text "run as dmestas" small at last box.s + (0,0.1) arrow from (X, 3*H+3*G - H/2) down G+0.05 arrow from (X, 2*H+2*G - H/2) down G+0.05 arrow from (X, 1*H+1*G - H/2) down G+0.05 # Details on right DX = 2.2 text "Download binary, create nats user, dirs," at (DX+1.8, 3*H+3*G+0.05) small ljust text "issue TLS cert (acme.sh DNS-01), install systemd unit" at (DX+1.8, 3*H+3*G-0.18) small ljust text "Create operator/accounts/users via nsc," at (DX+1.8, 2*H+2*G+0.05) small ljust text "generate .creds files, write operator.env" at (DX+1.8, 2*H+2*G-0.18) small ljust text "Render config, start nats.service," at (DX+1.8, 1*H+1*G+0.05) small ljust text "copy account JWTs to resolver dir" at (DX+1.8, 1*H+1*G-0.18) small ljust text "Create EDGESYNC and GO_LIBFOSSIL" at (DX+1.8, 0+0.05) small ljust text "JetStream streams" at (DX+1.8, 0-0.18) small ljust ``` ### Running a fresh install ```sh # On the server, as dmestas: cd ~/Projects/infra # Phase 1 (root) — idempotent export NATS_VERSION=2.10.22 sudo --preserve-env=NATS_VERSION bash nats/install.sh # Phase 2 (dmestas) — idempotent bash nats/bootstrap-creds.sh # Phase 3 (root) — idempotent sudo bash nats/start.sh # Phase 4 (dmestas) bash nats/provision-streams.sh ``` **Prerequisites for Phase 1:** - `/etc/nats/cf-token.env` must exist with `CF_Token=""` and mode 0600 **Prerequisites for Phase 2:** - nsc installed (`~/.local/bin/nsc`) - Phase 1 complete --- ## Firewall UFW is used (not Hetzner Cloud firewall). ``` sudo ufw allow 4222/tcp # NATS TLS (already open) sudo ufw status ``` Port 4222 is the only NATS-related port that needs to be open externally. Z 5ff27ea1617fb6b418ea5969a43c8bfe