D 2026-03-15T20:27:27.551
L NATS\sServer\s—\sSetup\s&\sArchitecture
N text/x-markdown
P ca9cc569520239ac7f73ae44224891038f2095cb0ac5d987dbc78e0d6814ddd1
U dmestas
W 6495
# NATS Server — Setup & Architecture
NATS 2.10 JetStream server for the **Fossil NATS Sync** project. Provides durable
pub/sub messaging between Fossil repositories and their leaf-node agents.
Public endpoint: `tls://sync.craftdesign.group:4222`
---
## Architecture
``` pikchr
right
# External clients
C1: box "Leaf Node" color white fill 0x3a5a3a width 1.2 height 0.4 rad 4px
move 0.1
arrow "TLS :4222" above color 0x555555 thick
move 0.1
NS: box "nats-server" color white fill 0x2a4a8a width 1.4 height 0.4 rad 4px
move 0.1
arrow color 0x555555
move 0.1
AC: box "Account JWT" fill 0xddeeff width 1.4 height 0.4 rad 4px
move 0.1
arrow color 0x555555
move 0.1
JS: box "JetStream" color white fill 0x2a6b8a width 1.3 height 0.4 rad 4px
# Second client below
C2: box "Admin CLI" color white fill 0x3a5a3a width 1.2 height 0.4 rad 4px \
at C1.s + (0, -0.7)
arrow from C2.e right 0.1 then to NS.sw color 0x555555
# TLS cert path below nats-server
down
TLS: box "acme.sh" fill 0xd0f0d0 width 1.0 height 0.35 rad 4px at NS.s + (0, -0.7)
arrow from TLS.n to NS.s color 0x888888 <- "cert" rjust small
CF: box "Cloudflare DNS" fill 0xd0f0d0 width 1.4 height 0.35 rad 4px at TLS.e + (0.6, 0)
arrow <-> from TLS.e to CF.w color 0x888888 "DNS-01" above small
# Streams below JetStream
ES: box "EDGESYNC" fill 0xddeeff width 1.3 height 0.35 rad 4px at JS.s + (0, -0.6)
GL: box "GO_LIBFOSSIL" fill 0xddeeff width 1.3 height 0.35 rad 4px at JS.s + (0, -1.1)
arrow from JS.s to ES.n color 0x888888
arrow from JS.s down 0.05 then to GL.n color 0x888888
```
---
## What Is Installed
| Component | Version | Path |
|-----------|---------|------|
| nats-server | 2.10.22 | `/usr/local/bin/nats-server` |
| nats CLI | 0.1.4 | `/usr/local/bin/nats` |
| nsc | 2.9.0 | `~/.local/bin/nsc` (dmestas) |
| acme.sh | latest | `/root/.acme.sh/acme.sh` |
| TLS cert | Let's Encrypt ECC | `/var/nats/tls/sync.craftdesign.group.{crt,key}` |
### Key Paths
| Path | Purpose |
|------|---------|
| `/usr/local/bin/nats-server` | Server binary |
| `/etc/nats/nats-server.conf` | Live rendered config |
| `/etc/nats/nats-server.conf.template` | Config template (envsubst vars) |
| `/etc/nats/operator.env` | Operator JWT (root-owned, 0600) |
| `/etc/nats/cf-token.env` | Cloudflare API token (root-owned, 0600) |
| `/var/nats/jetstream/` | JetStream message storage |
| `/var/nats/jwt/` | Account JWT resolver directory |
| `/var/nats/tls/` | TLS cert + key |
| `~/Projects/infra/nats/creds/` | `.creds` files for each user |
| `~/.local/share/nats/nsc/` | nsc operator/account/key store (dmestas) |
| `/etc/systemd/system/nats.service` | Systemd unit |
---
## Auth Hierarchy
``` pikchr
down
OP: box "Operator: fossil-sync" fill 0xfff0cc width 2.4 height 0.45 rad 5px
move 0.3
# Three accounts side by side
SYS: box "SYS" fill 0xd0eeff width 0.9 height 0.4 rad 4px at OP.s + (-1.5, -0.5)
ES: box "edgesync" fill 0xd0f0d0 width 1.1 height 0.4 rad 4px at OP.s + (0, -0.5)
GL: box "go-libfossil" fill 0xd0f0d0 width 1.3 height 0.4 rad 4px at OP.s + (1.6, -0.5)
line from OP.s to SYS.n color 0x888888
line from OP.s to ES.n color 0x888888
line from OP.s to GL.n color 0x888888
# Users under edgesync
ESA: box "admin" fill 0xeeffee width 0.9 height 0.35 rad 4px at ES.s + (-0.55, -0.55)
ESB: box "bridge" fill 0xeeffee width 0.9 height 0.35 rad 4px at ES.s + (0.55, -0.55)
line from ES.s to ESA.n color 0x888888
line from ES.s to ESB.n color 0x888888
# Users under go-libfossil
GLA: box "admin" fill 0xeeffee width 0.9 height 0.35 rad 4px at GL.s + (-0.55, -0.55)
GLB: box "bridge" fill 0xeeffee width 0.9 height 0.35 rad 4px at GL.s + (0.55, -0.55)
line from GL.s to GLA.n color 0x888888
line from GL.s to GLB.n color 0x888888
# Annotations
text "pub/sub: >" small at ESA.s + (0, -0.2)
text "fossil.edgesync.>" small at ESB.s + (0, -0.2)
text "pub/sub: >" small at GLA.s + (0, -0.2)
text "fossil.go-libfossil.>" small at GLB.s + (0, -0.2)
```
- **admin** users: full `>` pub/sub (local provisioning only)
- **bridge** users: scoped to `fossil.<account>.>` (bridge daemon)
- **SYS** account: NATS system events (server-internal use)
---
## JetStream Streams
| Stream | Subject filter | Retention | Max size |
|--------|---------------|-----------|---------|
| `EDGESYNC` | `fossil.edgesync.>` | 7 days | 1 GB |
| `GO_LIBFOSSIL` | `fossil.go-libfossil.>` | 7 days | 1 GB |
Storage: file (`/var/nats/jetstream/`)
Replicas: 1 (single-node)
Discard policy: old (drop oldest when full)
Dupe window: 2 minutes
---
## Install Scripts (4 Phases)
All scripts live in `nats/` in the `infra` Fossil repo.
``` pikchr
down
P1: box "Phase 1: install.sh (root)" fill 0xfff0cc width 3.2 height 0.45 rad 5px
text "binary · nats user · dirs · TLS cert · systemd unit" small at P1.s + (0, -0.18)
arrow from P1.s down 0.28 color 0x888888
P2: box "Phase 2: bootstrap-creds.sh (dmestas)" fill 0xd0f0d0 width 3.2 height 0.45 rad 5px
text "nsc operator · accounts · users · .creds files · operator.env" small at P2.s + (0, -0.18)
arrow from P2.s down 0.28 color 0x888888
P3: box "Phase 3: start.sh (root)" fill 0xd0eeff width 3.2 height 0.45 rad 5px
text "render config · start service · copy account JWTs" small at P3.s + (0, -0.18)
arrow from P3.s down 0.28 color 0x888888
P4: box "Phase 4: provision-streams.sh (dmestas)" fill 0xd0f0ff width 3.2 height 0.45 rad 5px
text "create EDGESYNC and GO_LIBFOSSIL streams" small at P4.s + (0, -0.18)
```
### Running a fresh install
```sh
# On the server, as dmestas:
cd ~/Projects/infra
# Phase 1 (root) — idempotent
export NATS_VERSION=2.10.22
sudo --preserve-env=NATS_VERSION bash nats/install.sh
# Phase 2 (dmestas) — idempotent
bash nats/bootstrap-creds.sh
# Phase 3 (root) — idempotent
sudo bash nats/start.sh
# Phase 4 (dmestas)
bash nats/provision-streams.sh
```
**Prerequisites for Phase 1:**
- `/etc/nats/cf-token.env` must exist with `CF_Token="<token>"` and mode 0600
**Prerequisites for Phase 2:**
- nsc installed (`~/.local/bin/nsc`)
- Phase 1 complete
---
## Firewall
UFW is used (not Hetzner Cloud firewall).
```
sudo ufw allow 4222/tcp # NATS TLS (already open)
sudo ufw status
```
Port 4222 is the only NATS-related port that needs to be open externally.
Z d2cb1ed55dd293358df6c8502956281a