Changes To NATS Server — Setup & Architecture

# NATS Server — Setup & Architecture

NATS 2.10 JetStream server for the **Fossil NATS Sync** project. Provides durable
pub/sub messaging between Fossil repositories and their leaf-node agents.

Public endpoint: `tls://sync.craftdesign.group:4222`

---

## Architecture

```pikchr
scale = 0.85

# External clients (left column)
box "External Leaf Node" "(e.g. dev laptop)" fit rad 6px at (0, 2.4)
box "External Leaf Node" "(e.g. CI runner)"  fit rad 6px at (0, 1.2)
box "Local Admin"        "(nats CLI)"         fit rad 6px at (0, 0.0)

# Arrow labels
arrow from 1st box.e right 0.3 then right 0.9; "TLS :4222" above
arrow from 2nd box.e right 0.3 then right 0.9; "TLS :4222" above
arrow from 3rd box.e right 0.3 then right 0.9; "TLS :4222" above

# NATS server box
NX = 2.8
NY = 1.2
box wid 2.2 ht 2.8 rad 6px at (NX, NY) "nats-server 2.10" "(vps-nbg1)" big bold fill lightyellow

# JetStream
box "JetStream" fit rad 4px at (NX, 1.8) fill lightblue
arrow from (NX, 1.55) down 0.25
box "/var/nats/jetstream" fit rad 4px at (NX, 1.2) fill lightyellow

# JWT resolver
arrow from (NX, 0.95) down 0.25
box "/var/nats/jwt" fit rad 4px at (NX, 0.6) fill lightyellow

# Accounts (right column)
RX = 5.4
box "Account: edgesync"     fit rad 6px at (RX, 2.2) fill lightgreen
box "  users: admin, bridge" fit rad 4px at (RX, 1.8) fill honeydew
box "Account: go-libfossil"  fit rad 6px at (RX, 1.1) fill lightgreen
box "  users: admin, bridge" fit rad 4px at (RX, 0.7) fill honeydew
box "Account: SYS"           fit rad 6px at (RX, 0.1) fill lightyellow

arrow from (NX+1.1, 2.0) right 0.3; "" above
arrow from (NX+1.1, 1.0) right 0.3; "" above

# TLS / cert renewal (bottom)
box "acme.sh" fit rad 6px at (1.4, -0.8) fill lightcyan
box "Cloudflare DNS" fit rad 6px at (3.4, -0.8) fill lightcyan
arrow from (1.4, -0.55) up 0.2 then to (NX-0.4, -0.1) "cert" above
arrow <-> from (1.4+0.7, -0.8) right 0.6; "DNS-01" above

# systemd (top)
box "systemd nats.service" fit rad 6px at (NX, 3.2) fill lavender
arrow from (NX, 3.0) down 0.15
```

---

## What Is Installed

| Component | Version | Path |
|-----------|---------|------|
| nats-server | 2.10.22 | `/usr/local/bin/nats-server` |
| nats CLI | 0.1.4 | `/usr/local/bin/nats` |
| nsc | 2.9.0 | `~/.local/bin/nsc` (dmestas) |
| acme.sh | latest | `/root/.acme.sh/acme.sh` |
| TLS cert | Let's Encrypt ECC | `/var/nats/tls/sync.craftdesign.group.{crt,key}` |

### Key Paths

| Path | Purpose |
|------|---------|
| `/usr/local/bin/nats-server` | Server binary |
| `/etc/nats/nats-server.conf` | Live rendered config |
| `/etc/nats/nats-server.conf.template` | Config template (envsubst vars) |
| `/etc/nats/operator.env` | Operator JWT (root-owned, 0600) |
| `/etc/nats/cf-token.env` | Cloudflare API token (root-owned, 0600) |
| `/var/nats/jetstream/` | JetStream message storage |
| `/var/nats/jwt/` | Account JWT resolver directory |
| `/var/nats/tls/` | TLS cert + key |
| `~/Projects/infra/nats/creds/` | `.creds` files for each user |
| `~/.local/share/nats/nsc/` | nsc operator/account/key store (dmestas) |
| `/etc/systemd/system/nats.service` | Systemd unit |

---

## Auth Hierarchy

```pikchr
scale = 0.8

# Operator
box "Operator: fossil-sync" wid 3.2 ht 0.5 rad 6px at (2.5, 3.0) fill lightyellow bold

# Accounts row
box "SYS" wid 1.2 ht 0.5 rad 6px at (0.8, 2.0) fill lightcyan
box "edgesync" wid 1.4 ht 0.5 rad 6px at (2.5, 2.0) fill lightgreen
box "go-libfossil" wid 1.6 ht 0.5 rad 6px at (4.4, 2.0) fill lightgreen

line from (2.5, 2.75) to (0.8, 2.25)
line from (2.5, 2.75) to (2.5, 2.25)
line from (2.5, 2.75) to (4.4, 2.25)

# Users under edgesync
box "admin" wid 1.0 ht 0.4 rad 4px at (1.9, 1.0) fill honeydew
box "bridge" wid 1.0 ht 0.4 rad 4px at (3.1, 1.0) fill honeydew
line from (2.5, 1.75) to (1.9, 1.2)
line from (2.5, 1.75) to (3.1, 1.2)

# Users under go-libfossil
box "admin" wid 1.0 ht 0.4 rad 4px at (3.8, 1.0) fill honeydew
box "bridge" wid 1.0 ht 0.4 rad 4px at (5.0, 1.0) fill honeydew
line from (4.4, 1.75) to (3.8, 1.2)
line from (4.4, 1.75) to (5.0, 1.2)

# Labels
text "pub/sub: >" at (2.3, 0.55) small
text "pub/sub: fossil.edgesync.>" at (2.5, 0.55) small
text "pub/sub: >" at (4.2, 0.55) small
text "pub/sub: fossil.go-libfossil.>" at (4.4, 0.55) small
```

- **admin** users: full `>` pub/sub (local provisioning only)
- **bridge** users: scoped to `fossil.<account>.>` (bridge daemon)
- **SYS** account: NATS system events (server-internal use)

---

## JetStream Streams

| Stream | Subject filter | Retention | Max size |
|--------|---------------|-----------|---------|
| `EDGESYNC` | `fossil.edgesync.>` | 7 days | 1 GB |
| `GO_LIBFOSSIL` | `fossil.go-libfossil.>` | 7 days | 1 GB |

Storage: file (`/var/nats/jetstream/`)
Replicas: 1 (single-node)
Discard policy: old (drop oldest when full)
Dupe window: 2 minutes

---

## Install Scripts (4 Phases)

All scripts live in `nats/` in the `infra` Fossil repo.

```pikchr
scale = 0.85
X = 0.0
W = 3.0
H = 0.7
G = 0.3

box "Phase 1: install.sh" wid W ht H rad 6px at (X, 3*H+3*G) fill lightyellow
  text "run as root" small at last box.s + (0,0.1)
box "Phase 2: bootstrap-creds.sh" wid W ht H rad 6px at (X, 2*H+2*G) fill lightgreen
  text "run as dmestas" small at last box.s + (0,0.1)
box "Phase 3: start.sh" wid W ht H rad 6px at (X, 1*H+1*G) fill lightblue
  text "run as root" small at last box.s + (0,0.1)
box "Phase 4: provision-streams.sh" wid W ht H rad 6px at (X, 0) fill lightcyan
  text "run as dmestas" small at last box.s + (0,0.1)

arrow from (X, 3*H+3*G - H/2) down G+0.05
arrow from (X, 2*H+2*G - H/2) down G+0.05
arrow from (X, 1*H+1*G - H/2) down G+0.05

# Details on right
DX = 2.2
text "Download binary, create nats user, dirs," at (DX+1.8, 3*H+3*G+0.05) small ljust
text "issue TLS cert (acme.sh DNS-01), install systemd unit" at (DX+1.8, 3*H+3*G-0.18) small ljust
text "Create operator/accounts/users via nsc," at (DX+1.8, 2*H+2*G+0.05) small ljust
text "generate .creds files, write operator.env" at (DX+1.8, 2*H+2*G-0.18) small ljust
text "Render config, start nats.service," at (DX+1.8, 1*H+1*G+0.05) small ljust
text "copy account JWTs to resolver dir" at (DX+1.8, 1*H+1*G-0.18) small ljust
text "Create EDGESYNC and GO_LIBFOSSIL" at (DX+1.8, 0+0.05) small ljust
text "JetStream streams" at (DX+1.8, 0-0.18) small ljust
```

### Running a fresh install

```sh
# On the server, as dmestas:
cd ~/Projects/infra

# Phase 1 (root) — idempotent
export NATS_VERSION=2.10.22
sudo --preserve-env=NATS_VERSION bash nats/install.sh

# Phase 2 (dmestas) — idempotent
bash nats/bootstrap-creds.sh

# Phase 3 (root) — idempotent
sudo bash nats/start.sh

# Phase 4 (dmestas)
bash nats/provision-streams.sh
```

**Prerequisites for Phase 1:**
- `/etc/nats/cf-token.env` must exist with `CF_Token="<token>"` and mode 0600

**Prerequisites for Phase 2:**
- nsc installed (`~/.local/bin/nsc`)
- Phase 1 complete

---

## Firewall

UFW is used (not Hetzner Cloud firewall).

```
sudo ufw allow 4222/tcp   # NATS TLS (already open)
sudo ufw status
```

Port 4222 is the only NATS-related port that needs to be open externally.