# NATS Server Setup — Design Spec

**Date:** 2026-03-15
**Status:** Approved
**Context:** Infrastructure foundation for Fossil NATS Sync (Phase E — Leaf Agent)

---

## Overview

Configure a NATS 2.10 server with JetStream on the Hetzner VPS as the messaging hub for the Fossil NATS Sync project. The server must be reachable by both Tailscale-connected leaf nodes (MacBook, Mac Mini) and external public clients. Configuration is version-controlled in a new `infra` Fossil repo for reproducible setup and future cluster expansion.

---

## Architecture

```
External clients / leaf agents
        │
        │ tls://sync.craftdesign.group:4222
        ▼
nats-server 2.10 (systemd, linux/arm64)
  listen: 0.0.0.0:4222  (TLS — external + Tailscale + local admin)
    │
    ├── /var/nats/jetstream   JetStream persistent storage
    ├── /var/nats/jwt         JWT resolver store
    └── /var/nats/tls         TLS cert + key (acme.sh managed)

Local admin tools (nsc, nats CLI) connect to tls://localhost:4222 --tlsskipverify
```

**Single listener:** NATS 2.10 does not support two independent TCP listeners with different TLS policies in one server process. All connections — external, Tailscale, and local admin — use the single TLS listener on port 4222. Local admin tools (`nsc push`, `nats` CLI) connect to `tls://localhost:4222` with `--tlsskipverify` (acceptable for loopback connections where network interception is impossible).

---

## NATS Version

**Target version: 2.10.x** (latest 2.10 stable at install time)

`install.sh` pins a `NATS_VERSION` variable (e.g., `2.10.22`) and downloads the `linux-arm64` binary:

```
https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/nats-server-v${NATS_VERSION}-linux-arm64.tar.gz
```

The `Dockerfile` pins the same minor version: `FROM nats:2.10-alpine`. Both are updated together when upgrading.

---

## Config Repo (`infra`)

New Fossil repo: `/var/fossil/repos/infra.fossil`, browsable at `fossil.craftdesign.group/infra/`.

```
infra/
├── nats/
│   ├── nats-server.conf.template  # Config template (contains $OPERATOR_JWT placeholder)
│   ├── nats.service               # Systemd unit
│   ├── install.sh                 # Phase 1: binary, dirs, TLS, unit file (does NOT start service)
│   ├── bootstrap-creds.sh         # Phase 2: one-time nsc operator/account/user setup
│   ├── start.sh                   # Phase 3: render config, enable and start service, nsc push
│   ├── provision-streams.sh       # Phase 4: idempotent JetStream stream creation
│   ├── Dockerfile                 # Docker surface for future cluster nodes
│   ├── creds/
│   │   └── README.md              # How to regenerate operator/account/user creds with nsc
│   └── README.md
└── README.md
```

`creds/` is listed in `.fossil-ignore` — NKey seeds and JWT files are never committed. `/etc/nats/nats-server.conf` (the rendered live config) is also never committed.

---

## Install Phases (explicit ordering)

**Phase 1 — `install.sh`**: installs binary, creates dirs, obtains TLS cert, copies config template and unit file. Does **not** start the service (operator JWT not yet available).

**Phase 2 — `bootstrap-creds.sh`**: one-time `nsc` setup. Creates operator, accounts, users. Writes `/etc/nats/operator.env` containing `OPERATOR_JWT=<value>`.

**Phase 3 — `start.sh`**: renders config template → live config, enables and starts the service, pushes JWTs to the running resolver.

**Phase 4 — `provision-streams.sh`**: creates JetStream streams after the server is running. Idempotent and non-interactive.

---

## TLS Provisioning

**Mechanism: `acme.sh` with Cloudflare DNS-01 challenge**

Traefik already holds ports 80/443, making HTTP-01 challenges impossible for non-Coolify services. DNS-01 via the Cloudflare API sidesteps this completely.

`install.sh` handles TLS setup:

```bash
# Install acme.sh (if not present)
curl https://get.acme.sh | sh -s email=admin@craftdesign.group

# cf-token.env must already exist at /etc/nats/cf-token.env (operator places it before running install.sh)
# It exports: CF_Token="<cloudflare-api-token>" with Zone:DNS:Edit permission for craftdesign.group
source /etc/nats/cf-token.env

~/.acme.sh/acme.sh --issue --dns dns_cf \
  -d sync.craftdesign.group \
  --key-file        /var/nats/tls/sync.craftdesign.group.key \
  --fullchain-file  /var/nats/tls/sync.craftdesign.group.crt \
  --reloadcmd       "systemctl kill --signal=SIGHUP nats"

# Set ownership so nats-server (running as nats user) can read the key
chown nats:nats /var/nats/tls/sync.craftdesign.group.key /var/nats/tls/sync.craftdesign.group.crt
chmod 0640 /var/nats/tls/sync.craftdesign.group.key /var/nats/tls/sync.craftdesign.group.crt
```

- `--fullchain-file` ensures the cert includes the Let's Encrypt intermediate
- `--reloadcmd` sends `SIGHUP` to `nats-server` after renewal; NATS reloads TLS certs on SIGHUP without dropping connections
- The Cloudflare API token requires `Zone:DNS:Edit` for `craftdesign.group`; stored at `/etc/nats/cf-token.env` (mode 0600, root-owned) — never committed to the repo
- `/etc/nats/cf-token.env` is also used by `acme.sh`'s cron renewal job, so it must persist on disk

---

## `install.sh` Responsibilities (idempotent, Phase 1)

1. Verify `NATS_VERSION` is set; download and install `nats-server` binary to `/usr/local/bin/nats-server`
2. Create `nats` system user (no login shell) if not present
3. Create directories:
   - `/var/nats/jetstream` — owned by `nats:nats`, mode 0750
   - `/var/nats/jwt`       — owned by `nats:nats`, mode 0750
   - `/var/nats/tls`       — owned by `nats:nats`, mode 0750
   - `/etc/nats/`          — owned by root, mode 0755
4. Verify `/etc/nats/cf-token.env` exists and is mode 0600; fail with clear error if not
5. Copy `nats-server.conf.template` to `/etc/nats/nats-server.conf.template`
6. Install `acme.sh` if not present; issue TLS cert via DNS-01 (skips if cert already valid); set ownership `nats:nats` on cert and key files
7. Install `nats.service` to `/etc/systemd/system/`; `systemctl daemon-reload`
8. Print: "Phase 1 complete. Run bootstrap-creds.sh next."

The service is **not** started by `install.sh`.

---

## `bootstrap-creds.sh` Responsibilities (Phase 2, one-time)

Requires `nsc` CLI (install via `curl -fsSL https://install.nats.io/nsc | sh`).

No server connection is needed at this stage — `nsc` works purely locally.

```bash
nsc add operator --name fossil-sync

# Use localhost TLS for all local nsc/nats CLI operations
nsc edit operator --account-jwt-server-url tls://localhost:4222

nsc add account --name edgesync
nsc add account --name go-libfossil

# Admin user per account — full pub/sub for local stream provisioning only
nsc add user --account edgesync     --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account go-libfossil --name admin --allow-pub '>' --allow-sub '>'

# Bridge user — scoped to project subjects
nsc add user --account edgesync \
  --name bridge \
  --allow-pub  'fossil.edgesync.>' \
  --allow-sub  'fossil.edgesync.>'

nsc add user --account go-libfossil \
  --name bridge \
  --allow-pub  'fossil.go-libfossil.>' \
  --allow-sub  'fossil.go-libfossil.>'

# Generate .creds files
nsc generate creds --account edgesync     --name bridge > creds/edgesync-bridge.creds
nsc generate creds --account go-libfossil --name bridge > creds/go-libfossil-bridge.creds
nsc generate creds --account edgesync     --name admin  > creds/edgesync-admin.creds
nsc generate creds --account go-libfossil --name admin  > creds/go-libfossil-admin.creds

# Extract operator JWT using nsc generate config (the supported extraction method)
OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null \
  | grep '^operator:' \
  | awk '{print $2}')

echo "OPERATOR_JWT=${OPERATOR_JWT}" > /etc/nats/operator.env
chmod 0600 /etc/nats/operator.env

echo "Phase 2 complete. Run start.sh next."
```

`nsc` local store lives at `~/.local/share/nats/nsc/` (default). `creds/README.md` documents the full `nsc` command sequence for recreating the credential tree on a fresh machine.

### Adding a new leaf node (post-setup)

```bash
nsc add user --account <project> \
  --name leaf-<hostname> \
  --allow-pub  'fossil.<project>.igot,fossil.<project>.file,fossil.<project>.delta' \
  --allow-sub  'fossil.<project>.gimme,fossil.<project>.file,fossil.<project>.meta,fossil.<project>.events'

nsc generate creds --account <project> --name leaf-<hostname> > leaf-<hostname>.creds

nsc push --all --server tls://localhost:4222 --tlsskipverify

# Distribute .creds to the leaf node via scp over Tailscale
```

---

## `start.sh` Responsibilities (Phase 3)

Renders config from template (preserving the template file), starts the service, pushes JWTs.

```bash
source /etc/nats/operator.env   # loads OPERATOR_JWT

# Render template → live config (template file is never modified)
envsubst < /etc/nats/nats-server.conf.template > /etc/nats/nats-server.conf

systemctl enable --now nats

# Wait for server to be ready
sleep 2

# Push account JWTs to the running resolver
nsc push --all --server tls://localhost:4222 --tlsskipverify

echo "Phase 3 complete. Run provision-streams.sh next."
```

Re-running `start.sh` is safe: `envsubst` always reads from the `.template` file.

---

## `provision-streams.sh` Responsibilities (Phase 4, idempotent)

Uses `tls://localhost:4222 --tlsskipverify` and admin creds. The `--defaults` flag suppresses all interactive prompts.

```bash
NATS_SERVER="tls://localhost:4222"
NATS_TLS_FLAGS="--tlsskipverify"

for PROJECT in edgesync go-libfossil; do
  STREAM=$(echo "$PROJECT" | tr '[:lower:]' '[:upper:]' | tr '-' '_')
  CREDS="creds/${PROJECT}-admin.creds"

  if nats stream info "$STREAM" \
      --server "$NATS_SERVER" $NATS_TLS_FLAGS \
      --creds  "$CREDS" &>/dev/null; then
    echo "Stream $STREAM already exists, skipping."
    continue
  fi

  nats stream add "$STREAM" \
    --server    "$NATS_SERVER" $NATS_TLS_FLAGS \
    --creds     "$CREDS" \
    --subjects  "fossil.${PROJECT}.>" \
    --retention limits \
    --max-age   168h \
    --max-bytes 1073741824 \
    --storage   file \
    --replicas  1 \
    --defaults

  echo "Stream $STREAM created."
done
```

---

## NATS Server Config Template (`nats-server.conf.template`)

```
server_name: vps-nbg1

listen: 0.0.0.0:4222

tls {
  cert_file: "/var/nats/tls/sync.craftdesign.group.crt"
  key_file:  "/var/nats/tls/sync.craftdesign.group.key"
}

jetstream {
  store_dir: "/var/nats/jetstream"
}

# JWT resolver — accounts pushed via `nsc push --all`
resolver: {
  type:         full
  dir:          "/var/nats/jwt"
  allow_delete: false
  interval:     "2m"
}

# Operator JWT — substituted by start.sh via envsubst
operator: $OPERATOR_JWT

# Cluster placeholder — populate when adding a second node
# cluster {
#   name: fossil-sync
#   listen: 0.0.0.0:6222
#   routes: [ nats://node2.internal:6222 ]
# }
```

---

## Systemd Unit (`nats.service`)

```ini
[Unit]
Description=NATS Server
After=network-online.target
Wants=network-online.target

[Service]
User=nats
Group=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats-server.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
```

---

## Auth & Security

### NKeys + JWT (three-tier)

```
Operator: fossil-sync
├── Account: edgesync
│   ├── User: admin    (pub/sub > — local provisioning on VPS only)
│   ├── User: bridge   (pub+sub fossil.edgesync.>)
│   └── User: leaf-<hostname>
└── Account: go-libfossil
    ├── User: admin
    ├── User: bridge   (pub+sub fossil.go-libfossil.>)
    └── User: leaf-<hostname>
```

- **Operator** is the root of trust — NKey seed lives in `nsc` local store on the VPS
- **Accounts** map 1:1 to Fossil projects; each is isolated — a user in one account cannot see another account's subjects
- **Admin users** have full pub/sub (`>`); their `.creds` files never leave the VPS; they connect to `tls://localhost:4222 --tlsskipverify`
- **Bridge and leaf users** have scoped permissions; their `.creds` files are distributed to the appropriate machines via `scp` over Tailscale
- Adding a new project: `nsc add account <name>` + `nsc push --all` — no server restart
- Revoking a leaf: delete its user JWT from the resolver; connection drops at next token refresh (default 2 min)

### Future migration path

Account names (`edgesync`, `go-libfossil`) will be replaced with opaque IDs when multi-tenant scale requires it. The JWT resolver supports this without protocol changes — it is a config and tooling update only.

### Subject permissions

```
Leaf user:
  publish:   fossil.<project>.igot, fossil.<project>.file, fossil.<project>.delta
  subscribe: fossil.<project>.gimme, fossil.<project>.file, fossil.<project>.meta, fossil.<project>.events

Bridge user:
  publish:   fossil.<project>.>
  subscribe: fossil.<project>.>

Admin user:
  publish:   >
  subscribe: >
  (used only on VPS via tls://localhost:4222 --tlsskipverify)
```

---

## Dockerfile

Same minor version as systemd install. The `$OPERATOR_JWT` placeholder must be rendered before building — the `Dockerfile` is intended for use with a pre-rendered `nats-server.conf` (not the `.template` file). Build workflow for Docker deployments:

```bash
source /etc/nats/operator.env
envsubst < nats-server.conf.template > nats-server.conf-rendered
docker build --build-arg CONF=nats-server.conf-rendered -t fossil-nats:2.10 .
```

```dockerfile
FROM nats:2.10-alpine
ARG CONF=nats-server.conf-rendered
COPY ${CONF} /etc/nats/nats-server.conf
# TLS certs and creds are mounted at runtime — never baked into the image
CMD ["nats-server", "-c", "/etc/nats/nats-server.conf"]
```

---

## DNS & Firewall

### DNS

Add an A record in Cloudflare for `craftdesign.group`:

| Name | Type | Value | Proxy |
|------|------|-------|-------|
| `sync` | A | `<hetzner-public-ip>` | **DNS only (grey cloud)** |

Must be grey cloud — NATS is raw TCP; Cloudflare cannot proxy it.

### Hetzner Cloud Firewall

Add an inbound rule to the VPS firewall:

| Protocol | Port | Source |
|----------|------|--------|
| TCP | 4222 | 0.0.0.0/0, ::/0 |

---

## JetStream Streams

| Stream | Subjects | Retention | Storage |
|--------|----------|-----------|---------|
| `EDGESYNC` | `fossil.edgesync.>` | Limits (7 days / 1 GB) | File |
| `GO_LIBFOSSIL` | `fossil.go-libfossil.>` | Limits (7 days / 1 GB) | File |

---

## Operational Reference

### Service management

```bash
sudo systemctl status nats
sudo journalctl -u nats -f
sudo systemctl kill --signal=SIGHUP nats   # reload config/certs — no connection drop
```

### Verify public connectivity

```bash
nats server ping --server tls://sync.craftdesign.group:4222
nats account info --server tls://sync.craftdesign.group:4222 --creds creds/edgesync-bridge.creds
```

### Verify local admin connectivity

```bash
nats server ping --server tls://localhost:4222 --tlsskipverify
nats stream ls   --server tls://localhost:4222 --tlsskipverify --creds creds/edgesync-admin.creds
```

### Re-push JWTs after credential changes

```bash
nsc push --all --server tls://localhost:4222 --tlsskipverify
```

---

## Out of Scope

- NATS clustering — cluster config hook (commented placeholder) present; implementation deferred
- NATS WebSocket endpoint — deferred until browser/WASM leaf is implemented
- Monitoring / Prometheus scraping — deferred