infra: NATS Server Setup

Date: 2026-03-15 Status: Approved Context: Infrastructure foundation for Fossil NATS Sync (Phase E — Leaf Agent)

Overview

Configure a NATS 2.10 server with JetStream on the Hetzner VPS as the messaging hub for the Fossil NATS Sync project. The server must be reachable by both Tailscale-connected leaf nodes (MacBook, Mac Mini) and external public clients. Configuration is version-controlled in a new infra Fossil repo for reproducible setup and future cluster expansion.

Architecture

External clients / leaf agents
        │
        │ tls://sync.craftdesign.group:4222
        ▼
nats-server 2.10 (systemd, linux/arm64)
  listen: 0.0.0.0:4222  (TLS — external + Tailscale + local admin)
    │
    ├── /var/nats/jetstream   JetStream persistent storage
    ├── /var/nats/jwt         JWT resolver store
    └── /var/nats/tls         TLS cert + key (acme.sh managed)

Local admin tools (nsc, nats CLI) connect to tls://localhost:4222 --tlsskipverify

Single listener: NATS 2.10 does not support two independent TCP listeners with different TLS policies in one server process. All connections — external, Tailscale, and local admin — use the single TLS listener on port 4222. Local admin tools (nsc push, nats CLI) connect to tls://localhost:4222 with --tlsskipverify (acceptable for loopback connections where network interception is impossible).

NATS Version

Target version: 2.10.x (latest 2.10 stable at install time)

install.sh pins a NATS_VERSION variable (e.g., 2.10.22) and downloads the linux-arm64 binary:

https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/nats-server-v${NATS_VERSION}-linux-arm64.tar.gz

The Dockerfile pins the same minor version: FROM nats:2.10-alpine. Both are updated together when upgrading.

Config Repo (`infra`)

New Fossil repo: /var/fossil/repos/infra.fossil, browsable at fossil.craftdesign.group/infra/.

infra/
├── nats/
│   ├── nats-server.conf.template  # Config template (contains $OPERATOR_JWT placeholder)
│   ├── nats.service               # Systemd unit
│   ├── install.sh                 # Phase 1: binary, dirs, TLS, unit file (does NOT start service)
│   ├── bootstrap-creds.sh         # Phase 2: one-time nsc operator/account/user setup
│   ├── start.sh                   # Phase 3: render config, enable and start service, nsc push
│   ├── provision-streams.sh       # Phase 4: idempotent JetStream stream creation
│   ├── Dockerfile                 # Docker surface for future cluster nodes
│   ├── creds/
│   │   └── README.md              # How to regenerate operator/account/user creds with nsc
│   └── README.md
└── README.md

creds/ is listed in .fossil-ignore — NKey seeds and JWT files are never committed. /etc/nats/nats-server.conf (the rendered live config) is also never committed.

Install Phases (explicit ordering)

Phase 1 — install.sh: installs binary, creates dirs, obtains TLS cert, copies config template and unit file. Does not start the service (operator JWT not yet available).

Phase 2 — bootstrap-creds.sh: one-time nsc setup. Creates operator, accounts, users. Writes /etc/nats/operator.env containing OPERATOR_JWT=<value>.

Phase 3 — start.sh: renders config template → live config, enables and starts the service, pushes JWTs to the running resolver.

Phase 4 — provision-streams.sh: creates JetStream streams after the server is running. Idempotent and non-interactive.

TLS Provisioning

Mechanism: acme.sh with Cloudflare DNS-01 challenge

Traefik already holds ports 80/443, making HTTP-01 challenges impossible for non-Coolify services. DNS-01 via the Cloudflare API sidesteps this completely.

install.sh handles TLS setup:

# Install acme.sh (if not present)
curl https://get.acme.sh | sh -s [email protected]

# cf-token.env must already exist at /etc/nats/cf-token.env (operator places it before running install.sh)
# It exports: CF_Token="<cloudflare-api-token>" with Zone:DNS:Edit permission for craftdesign.group
source /etc/nats/cf-token.env

~/.acme.sh/acme.sh --issue --dns dns_cf \
  -d sync.craftdesign.group \
  --key-file        /var/nats/tls/sync.craftdesign.group.key \
  --fullchain-file  /var/nats/tls/sync.craftdesign.group.crt \
  --reloadcmd       "systemctl kill --signal=SIGHUP nats"

# Set ownership so nats-server (running as nats user) can read the key
chown nats:nats /var/nats/tls/sync.craftdesign.group.key /var/nats/tls/sync.craftdesign.group.crt
chmod 0640 /var/nats/tls/sync.craftdesign.group.key /var/nats/tls/sync.craftdesign.group.crt

--fullchain-file ensures the cert includes the Let's Encrypt intermediate
--reloadcmd sends SIGHUP to nats-server after renewal; NATS reloads TLS certs on SIGHUP without dropping connections
The Cloudflare API token requires Zone:DNS:Edit for craftdesign.group; stored at /etc/nats/cf-token.env (mode 0600, root-owned) — never committed to the repo
/etc/nats/cf-token.env is also used by acme.sh's cron renewal job, so it must persist on disk

`install.sh` Responsibilities (idempotent, Phase 1)

Verify NATS_VERSION is set; download and install nats-server binary to /usr/local/bin/nats-server
Create nats system user (no login shell) if not present
Create directories:
- /var/nats/jetstream — owned by nats:nats, mode 0750
- /var/nats/jwt — owned by nats:nats, mode 0750
- /var/nats/tls — owned by nats:nats, mode 0750
- /etc/nats/ — owned by root, mode 0755
Verify /etc/nats/cf-token.env exists and is mode 0600; fail with clear error if not
Copy nats-server.conf.template to /etc/nats/nats-server.conf.template
Install acme.sh if not present; issue TLS cert via DNS-01 (skips if cert already valid); set ownership nats:nats on cert and key files
Install nats.service to /etc/systemd/system/; systemctl daemon-reload
Print: "Phase 1 complete. Run bootstrap-creds.sh next."

The service is not started by install.sh.

`bootstrap-creds.sh` Responsibilities (Phase 2, one-time)

Requires nsc CLI (install via curl -fsSL https://install.nats.io/nsc | sh).

No server connection is needed at this stage — nsc works purely locally.

nsc add operator --name fossil-sync

# Use localhost TLS for all local nsc/nats CLI operations
nsc edit operator --account-jwt-server-url tls://localhost:4222

nsc add account --name edgesync
nsc add account --name go-libfossil

# Admin user per account — full pub/sub for local stream provisioning only
nsc add user --account edgesync     --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account go-libfossil --name admin --allow-pub '>' --allow-sub '>'

# Bridge user — scoped to project subjects
nsc add user --account edgesync \
  --name bridge \
  --allow-pub  'fossil.edgesync.>' \
  --allow-sub  'fossil.edgesync.>'

nsc add user --account go-libfossil \
  --name bridge \
  --allow-pub  'fossil.go-libfossil.>' \
  --allow-sub  'fossil.go-libfossil.>'

# Generate .creds files
nsc generate creds --account edgesync     --name bridge > creds/edgesync-bridge.creds
nsc generate creds --account go-libfossil --name bridge > creds/go-libfossil-bridge.creds
nsc generate creds --account edgesync     --name admin  > creds/edgesync-admin.creds
nsc generate creds --account go-libfossil --name admin  > creds/go-libfossil-admin.creds

# Extract operator JWT using nsc generate config (the supported extraction method)
OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null \
  | grep '^operator:' \
  | awk '{print $2}')

echo "OPERATOR_JWT=${OPERATOR_JWT}" > /etc/nats/operator.env
chmod 0600 /etc/nats/operator.env

echo "Phase 2 complete. Run start.sh next."

nsc local store lives at ~/.local/share/nats/nsc/ (default). creds/README.md documents the full nsc command sequence for recreating the credential tree on a fresh machine.

Adding a new leaf node (post-setup)

nsc add user --account <project> \
  --name leaf-<hostname> \
  --allow-pub  'fossil.<project>.igot,fossil.<project>.file,fossil.<project>.delta' \
  --allow-sub  'fossil.<project>.gimme,fossil.<project>.file,fossil.<project>.meta,fossil.<project>.events'

nsc generate creds --account <project> --name leaf-<hostname> > leaf-<hostname>.creds

nsc push --all --server tls://localhost:4222 --tlsskipverify

# Distribute .creds to the leaf node via scp over Tailscale

`start.sh` Responsibilities (Phase 3)

Renders config from template (preserving the template file), starts the service, pushes JWTs.

source /etc/nats/operator.env   # loads OPERATOR_JWT

# Render template → live config (template file is never modified)
envsubst < /etc/nats/nats-server.conf.template > /etc/nats/nats-server.conf

systemctl enable --now nats

# Wait for server to be ready
sleep 2

# Push account JWTs to the running resolver
nsc push --all --server tls://localhost:4222 --tlsskipverify

echo "Phase 3 complete. Run provision-streams.sh next."

Re-running start.sh is safe: envsubst always reads from the .template file.

`provision-streams.sh` Responsibilities (Phase 4, idempotent)

Uses tls://localhost:4222 --tlsskipverify and admin creds. The --defaults flag suppresses all interactive prompts.

NATS_SERVER="tls://localhost:4222"
NATS_TLS_FLAGS="--tlsskipverify"

for PROJECT in edgesync go-libfossil; do
  STREAM=$(echo "$PROJECT" | tr '[:lower:]' '[:upper:]' | tr '-' '_')
  CREDS="creds/${PROJECT}-admin.creds"

  if nats stream info "$STREAM" \
      --server "$NATS_SERVER" $NATS_TLS_FLAGS \
      --creds  "$CREDS" &>/dev/null; then
    echo "Stream $STREAM already exists, skipping."
    continue
  fi

  nats stream add "$STREAM" \
    --server    "$NATS_SERVER" $NATS_TLS_FLAGS \
    --creds     "$CREDS" \
    --subjects  "fossil.${PROJECT}.>" \
    --retention limits \
    --max-age   168h \
    --max-bytes 1073741824 \
    --storage   file \
    --replicas  1 \
    --defaults

  echo "Stream $STREAM created."
done

NATS Server Config Template (`nats-server.conf.template`)

server_name: vps-nbg1

listen: 0.0.0.0:4222

tls {
  cert_file: "/var/nats/tls/sync.craftdesign.group.crt"
  key_file:  "/var/nats/tls/sync.craftdesign.group.key"
}

jetstream {
  store_dir: "/var/nats/jetstream"
}

# JWT resolver — accounts pushed via `nsc push --all`
resolver: {
  type:         full
  dir:          "/var/nats/jwt"
  allow_delete: false
  interval:     "2m"
}

# Operator JWT — substituted by start.sh via envsubst
operator: $OPERATOR_JWT

# Cluster placeholder — populate when adding a second node
# cluster {
#   name: fossil-sync
#   listen: 0.0.0.0:6222
#   routes: [ nats://node2.internal:6222 ]
# }

Systemd Unit (`nats.service`)

[Unit]
Description=NATS Server
After=network-online.target
Wants=network-online.target

[Service]
User=nats
Group=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats-server.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target

Auth & Security

NKeys + JWT (three-tier)

Operator: fossil-sync
├── Account: edgesync
│   ├── User: admin    (pub/sub > — local provisioning on VPS only)
│   ├── User: bridge   (pub+sub fossil.edgesync.>)
│   └── User: leaf-<hostname>
└── Account: go-libfossil
    ├── User: admin
    ├── User: bridge   (pub+sub fossil.go-libfossil.>)
    └── User: leaf-<hostname>

Operator is the root of trust — NKey seed lives in nsc local store on the VPS
Accounts map 1:1 to Fossil projects; each is isolated — a user in one account cannot see another account's subjects
Admin users have full pub/sub (>); their .creds files never leave the VPS; they connect to tls://localhost:4222 --tlsskipverify
Bridge and leaf users have scoped permissions; their .creds files are distributed to the appropriate machines via scp over Tailscale
Adding a new project: nsc add account <name> + nsc push --all — no server restart
Revoking a leaf: delete its user JWT from the resolver; connection drops at next token refresh (default 2 min)

Future migration path

Account names (edgesync, go-libfossil) will be replaced with opaque IDs when multi-tenant scale requires it. The JWT resolver supports this without protocol changes — it is a config and tooling update only.

Subject permissions

Leaf user:
  publish:   fossil.<project>.igot, fossil.<project>.file, fossil.<project>.delta
  subscribe: fossil.<project>.gimme, fossil.<project>.file, fossil.<project>.meta, fossil.<project>.events

Bridge user:
  publish:   fossil.<project>.>
  subscribe: fossil.<project>.>

Admin user:
  publish:   >
  subscribe: >
  (used only on VPS via tls://localhost:4222 --tlsskipverify)

Dockerfile

Same minor version as systemd install. The $OPERATOR_JWT placeholder must be rendered before building — the Dockerfile is intended for use with a pre-rendered nats-server.conf (not the .template file). Build workflow for Docker deployments:

source /etc/nats/operator.env
envsubst < nats-server.conf.template > nats-server.conf-rendered
docker build --build-arg CONF=nats-server.conf-rendered -t fossil-nats:2.10 .

FROM nats:2.10-alpine
ARG CONF=nats-server.conf-rendered
COPY ${CONF} /etc/nats/nats-server.conf
# TLS certs and creds are mounted at runtime — never baked into the image
CMD ["nats-server", "-c", "/etc/nats/nats-server.conf"]

DNS & Firewall

DNS

Add an A record in Cloudflare for craftdesign.group:

Name	Type	Value	Proxy
`sync`	A	`<hetzner-public-ip>`	DNS only (grey cloud)

Must be grey cloud — NATS is raw TCP; Cloudflare cannot proxy it.

Hetzner Cloud Firewall

Add an inbound rule to the VPS firewall:

Protocol	Port	Source
TCP	4222	0.0.0.0/0, ::/0

JetStream Streams

Stream	Subjects	Retention	Storage
`EDGESYNC`	`fossil.edgesync.>`	Limits (7 days / 1 GB)	File
`GO_LIBFOSSIL`	`fossil.go-libfossil.>`	Limits (7 days / 1 GB)	File

Operational Reference

Service management

sudo systemctl status nats
sudo journalctl -u nats -f
sudo systemctl kill --signal=SIGHUP nats   # reload config/certs — no connection drop

Verify public connectivity

nats server ping --server tls://sync.craftdesign.group:4222
nats account info --server tls://sync.craftdesign.group:4222 --creds creds/edgesync-bridge.creds

Verify local admin connectivity

nats server ping --server tls://localhost:4222 --tlsskipverify
nats stream ls   --server tls://localhost:4222 --tlsskipverify --creds creds/edgesync-admin.creds

Re-push JWTs after credential changes

nsc push --all --server tls://localhost:4222 --tlsskipverify

Out of Scope

NATS clustering — cluster config hook (commented placeholder) present; implementation deferred
NATS WebSocket endpoint — deferred until browser/WASM leaf is implemented
Monitoring / Prometheus scraping — deferred

NATS Server Setup — Design Spec