D 2026-03-15T20:21:57.866
L NATS\sServer\s—\sOperations\sRunbook
N text/x-markdown
U dmestas
W 5736
# NATS Server — Operations Runbook

Day-to-day operations for the NATS server on `vps-nbg1`.
See [NATS Server — Setup & Architecture](NATS Server — Setup & Architecture) for the full design.

---

## Service Management

```sh
# Status
sudo systemctl status nats

# Start / stop / restart
sudo systemctl start nats
sudo systemctl stop nats
sudo systemctl restart nats

# Reload TLS cert only (no restart needed after renewal)
sudo systemctl kill --signal=SIGHUP nats

# Logs (live)
sudo journalctl -u nats -f

# Logs (last 100 lines)
sudo journalctl -u nats -n 100 --no-pager
```

---

## Verify Health

```sh
CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
SERVER=tls://sync.craftdesign.group:4222

# List streams
nats stream ls --server $SERVER --creds $CREDS

# Stream details
nats stream info EDGESYNC --server $SERVER --creds $CREDS

# Publish a test message
nats pub fossil.edgesync.smoke "test" --server $SERVER --creds $CREDS

# Subscribe (run in a second terminal)
nats sub 'fossil.edgesync.>' --server $SERVER --creds $CREDS
```

---

## TLS Certificate Renewal

Certificates are issued by Let's Encrypt via acme.sh DNS-01 (Cloudflare).
acme.sh installs a cron job under root that auto-renews 30 days before expiry.
On renewal, it runs `systemctl kill --signal=SIGHUP nats` to reload the cert live.

### Manual renewal

```sh
source /etc/nats/cf-token.env   # sets CF_Token (root only)
export CF_Token
/root/.acme.sh/acme.sh --renew -d sync.craftdesign.group --ecc --server letsencrypt
```

### Check cert expiry

```sh
openssl x509 -in /var/nats/tls/sync.craftdesign.group.crt -noout -dates
```

---

## Adding a New Account

Run as `dmestas`:

```sh
ACCOUNT=my-new-account

nsc add account --name "$ACCOUNT"
nsc edit account --name "$ACCOUNT" --js-tier 0 \
  --js-streams -1 --js-consumer -1 --js-disk-storage -1 --js-mem-storage -1

# Add users
nsc add user --account "$ACCOUNT" --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account "$ACCOUNT" --name bridge \
  --allow-pub  "fossil.${ACCOUNT}.>" \
  --allow-sub  "fossil.${ACCOUNT}.>"

# Export creds
nsc generate creds --account "$ACCOUNT" --name admin  > ~/Projects/infra/nats/creds/${ACCOUNT}-admin.creds
nsc generate creds --account "$ACCOUNT" --name bridge > ~/Projects/infra/nats/creds/${ACCOUNT}-bridge.creds
chmod 0600 ~/Projects/infra/nats/creds/${ACCOUNT}-*.creds

# Copy updated account JWT to resolver dir (requires public key)
PUBKEY=$(nsc describe account --name "$ACCOUNT" -R 2>/dev/null | grep 'Account ID' | awk '{print $NF}')
sudo cp ~/.local/share/nats/nsc/stores/fossil-sync/accounts/${ACCOUNT}/${ACCOUNT}.jwt \
        /var/nats/jwt/${PUBKEY}.jwt
sudo chown nats:nats /var/nats/jwt/${PUBKEY}.jwt

# Restart to pick up new account
sudo systemctl restart nats
```

Then create a JetStream stream for the account using `provision-streams.sh` as a template.

---

## Adding a Second Node (Clustering)

The config template has a commented `cluster {}` block. When adding a second node:

1. Uncomment and fill in the cluster block in `nats-server.conf.template`
2. Change `--js-tier 0` to `--js-tier 1` or `--js-tier 3` in account JWTs (R1/R3 replication)
3. Copy the operator keys and account JWTs to the second node
4. Re-run start.sh on both nodes

---

## Known Gotchas

These were found during the initial live install (2026-03-15):

| Issue | Root cause | Fix |
|-------|-----------|-----|
| `CF_Token` not passed to acme.sh subprocess | `source` doesn't export | `export CF_Token` after source |
| ZeroSSL returns 24h retry-after | Default CA rate limit | Use `--server letsencrypt` |
| `nsc list operators` output piped to grep silently fails | nsc writes to stderr | Pipe `2>&1` not `2>/dev/null` |
| Same, but grep skips matches | ANSI color codes in output | Strip with `sed 's/\x1b\[[0-9;]*m//g'` |
| `nsc generate config` fails: "system account not set" | Must create SYS account first | `nsc add account --name SYS && nsc edit operator --system-account SYS` |
| `nsc push` fails: scheme "tls" not supported | nsc push only accepts nats/http/https | Use file-copy approach instead |
| `nats stream add --max-msgs -1` fails | `-1` parsed as a flag | Use `--max-msgs=-1` (equals notation) |
| JetStream not enabled for account (10039) | `--js-tier 1` requires clustered server for R1 | Use `--js-tier 0` (global tier) on single node |
| Account JWT changes not picked up | SIGHUP only reloads TLS certs | `systemctl restart nats` after JWT changes |
| `system_account: ""` in startup log | Cosmetic only — server loads account on-demand | No fix needed; add `system_account: <key>` to conf for clarity |

---

## Credentials Map

| File | Account | User | Permissions |
|------|---------|------|-------------|
| `creds/edgesync-admin.creds` | edgesync | admin | pub/sub `>` |
| `creds/edgesync-bridge.creds` | edgesync | bridge | pub/sub `fossil.edgesync.>` |
| `creds/go-libfossil-admin.creds` | go-libfossil | admin | pub/sub `>` |
| `creds/go-libfossil-bridge.creds` | go-libfossil | bridge | pub/sub `fossil.go-libfossil.>` |

Creds files are in `~/Projects/infra/nats/creds/` (gitignored / fossil-ignored).
The operator signing keys are in `~/.local/share/nats/nsc/keys/` — back these up.

---

## Secrets Locations

| Secret | Location | Owner | Mode |
|--------|---------|-------|------|
| Cloudflare API token | `/etc/nats/cf-token.env` | root | 0600 |
| Operator JWT | `/etc/nats/operator.env` | root | 0600 |
| nsc key store | `~/.local/share/nats/nsc/keys/` | dmestas | 0700 |
| TLS private key | `/var/nats/tls/sync.craftdesign.group.key` | nats | 0640 |

**Back up the nsc key store.** Loss of the operator signing key means you cannot
issue new accounts or users and must rebuild the PKI from scratch.

Z cacb5326285bf8c2c7e50e874a91b329