D 2026-03-15T20:21:57.866
L NATS\sServer\s—\sOperations\sRunbook
N text/x-markdown
U dmestas
W 5736
# NATS Server — Operations Runbook
Day-to-day operations for the NATS server on `vps-nbg1`.
See [NATS Server — Setup & Architecture](NATS Server — Setup & Architecture) for the full design.
---
## Service Management
```sh
# Status
sudo systemctl status nats
# Start / stop / restart
sudo systemctl start nats
sudo systemctl stop nats
sudo systemctl restart nats
# Reload TLS cert only (no restart needed after renewal)
sudo systemctl kill --signal=SIGHUP nats
# Logs (live)
sudo journalctl -u nats -f
# Logs (last 100 lines)
sudo journalctl -u nats -n 100 --no-pager
```
---
## Verify Health
```sh
CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
SERVER=tls://sync.craftdesign.group:4222
# List streams
nats stream ls --server $SERVER --creds $CREDS
# Stream details
nats stream info EDGESYNC --server $SERVER --creds $CREDS
# Publish a test message
nats pub fossil.edgesync.smoke "test" --server $SERVER --creds $CREDS
# Subscribe (run in a second terminal)
nats sub 'fossil.edgesync.>' --server $SERVER --creds $CREDS
```
---
## TLS Certificate Renewal
Certificates are issued by Let's Encrypt via acme.sh DNS-01 (Cloudflare).
acme.sh installs a cron job under root that auto-renews 30 days before expiry.
On renewal, it runs `systemctl kill --signal=SIGHUP nats` to reload the cert live.
### Manual renewal
```sh
source /etc/nats/cf-token.env # sets CF_Token (root only)
export CF_Token
/root/.acme.sh/acme.sh --renew -d sync.craftdesign.group --ecc --server letsencrypt
```
### Check cert expiry
```sh
openssl x509 -in /var/nats/tls/sync.craftdesign.group.crt -noout -dates
```
---
## Adding a New Account
Run as `dmestas`:
```sh
ACCOUNT=my-new-account
nsc add account --name "$ACCOUNT"
nsc edit account --name "$ACCOUNT" --js-tier 0 \
--js-streams -1 --js-consumer -1 --js-disk-storage -1 --js-mem-storage -1
# Add users
nsc add user --account "$ACCOUNT" --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account "$ACCOUNT" --name bridge \
--allow-pub "fossil.${ACCOUNT}.>" \
--allow-sub "fossil.${ACCOUNT}.>"
# Export creds
nsc generate creds --account "$ACCOUNT" --name admin > ~/Projects/infra/nats/creds/${ACCOUNT}-admin.creds
nsc generate creds --account "$ACCOUNT" --name bridge > ~/Projects/infra/nats/creds/${ACCOUNT}-bridge.creds
chmod 0600 ~/Projects/infra/nats/creds/${ACCOUNT}-*.creds
# Copy updated account JWT to resolver dir (requires public key)
PUBKEY=$(nsc describe account --name "$ACCOUNT" -R 2>/dev/null | grep 'Account ID' | awk '{print $NF}')
sudo cp ~/.local/share/nats/nsc/stores/fossil-sync/accounts/${ACCOUNT}/${ACCOUNT}.jwt \
/var/nats/jwt/${PUBKEY}.jwt
sudo chown nats:nats /var/nats/jwt/${PUBKEY}.jwt
# Restart to pick up new account
sudo systemctl restart nats
```
Then create a JetStream stream for the account using `provision-streams.sh` as a template.
---
## Adding a Second Node (Clustering)
The config template has a commented `cluster {}` block. When adding a second node:
1. Uncomment and fill in the cluster block in `nats-server.conf.template`
2. Change `--js-tier 0` to `--js-tier 1` or `--js-tier 3` in account JWTs (R1/R3 replication)
3. Copy the operator keys and account JWTs to the second node
4. Re-run start.sh on both nodes
---
## Known Gotchas
These were found during the initial live install (2026-03-15):
| Issue | Root cause | Fix |
|-------|-----------|-----|
| `CF_Token` not passed to acme.sh subprocess | `source` doesn't export | `export CF_Token` after source |
| ZeroSSL returns 24h retry-after | Default CA rate limit | Use `--server letsencrypt` |
| `nsc list operators` output piped to grep silently fails | nsc writes to stderr | Pipe `2>&1` not `2>/dev/null` |
| Same, but grep skips matches | ANSI color codes in output | Strip with `sed 's/\x1b\[[0-9;]*m//g'` |
| `nsc generate config` fails: "system account not set" | Must create SYS account first | `nsc add account --name SYS && nsc edit operator --system-account SYS` |
| `nsc push` fails: scheme "tls" not supported | nsc push only accepts nats/http/https | Use file-copy approach instead |
| `nats stream add --max-msgs -1` fails | `-1` parsed as a flag | Use `--max-msgs=-1` (equals notation) |
| JetStream not enabled for account (10039) | `--js-tier 1` requires clustered server for R1 | Use `--js-tier 0` (global tier) on single node |
| Account JWT changes not picked up | SIGHUP only reloads TLS certs | `systemctl restart nats` after JWT changes |
| `system_account: ""` in startup log | Cosmetic only — server loads account on-demand | No fix needed; add `system_account: <key>` to conf for clarity |
---
## Credentials Map
| File | Account | User | Permissions |
|------|---------|------|-------------|
| `creds/edgesync-admin.creds` | edgesync | admin | pub/sub `>` |
| `creds/edgesync-bridge.creds` | edgesync | bridge | pub/sub `fossil.edgesync.>` |
| `creds/go-libfossil-admin.creds` | go-libfossil | admin | pub/sub `>` |
| `creds/go-libfossil-bridge.creds` | go-libfossil | bridge | pub/sub `fossil.go-libfossil.>` |
Creds files are in `~/Projects/infra/nats/creds/` (gitignored / fossil-ignored).
The operator signing keys are in `~/.local/share/nats/nsc/keys/` — back these up.
---
## Secrets Locations
| Secret | Location | Owner | Mode |
|--------|---------|-------|------|
| Cloudflare API token | `/etc/nats/cf-token.env` | root | 0600 |
| Operator JWT | `/etc/nats/operator.env` | root | 0600 |
| nsc key store | `~/.local/share/nats/nsc/keys/` | dmestas | 0700 |
| TLS private key | `/var/nats/tls/sync.craftdesign.group.key` | nats | 0640 |
**Back up the nsc key store.** Loss of the operator signing key means you cannot
issue new accounts or users and must rebuild the PKI from scratch.
Z cacb5326285bf8c2c7e50e874a91b329