NATS Server — Operations Runbook

Day-to-day operations for the NATS server on vps-nbg1. See NATS Server — Setup & Architecture for the full design.

Service Management

# Status
sudo systemctl status nats

# Start / stop / restart
sudo systemctl start nats
sudo systemctl stop nats
sudo systemctl restart nats

# Reload TLS cert only (no restart needed after renewal)
sudo systemctl kill --signal=SIGHUP nats

# Logs (live)
sudo journalctl -u nats -f

# Logs (last 100 lines)
sudo journalctl -u nats -n 100 --no-pager

Verify Health

CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
SERVER=tls://sync.craftdesign.group:4222

# List streams
nats stream ls --server $SERVER --creds $CREDS

# Stream details
nats stream info EDGESYNC --server $SERVER --creds $CREDS

# Publish a test message
nats pub fossil.edgesync.smoke "test" --server $SERVER --creds $CREDS

# Subscribe (run in a second terminal)
nats sub 'fossil.edgesync.>' --server $SERVER --creds $CREDS

TLS Certificate Renewal

Certificates are issued by Let's Encrypt via acme.sh DNS-01 (Cloudflare). acme.sh installs a cron job under root that auto-renews 30 days before expiry. On renewal, it runs systemctl kill --signal=SIGHUP nats to reload the cert live.

Manual renewal

source /etc/nats/cf-token.env   # sets CF_Token (root only)
export CF_Token
/root/.acme.sh/acme.sh --renew -d sync.craftdesign.group --ecc --server letsencrypt

Check cert expiry

openssl x509 -in /var/nats/tls/sync.craftdesign.group.crt -noout -dates

Adding a New Account

Run as dmestas:

ACCOUNT=my-new-account

nsc add account --name "$ACCOUNT"
nsc edit account --name "$ACCOUNT" --js-tier 0 \
  --js-streams -1 --js-consumer -1 --js-disk-storage -1 --js-mem-storage -1

# Add users
nsc add user --account "$ACCOUNT" --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account "$ACCOUNT" --name bridge \
  --allow-pub  "fossil.${ACCOUNT}.>" \
  --allow-sub  "fossil.${ACCOUNT}.>"

# Export creds
nsc generate creds --account "$ACCOUNT" --name admin  > ~/Projects/infra/nats/creds/${ACCOUNT}-admin.creds
nsc generate creds --account "$ACCOUNT" --name bridge > ~/Projects/infra/nats/creds/${ACCOUNT}-bridge.creds
chmod 0600 ~/Projects/infra/nats/creds/${ACCOUNT}-*.creds

# Copy updated account JWT to resolver dir (requires public key)
PUBKEY=$(nsc describe account --name "$ACCOUNT" -R 2>/dev/null | grep 'Account ID' | awk '{print $NF}')
sudo cp ~/.local/share/nats/nsc/stores/fossil-sync/accounts/${ACCOUNT}/${ACCOUNT}.jwt \
        /var/nats/jwt/${PUBKEY}.jwt
sudo chown nats:nats /var/nats/jwt/${PUBKEY}.jwt

# Restart to pick up new account
sudo systemctl restart nats

Then create a JetStream stream for the account using provision-streams.sh as a template.

Adding a Second Node (Clustering)

The config template has a commented cluster {} block. When adding a second node:

Uncomment and fill in the cluster block in nats-server.conf.template
Change --js-tier 0 to --js-tier 1 or --js-tier 3 in account JWTs (R1/R3 replication)
Copy the operator keys and account JWTs to the second node
Re-run start.sh on both nodes

Known Gotchas

These were found during the initial live install (2026-03-15):

Issue	Root cause	Fix
`CF_Token` not passed to acme.sh subprocess	`source` doesn't export	`export CF_Token` after source
ZeroSSL returns 24h retry-after	Default CA rate limit	Use `--server letsencrypt`
`nsc list operators` output piped to grep silently fails	nsc writes to stderr	Pipe `2>&1` not `2>/dev/null`
Same, but grep skips matches	ANSI color codes in output	Strip with `sed 's/\x1b\[[0-9;]*m//g'`
`nsc generate config` fails: "system account not set"	Must create SYS account first	`nsc add account --name SYS && nsc edit operator --system-account SYS`
`nsc push` fails: scheme "tls" not supported	nsc push only accepts nats/http/https	Use file-copy approach instead
`nats stream add --max-msgs -1` fails	`-1` parsed as a flag	Use `--max-msgs=-1` (equals notation)
JetStream not enabled for account (10039)	`--js-tier 1` requires clustered server for R1	Use `--js-tier 0` (global tier) on single node
Account JWT changes not picked up	SIGHUP only reloads TLS certs	`systemctl restart nats` after JWT changes
`system_account: ""` in startup log	Cosmetic only — server loads account on-demand	No fix needed; add `system_account: <key>` to conf for clarity

Credentials Map

File	Account	User	Permissions
`creds/edgesync-admin.creds`	edgesync	admin	pub/sub `>`
`creds/edgesync-bridge.creds`	edgesync	bridge	pub/sub `fossil.edgesync.>`
`creds/go-libfossil-admin.creds`	go-libfossil	admin	pub/sub `>`
`creds/go-libfossil-bridge.creds`	go-libfossil	bridge	pub/sub `fossil.go-libfossil.>`

Creds files are in ~/Projects/infra/nats/creds/ (gitignored / fossil-ignored). The operator signing keys are in ~/.local/share/nats/nsc/keys/ — back these up.

Secrets Locations

Secret	Location	Owner	Mode
Cloudflare API token	`/etc/nats/cf-token.env`	root	0600
Operator JWT	`/etc/nats/operator.env`	root	0600
nsc key store	`~/.local/share/nats/nsc/keys/`	dmestas	0700
TLS private key	`/var/nats/tls/sync.craftdesign.group.key`	nats	0640

Back up the nsc key store. Loss of the operator signing key means you cannot issue new accounts or users and must rebuild the PKI from scratch.