# NATS Server Setup Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Deploy a NATS 2.10 server with JetStream and NKeys+JWT auth on the Hetzner VPS, reachable publicly at `tls://sync.craftdesign.group:4222`, with all configuration version-controlled in the `infra` Fossil repo.
**Architecture:** Four sequential install phases (install.sh → bootstrap-creds.sh → start.sh → provision-streams.sh) produce a running, authenticated NATS server. TLS is handled by `acme.sh` DNS-01 via Cloudflare. Auth uses a three-tier NKeys+JWT hierarchy (operator → accounts → users) managed by `nsc`.
**Tech Stack:** NATS Server 2.10 (linux/arm64), `nsc` CLI, `acme.sh`, Fossil SCM, systemd, bash.
---
## Chunk 1: Repo scaffolding and static config files
### Task 1: Create directory structure and static config files
**Files:**
- Create: `infra/nats/nats-server.conf.template`
- Create: `infra/nats/nats.service`
- Create: `infra/nats/Dockerfile`
- Create: `infra/nats/creds/README.md`
- Create: `infra/nats/README.md`
- Create: `infra/README.md`
Working directory for all tasks: `~/Projects/infra`
- [ ] **Step 1: Write `nats/nats-server.conf.template`**
```bash
cat > ~/Projects/infra/nats/nats-server.conf.template << 'EOF'
server_name: vps-nbg1
listen: 0.0.0.0:4222
tls {
cert_file: "/var/nats/tls/sync.craftdesign.group.crt"
key_file: "/var/nats/tls/sync.craftdesign.group.key"
}
jetstream {
store_dir: "/var/nats/jetstream"
}
# JWT resolver — accounts pushed via `nsc push --all`
resolver: {
type: full
dir: "/var/nats/jwt"
allow_delete: false
interval: "2m"
}
# Operator JWT — substituted by start.sh via envsubst
operator: $OPERATOR_JWT
# Cluster placeholder — populate when adding a second node
# cluster {
# name: fossil-sync
# listen: 0.0.0.0:6222
# routes: [ nats://node2.internal:6222 ]
# }
EOF
```
- [ ] **Step 2: Write `nats/nats.service`**
```bash
cat > ~/Projects/infra/nats/nats.service << 'EOF'
[Unit]
Description=NATS Server
After=network-online.target
Wants=network-online.target
[Service]
User=nats
Group=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats-server.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576
[Install]
WantedBy=multi-user.target
EOF
```
- [ ] **Step 3: Write `nats/Dockerfile`**
```bash
cat > ~/Projects/infra/nats/Dockerfile << 'EOF'
FROM nats:2.10-alpine
# Render nats-server.conf.template before building:
# source /etc/nats/operator.env
# envsubst < nats-server.conf.template > nats-server.conf-rendered
# docker build --build-arg CONF=nats-server.conf-rendered -t fossil-nats:2.10 .
ARG CONF=nats-server.conf-rendered
COPY ${CONF} /etc/nats/nats-server.conf
# TLS certs and creds are mounted at runtime — never baked into the image:
# docker run -v /var/nats/tls:/var/nats/tls:ro \
# -v /var/nats/jwt:/var/nats/jwt \
# -v /var/nats/jetstream:/var/nats/jetstream \
# -p 4222:4222 fossil-nats:2.10
CMD ["nats-server", "-c", "/etc/nats/nats-server.conf"]
EOF
```
- [ ] **Step 4: Write `nats/creds/README.md`**
```bash
cat > ~/Projects/infra/nats/creds/README.md << 'EOF'
# Credentials
This directory holds generated `.creds` files for NATS users. These files are
fossil-ignored and must never be committed.
## Regenerating the full credential tree
Prerequisites: `nsc` CLI installed (`curl -fsSL https://install.nats.io/nsc | sh`)
```bash
# 1. Add operator (only if nsc store is empty)
nsc add operator --name fossil-sync
nsc edit operator --account-jwt-server-url tls://localhost:4222
# 2. Add accounts
nsc add account --name edgesync
nsc add account --name go-libfossil
# 3. Add users
nsc add user --account edgesync --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account go-libfossil --name admin --allow-pub '>' --allow-sub '>'
nsc add user --account edgesync \
--name bridge \
--allow-pub 'fossil.edgesync.>' \
--allow-sub 'fossil.edgesync.>'
nsc add user --account go-libfossil \
--name bridge \
--allow-pub 'fossil.go-libfossil.>' \
--allow-sub 'fossil.go-libfossil.>'
# 4. Generate .creds files into this directory
nsc generate creds --account edgesync --name bridge > edgesync-bridge.creds
nsc generate creds --account go-libfossil --name bridge > go-libfossil-bridge.creds
nsc generate creds --account edgesync --name admin > edgesync-admin.creds
nsc generate creds --account go-libfossil --name admin > go-libfossil-admin.creds
# 5. Re-extract operator JWT
OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null \
| grep '^operator:' | awk '{print $2}')
echo "OPERATOR_JWT=${OPERATOR_JWT}" > /etc/nats/operator.env
chmod 0600 /etc/nats/operator.env
# 6. Push JWTs to running server
nsc push --all --server tls://localhost:4222 --tlsskipverify
```
## Adding a new leaf node
```bash
PROJECT=edgesync # or go-libfossil
HOSTNAME=m4-macbook
nsc add user --account "$PROJECT" \
--name "leaf-${HOSTNAME}" \
--allow-pub "fossil.${PROJECT}.igot,fossil.${PROJECT}.file,fossil.${PROJECT}.delta" \
--allow-sub "fossil.${PROJECT}.gimme,fossil.${PROJECT}.file,fossil.${PROJECT}.meta,fossil.${PROJECT}.events"
nsc generate creds --account "$PROJECT" --name "leaf-${HOSTNAME}" > "leaf-${HOSTNAME}.creds"
nsc push --all --server tls://localhost:4222 --tlsskipverify
# Transfer to leaf node over Tailscale
scp "leaf-${HOSTNAME}.creds" "${HOSTNAME}:/path/to/agent/creds/"
```
EOF
```
- [ ] **Step 5: Write `nats/README.md`**
```bash
cat > ~/Projects/infra/nats/README.md << 'EOF'
# NATS Server
NATS 2.10 with JetStream and NKeys+JWT auth.
- **Public endpoint:** `tls://sync.craftdesign.group:4222`
- **Local admin:** `tls://localhost:4222 --tlsskipverify`
- **JetStream streams:** EDGESYNC, GO_LIBFOSSIL (7 days / 1 GB each)
- **Spec:** `docs/superpowers/specs/2026-03-15-nats-server-setup-design.md`
## Install (fresh VPS)
```bash
# 0. Place Cloudflare API token first
sudo mkdir -p /etc/nats
echo 'CF_Token="<your-token>"' | sudo tee /etc/nats/cf-token.env
sudo chmod 0600 /etc/nats/cf-token.env
# 1-4. Run phases in order
export NATS_VERSION=2.10.22
sudo --preserve-env=NATS_VERSION bash nats/install.sh
bash nats/bootstrap-creds.sh
sudo bash nats/start.sh
bash nats/provision-streams.sh
```
## Service management
```bash
sudo systemctl status nats
sudo journalctl -u nats -f
sudo systemctl kill --signal=SIGHUP nats # reload certs/config, no connection drop
```
## Verify
```bash
nats server ping --server tls://sync.craftdesign.group:4222
nats server ping --server tls://localhost:4222 --tlsskipverify
```
EOF
```
- [ ] **Step 6: Write `infra/README.md`**
```bash
cat > ~/Projects/infra/README.md << 'EOF'
# infra
Infrastructure configuration for craftdesign.group services.
| Service | Directory | Endpoint |
|---------|-----------|----------|
| NATS | `nats/` | tls://sync.craftdesign.group:4222 |
EOF
```
- [ ] **Step 7: Create `.fossil-ignore` to prevent credential staging**
```bash
cat > ~/Projects/infra/.fossil-ignore << 'EOF'
nats/creds/*.creds
/etc/nats/nats-server.conf
EOF
```
- [ ] **Step 8: Add files to Fossil and commit**
```bash
cd ~/Projects/infra
fossil add nats/ README.md .fossil-ignore
fossil commit -m "Add NATS repo scaffolding and static config files"
```
Expected: `New_Version: <hash>`
---
## Chunk 2: install.sh
### Task 2: Write and validate `install.sh` (Phase 1)
**Files:**
- Create: `infra/nats/install.sh`
`install.sh` must be idempotent — safe to re-run on a machine where NATS is already partially installed.
- [ ] **Step 1: Write `nats/install.sh`**
```bash
cat > ~/Projects/infra/nats/install.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
# Ensure acme.sh and cron renewal run under root's home, not the invoking user's
HOME=/root
# ── Configuration ────────────────────────────────────────────────────────────
NATS_VERSION="${NATS_VERSION:?NATS_VERSION must be set, e.g. export NATS_VERSION=2.10.22}"
NATS_BIN=/usr/local/bin/nats-server
NATS_CONF_DIR=/etc/nats
NATS_DATA_DIR=/var/nats
CF_TOKEN_FILE="${NATS_CONF_DIR}/cf-token.env"
DOMAIN=sync.craftdesign.group
RELOAD_CMD="systemctl kill --signal=SIGHUP nats"
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "[install.sh] $*"; }
die() { echo "[install.sh] ERROR: $*" >&2; exit 1; }
# ── Step 1: Download and install nats-server binary ──────────────────────────
if [[ -x "$NATS_BIN" ]] && "$NATS_BIN" --version 2>/dev/null | grep -q "v${NATS_VERSION}$"; then
log "nats-server $NATS_VERSION already installed, skipping download."
else
log "Downloading nats-server v${NATS_VERSION} linux/arm64..."
ARCHIVE="nats-server-v${NATS_VERSION}-linux-arm64.tar.gz"
URL="https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/${ARCHIVE}"
TMP=$(mktemp -d)
curl -fsSL "$URL" -o "${TMP}/${ARCHIVE}"
tar -xzf "${TMP}/${ARCHIVE}" -C "$TMP"
install -m 0755 "${TMP}/nats-server-v${NATS_VERSION}-linux-arm64/nats-server" "$NATS_BIN"
rm -rf "$TMP"
log "nats-server installed to $NATS_BIN"
fi
# ── Step 2: Create nats system user ─────────────────────────────────────────
if ! id nats &>/dev/null; then
useradd --system --no-create-home --shell /usr/sbin/nologin nats
log "Created nats system user."
else
log "nats user already exists, skipping."
fi
# ── Step 3: Create data directories ─────────────────────────────────────────
for DIR in "${NATS_DATA_DIR}/jetstream" "${NATS_DATA_DIR}/jwt" "${NATS_DATA_DIR}/tls"; do
mkdir -p "$DIR"
chown nats:nats "$DIR"
chmod 0750 "$DIR"
log "Directory ready: $DIR"
done
# ── Step 4: Create /etc/nats and copy config template ───────────────────────
mkdir -p "$NATS_CONF_DIR"
chown root:root "$NATS_CONF_DIR"
chmod 0755 "$NATS_CONF_DIR"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cp "${SCRIPT_DIR}/nats-server.conf.template" "${NATS_CONF_DIR}/nats-server.conf.template"
log "Config template copied to ${NATS_CONF_DIR}/nats-server.conf.template"
# ── Step 5: Verify Cloudflare token file ────────────────────────────────────
[[ -f "$CF_TOKEN_FILE" ]] || die \
"Missing $CF_TOKEN_FILE. Create it with: echo 'CF_Token=\"<token>\"' > $CF_TOKEN_FILE && chmod 0600 $CF_TOKEN_FILE"
[[ "$(stat -c %a "$CF_TOKEN_FILE")" == "600" ]] || \
{ chmod 0600 "$CF_TOKEN_FILE"; log "Fixed permissions on $CF_TOKEN_FILE"; }
# ── Step 6: Install acme.sh and issue TLS cert ──────────────────────────────
ACME="${HOME}/.acme.sh/acme.sh"
if [[ ! -x "$ACME" ]]; then
log "Installing acme.sh..."
curl -fsSL https://get.acme.sh | sh -s [email protected]
fi
CERT_FILE="${NATS_DATA_DIR}/tls/${DOMAIN}.crt"
KEY_FILE="${NATS_DATA_DIR}/tls/${DOMAIN}.key"
# shellcheck source=/dev/null
source "$CF_TOKEN_FILE"
if "$ACME" --list 2>/dev/null | grep -q "$DOMAIN"; then
log "Certificate for $DOMAIN already exists in acme.sh store."
else
log "Issuing TLS certificate for $DOMAIN via DNS-01..."
"$ACME" --issue --dns dns_cf -d "$DOMAIN"
fi
# Always install cert to target paths (ensures files exist on re-runs)
"$ACME" --install-cert -d "$DOMAIN" \
--key-file "$KEY_FILE" \
--fullchain-file "$CERT_FILE" \
--reloadcmd "$RELOAD_CMD"
chown nats:nats "$CERT_FILE" "$KEY_FILE"
chmod 0640 "$CERT_FILE" "$KEY_FILE"
log "TLS cert and key written to $NATS_DATA_DIR/tls/ and owned by nats:nats"
# ── Step 7: Install systemd unit ────────────────────────────────────────────
cp "${SCRIPT_DIR}/nats.service" /etc/systemd/system/nats.service
systemctl daemon-reload
log "nats.service installed and daemon reloaded."
log ""
log "Phase 1 complete."
log "Next: run bootstrap-creds.sh (as dmestas, not root)"
SCRIPT
chmod +x ~/Projects/infra/nats/install.sh
```
- [ ] **Step 2: Validate script syntax with shellcheck**
```bash
shellcheck ~/Projects/infra/nats/install.sh
```
Expected: no output (no errors). If shellcheck is not installed: `sudo apt-get install -y shellcheck`.
- [ ] **Step 3: Verify NATS version exists for linux/arm64 before running**
```bash
export NATS_VERSION=2.10.22
curl -fsI "https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/nats-server-v${NATS_VERSION}-linux-arm64.tar.gz" | head -1
```
Expected: `HTTP/2 302` or `HTTP/1.1 302` (redirect to asset). If 404, check the [NATS releases page](https://github.com/nats-io/nats-server/releases) for the latest 2.10.x version and update `NATS_VERSION`.
- [ ] **Step 4: Place the Cloudflare API token file before running**
```bash
# Cloudflare token needs Zone:DNS:Edit permission for craftdesign.group
sudo mkdir -p /etc/nats
sudo bash -c 'echo "CF_Token=\"<YOUR_CF_TOKEN>\"" > /etc/nats/cf-token.env'
sudo chmod 0600 /etc/nats/cf-token.env
```
Replace `<YOUR_CF_TOKEN>` with the actual token from Cloudflare dashboard → My Profile → API Tokens.
- [ ] **Step 5: Run install.sh**
```bash
export NATS_VERSION=2.10.22
sudo --preserve-env=NATS_VERSION bash ~/Projects/infra/nats/install.sh
```
Expected output (key lines):
```
[install.sh] nats-server installed to /usr/local/bin/nats-server
[install.sh] Created nats system user.
[install.sh] Directory ready: /var/nats/jetstream
[install.sh] Directory ready: /var/nats/jwt
[install.sh] Directory ready: /var/nats/tls
[install.sh] Config template copied to /etc/nats/nats-server.conf.template
[install.sh] TLS cert and key owned by nats:nats
[install.sh] nats.service installed and daemon reloaded.
[install.sh] Phase 1 complete.
```
- [ ] **Step 6: Verify install artifacts**
```bash
# Binary installed and correct version
/usr/local/bin/nats-server --version
# nats user exists
id nats
# Directories exist with correct ownership
ls -la /var/nats/
# TLS cert exists and is readable by nats user
sudo -u nats cat /var/nats/tls/sync.craftdesign.group.crt | head -1
# Expected: -----BEGIN CERTIFICATE-----
# Unit file installed
systemctl status nats
# Expected: inactive (dead) — not started yet, that's correct
```
- [ ] **Step 7: Commit**
```bash
cd ~/Projects/infra
fossil add nats/install.sh
fossil commit -m "Add nats/install.sh (Phase 1)"
```
---
## Chunk 3: bootstrap-creds.sh and start.sh
### Task 3: Write `bootstrap-creds.sh` (Phase 2)
**Files:**
- Create: `infra/nats/bootstrap-creds.sh`
- [ ] **Step 1: Install nsc CLI (if not already installed)**
```bash
which nsc || curl -fsSL https://install.nats.io/nsc | sh
# Ensure nsc is on PATH
export PATH="${HOME}/.nsc/bin:${PATH}"
nsc --version
```
Expected: `nsc version 0.x.x` (any version).
- [ ] **Step 2: Write `nats/bootstrap-creds.sh`**
```bash
cat > ~/Projects/infra/nats/bootstrap-creds.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
# Run as dmestas (NOT root) — nsc stores creds in ~/.local/share/nats/nsc/
# Requires: nsc on PATH, /etc/nats/ writable by this user (or sudo for final write)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CREDS_DIR="${SCRIPT_DIR}/creds"
log() { echo "[bootstrap-creds.sh] $*"; }
die() { echo "[bootstrap-creds.sh] ERROR: $*" >&2; exit 1; }
command -v nsc &>/dev/null || die "nsc not found. Install with: curl -fsSL https://install.nats.io/nsc | sh; export PATH=\${HOME}/.nsc/bin:\${PATH}"
# ── Operator ─────────────────────────────────────────────────────────────────
if nsc list operators 2>/dev/null | grep -q 'fossil-sync'; then
log "Operator fossil-sync already exists, skipping creation."
else
log "Creating operator fossil-sync..."
nsc add operator --name fossil-sync
nsc edit operator --name fossil-sync \
--account-jwt-server-url tls://localhost:4222
fi
# ── Accounts ─────────────────────────────────────────────────────────────────
for ACCOUNT in edgesync go-libfossil; do
if nsc describe account --name "$ACCOUNT" &>/dev/null; then
log "Account $ACCOUNT already exists, skipping."
else
log "Creating account $ACCOUNT..."
nsc add account --name "$ACCOUNT"
fi
done
# ── Users ─────────────────────────────────────────────────────────────────────
# Admin users (local provisioning only — full pub/sub)
for ACCOUNT in edgesync go-libfossil; do
if nsc describe user --account "$ACCOUNT" --name admin &>/dev/null; then
log "User admin in $ACCOUNT already exists, skipping."
else
log "Creating admin user in account $ACCOUNT..."
nsc add user --account "$ACCOUNT" --name admin \
--allow-pub '>' --allow-sub '>'
fi
done
# Bridge users (scoped to project subjects)
if ! nsc describe user --account edgesync --name bridge &>/dev/null; then
nsc add user --account edgesync --name bridge \
--allow-pub 'fossil.edgesync.>' \
--allow-sub 'fossil.edgesync.>'
log "Created bridge user in edgesync."
fi
if ! nsc describe user --account go-libfossil --name bridge &>/dev/null; then
nsc add user --account go-libfossil --name bridge \
--allow-pub 'fossil.go-libfossil.>' \
--allow-sub 'fossil.go-libfossil.>'
log "Created bridge user in go-libfossil."
fi
# ── Generate .creds files ─────────────────────────────────────────────────────
mkdir -p "$CREDS_DIR"
log "Generating .creds files in ${CREDS_DIR}..."
nsc generate creds --account edgesync --name bridge > "${CREDS_DIR}/edgesync-bridge.creds"
nsc generate creds --account go-libfossil --name bridge > "${CREDS_DIR}/go-libfossil-bridge.creds"
nsc generate creds --account edgesync --name admin > "${CREDS_DIR}/edgesync-admin.creds"
nsc generate creds --account go-libfossil --name admin > "${CREDS_DIR}/go-libfossil-admin.creds"
chmod 0600 "${CREDS_DIR}"/*.creds
# ── Extract operator JWT ──────────────────────────────────────────────────────
log "Extracting operator JWT..."
OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null \
| grep '^operator:' \
| awk '{print $2}')
[[ -n "$OPERATOR_JWT" ]] || die "Failed to extract operator JWT from nsc generate config output."
OPERATOR_ENV=/etc/nats/operator.env
if [[ -w "$(dirname "$OPERATOR_ENV")" ]]; then
echo "OPERATOR_JWT=${OPERATOR_JWT}" > "$OPERATOR_ENV"
chmod 0600 "$OPERATOR_ENV"
log "Wrote OPERATOR_JWT to $OPERATOR_ENV"
else
# /etc/nats may be root-owned; write via sudo
echo "OPERATOR_JWT=${OPERATOR_JWT}" | sudo tee "$OPERATOR_ENV" > /dev/null
sudo chmod 0600 "$OPERATOR_ENV"
log "Wrote OPERATOR_JWT to $OPERATOR_ENV (via sudo)"
fi
log ""
log "Phase 2 complete."
log "Next: sudo bash nats/start.sh"
SCRIPT
chmod +x ~/Projects/infra/nats/bootstrap-creds.sh
```
- [ ] **Step 3: Validate with shellcheck**
```bash
shellcheck ~/Projects/infra/nats/bootstrap-creds.sh
```
Expected: no output.
- [ ] **Step 4: Run bootstrap-creds.sh**
```bash
# Run as dmestas (NOT root)
cd ~/Projects/infra
bash nats/bootstrap-creds.sh
```
Expected output (key lines):
```
[bootstrap-creds.sh] Creating operator fossil-sync...
[bootstrap-creds.sh] Creating account edgesync...
[bootstrap-creds.sh] Creating account go-libfossil...
[bootstrap-creds.sh] Generating .creds files in .../creds...
[bootstrap-creds.sh] Wrote OPERATOR_JWT to /etc/nats/operator.env
[bootstrap-creds.sh] Phase 2 complete.
```
- [ ] **Step 5: Verify credential artifacts**
```bash
# Four .creds files generated
ls -la ~/Projects/infra/nats/creds/
# Expected: edgesync-admin.creds, edgesync-bridge.creds, go-libfossil-admin.creds, go-libfossil-bridge.creds
# operator.env contains a JWT (long base64-like string)
sudo grep -o 'OPERATOR_JWT=.\{20\}' /etc/nats/operator.env
# Expected: OPERATOR_JWT=eyJ0eXAiOiJ... (truncated)
# nsc store populated
nsc list accounts
# Expected: edgesync, go-libfossil listed
```
- [ ] **Step 6: Commit**
```bash
cd ~/Projects/infra
fossil add nats/bootstrap-creds.sh
fossil commit -m "Add nats/bootstrap-creds.sh (Phase 2)"
```
---
### Task 4: Write `start.sh` (Phase 3)
**Files:**
- Create: `infra/nats/start.sh`
- [ ] **Step 1: Write `nats/start.sh`**
```bash
cat > ~/Projects/infra/nats/start.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
# Run as root (needs systemctl enable/start)
OPERATOR_ENV=/etc/nats/operator.env
CONF_TEMPLATE=/etc/nats/nats-server.conf.template
CONF_LIVE=/etc/nats/nats-server.conf
log() { echo "[start.sh] $*"; }
die() { echo "[start.sh] ERROR: $*" >&2; exit 1; }
[[ -f "$OPERATOR_ENV" ]] || die "Missing $OPERATOR_ENV — run bootstrap-creds.sh first."
[[ -f "$CONF_TEMPLATE" ]] || die "Missing $CONF_TEMPLATE — run install.sh first."
command -v envsubst &>/dev/null || die "envsubst not found (install gettext: apt-get install -y gettext-base)"
# ── Render config from template ───────────────────────────────────────────────
# shellcheck source=/dev/null
source "$OPERATOR_ENV"
[[ -n "${OPERATOR_JWT:-}" ]] || die "OPERATOR_JWT is empty in $OPERATOR_ENV"
envsubst < "$CONF_TEMPLATE" > "$CONF_LIVE"
log "Rendered $CONF_TEMPLATE → $CONF_LIVE"
# ── Enable and start nats ─────────────────────────────────────────────────────
systemctl enable --now nats
log "nats service enabled and started."
# ── Wait for server to be ready ───────────────────────────────────────────────
log "Waiting for NATS to accept connections..."
for i in $(seq 1 10); do
if ss -tlnp | grep -q ':4222'; then
log "Port 4222 is listening."
break
fi
sleep 1
[[ "$i" -eq 10 ]] && die "NATS did not start in 10 seconds. Check: journalctl -u nats -n 50"
done
# ── Push account JWTs to resolver ────────────────────────────────────────────
# Switch to dmestas to run nsc (nsc store is in dmestas home)
# nsc is at ~/.nsc/bin/nsc — must be on PATH explicitly for non-login shell
log "Pushing account JWTs to resolver..."
sudo -u dmestas bash -c \
'export PATH="${HOME}/.nsc/bin:${PATH}"; nsc push --all --server tls://localhost:4222 --tlsskipverify' || \
die "nsc push failed. Check nsc is installed for dmestas and accounts exist."
log ""
log "Phase 3 complete."
log "Next: bash nats/provision-streams.sh"
SCRIPT
chmod +x ~/Projects/infra/nats/start.sh
```
- [ ] **Step 2: Validate with shellcheck**
```bash
shellcheck ~/Projects/infra/nats/start.sh
```
Expected: no output.
- [ ] **Step 3: Ensure `gettext-base` is installed (provides `envsubst`)**
```bash
envsubst --version || sudo apt-get install -y gettext-base
```
- [ ] **Step 4: Run start.sh**
```bash
sudo bash ~/Projects/infra/nats/start.sh
```
Expected output (key lines):
```
[start.sh] Rendered /etc/nats/nats-server.conf.template → /etc/nats/nats-server.conf
[start.sh] nats service enabled and started.
[start.sh] Port 4222 is listening.
[start.sh] Pushing account JWTs to resolver...
[start.sh] Phase 3 complete.
```
- [ ] **Step 5: Verify NATS is running**
```bash
sudo systemctl status nats
# Expected: active (running)
sudo journalctl -u nats -n 20
# Expected: lines containing "Listening for client connections on 0.0.0.0:4222"
# Local TLS ping (no external DNS needed yet)
nats server ping --server tls://localhost:4222 --tlsskipverify
# Expected: nats-server (vps-nbg1) responded OK in X.XXms
```
- [ ] **Step 6: Commit**
```bash
cd ~/Projects/infra
fossil add nats/start.sh
fossil commit -m "Add nats/start.sh (Phase 3)"
```
---
## Chunk 4: provision-streams.sh, DNS, firewall, and end-to-end verification
### Task 5: Write `provision-streams.sh` (Phase 4)
**Files:**
- Create: `infra/nats/provision-streams.sh`
- [ ] **Step 1: Install `nats` CLI (if not already installed)**
```bash
# Skip if already installed
if command -v nats &>/dev/null; then
nats --version
else
# Install for linux/arm64 — verify version at https://github.com/nats-io/natscli/releases
NATS_CLI_VERSION=0.1.4
curl -fsSL "https://github.com/nats-io/natscli/releases/download/v${NATS_CLI_VERSION}/nats-${NATS_CLI_VERSION}-linux-arm64.zip" \
-o /tmp/nats-cli.zip
unzip -o /tmp/nats-cli.zip -d /tmp/nats-cli/
# Locate the binary explicitly
NATS_BIN=$(find /tmp/nats-cli -name nats -type f | head -1)
[[ -n "$NATS_BIN" ]] || { echo "ERROR: nats binary not found in zip archive"; exit 1; }
sudo install -m 0755 "$NATS_BIN" /usr/local/bin/nats
nats --version
fi
```
Expected: `nats: version 0.x.x`
- [ ] **Step 2: Write `nats/provision-streams.sh`**
```bash
cat > ~/Projects/infra/nats/provision-streams.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CREDS_DIR="${SCRIPT_DIR}/creds"
NATS_SERVER="tls://localhost:4222"
# Use an array to avoid unquoted variable expansion (shellcheck SC2086)
NATS_TLS_FLAGS=(--tlsskipverify)
log() { echo "[provision-streams.sh] $*"; }
die() { echo "[provision-streams.sh] ERROR: $*" >&2; exit 1; }
command -v nats &>/dev/null || die "nats CLI not found. Install from https://github.com/nats-io/natscli/releases"
for PROJECT in edgesync go-libfossil; do
STREAM=$(echo "$PROJECT" | tr '[:lower:]' '[:upper:]' | tr '-' '_')
CREDS="${CREDS_DIR}/${PROJECT}-admin.creds"
[[ -f "$CREDS" ]] || die "Missing $CREDS — run bootstrap-creds.sh first."
if nats stream info "$STREAM" \
--server "$NATS_SERVER" "${NATS_TLS_FLAGS[@]}" \
--creds "$CREDS" &>/dev/null; then
log "Stream $STREAM already exists, skipping."
continue
fi
log "Creating stream $STREAM (subjects: fossil.${PROJECT}.>)..."
# All fields specified explicitly to avoid interactive prompts in non-TTY contexts
nats stream add "$STREAM" \
--server "$NATS_SERVER" "${NATS_TLS_FLAGS[@]}" \
--creds "$CREDS" \
--subjects "fossil.${PROJECT}.>" \
--retention limits \
--max-age 168h \
--max-bytes 1073741824 \
--max-msgs -1 \
--max-msgs-per-subject -1 \
--max-msg-size -1 \
--max-consumers -1 \
--discard old \
--storage file \
--replicas 1 \
--dupe-window 2m \
--defaults
log "Stream $STREAM created."
done
log ""
log "Phase 4 complete. NATS is fully provisioned."
log ""
log "Verify streams:"
log " nats stream ls --server tls://localhost:4222 --tlsskipverify --creds ${CREDS_DIR}/edgesync-admin.creds"
SCRIPT
chmod +x ~/Projects/infra/nats/provision-streams.sh
```
- [ ] **Step 3: Validate with shellcheck**
```bash
shellcheck ~/Projects/infra/nats/provision-streams.sh
```
Expected: no output.
- [ ] **Step 4: Run provision-streams.sh**
```bash
cd ~/Projects/infra
bash nats/provision-streams.sh
```
Expected output:
```
[provision-streams.sh] Creating stream EDGESYNC (subjects: fossil.edgesync.>)...
[provision-streams.sh] Stream EDGESYNC created.
[provision-streams.sh] Creating stream GO_LIBFOSSIL (subjects: fossil.go-libfossil.>)...
[provision-streams.sh] Stream GO_LIBFOSSIL created.
[provision-streams.sh] Phase 4 complete. NATS is fully provisioned.
```
- [ ] **Step 5: Verify streams exist**
```bash
CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
nats stream ls --server tls://localhost:4222 --tlsskipverify --creds "$CREDS"
# Expected: EDGESYNC, GO_LIBFOSSIL listed
nats stream info EDGESYNC --server tls://localhost:4222 --tlsskipverify --creds "$CREDS"
# Expected: subjects fossil.edgesync.>, storage File, max age 168h, max bytes 1.0 GiB
```
- [ ] **Step 6: Commit**
```bash
cd ~/Projects/infra
fossil add nats/provision-streams.sh
fossil commit -m "Add nats/provision-streams.sh (Phase 4)"
```
---
### Task 6: DNS, Hetzner firewall, and end-to-end verification
**Files:** No new files — this task is manual configuration and verification.
- [ ] **Step 1: Add Hetzner Cloud firewall rule for port 4222**
In the [Hetzner Cloud Console](https://console.hetzner.cloud/):
1. Navigate to the VPS → Firewalls
2. Add inbound rule: Protocol TCP, Port 4222, Source 0.0.0.0/0 and ::/0
3. Apply the firewall
Or via `hcloud` CLI (if installed):
```bash
hcloud firewall add-rule <firewall-name> --direction in --protocol tcp --port 4222 --source-ips 0.0.0.0/0 --source-ips ::/0
```
- [ ] **Step 2: Add DNS A record in Cloudflare**
In the [Cloudflare dashboard](https://dash.cloudflare.com/) for `craftdesign.group`:
1. DNS → Add record
2. Type: A, Name: `sync`, IPv4: `<hetzner-vps-public-ip>`, Proxy status: **DNS only (grey cloud)**
> **Important:** The proxy toggle must be set to grey cloud (DNS only). NATS speaks raw TCP — Cloudflare's proxy only handles HTTP/HTTPS and cannot forward port 4222. An orange-cloud (proxied) record will cause all client connections to fail silently.
Get the public IP if needed:
```bash
curl -s https://ipinfo.io/ip
```
- [ ] **Step 3: Verify DNS propagation**
```bash
# Retry until the record resolves (may take 1–2 minutes)
until dig +short sync.craftdesign.group | grep -q '.'; do
echo "Waiting for DNS..."; sleep 10
done
dig +short sync.craftdesign.group
# Expected: <hetzner-vps-public-ip>
```
- [ ] **Step 4: Verify public TLS connectivity**
```bash
# Ping via public hostname
nats server ping --server tls://sync.craftdesign.group:4222
# Expected: nats-server (vps-nbg1) responded OK in X.XXms
# Account info via bridge creds
nats account info \
--server tls://sync.craftdesign.group:4222 \
--creds ~/Projects/infra/nats/creds/edgesync-bridge.creds
# Expected: account info for edgesync (no error)
```
- [ ] **Step 5: Verify TLS cert is valid (from external perspective)**
```bash
# Check cert details
openssl s_client -connect sync.craftdesign.group:4222 -showcerts </dev/null 2>/dev/null \
| openssl x509 -noout -subject -dates
# Expected: subject=CN=sync.craftdesign.group, not expired
```
- [ ] **Step 6: Test publish/subscribe round-trip**
> **Manual step — requires two simultaneous terminals.** Cannot be automated as a sequential script because `nats sub` blocks indefinitely.
In **terminal 1** (start first, leave running):
```bash
nats sub "fossil.edgesync.file" \
--server tls://sync.craftdesign.group:4222 \
--creds ~/Projects/infra/nats/creds/edgesync-bridge.creds
```
In **terminal 2** (after terminal 1 shows "Subscribing on fossil.edgesync.file"):
```bash
nats pub "fossil.edgesync.file" "hello from leaf" \
--server tls://sync.craftdesign.group:4222 \
--creds ~/Projects/infra/nats/creds/edgesync-bridge.creds
```
Expected: terminal 1 receives and prints `hello from leaf`. Press Ctrl-C to stop the subscriber.
- [ ] **Step 7: Verify JetStream persistence (publish then disconnect and replay)**
```bash
CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
SERVER="tls://sync.craftdesign.group:4222"
# Publish a message to the stream
nats pub "fossil.edgesync.test-persistence" "persistence-test" --server "$SERVER" --creds "$CREDS"
# Confirm the most recent message is stored (--last avoids hardcoding a sequence number)
nats stream get EDGESYNC --last --server "$SERVER" --creds "$CREDS"
# Expected: message data "persistence-test"
```
- [ ] **Step 8: Final commit — mark infra setup complete**
```bash
cd ~/Projects/infra
fossil commit -m "NATS server fully provisioned and verified"
```
---
## Appendix: Troubleshooting
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `nats-server` fails to start | Bad operator JWT in conf | Re-run `start.sh` with fresh `operator.env` |
| `nsc push` fails | Server not listening yet | Wait for port 4222, check `journalctl -u nats` |
| TLS handshake error locally | cert/key not owned by nats | `chown nats:nats /var/nats/tls/*` |
| DNS not resolving | Cloudflare proxy enabled | Set DNS record to grey cloud (DNS only) |
| Pub/sub auth error | JWT not pushed to resolver | `nsc push --all --server tls://localhost:4222 --tlsskipverify` |
| acme.sh DNS challenge fails | CF_Token missing or wrong scope | Verify token has `Zone:DNS:Edit` for `craftdesign.group` |