For agentic workers: REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Deploy a NATS 2.10 server with JetStream and NKeys+JWT auth on the Hetzner VPS, reachable publicly at tls://sync.craftdesign.group:4222, with all configuration version-controlled in the infra Fossil repo.

Architecture: Four sequential install phases (install.sh → bootstrap-creds.sh → start.sh → provision-streams.sh) produce a running, authenticated NATS server. TLS is handled by acme.sh DNS-01 via Cloudflare. Auth uses a three-tier NKeys+JWT hierarchy (operator → accounts → users) managed by nsc.

Tech Stack: NATS Server 2.10 (linux/arm64), nsc CLI, acme.sh, Fossil SCM, systemd, bash.

Chunk 1: Repo scaffolding and static config files

Task 1: Create directory structure and static config files

Files:

Create: infra/nats/nats-server.conf.template
Create: infra/nats/nats.service
Create: infra/nats/Dockerfile
Create: infra/nats/creds/README.md
Create: infra/nats/README.md
Create: infra/README.md

Working directory for all tasks: ~/Projects/infra

[ ] Step 1: Write nats/nats-server.conf.template

cat > ~/Projects/infra/nats/nats-server.conf.template << 'EOF'
server_name: vps-nbg1

listen: 0.0.0.0:4222

tls {
  cert_file: "/var/nats/tls/sync.craftdesign.group.crt"
  key_file:  "/var/nats/tls/sync.craftdesign.group.key"
}

jetstream {
  store_dir: "/var/nats/jetstream"
}

# JWT resolver — accounts pushed via `nsc push --all`
resolver: {
  type:         full
  dir:          "/var/nats/jwt"
  allow_delete: false
  interval:     "2m"
}

# Operator JWT — substituted by start.sh via envsubst
operator: $OPERATOR_JWT

# Cluster placeholder — populate when adding a second node
# cluster {
#   name: fossil-sync
#   listen: 0.0.0.0:6222
#   routes: [ nats://node2.internal:6222 ]
# }
EOF

[ ] Step 2: Write nats/nats.service

cat > ~/Projects/infra/nats/nats.service << 'EOF'
[Unit]
Description=NATS Server
After=network-online.target
Wants=network-online.target

[Service]
User=nats
Group=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats-server.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
EOF

[ ] Step 3: Write nats/Dockerfile

cat > ~/Projects/infra/nats/Dockerfile << 'EOF'
FROM nats:2.10-alpine
# Render nats-server.conf.template before building:
#   source /etc/nats/operator.env
#   envsubst < nats-server.conf.template > nats-server.conf-rendered
#   docker build --build-arg CONF=nats-server.conf-rendered -t fossil-nats:2.10 .
ARG CONF=nats-server.conf-rendered
COPY ${CONF} /etc/nats/nats-server.conf
# TLS certs and creds are mounted at runtime — never baked into the image:
#   docker run -v /var/nats/tls:/var/nats/tls:ro \
#              -v /var/nats/jwt:/var/nats/jwt \
#              -v /var/nats/jetstream:/var/nats/jetstream \
#              -p 4222:4222 fossil-nats:2.10
CMD ["nats-server", "-c", "/etc/nats/nats-server.conf"]
EOF

[ ] Step 4: Write nats/creds/README.md

cat > ~/Projects/infra/nats/creds/README.md << 'EOF'
# Credentials

This directory holds generated `.creds` files for NATS users. These files are
fossil-ignored and must never be committed.

## Regenerating the full credential tree

Prerequisites: `nsc` CLI installed (`curl -fsSL https://install.nats.io/nsc | sh`)

bash

1. Add operator (only if nsc store is empty)

nsc add operator --name fossil-sync nsc edit operator --account-jwt-server-url tls://localhost:4222

2. Add accounts

nsc add account --name edgesync nsc add account --name go-libfossil

3. Add users

nsc add user --account edgesync --name admin --allow-pub '>' --allow-sub '>' nsc add user --account go-libfossil --name admin --allow-pub '>' --allow-sub '>'

nsc add user --account edgesync --name bridge --allow-pub 'fossil.edgesync.>' --allow-sub 'fossil.edgesync.>'

nsc add user --account go-libfossil --name bridge --allow-pub 'fossil.go-libfossil.>' --allow-sub 'fossil.go-libfossil.>'

4. Generate .creds files into this directory

nsc generate creds --account edgesync --name bridge > edgesync-bridge.creds nsc generate creds --account go-libfossil --name bridge > go-libfossil-bridge.creds nsc generate creds --account edgesync --name admin > edgesync-admin.creds nsc generate creds --account go-libfossil --name admin > go-libfossil-admin.creds

5. Re-extract operator JWT

OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null | grep '^operator:' | awk '{print $2}') echo "OPERATOR_JWT=${OPERATOR_JWT}" > /etc/nats/operator.env chmod 0600 /etc/nats/operator.env

6. Push JWTs to running server

nsc push --all --server tls://localhost:4222 --tlsskipverify ```

Adding a new leaf node

PROJECT=edgesync   # or go-libfossil
HOSTNAME=m4-macbook

nsc add user --account "$PROJECT" \
  --name "leaf-${HOSTNAME}" \
  --allow-pub  "fossil.${PROJECT}.igot,fossil.${PROJECT}.file,fossil.${PROJECT}.delta" \
  --allow-sub  "fossil.${PROJECT}.gimme,fossil.${PROJECT}.file,fossil.${PROJECT}.meta,fossil.${PROJECT}.events"

nsc generate creds --account "$PROJECT" --name "leaf-${HOSTNAME}" > "leaf-${HOSTNAME}.creds"
nsc push --all --server tls://localhost:4222 --tlsskipverify

# Transfer to leaf node over Tailscale
scp "leaf-${HOSTNAME}.creds" "${HOSTNAME}:/path/to/agent/creds/"

EOF ```

[ ] Step 5: Write nats/README.md

cat > ~/Projects/infra/nats/README.md << 'EOF'
# NATS Server

NATS 2.10 with JetStream and NKeys+JWT auth.

- **Public endpoint:** `tls://sync.craftdesign.group:4222`
- **Local admin:** `tls://localhost:4222 --tlsskipverify`
- **JetStream streams:** EDGESYNC, GO_LIBFOSSIL (7 days / 1 GB each)
- **Spec:** `docs/superpowers/specs/2026-03-15-nats-server-setup-design.md`

## Install (fresh VPS)

bash

0. Place Cloudflare API token first

sudo mkdir -p /etc/nats echo 'CF_Token=""' | sudo tee /etc/nats/cf-token.env sudo chmod 0600 /etc/nats/cf-token.env

1-4. Run phases in order

export NATS_VERSION=2.10.22 sudo --preserve-env=NATS_VERSION bash nats/install.sh bash nats/bootstrap-creds.sh sudo bash nats/start.sh bash nats/provision-streams.sh ```

Service management

sudo systemctl status nats
sudo journalctl -u nats -f
sudo systemctl kill --signal=SIGHUP nats   # reload certs/config, no connection drop

Verify

nats server ping --server tls://sync.craftdesign.group:4222
nats server ping --server tls://localhost:4222 --tlsskipverify

EOF ```

[ ] Step 6: Write infra/README.md

cat > ~/Projects/infra/README.md << 'EOF'
# infra

Infrastructure configuration for craftdesign.group services.

| Service | Directory | Endpoint |
|---------|-----------|----------|
| NATS    | `nats/`   | tls://sync.craftdesign.group:4222 |
EOF

[ ] Step 7: Create .fossil-ignore to prevent credential staging

cat > ~/Projects/infra/.fossil-ignore << 'EOF'
nats/creds/*.creds
/etc/nats/nats-server.conf
EOF

[ ] Step 8: Add files to Fossil and commit

cd ~/Projects/infra
fossil add nats/ README.md .fossil-ignore
fossil commit -m "Add NATS repo scaffolding and static config files"

Expected: New_Version: <hash>

Chunk 2: install.sh

Task 2: Write and validate `install.sh` (Phase 1)

Files:

Create: infra/nats/install.sh

install.sh must be idempotent — safe to re-run on a machine where NATS is already partially installed.

[ ] Step 1: Write nats/install.sh

cat > ~/Projects/infra/nats/install.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail

# Ensure acme.sh and cron renewal run under root's home, not the invoking user's
HOME=/root

# ── Configuration ────────────────────────────────────────────────────────────
NATS_VERSION="${NATS_VERSION:?NATS_VERSION must be set, e.g. export NATS_VERSION=2.10.22}"
NATS_BIN=/usr/local/bin/nats-server
NATS_CONF_DIR=/etc/nats
NATS_DATA_DIR=/var/nats
CF_TOKEN_FILE="${NATS_CONF_DIR}/cf-token.env"
DOMAIN=sync.craftdesign.group
RELOAD_CMD="systemctl kill --signal=SIGHUP nats"

# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "[install.sh] $*"; }
die() { echo "[install.sh] ERROR: $*" >&2; exit 1; }

# ── Step 1: Download and install nats-server binary ──────────────────────────
if [[ -x "$NATS_BIN" ]] && "$NATS_BIN" --version 2>/dev/null | grep -q "v${NATS_VERSION}$"; then
  log "nats-server $NATS_VERSION already installed, skipping download."
else
  log "Downloading nats-server v${NATS_VERSION} linux/arm64..."
  ARCHIVE="nats-server-v${NATS_VERSION}-linux-arm64.tar.gz"
  URL="https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/${ARCHIVE}"
  TMP=$(mktemp -d)
  curl -fsSL "$URL" -o "${TMP}/${ARCHIVE}"
  tar -xzf "${TMP}/${ARCHIVE}" -C "$TMP"
  install -m 0755 "${TMP}/nats-server-v${NATS_VERSION}-linux-arm64/nats-server" "$NATS_BIN"
  rm -rf "$TMP"
  log "nats-server installed to $NATS_BIN"
fi

# ── Step 2: Create nats system user ─────────────────────────────────────────
if ! id nats &>/dev/null; then
  useradd --system --no-create-home --shell /usr/sbin/nologin nats
  log "Created nats system user."
else
  log "nats user already exists, skipping."
fi

# ── Step 3: Create data directories ─────────────────────────────────────────
for DIR in "${NATS_DATA_DIR}/jetstream" "${NATS_DATA_DIR}/jwt" "${NATS_DATA_DIR}/tls"; do
  mkdir -p "$DIR"
  chown nats:nats "$DIR"
  chmod 0750 "$DIR"
  log "Directory ready: $DIR"
done

# ── Step 4: Create /etc/nats and copy config template ───────────────────────
mkdir -p "$NATS_CONF_DIR"
chown root:root "$NATS_CONF_DIR"
chmod 0755 "$NATS_CONF_DIR"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cp "${SCRIPT_DIR}/nats-server.conf.template" "${NATS_CONF_DIR}/nats-server.conf.template"
log "Config template copied to ${NATS_CONF_DIR}/nats-server.conf.template"

# ── Step 5: Verify Cloudflare token file ────────────────────────────────────
[[ -f "$CF_TOKEN_FILE" ]] || die \
  "Missing $CF_TOKEN_FILE. Create it with: echo 'CF_Token=\"<token>\"' > $CF_TOKEN_FILE && chmod 0600 $CF_TOKEN_FILE"
[[ "$(stat -c %a "$CF_TOKEN_FILE")" == "600" ]] || \
  { chmod 0600 "$CF_TOKEN_FILE"; log "Fixed permissions on $CF_TOKEN_FILE"; }

# ── Step 6: Install acme.sh and issue TLS cert ──────────────────────────────
ACME="${HOME}/.acme.sh/acme.sh"
if [[ ! -x "$ACME" ]]; then
  log "Installing acme.sh..."
  curl -fsSL https://get.acme.sh | sh -s [email protected]
fi

CERT_FILE="${NATS_DATA_DIR}/tls/${DOMAIN}.crt"
KEY_FILE="${NATS_DATA_DIR}/tls/${DOMAIN}.key"

# shellcheck source=/dev/null
source "$CF_TOKEN_FILE"

if "$ACME" --list 2>/dev/null | grep -q "$DOMAIN"; then
  log "Certificate for $DOMAIN already exists in acme.sh store."
else
  log "Issuing TLS certificate for $DOMAIN via DNS-01..."
  "$ACME" --issue --dns dns_cf -d "$DOMAIN"
fi

# Always install cert to target paths (ensures files exist on re-runs)
"$ACME" --install-cert -d "$DOMAIN" \
  --key-file        "$KEY_FILE" \
  --fullchain-file  "$CERT_FILE" \
  --reloadcmd       "$RELOAD_CMD"

chown nats:nats "$CERT_FILE" "$KEY_FILE"
chmod 0640 "$CERT_FILE" "$KEY_FILE"
log "TLS cert and key written to $NATS_DATA_DIR/tls/ and owned by nats:nats"

# ── Step 7: Install systemd unit ────────────────────────────────────────────
cp "${SCRIPT_DIR}/nats.service" /etc/systemd/system/nats.service
systemctl daemon-reload
log "nats.service installed and daemon reloaded."

log ""
log "Phase 1 complete."
log "Next: run bootstrap-creds.sh (as dmestas, not root)"
SCRIPT
chmod +x ~/Projects/infra/nats/install.sh

[ ] Step 2: Validate script syntax with shellcheck

shellcheck ~/Projects/infra/nats/install.sh

Expected: no output (no errors). If shellcheck is not installed: sudo apt-get install -y shellcheck.

[ ] Step 3: Verify NATS version exists for linux/arm64 before running

export NATS_VERSION=2.10.22
curl -fsI "https://github.com/nats-io/nats-server/releases/download/v${NATS_VERSION}/nats-server-v${NATS_VERSION}-linux-arm64.tar.gz" | head -1

Expected: HTTP/2 302 or HTTP/1.1 302 (redirect to asset). If 404, check the NATS releases page for the latest 2.10.x version and update NATS_VERSION.

[ ] Step 4: Place the Cloudflare API token file before running

# Cloudflare token needs Zone:DNS:Edit permission for craftdesign.group
sudo mkdir -p /etc/nats
sudo bash -c 'echo "CF_Token=\"<YOUR_CF_TOKEN>\"" > /etc/nats/cf-token.env'
sudo chmod 0600 /etc/nats/cf-token.env

Replace <YOUR_CF_TOKEN> with the actual token from Cloudflare dashboard → My Profile → API Tokens.

[ ] Step 5: Run install.sh

export NATS_VERSION=2.10.22
sudo --preserve-env=NATS_VERSION bash ~/Projects/infra/nats/install.sh

Expected output (key lines):

[install.sh] nats-server installed to /usr/local/bin/nats-server
[install.sh] Created nats system user.
[install.sh] Directory ready: /var/nats/jetstream
[install.sh] Directory ready: /var/nats/jwt
[install.sh] Directory ready: /var/nats/tls
[install.sh] Config template copied to /etc/nats/nats-server.conf.template
[install.sh] TLS cert and key owned by nats:nats
[install.sh] nats.service installed and daemon reloaded.
[install.sh] Phase 1 complete.

[ ] Step 6: Verify install artifacts

# Binary installed and correct version
/usr/local/bin/nats-server --version

# nats user exists
id nats

# Directories exist with correct ownership
ls -la /var/nats/

# TLS cert exists and is readable by nats user
sudo -u nats cat /var/nats/tls/sync.craftdesign.group.crt | head -1
# Expected: -----BEGIN CERTIFICATE-----

# Unit file installed
systemctl status nats
# Expected: inactive (dead) — not started yet, that's correct

[ ] Step 7: Commit

cd ~/Projects/infra
fossil add nats/install.sh
fossil commit -m "Add nats/install.sh (Phase 1)"

Chunk 3: bootstrap-creds.sh and start.sh

Task 3: Write `bootstrap-creds.sh` (Phase 2)

Files:

Create: infra/nats/bootstrap-creds.sh
[ ] Step 1: Install nsc CLI (if not already installed)

which nsc || curl -fsSL https://install.nats.io/nsc | sh
# Ensure nsc is on PATH
export PATH="${HOME}/.nsc/bin:${PATH}"
nsc --version

Expected: nsc version 0.x.x (any version).

[ ] Step 2: Write nats/bootstrap-creds.sh

cat > ~/Projects/infra/nats/bootstrap-creds.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail

# Run as dmestas (NOT root) — nsc stores creds in ~/.local/share/nats/nsc/
# Requires: nsc on PATH, /etc/nats/ writable by this user (or sudo for final write)

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CREDS_DIR="${SCRIPT_DIR}/creds"

log() { echo "[bootstrap-creds.sh] $*"; }
die() { echo "[bootstrap-creds.sh] ERROR: $*" >&2; exit 1; }

command -v nsc &>/dev/null || die "nsc not found. Install with: curl -fsSL https://install.nats.io/nsc | sh; export PATH=\${HOME}/.nsc/bin:\${PATH}"

# ── Operator ─────────────────────────────────────────────────────────────────
if nsc list operators 2>/dev/null | grep -q 'fossil-sync'; then
  log "Operator fossil-sync already exists, skipping creation."
else
  log "Creating operator fossil-sync..."
  nsc add operator --name fossil-sync
  nsc edit operator --name fossil-sync \
    --account-jwt-server-url tls://localhost:4222
fi

# ── Accounts ─────────────────────────────────────────────────────────────────
for ACCOUNT in edgesync go-libfossil; do
  if nsc describe account --name "$ACCOUNT" &>/dev/null; then
    log "Account $ACCOUNT already exists, skipping."
  else
    log "Creating account $ACCOUNT..."
    nsc add account --name "$ACCOUNT"
  fi
done

# ── Users ─────────────────────────────────────────────────────────────────────
# Admin users (local provisioning only — full pub/sub)
for ACCOUNT in edgesync go-libfossil; do
  if nsc describe user --account "$ACCOUNT" --name admin &>/dev/null; then
    log "User admin in $ACCOUNT already exists, skipping."
  else
    log "Creating admin user in account $ACCOUNT..."
    nsc add user --account "$ACCOUNT" --name admin \
      --allow-pub '>' --allow-sub '>'
  fi
done

# Bridge users (scoped to project subjects)
if ! nsc describe user --account edgesync --name bridge &>/dev/null; then
  nsc add user --account edgesync --name bridge \
    --allow-pub  'fossil.edgesync.>' \
    --allow-sub  'fossil.edgesync.>'
  log "Created bridge user in edgesync."
fi

if ! nsc describe user --account go-libfossil --name bridge &>/dev/null; then
  nsc add user --account go-libfossil --name bridge \
    --allow-pub  'fossil.go-libfossil.>' \
    --allow-sub  'fossil.go-libfossil.>'
  log "Created bridge user in go-libfossil."
fi

# ── Generate .creds files ─────────────────────────────────────────────────────
mkdir -p "$CREDS_DIR"
log "Generating .creds files in ${CREDS_DIR}..."

nsc generate creds --account edgesync     --name bridge > "${CREDS_DIR}/edgesync-bridge.creds"
nsc generate creds --account go-libfossil --name bridge > "${CREDS_DIR}/go-libfossil-bridge.creds"
nsc generate creds --account edgesync     --name admin  > "${CREDS_DIR}/edgesync-admin.creds"
nsc generate creds --account go-libfossil --name admin  > "${CREDS_DIR}/go-libfossil-admin.creds"
chmod 0600 "${CREDS_DIR}"/*.creds

# ── Extract operator JWT ──────────────────────────────────────────────────────
log "Extracting operator JWT..."
OPERATOR_JWT=$(nsc generate config --nats-resolver 2>/dev/null \
  | grep '^operator:' \
  | awk '{print $2}')

[[ -n "$OPERATOR_JWT" ]] || die "Failed to extract operator JWT from nsc generate config output."

OPERATOR_ENV=/etc/nats/operator.env
if [[ -w "$(dirname "$OPERATOR_ENV")" ]]; then
  echo "OPERATOR_JWT=${OPERATOR_JWT}" > "$OPERATOR_ENV"
  chmod 0600 "$OPERATOR_ENV"
  log "Wrote OPERATOR_JWT to $OPERATOR_ENV"
else
  # /etc/nats may be root-owned; write via sudo
  echo "OPERATOR_JWT=${OPERATOR_JWT}" | sudo tee "$OPERATOR_ENV" > /dev/null
  sudo chmod 0600 "$OPERATOR_ENV"
  log "Wrote OPERATOR_JWT to $OPERATOR_ENV (via sudo)"
fi

log ""
log "Phase 2 complete."
log "Next: sudo bash nats/start.sh"
SCRIPT
chmod +x ~/Projects/infra/nats/bootstrap-creds.sh

[ ] Step 3: Validate with shellcheck

shellcheck ~/Projects/infra/nats/bootstrap-creds.sh

Expected: no output.

[ ] Step 4: Run bootstrap-creds.sh

# Run as dmestas (NOT root)
cd ~/Projects/infra
bash nats/bootstrap-creds.sh

Expected output (key lines):

[bootstrap-creds.sh] Creating operator fossil-sync...
[bootstrap-creds.sh] Creating account edgesync...
[bootstrap-creds.sh] Creating account go-libfossil...
[bootstrap-creds.sh] Generating .creds files in .../creds...
[bootstrap-creds.sh] Wrote OPERATOR_JWT to /etc/nats/operator.env
[bootstrap-creds.sh] Phase 2 complete.

[ ] Step 5: Verify credential artifacts

# Four .creds files generated
ls -la ~/Projects/infra/nats/creds/
# Expected: edgesync-admin.creds, edgesync-bridge.creds, go-libfossil-admin.creds, go-libfossil-bridge.creds

# operator.env contains a JWT (long base64-like string)
sudo grep -o 'OPERATOR_JWT=.\{20\}' /etc/nats/operator.env
# Expected: OPERATOR_JWT=eyJ0eXAiOiJ... (truncated)

# nsc store populated
nsc list accounts
# Expected: edgesync, go-libfossil listed

[ ] Step 6: Commit

cd ~/Projects/infra
fossil add nats/bootstrap-creds.sh
fossil commit -m "Add nats/bootstrap-creds.sh (Phase 2)"

Task 4: Write `start.sh` (Phase 3)

Files:

Create: infra/nats/start.sh
[ ] Step 1: Write nats/start.sh

cat > ~/Projects/infra/nats/start.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail

# Run as root (needs systemctl enable/start)
OPERATOR_ENV=/etc/nats/operator.env
CONF_TEMPLATE=/etc/nats/nats-server.conf.template
CONF_LIVE=/etc/nats/nats-server.conf

log() { echo "[start.sh] $*"; }
die() { echo "[start.sh] ERROR: $*" >&2; exit 1; }

[[ -f "$OPERATOR_ENV" ]]    || die "Missing $OPERATOR_ENV — run bootstrap-creds.sh first."
[[ -f "$CONF_TEMPLATE" ]]   || die "Missing $CONF_TEMPLATE — run install.sh first."
command -v envsubst &>/dev/null || die "envsubst not found (install gettext: apt-get install -y gettext-base)"

# ── Render config from template ───────────────────────────────────────────────
# shellcheck source=/dev/null
source "$OPERATOR_ENV"
[[ -n "${OPERATOR_JWT:-}" ]] || die "OPERATOR_JWT is empty in $OPERATOR_ENV"

envsubst < "$CONF_TEMPLATE" > "$CONF_LIVE"
log "Rendered $CONF_TEMPLATE → $CONF_LIVE"

# ── Enable and start nats ─────────────────────────────────────────────────────
systemctl enable --now nats
log "nats service enabled and started."

# ── Wait for server to be ready ───────────────────────────────────────────────
log "Waiting for NATS to accept connections..."
for i in $(seq 1 10); do
  if ss -tlnp | grep -q ':4222'; then
    log "Port 4222 is listening."
    break
  fi
  sleep 1
  [[ "$i" -eq 10 ]] && die "NATS did not start in 10 seconds. Check: journalctl -u nats -n 50"
done

# ── Push account JWTs to resolver ────────────────────────────────────────────
# Switch to dmestas to run nsc (nsc store is in dmestas home)
# nsc is at ~/.nsc/bin/nsc — must be on PATH explicitly for non-login shell
log "Pushing account JWTs to resolver..."
sudo -u dmestas bash -c \
  'export PATH="${HOME}/.nsc/bin:${PATH}"; nsc push --all --server tls://localhost:4222 --tlsskipverify' || \
  die "nsc push failed. Check nsc is installed for dmestas and accounts exist."

log ""
log "Phase 3 complete."
log "Next: bash nats/provision-streams.sh"
SCRIPT
chmod +x ~/Projects/infra/nats/start.sh

[ ] Step 2: Validate with shellcheck

shellcheck ~/Projects/infra/nats/start.sh

Expected: no output.

[ ] Step 3: Ensure gettext-base is installed (provides envsubst)

envsubst --version || sudo apt-get install -y gettext-base

[ ] Step 4: Run start.sh

sudo bash ~/Projects/infra/nats/start.sh

Expected output (key lines):

[start.sh] Rendered /etc/nats/nats-server.conf.template → /etc/nats/nats-server.conf
[start.sh] nats service enabled and started.
[start.sh] Port 4222 is listening.
[start.sh] Pushing account JWTs to resolver...
[start.sh] Phase 3 complete.

[ ] Step 5: Verify NATS is running

sudo systemctl status nats
# Expected: active (running)

sudo journalctl -u nats -n 20
# Expected: lines containing "Listening for client connections on 0.0.0.0:4222"

# Local TLS ping (no external DNS needed yet)
nats server ping --server tls://localhost:4222 --tlsskipverify
# Expected: nats-server (vps-nbg1) responded OK in X.XXms

[ ] Step 6: Commit

cd ~/Projects/infra
fossil add nats/start.sh
fossil commit -m "Add nats/start.sh (Phase 3)"

Chunk 4: provision-streams.sh, DNS, firewall, and end-to-end verification

Task 5: Write `provision-streams.sh` (Phase 4)

Files:

Create: infra/nats/provision-streams.sh
[ ] Step 1: Install nats CLI (if not already installed)

# Skip if already installed
if command -v nats &>/dev/null; then
  nats --version
else
  # Install for linux/arm64 — verify version at https://github.com/nats-io/natscli/releases
  NATS_CLI_VERSION=0.1.4
  curl -fsSL "https://github.com/nats-io/natscli/releases/download/v${NATS_CLI_VERSION}/nats-${NATS_CLI_VERSION}-linux-arm64.zip" \
    -o /tmp/nats-cli.zip
  unzip -o /tmp/nats-cli.zip -d /tmp/nats-cli/
  # Locate the binary explicitly
  NATS_BIN=$(find /tmp/nats-cli -name nats -type f | head -1)
  [[ -n "$NATS_BIN" ]] || { echo "ERROR: nats binary not found in zip archive"; exit 1; }
  sudo install -m 0755 "$NATS_BIN" /usr/local/bin/nats
  nats --version
fi

Expected: nats: version 0.x.x

[ ] Step 2: Write nats/provision-streams.sh

cat > ~/Projects/infra/nats/provision-streams.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CREDS_DIR="${SCRIPT_DIR}/creds"
NATS_SERVER="tls://localhost:4222"
# Use an array to avoid unquoted variable expansion (shellcheck SC2086)
NATS_TLS_FLAGS=(--tlsskipverify)

log() { echo "[provision-streams.sh] $*"; }
die() { echo "[provision-streams.sh] ERROR: $*" >&2; exit 1; }

command -v nats &>/dev/null || die "nats CLI not found. Install from https://github.com/nats-io/natscli/releases"

for PROJECT in edgesync go-libfossil; do
  STREAM=$(echo "$PROJECT" | tr '[:lower:]' '[:upper:]' | tr '-' '_')
  CREDS="${CREDS_DIR}/${PROJECT}-admin.creds"

  [[ -f "$CREDS" ]] || die "Missing $CREDS — run bootstrap-creds.sh first."

  if nats stream info "$STREAM" \
      --server "$NATS_SERVER" "${NATS_TLS_FLAGS[@]}" \
      --creds  "$CREDS" &>/dev/null; then
    log "Stream $STREAM already exists, skipping."
    continue
  fi

  log "Creating stream $STREAM (subjects: fossil.${PROJECT}.>)..."
  # All fields specified explicitly to avoid interactive prompts in non-TTY contexts
  nats stream add "$STREAM" \
    --server            "$NATS_SERVER" "${NATS_TLS_FLAGS[@]}" \
    --creds             "$CREDS" \
    --subjects          "fossil.${PROJECT}.>" \
    --retention         limits \
    --max-age           168h \
    --max-bytes         1073741824 \
    --max-msgs          -1 \
    --max-msgs-per-subject -1 \
    --max-msg-size      -1 \
    --max-consumers     -1 \
    --discard           old \
    --storage           file \
    --replicas          1 \
    --dupe-window       2m \
    --defaults

  log "Stream $STREAM created."
done

log ""
log "Phase 4 complete. NATS is fully provisioned."
log ""
log "Verify streams:"
log "  nats stream ls --server tls://localhost:4222 --tlsskipverify --creds ${CREDS_DIR}/edgesync-admin.creds"
SCRIPT
chmod +x ~/Projects/infra/nats/provision-streams.sh

[ ] Step 3: Validate with shellcheck

shellcheck ~/Projects/infra/nats/provision-streams.sh

Expected: no output.

[ ] Step 4: Run provision-streams.sh

cd ~/Projects/infra
bash nats/provision-streams.sh

Expected output:

[provision-streams.sh] Creating stream EDGESYNC (subjects: fossil.edgesync.>)...
[provision-streams.sh] Stream EDGESYNC created.
[provision-streams.sh] Creating stream GO_LIBFOSSIL (subjects: fossil.go-libfossil.>)...
[provision-streams.sh] Stream GO_LIBFOSSIL created.
[provision-streams.sh] Phase 4 complete. NATS is fully provisioned.

[ ] Step 5: Verify streams exist

CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
nats stream ls --server tls://localhost:4222 --tlsskipverify --creds "$CREDS"
# Expected: EDGESYNC, GO_LIBFOSSIL listed

nats stream info EDGESYNC --server tls://localhost:4222 --tlsskipverify --creds "$CREDS"
# Expected: subjects fossil.edgesync.>, storage File, max age 168h, max bytes 1.0 GiB

[ ] Step 6: Commit

cd ~/Projects/infra
fossil add nats/provision-streams.sh
fossil commit -m "Add nats/provision-streams.sh (Phase 4)"

Task 6: DNS, Hetzner firewall, and end-to-end verification

Files: No new files — this task is manual configuration and verification.

[ ] Step 1: Add Hetzner Cloud firewall rule for port 4222

In the Hetzner Cloud Console:

Navigate to the VPS → Firewalls
Add inbound rule: Protocol TCP, Port 4222, Source 0.0.0.0/0 and ::/0
Apply the firewall

Or via hcloud CLI (if installed):

hcloud firewall add-rule <firewall-name> --direction in --protocol tcp --port 4222 --source-ips 0.0.0.0/0 --source-ips ::/0

[ ] Step 2: Add DNS A record in Cloudflare

In the Cloudflare dashboard for craftdesign.group:

DNS → Add record
Type: A, Name: sync, IPv4: <hetzner-vps-public-ip>, Proxy status: DNS only (grey cloud)

Important: The proxy toggle must be set to grey cloud (DNS only). NATS speaks raw TCP — Cloudflare's proxy only handles HTTP/HTTPS and cannot forward port 4222. An orange-cloud (proxied) record will cause all client connections to fail silently.

Get the public IP if needed:

curl -s https://ipinfo.io/ip

[ ] Step 3: Verify DNS propagation

# Retry until the record resolves (may take 1–2 minutes)
until dig +short sync.craftdesign.group | grep -q '.'; do
  echo "Waiting for DNS..."; sleep 10
done
dig +short sync.craftdesign.group
# Expected: <hetzner-vps-public-ip>

[ ] Step 4: Verify public TLS connectivity

# Ping via public hostname
nats server ping --server tls://sync.craftdesign.group:4222
# Expected: nats-server (vps-nbg1) responded OK in X.XXms

# Account info via bridge creds
nats account info \
  --server tls://sync.craftdesign.group:4222 \
  --creds ~/Projects/infra/nats/creds/edgesync-bridge.creds
# Expected: account info for edgesync (no error)

[ ] Step 5: Verify TLS cert is valid (from external perspective)

# Check cert details
openssl s_client -connect sync.craftdesign.group:4222 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates
# Expected: subject=CN=sync.craftdesign.group, not expired

[ ] Step 6: Test publish/subscribe round-trip

Manual step — requires two simultaneous terminals. Cannot be automated as a sequential script because nats sub blocks indefinitely.

In terminal 1 (start first, leave running):

nats sub "fossil.edgesync.file" \
  --server tls://sync.craftdesign.group:4222 \
  --creds ~/Projects/infra/nats/creds/edgesync-bridge.creds

In terminal 2 (after terminal 1 shows "Subscribing on fossil.edgesync.file"):

nats pub "fossil.edgesync.file" "hello from leaf" \
  --server tls://sync.craftdesign.group:4222 \
  --creds ~/Projects/infra/nats/creds/edgesync-bridge.creds

Expected: terminal 1 receives and prints hello from leaf. Press Ctrl-C to stop the subscriber.

[ ] Step 7: Verify JetStream persistence (publish then disconnect and replay)

CREDS=~/Projects/infra/nats/creds/edgesync-admin.creds
SERVER="tls://sync.craftdesign.group:4222"

# Publish a message to the stream
nats pub "fossil.edgesync.test-persistence" "persistence-test" --server "$SERVER" --creds "$CREDS"

# Confirm the most recent message is stored (--last avoids hardcoding a sequence number)
nats stream get EDGESYNC --last --server "$SERVER" --creds "$CREDS"
# Expected: message data "persistence-test"

[ ] Step 8: Final commit — mark infra setup complete

cd ~/Projects/infra
fossil commit -m "NATS server fully provisioned and verified"

Appendix: Troubleshooting

Symptom	Likely cause	Fix
`nats-server` fails to start	Bad operator JWT in conf	Re-run `start.sh` with fresh `operator.env`
`nsc push` fails	Server not listening yet	Wait for port 4222, check `journalctl -u nats`
TLS handshake error locally	cert/key not owned by nats	`chown nats:nats /var/nats/tls/*`
DNS not resolving	Cloudflare proxy enabled	Set DNS record to grey cloud (DNS only)
Pub/sub auth error	JWT not pushed to resolver	`nsc push --all --server tls://localhost:4222 --tlsskipverify`
acme.sh DNS challenge fails	CF_Token missing or wrong scope	Verify token has `Zone:DNS:Edit` for `craftdesign.group`

Chunk 1: Repo scaffolding and static config files

Task 1: Create directory structure and static config files

1. Add operator (only if nsc store is empty)

2. Add accounts

3. Add users

4. Generate .creds files into this directory

5. Re-extract operator JWT

6. Push JWTs to running server

Adding a new leaf node

0. Place Cloudflare API token first

1-4. Run phases in order

Service management

Verify

Chunk 2: install.sh

Task 2: Write and validate install.sh (Phase 1)

Chunk 3: bootstrap-creds.sh and start.sh

Task 3: Write bootstrap-creds.sh (Phase 2)

Task 4: Write start.sh (Phase 3)

Chunk 4: provision-streams.sh, DNS, firewall, and end-to-end verification

Task 5: Write provision-streams.sh (Phase 4)

Task 6: DNS, Hetzner firewall, and end-to-end verification

Appendix: Troubleshooting

Task 2: Write and validate `install.sh` (Phase 1)

Task 3: Write `bootstrap-creds.sh` (Phase 2)

Task 4: Write `start.sh` (Phase 3)

Task 5: Write `provision-streams.sh` (Phase 4)