hermes-skills/skills/devops/hermes-backup/SKILL.md

---
name: hermes-backup
description: "Full Hermes backup to Wasabi S3 — config, sessions, profiles, keys, and restore script."
version: 1.7.0
author: ShoNuff
tags: [hermes, backup, wasabi, s3, disaster-recovery, live-sync, warm-standby]
---

# Hermes Backup

Regularly backs up the entire Hermes environment so you can restore on a new server.

## Pitfalls

- **Do not invent fallback keys.** A top-level `fallback_providers: [...]` entry is not a valid Hermes setting and may be silently ignored. Current Hermes uses `model.fallbacks` for main-agent fallbacks and `delegation.fallback` for child-agent fallbacks. Treat the installed schema and `hermes config check` as authoritative. After editing, parse the YAML, run `hermes config check`, restart the gateway if required, and perform a real fallback inference test. See `references/hermes-config-discipline.md`.
- **Never use `/tmp` as staging directory** — On most Linux systems, `/tmp` is a **tmpfs** (RAM-backed, usually ~1GB). Hermes backups include state.db and profiles/ which easily exceed this. Stage backups on root disk: `BACKUP_DIR="/root/.hermes/.backups/hermes-backup-$(date +%F)"`.
- **Check `/tmp` usage before debugging** — If the backup script exits with `No space left on device` but `df -h /` shows free space, run `df -h /tmp`. If it's a tmpfs at 100%, the fix is moving the staging dir off tmpfs, not clearing disk space.
- **The backup archive is created from the staging directory** — If you move `BACKUP_DIR` off tmpfs, also move `ARCHIVE` off tmpfs to match. The `tar czf` runs from the staging dir's parent, so both paths need consistent disk backing.
- **Fix the config.yaml temp_dir too** — The backup Python script (e.g. `wisp-backup.py`) likely has its own `temp_dir` in config.yaml. If you only change the bash wrapper's staging dir, the Python script still writes intermediate files to `/tmp`. Patch both.
- **`tar` `cd` path must match staging dir parent** — If `BACKUP_DIR` moved off `/tmp` to `~/.hermes/.backups/`, the `tar czf` command must `cd "$(dirname "$BACKUP_DIR")"` not `cd /tmp`. Otherwise tar produces `Cannot stat: No such file`.
- **Wasabi bucket names are globally unique** — `hermes-backups` may be taken by another account. Use alternatives like `hermes-vps-backups`.
- **IAM policy needs `PutBucketVersioning` explicitly** — Without it, `put-bucket-versioning` returns `Access Denied` even if other S3 permissions work.
- **Policy changes take 30-60s to propagate** — If a CreateBucket or PutBucketVersioning call fails with AccessDenied immediately after policy update, retry after a short wait.
- **ListBuckets returns AccessDenied — this is expected** — The IAM user's policy scopes to specific bucket ARNs, not `s3:ListAllMyBuckets`. Test by reading a specific bucket by name instead.
- **After migration, verify S3 access immediately** — Run `aws s3 ls s3://<bucket-name>/ --endpoint-url https://s3.us-east-1.wasabisys.com/`. If you get `InvalidAccessKeyId`, the credentials file didn't survive the transfer. Check `~/.aws/credentials` exists and is readable.
- **The cold spare / standby section was moved** — the old "Failover to cold spare (~2 min recovery)" was replaced by the automatic warm standby restore below. The old manual procedure (`curl ... install.sh && aws s3 cp latest-hermes.tar.gz && systemctl restart`) is deprecated in favor of the systemd oneshot pattern.
- **Live sync and warm standby are independent layers** — The 15-min live sync pushes TO S3. The warm standby restore pulls FROM S3. They don't depend on each other's scripts or configs.
- **Secret files consolidated into standalone files (`.hetzner_token`, `.netcup_api_key`) are fragile** — They get left behind during migration, forgotten during key rotation, and break scripts silently. Best practice is to consolidate all secrets into `~/.hermes/.env` (chmod 600) and remove the standalone files. See `references/secret-management.md` for the standard workflow.
- **Post-migration, check that standalone token files were properly merged into `.env`** — `grep -r "api_key\|token\|password\|secret" ~/.hermes/config.yaml` (expect refs only, not values). If scripts reference deleted files, patch them to read from `.env` instead.
- **System-level systemd services are NOT backed up** — The backup script only copies from `~/.config/systemd/user/hermes-gateway*.service` (user-scoped). All services at `/etc/systemd/system/` — `hermes.service`, `hermes-assistant.service`, `hermes-browser.service`, `shark-game.service`, `mysql-tunnel.service`, `ollama.service` — are silently missing from backups. A full restore on a new server loses every systemd unit except the Anita gateway. **FIXED 2026-07-08:** backup script now copies from `/etc/systemd/system/hermes*.service` plus mysql-tunnel, ollama, and shark-game. Restore script uses `sudo` to write to `/etc/systemd/system/` and runs `systemctl daemon-reload`. If you add new systemd services, add `cp /etc/systemd/system/<new-service>.service "$BACKUP_DIR/systemd/"` to both the backup and embedded restore scripts.
- **Caddy reverse proxy config is NOT backed up** — `/etc/caddy/Caddyfile` is the single point of failure for all web routing. A restore on a new server would have Hermes running but all Caddy-proxied services unreachable. **FIXED 2026-07-08:** backup script now copies `/etc/caddy/Caddyfile` to `$BACKUP_DIR/caddy/`, and the live-sync script pushes it to `s3://hermes-vps-backups/live/caddy/Caddyfile`. The embedded restore script restores to `/etc/caddy/` with `sudo`.
- **Only wisp_rsa SSH keys are backed up** — The backup script explicitly copies `$HOME/.ssh/wisp_rsa` and `.pub` but ignores `itpp-infra`, `authorized_keys`, and any other keys. The itpp-infra key is used by the mysql-tunnel systemd service — restore without it means crippled database access. Fix: extend the SSH key loop to include `itpp-infra` and `authorized_keys`.
- **Standalone credential files outside the profiles/ dir are not backed up** — Files like `/root/.hermes/config/migration-creds.txt`, `/root/.hermes/references/dre-temp-passwords.txt` are invisible to the backup pipeline. The profiles/ dir is covered via rsync, but root-level credential files are only backed up if they match hardcoded names. **FIXED 2026-07-08:** `migration-creds.txt` now explicitly backed up to `$BACKUP_DIR/config/migration-creds.txt` and restored with `chmod 600`. Run `find /root/.hermes -maxdepth 3 \( -name '*.env' -o -name '*cred*' -o -name '*pass*' -o -name '*secret*' -o -name '*token*' \)` to audit remaining gaps and add them explicitly.
- **Private keys with wrong permissions (644 instead of 600)** — `/root/wg-server.key`, `/root/wg-ccr.key` are WireGuard private keys world-readable. `/root/.hermes/references/dre-temp-passwords.txt` is 644 (passwords readable by all users). Find them with `find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' -o -name '*secret*' ) -not -perm 600`. Fix: `chmod 600` on each.
- **Stale full backups are silent** — The daily full backup cron can stop running (e.g. last ran 3+ days ago). No alert fires when a daily backup is missed. The live sync (every 15 min) continues running, masking the full backup's absence. Check recent full backup dates with `aws s3 ls s3://$BUCKET/$DEST_DIR/ --endpoint-url $ENDPOINT | tail -5`. **FIXED 2026-07-10:** Daily watchdog script at `/root/.hermes/scripts/backup-audit-check.sh` runs from system crontab at 2 AM UTC (1h after backup). Logs to syslog tag `backup-audit`. Full reference: `references/backup-audit-watchdog.md`.
- **`~/.aws/config` is not explicitly backed up** — The backup script copies `~/.aws/credentials` but not `~/.aws/config`. While it only contains the region setting (43 bytes, 644 perms), it's still a gap if you're automating complete restore. Add `cp "$HOME/.aws/config" "$BACKUP_DIR/aws/" 2>/dev/null || true`.
- **All backup scripts MUST source the AWS CLI venv for cron** — Cron has a minimal PATH. Without `source /opt/awscli-venv/bin/activate`, `aws` is not found and the upload step silently fails while the local archive is created successfully. This gives a false sense of security. `hermes-backup.sh` was missing this activation (fixed Jul 11, 2026). `root-essentials-backup.sh` and `hermes-live-sync.sh` had it correctly. Audit all scripts with `grep -L 'awscli-venv' /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh`.
- **`root-essentials-backup.sh` has a wrong Caddyfile path** — The `tar --append` on line 30 uses `-C /etc systemd/system/caddy/Caddyfile` which resolves to `/etc/systemd/system/caddy/Caddyfile` (does not exist). Correct path: `-C /etc caddy/Caddyfile` → `/etc/caddy/Caddyfile`. The error is hidden by `2>/dev/null || true`, so the tarball silently lacks the Caddyfile.
- **`root-essentials-backup.sh` uses `/tmp` for staging** — The archive is written to `/tmp/root-essentials-<date>.tar.gz` before upload. On this installation `/tmp` has 7.5G free so it works, but the general guidance is to avoid tmpfs for backup staging. If the archive grows beyond tmpfs capacity, the script fails mid-backup.
- **`root-essentials-backup.sh` can silently fail the S3 upload while local tarballs succeed** — The script has `set -euo pipefail`, so if the `aws s3 cp` upload step fails, the script exits immediately without cleaning up. This leaves stranded local tarballs in `/tmp/root-essentials-*.tar.gz` while S3 shows stale or missing files. The journal/syslog output (piped through `logger`) may show only the first echo line ("Creating backup...") with nothing after, because the upload failure prevents later echo lines from executing. The backup-audit watchdog only checks the full-backup bucket (`hermes-vps-backups/hermes-full-backup/`), not the root-backup prefix, so root-essentials failures go undetected. **Diagnostic procedure when root-backup/ looks stale on S3:**
  1. `ls -la /tmp/root-essentials-*.tar.gz` — if recent tarballs exist locally, the upload step is the fault
  2. `journalctl -t root-essentials-backup --no-pager -n 10` — if output stops at "Creating backup...", the script died after tar but before or during upload
  3. `grep 'root-essentials-backup' /var/log/syslog` — same check via syslog if journal isn't available
  4. Run the script manually to see the actual error: `bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1`
- **Empty purpose-specific buckets are a silent gap** — `itpropartner-system-configs` and `itpropartner-docker-volumes` exist on Wasabi (confirmed Jul 11, 2026) but contain 0 objects. The sync scripts (`hermes-system-config-sync.sh`, `hermes-docker-sync.sh`) are either not running or failing silently. Their output goes to cron's default mail spool, which may not be monitored. Verify with `aws s3 ls s3://<bucket>/ --recursive --summarize --endpoint-url ...`.

## Backup completeness audit

Run this after any backup script change to verify nothing critical is missing:

```bash
echo "=== Systemd services ==="
echo "User (backed up):"
ls ~/.config/systemd/user/hermes-gateway*.service 2>/dev/null || echo "(none)"
echo "System (NOT backed up):"
ls /etc/systemd/system/*.service 2>/dev/null || echo "(none)"
echo "=== Caddy config ==="
[ -f /etc/caddy/Caddyfile ] && echo "EXISTS (NOT backed up)" || echo "MISSING"
echo "=== SSH keys ==="
echo "Backed up:" && ls ~/.ssh/wisp_rsa* 2>/dev/null || echo "(none)"
echo "NOT backed up:" && for f in ~/.ssh/itpp-infra ~/.ssh/itpp-infra.pub ~/.ssh/authorized_keys; do
  [ -f "$f" ] && echo "  $f" || true; done
echo "=== WireGuard keys ==="
find /root -maxdepth 2 -name '*.key' 2>/dev/null || echo "(none)"
echo "=== Standalone creds outside backup scope ==="
find /root/.hermes -maxdepth 3 ( -name '*.env' -o -name '*cred*' -o -name '*pass*' \
  -o -name '*secret*' -o -name '*token*' ) -not -path '*/profiles/*' 2>/dev/null
echo "=== Permission audit (should all be 600) ==="
find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' \
  -o -name '*secret*' ) -not -perm 600 2>/dev/null || echo "(all correct)"
echo "=== Last full backup dates ==="
source /opt/awscli-venv/bin/activate 2>/dev/null && \
  aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com 2>&1 | tail -5
```

## Post-fix verification (run after script changes)

After modifying the backup or live-sync scripts, verify the new items are actually picked up:

```bash
# 1. Dry-run the backup script's new sections
echo "=== systemd services to be backed up ==="
ls /etc/systemd/system/hermes*.service /etc/systemd/system/mysql-tunnel.service \
   /etc/systemd/system/ollama.service /etc/systemd/system/shark-game.service 2>/dev/null
echo "---"
echo "=== Caddyfile to be backed up ==="
stat /etc/caddy/Caddyfile 2>/dev/null && echo "EXISTS" || echo "MISSING"
echo "---"
echo "=== migration-creds.txt to be backed up ==="
stat /root/.hermes/migration-creds.txt 2>/dev/null && echo "EXISTS" || echo "MISSING"

# 2. Verify live-sync puts Caddyfile on S3
source /opt/awscli-venv/bin/activate
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com/

# 3. Verify the embedded restore.sh heredoc is correct
grep -n 'sudo mkdir -p /etc/systemd/system' /root/.hermes/scripts/hermes-backup.sh
grep -n 'sudo mkdir -p /etc/caddy' /root/.hermes/scripts/hermes-backup.sh
grep -n 'migration-creds.txt' /root/.hermes/scripts/hermes-backup.sh

# 4. Syntax check
bash -n /root/.hermes/scripts/hermes-backup.sh && echo "backup.sh: OK"
bash -n /root/.hermes/scripts/hermes-live-sync.sh && echo "live-sync.sh: OK"

# 5. Venv activation audit — ALL backup/sync scripts must source the venv for cron
echo "=== Scripts missing venv activation (will fail in cron) ==="
for script in /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh; do
    grep -q 'awscli-venv' "$script" 2>/dev/null || echo "  MISSING: $script"
done
```

## SiteGround SFTP backup

SiteGround websites are backed up via SFTP to `s3://hermes-vps-backups/siteground/<site>/`. The connection uses an encrypted RSA key with passphrase. See `references/siteground-backup-pattern.md` for credentials, connection method, exclusion recommendations, and the database export strategy.

**Pitfall — Fleet-wide SFTP backup times out:** Attempting to backup all 21 SiteGround WordPress sites in a single run timed out at 600 seconds (Jul 10, 2026). SFTP over shared hosting is too slow for bulk migration. Two alternatives: (1) Back up individual sites in batches of 3-5, or (2) Use MainWP + WPvivid Pro (already configured, backs up 15 sites daily to Wasabi S3) as the primary migration path and rely on SFTP only for sites not in MainWP.

**Pitfall — SFTP root is the site directory:** SiteGround's SFTP user sees WordPress site directories in the root (`/`), not under `/home/`. Listing `/home/` returns `OSError: list failed`. Always list `/` first to discover the actual site directories.

## What's backed up

| Item | Why |
|------|-----|
| `config.yaml`, `.env`, `auth.json`, `migration-creds.txt` | All config + API keys + migration secrets |
| `state.db` + `sessions/` | Full conversation history (~1.1GB) |
| `skills/` | All installed/created skills |
| `scripts/` | Custom scripts (backup, feed digest, etc.) |
| `profiles/` | Anita's profile + any others |
| `cron/` | Job definitions |
| SSH keys | WISP backup keys |
| Mail passwords | Himalaya creds |
| AWS/Wasabi creds | For S3 access |
| Systemd services | All Hermes services from `/etc/systemd/system/` |
| Caddyfile | Reverse proxy config from `/etc/caddy/` |

## Included in backup

A `restore.sh` script is included — extract the tarball on a new server and run it.

## Verified S3 endpoint

All buckets use:
```
--endpoint-url https://s3.us-east-1.wasabisys.com
```

## Live buckets (created 2026-07-04, versioning ON, all 5 confirmed Jul 10)

| Bucket | Purpose | Status |
|---|---|---|
| `hermes-vps-backups` | Hermes full backup (config, sessions, profiles, keys) + live sync + full archive tarballs + standby scripts | Active since Jul 4 |
| `mikrotik-ccr-backups` | Router config exports (home gateway, wisp gateway) | Active since Jul 4 |
| `itpropartner-system-configs` | System configs: Caddyfile, systemd services, ops scripts, SSH keys, env files | 🟡 Bucket exists (confirmed Jul 11) but **empty** — sync script not running |
| `itpropartner-docker-volumes` | Docker volume data: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama | 🟡 Bucket exists (confirmed Jul 11) but **empty** — sync script not running |
| `itpropartner-backups` | Legacy files (signature images, misc) — data mostly migrated to purpose-specific buckets | Active since Jul 4 |

**Note:** `hermes-backups` (simple name) is taken globally on Wasabi. Always try an alternative like `hermes-vps-backups` or append a qualifier. When adding new buckets to the Wasabi IAM policy, the existing `hermes` policy uses two Statement blocks (one for bucket-level actions like ListBucket/GetBucketVersioning, one for object-level actions like GetObject/PutObject/DeleteObject). Add the new bucket ARN (with and without `/*`) to BOTH blocks. Test with `aws s3 ls`, `aws s3 cp` (PUT), `aws s3 cp ... -` (GET/readback), and `aws s3 rm` (DELETE) — probe files must be cleaned up after.

## Current cron status

- **Hermes live sync**: Every 15 min via `hermes-live-sync.sh` (no_agent, silent) → `s3://hermes-vps-backups/live/`
- **Hermes system config sync**: Every 15 min via `hermes-system-config-sync.sh` (no_agent, silent) → `s3://itpropartner-system-configs/`
- **Hermes docker volume sync**: Every 15 min via `hermes-docker-sync.sh` (no_agent, silent) → `s3://itpropartner-docker-volumes/`
- **Hermes full backup**: Daily at 1:00 AM UTC via **system crontab** (NOT Hermes cron — blocked by gateway lifecycle guard #30719). Script: `hermes-backup.sh`.
- **Full backup audit watchdog**: Daily at 2:00 AM UTC via system crontab. Script: `backup-audit-check.sh`.
- **Root essentials backup**: Daily at 3:00 AM UTC via system crontab (42MB tarball). Script: `root-essentials-backup.sh`. S3: `s3://hermes-vps-backups/root-backup/`. Includes /root configs, skills, scripts, references, profiles, SSH keys, systemd services, Caddyfile, and /var/www. Corruption check: downloads + `tar tzf` verify. Restore: extract + live sync for state.db = ~10 min recovery. **Fastest recovery path.**
- **Router backup**: Daily at 6:00 AM UTC (script: `run-wisp-backup.sh` → `wisp-backup/wisp-backup.py`)
- **Hetzner snapshots**: Weekly Monday 5:00 AM UTC (script: `snapshot-hetzner.py`)

## Live sync (every 15m)

A lightweight `aws s3 sync` that mirrors the entire `.hermes/` directory (minus bulky caches like `audio_cache/`, `image_cache/`, `cache/`, `sandboxes/`, `kanban/`, `node/`, `bin/`, `logs/`, `*.lock`) to `s3://hermes-vps-backups/live/`.

**Design principles:**
- **Silent on success** — the no_agent cron script suppresses output when nothing changed. Only failures produce output.
- **`aws s3 sync` handles deltas** — only changed files upload. Typically <5 seconds per run.
- **RPO ~15 minutes** — worst case you lose 15 min of state.db changes.

**Script** (`hermes-live-sync.sh`):
```bash
#!/bin/bash
HERMES_HOME="${HERMES_HOME:-$HOME/.hermes}"
ENDPOINT="https://s3.us-east-1.wasabisys.com"
BUCKET="hermes-vps-backups"
DEST_PREFIX="live"

if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi

aws s3 sync "$HERMES_HOME" "s3://$BUCKET/$DEST_PREFIX/" \
    --endpoint-url "$ENDPOINT" \
    --exclude "audio_cache/*" \
    --exclude "image_cache/*" \
    --exclude "cache/*" \
    --exclude "sandboxes/*" \
    --exclude "kanban/*" \
    --exclude "node/*" \
    --exclude "bin/*" \
    --exclude "logs/*" \
    --exclude "*.lock" \
    --exclude ".skills_prompt_snapshot.json" \
    --exclude ".update_check" \
    2>&1 | grep -v "^$" || true

# NOTE: Docker volumes moved to hermes-docker-sync.sh (itpropartner-docker-volumes)
# NOTE: Caddyfile moved to hermes-system-config-sync.sh (itpropartner-system-configs)
exit 0
```

## Purpose-specific sync scripts

The monolithic `hermes-live-sync.sh` was split into purpose-specific scripts — see `references/purpose-specific-buckets.md` for details on the normalization and the script contents.

### `hermes-system-config-sync.sh` → `s3://itpropartner-system-configs/`
Exports: Caddyfile, systemd service files (hermes*, mysql-tunnel, ollama, shark-game), ops scripts, SSH keys (non-known_hosts), .env + AWS credentials.

### `hermes-docker-sync.sh` → `s3://itpropartner-docker-volumes/`
Exports: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama volume data — each under its own prefix.

### DB Dump Automation (KNOWN GAP — see server-dr-plans-v3.md)
MariaDB on wphost02 (Apex) and fleettracker360 are NOT backed up via scheduled dump to S3. This is a known gap flagged in the Jul 10 DR audit. The backup standard declares daily DB dumps with 90-day retention, but this is not yet deployed.

### Docker Volume Backup (KNOWN GAP — see server-dr-plans-v3.md)
Docker volume backups declared in the provisioning standard but not yet automated. All Docker hosts need volume backup scripts pushed to S3 per-server buckets.

### Memory Consolidation Backup (NEW — Jul 9, 2026)
Memory is auto-consolidated every 10 min with backup-before-prune guarantee:
- Backup to: `s3://hermes-vps-backups/live/memories/memory-backup.json`
- History: `s3://hermes-vps-backups/live/memories/history/memory-backup-<UTC>.json`
- Local retention: 48h under `~/.hermes/memories/.consolidate-backups/`
- Restore procedure: See `devops/disaster-recovery-audit` skill's Memory Consolidation Safety section

### Creating the cron jobs
```bash
hermes cron create \
  --name "hermes-live-sync" \
  --schedule "every 15m" \
  --script hermes-live-sync.sh \
  --no_agent \
  --deliver local
```

## Failover to cold spare (automatic warm standby)

When a cold/spare server exists (e.g. an old primary that was migrated off), a systemd oneshot service syncs the latest live state from S3 before Hermes starts. No manual steps needed — boot it and it picks up from the last 15-min checkpoint.

**The restore script** (`hermes-standby-restore.sh`):
1. Waits for network (up to 30s)
2. Pings the **live netcup box** (152.53.192.33) first — **if it responds, exits 0 without starting Hermes** (split-brain guard)
3. Loads AWS CLI
4. Runs `aws s3 sync` from `s3://hermes-vps-backups/live/` to `~/.hermes/` (same exclusion patterns)
5. Starts Hermes gateway

**Split-brain prevention is mandatory.** The standby box and live box share the same Telegram bot token. If both run Hermes simultaneously, messages get lost and cron jobs fire from both. The ping check happens BEFORE the S3 sync or Hermes start.

Two failover mechanisms exist — **boot-time** (systemd oneshot) and **periodic** (cron watchdog). Both check the live box before acting.

**Boot-time** (`hermes-standby-restore.sh` via systemd):
- Runs once on boot, before Hermes starts
- Pings live box once — if alive, exits 0 (stays dormant)
- If dead, syncs S3 state and starts Hermes
- Designed for cold-start: power on the standby and it picks up

**Periodic watchdog** (`hermes-standby-watchdog.sh` via system crontab):
- Runs every 10 min (system crontab, not Hermes cron — Hermes isn't running on standby)
- Pings live box 3 times immediately
- If all fail, runs 4 confirmation cycles at 60s intervals (~3.5 min total)
- If still dead → sends **Telegram alert** + **email alert**, syncs S3, starts Hermes
- Telegram alert format: `🚨 Hermes Failover — app1 (netcup) is offline\n\nTimestamp: ...\nAction: S3 sync → Hermes gateway start`
- Email alert: same content via SMTP (mail.germainebrown.com:587) with password from `/root/.config/himalaya/g-germainebrown.pass`
- This catches the case where the live box dies *while running* (not just on boot)

**Pre-register the standby SSH key** in your cloud provider's project. Without it, you can't deploy the script. On Hetzner: `POST /v1/ssh_keys` with your public key.

**The systemd unit** (`hermes-standby.service`):
```ini
[Unit]
Description=Hermes Standby Restore — pull latest state from S3 before Hermes starts
Before=hermes.service
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/root/.hermes/scripts/hermes-standby-restore.sh
RemainAfterExit=true
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Deploy on standby box
```bash
wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
chmod +x /root/.hermes/scripts/hermes-standby-restore.sh
cp /root/.hermes/scripts/hermes-standby.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable hermes-standby.service
```

### Pre-requisites for fast failover
- S3 access keys available externally (password manager, not on the server being restored)
- AWS CLI installed and configured on the standby box
- Cold spare server already provisioned and powered on
- SSH key registered in your cloud provider's project **before** attempting standby deployment

### Standby deployment without SSH access (Hetzner rescue mode)

If you have no SSH access to the standby box (no key deployed), use Hetzner rescue mode:

1. **Register your SSH key** in the Hetzner project via API or Cloud Console — rescue mode accepts numeric key IDs, not raw keys:
   ```bash
   curl -s -X POST https://api.hetzner.cloud/v1/ssh_keys \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name":"my-key","public_key":"ssh-ed25519 AAAA..."}'
   ```

2. **Enable rescue mode with that key ID** (the numeric ID from the response, not a string):
   ```bash
   curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/enable_rescue \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"type":"linux64","ssh_keys":[114708824]}'
   ```

3. **Power on** — boots into rescue with your key authorized as root:
   ```bash
   curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/poweron \
     -H "Authorization: Bearer $TOKEN" -d '{}'
   ```

4. **SSH into rescue** as root:
   ```bash
   ssh -i ~/.ssh/your_key root@$SERVER_IP
   ```

5. **Mount the main partition** and inject your SSH key into the OS:
   ```bash
   mkdir -p /mnt/root
   mount /dev/sda1 /mnt/root
   mkdir -p /mnt/root/root/.ssh
   echo "$PUBKEY" >> /mnt/root/root/.ssh/authorized_keys
   chmod 600 /mnt/root/root/.ssh/authorized_keys
   ```

6. **Disable the old Hermes service** on the mounted disk so it doesn't autostart:
   ```bash
   mount -t proc none /mnt/root/proc   # needed for chroot systemctl
   chroot /mnt/root systemctl disable hermes-gateway.service
   ```

7. **Deploy your standby scripts**, then disable rescue + reboot via API:
   ```bash
   curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/disable_rescue \
     -H "Authorization: Bearer $TOKEN" -d '{}'
   curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/reboot \
     -H "Authorization: Bearer $TOKEN" -d '{}'
   ```

**Pitfalls:**
- **Register the SSH key FIRST.** `enable_rescue` needs a numeric `ssh_key_id`. If you get `"SSH key not found"` (404), the key string was passed instead of the integer ID, or the key isn't in the project yet.
- **Rescue mode also sets a root password** — the `enable_rescue` response includes a `root_password` field. If SSH key auth fails, use root + that password.
- **After disable_rescue + reboot, wait ~40s** for the normal OS to boot before attempting SSH.
- **Hetzner Cloud API hostname != OS hostname** — Renaming via `PUT /v1/servers/{id} {"name":"new-name"}` only changes Hetzner's label. The OS still has the old `/etc/hostname`. Fix after first SSH: `hostnamectl set-hostname new-name.fqdn`. Verify with both `hostname` and `hostname -f`.
- **chroot systemctl still works without /proc** but warns "This is not a supported mode of operation." The `enable`/`disable` commands succeed despite the warning. Mount `/proc` if you want clean output.
- **Check server API status before attempting SSH** — `GET /v1/servers/{id}` returns a `status` field. If it's `off`, re-power-on and wait.

## DR issue log

Track all audit findings, root causes, fixes, and verification in `references/dr-issue-log.md`. Every DR audit produces a dated entry.

### DR issue log formatting rules

Germaine views these files on mobile. Pipe tables (`| col | col |`) render unreadably on small screens.

**DR issue log format:**

```
**Fixed** [OK]
- **DR-001** [HIGH] Short title -> what was done

**Resolved** [INFO]
- **DR-009** Short title -> what was done

**Investigating** [PENDING]
- **DR-006** [HIGH] Short title -> what was done
```

Individual issue entries use:
```
**Problem**
Description...

**Root Cause**
Why it happened...

**Fix**
What was applied...

**Verification**
- [OK] Evidence item 1
- [OK] Evidence item 2
```

Sub-status tags use `[OK]`, `[PENDING]`, `[HIGH]`, `[MED]`, `[INFO]` — never use emoji (they garble on some mobile viewers). Use `->` not `→`. Use `--` not `—`. Keep it pure ASCII so any text editor or terminal can display it cleanly. Format per issue: problem, root cause, fix, status, verified-by.

## Periodic backup health audit

Run the **S3 bucket audit procedure** to verify all three backup buckets are healthy — versioning on, files being written, no silent pipeline failures.

The procedure covers:
- Versioning status check on every bucket
- File counts and total size
- Staleness detection (objects from today vs >48h gap)
- Live sync freshness verification (count today's objects in the live/ prefix)
- Full backup archive age check
- Common failure patterns with fixes

See `references/backup-audit-procedure.md` for the full quick-audit script and per-bucket methodology.

**Weekly audit recommended.** The MikroTik router backup pipeline once went 3 days without producing configs before being noticed — routine audits catch these before data loss becomes a risk.

## Verify backup costs from invoices

To confirm backups are running within expected resource usage, extract Hetzner invoices:

```bash
apt-get install -y poppler-utils
pdftotext /path/to/invoice.pdf -
```

Hetzner invoice PDFs line items include: server type counts, unit prices, backup percentage (20% of instance price), and snapshot storage (GB-months). Also check Hetzner invoice line items: if backup charges appear but aren't on the right products, the backup config may need updating.

## Setup

1. Create a Wasabi IAM user (subuser, not root). See `infrastructure-automation` skill's `references/wasabi-iam-setup.md` for the full policy and common pitfalls.
2. Create the Wasabi bucket (names are globally unique — if taken, try alternatives like `hermes-vps-backups`).
3. Edit `/root/.hermes/scripts/hermes-backup.sh`:
   - `BUCKET`: set to the actual bucket name
   - `BACKUP_DIR` and `ARCHIVE`: MUST use a root-disk path (e.g. `~/.hermes/.backups/`), NOT `/tmp` (tmpfs is RAM-backed and fills up)
   - The `tar czf` command runs from `$(dirname "$BACKUP_DIR")` — ensure the script's `cd` target matches the staging dir's actual parent, not `/tmp`
4. Ensure `~/.aws/credentials` exists with the IAM user's access key + secret key, `chmod 600`
5. **Test:** Run `bash /root/.hermes/scripts/hermes-backup-worker.sh` or simulate with a dry `aws s3 ls` on each bucket (see Pitfalls section about ListBuckets). The first real backup fires at the next cron tick.
6. Create the live sync cron job (see Live sync section above)
7. If a cold spare exists, deploy the standby restore (see Failover section above)
8. Cron job handles daily runs automatically

## Secret consolidation workflow

All Hermes secrets should live in a single `~/.hermes/.env` file (chmod 600). Standalone secret files (`scripts/.hetzner_token`, `scripts/.netcup_api_key`, etc.) get lost during migration and forgotten during key rotation.

**Standard consolidation steps:**

1. Read existing `.env` and load into a dict
2. Read each standalone token file
3. Add them to the dict with descriptive names (`HETZNER_API_TOKEN`, `NETCUP_API_KEY`, `ADMIN_AI_API_KEY`, etc.)
4. Write the full dict back to `.env`
5. `chmod 600 ~/.hermes/.env`
6. `rm` the standalone files
7. Patch any scripts that referenced the old file paths to source from `.env` instead
8. Verify: `stat -c '%a' ~/.hermes/.env` → should show `600`

**Custom-provider credential rule:** Do not assume an admin-ai key is still in `config.yaml`, `.env`, or `auth.json`. Inspect all three without exposing values. `auth.json` may retain only a fingerprint after the actual provider and secret were removed. Store the recoverable secret in `.env` using a descriptive variable such as `ADMIN_AI_API_KEY`, configure the provider through supported Hermes fields, and verify `/models` plus a real completion before calling it restored. If auxiliary tasks use a separate key field, verify that copy independently.

## Post-migration verification

After restoring Hermes on a new server, verify these in order:

### S3/Wasabi access
```bash
source /opt/awscli-venv/bin/activate
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
aws s3 ls s3://hermes-vps-backups/live/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | head -10
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5
aws s3 ls s3://itpropartner-system-configs/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://itpropartner-docker-volumes/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://itpropartner-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
```

**Pitfall — stale credentials:** `aws configure list` can show credentials present but `aws s3 ls` returns `InvalidAccessKeyId`. This means the IAM key was rotated on the old server but `~/.aws/credentials` on this server has the stale version. Get the current key from the password manager, update the file, and retest. The file is a simple 3-line INI — just `aws_access_key_id` and `aws_secret_access_key` under `[default]`.

### LLM provider availability
If the provider is a proxy (e.g., admin-ai routing to multiple backends), verify OpenRouter models are reachable:
```bash
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"
```
Check for `openrouter/*` entries in the output. The proxy should expose 80+ models including OpenAI, Claude, Gemini, DeepSeek, etc.

**Pitfall — mismatched API keys:** The API key appears in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. If the proxy rotated its key, both must be updated. The curl test uses the provider key (first hit), but the vision auxiliary provider may have the old key separately.

### IMAP connectivity check
If email triage jobs exist, verify IMAP server reachable proactively:
```bash
python3 -c "
import ssl, socket
ctx = ssl.create_default_context()
s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
print(f'IMAP: {ss.version()}')
ss.close()
"
```
Also verify password files present: `ls ~/.config/himalaya/*.pass 2>/dev/null`

### Heremes gateway health
```bash
hermes gateway status
journalctl -u hermes -n 10 --no-pager
```

### Cron job restoration
```bash
hermes cron list
```
Should show all jobs. If empty, wait 30-60s for scheduler to discover them from state.db.

## Restore on new server (full replacement)

```bash
# Install Hermes first
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

# Download latest backup
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-YYYY-MM-DD.tar.gz . \
    --endpoint-url https://s3.us-east-1.wasabisys.com

# Extract and restore
tar xzf hermes-full-backup-YYYY-MM-DD.tar.gz
cd hermes-backup-YYYY-MM-DD
bash restore.sh

# Then set up live sync and warm standby (see sections above)
# hermes cron create --name "hermes-live-sync" --schedule "every 15m" --script hermes-live-sync.sh --no_agent --deliver local

# Start gateways
hermes gateway restart
hermes profile use anita && anita gateway restart  # if Anita exists
```