Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,302 @@
+---
+name: hermes-standby-deployment
+description: "Deploy a warm standby/failover Hermes box on another server — boot-time restore systemd service, periodic health check cron, Telegram + email alerts on failover."
+version: 1.2.0
+author: ShoNuff
+license: MIT
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [dr, failover, standby, hetzner, s3, backup]
+    related_skills: [hermes-backup, infrastructure-automation]
+---
+
+# Hermes Standby Deployment
+
+Use this when setting up a secondary Hermes server that auto-failovers if the primary dies. The standby stays dormant (Hermes not running) until the primary is confirmed dead.
+
+## Architecture
+
+```
+Primary (netcup)                         Standby (Hetzner)
+┌─────────────────────┐                 ┌──────────────────────────┐
+│  Live Hermes        │      ping       │  Dormant — Hermes OFF    │
+│  15-min sync → S3   │◄──────────────►│  systemd: check on boot  │
+│  Telegram connected  │  152.53.192.33 │  cron: check every 10 min │
+└─────────────────────┘                 └──────────────────────────┘
+         │                                        │
+         ▼                                        ▼
+  s3://hermes-vps-backups/live/
+         │
+         └── If primary dies → standby syncs S3, starts Hermes, alerts via Telegram + email
+```
+
+## Prerequisites
+
+- A Linux server (the standby) — Hetzner CPX11 (2C/2G) works, Ubuntu/Debian
+- The infrastructure SSH key (`itpp-infra`) registered in the Hetzner Cloud project
+- AWS CLI installed on the standby (`pip3 install awscli`)
+- Wasabi S3 credentials set up in `~/.aws/credentials`
+- Hermes Agent installed (`pip3 install hermes-agent`)
+- Telegram bot token + chat ID
+
+## Files
+
+All DR artifacts stored on S3 at `s3://hermes-vps-backups/standby/`:
+
+| File | Purpose |
+|---|---|
+| `DR-PLAN.md` | Full disaster recovery plan |
+| `server-inventory.md` | Full server inventory (all Hetzner + netcup boxes) |
+| `hermes-standby-restore.sh` | Boot-time failover — systemd runs this before Hermes would start |
+| `hermes-standby-watchdog.sh` | Periodic failover — system crontab every 10 min |
+| `hermes-standby.service` | systemd unit for boot restore |
+| `recovery-bundle-YYYY-MM-DD.md` | Full data dump including conversation history |
+
+## SSH Key
+
+The designated infrastructure key is **`itpp-infra`** (Ed25519, fingerprint `e2:42:9c:e3:d6:ed:db:cd:b6:6a:f7:bc:7d:a4:19:92`). It is deployed to all access-managed servers. On this installation it lives at `/root/.ssh/itpp-infra` and `/root/.ssh/itpp-infra.pub`. Key ID `114709791` in the Hetzner Cloud project.
+
+Previous key `wisp_rsa` is superseded but kept locally as fallback.
+
+## Step-by-Step Deployment
+
+### 1. Register SSH key in Hetzner project
+
+Skip if `itpp-infra` is already registered — check via API:
+
+```python
+import urllib.request, json
+token = open('/root/.hermes/scripts/.hetzner_token').read().strip()
+headers = {'Authorization': f'Bearer {token}'}
+req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', headers=headers)
+for k in json.loads(urllib.request.urlopen(req, timeout=10).read()).get('ssh_keys', []):
+    print(f'{k["name"]} (ID {k["id"]})')
+```
+
+If not present, register:
+
+```python
+with open('/root/.ssh/itpp-infra.pub') as f:
+    pubkey = f.read().strip()
+data = json.dumps({"name": "itpp-infra", "public_key": pubkey})
+req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', data=data.encode(), headers={
+    **headers, 'Content-Type': 'application/json'}, method='POST')
+result = json.loads(urllib.request.urlopen(req, timeout=15).read())
+key_id = result['ssh_key']['id']
+print(f'Key registered: ID {key_id}')
+```
+
+Save the returned SSH key ID (e.g. `114709791`) for use in rescue mode.
+
+### 2. Enable rescue mode with SSH key, then power on
+
+**Note: Works reliably on CPX11 and CPX21. Does NOT work on CPX41 — see Pitfalls below.**
+
+```python
+data = json.dumps({"type": "linux64", "ssh_keys": [SSH_KEY_ID]})
+req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/enable_rescue', data=data.encode(), headers=headers, method='POST')
+urllib.request.urlopen(req, timeout=15)
+req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/poweron', data=b'{}', headers=headers, method='POST')
+urllib.request.urlopen(req, timeout=15)
+```
+
+### 3. Inject SSH key via rescue
+
+```bash
+mount /dev/sda1 /mnt/root
+mkdir -p /mnt/root/root/.ssh
+echo '<public-key>' >> /mnt/root/root/.ssh/authorized_keys
+chmod 600 /mnt/root/root/.ssh/authorized_keys
+
+# If ippadmin user exists:
+mkdir -p /mnt/root/home/ippadmin/.ssh
+echo '<public-key>' >> /mnt/root/home/ippadmin/.ssh/authorized_keys
+chown -R 1000:1000 /mnt/root/home/ippadmin/.ssh
+```
+
+### 4. Disable Hermes autostart + install standby service
+
+```bash
+systemctl disable hermes-gateway.service
+
+wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
+  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
+chmod +x /root/.hermes/scripts/hermes-standby-restore.sh
+
+wget -O /root/.hermes/scripts/hermes-standby-watchdog.sh \
+  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-watchdog.sh
+chmod +x /root/.hermes/scripts/hermes-standby-watchdog.sh
+
+wget -O /etc/systemd/system/hermes-standby.service \
+  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby.service
+
+systemctl daemon-reload
+systemctl enable hermes-standby.service
+```
+
+### 5. Add watchdog crontab
+
+```bash
+echo "*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh" | crontab -
+```
+
+### 6. Wasabi credentials
+
+```bash
+mkdir -p ~/.aws
+cat > ~/.aws/credentials << 'EOF'
+[default]
+aws_access_key_id = <ACCESS_KEY>
+aws_secret_access_key = <SECRET_KEY>
+EOF
+cat > ~/.aws/config << 'EOF'
+[default]
+region = us-east-1
+output = json
+EOF
+```
+
+### 7. Telegram .env
+
+```bash
+cat > /root/.hermes/.env << 'EOF'
+TELEGRAM_BOT_TOKEN=<TOKEN>
+TELEGRAM_HOME_CHANNEL=<CHAT_ID>
+TELEGRAM_ALLOWED_USERS=<CHAT_ID>
+TERMINAL_TIMEOUT=60
+TERMINAL_LIFETIME_SECONDS=300
+EOF
+```
+
+### 8. Reboot normal
+
+```bash
+req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/disable_rescue', data=b'{}', headers=headers, method='POST')
+urllib.request.urlopen(req)
+req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/reboot', data=b'{}', headers=headers, method='POST')
+urllib.request.urlopen(req)
+```
+
+### 9. Verify
+
+```bash
+systemctl status hermes-standby   # "active (exited)" + "Live box is REACHABLE"
+systemctl is-enabled hermes-gateway  # "disabled"
+crontab -l  # shows watchdog script
+```
+
+## Failover Logic
+
+### Boot-time (systemd)
+1. Wait for network
+2. Ping primary 3 times
+3. If primary reachable → exit (stay dormant)
+4. If primary dead → sync S3, start Hermes
+
+### Periodic (system crontab, every 5 min)
+1. If Hermes already running on standby → exit (already failed over)
+2. Ping primary 3 times immediately
+3. If all fail → confirm over 4 cycles at 30s intervals (~2 min)
+4. If any succeeds → abort (primary recovered)
+5. If all 4 cycles fail → send Telegram + email alert, sync S3, start Hermes
+
+**Update (July 8, 2026):** Timing changed from every 10 min / 3.5 min confirmation to every 5 min / 2 min confirmation for faster failover detection. Update both the cron schedule (`*/5 * * * *`) and the watchdog script's `sleep` value (30s instead of 60s) when deploying new standby servers.
+
+## Periodic Audit Checklist
+
+When auditing a deployed warm standby (e.g., DR readiness review), run through these checks:
+
+### 1. Power State
+- [ ] Query Hetzner API: `GET /v1/servers/{id}` — note `status` field. The server should be `running` (warm standby: OS up, Hermes off) or `off` (cold standby: boots on demand). Confirm which state the DR plan expects.
+- [ ] SSH in and confirm Hermes gateway is **inactive**: `systemctl is-active hermes-gateway` should print `inactive`.
+- [ ] Check actual uptime: `uptime`. If the server has been up for weeks, confirm the intent is warm (always-on) vs cold (boot-on-demand).
+
+### 2. Standby Systemd Service
+- [ ] `systemctl is-active hermes-standby` — should be `active (exited)`.
+- [ ] Check the last restore log: `tail -5 /var/log/hermes-standby-restore.log` — should show `"Live box is REACHABLE — staying dormant"`.
+- [ ] If the log is empty or shows errors, the restore script may need investigation.
+
+### 3. Watchdog Cron
+- [ ] `crontab -l` — should contain `*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh` (5 min interval, was */10 before Jul 8 2026 update).
+- [ ] Confirm the watchdog script's confirmation sleep is 30s (not 60s): `grep 'sleep' /root/.hermes/scripts/hermes-standby-watchdog.sh`.
+- [ ] Check the watchdog log: `cat /var/log/hermes-standby-watchdog.log` — should be empty if the live box has been healthy. Any entries are prior failure events.
+- [ ] Verify the watchdog script has `chmod +x` and readable credentials.
+
+### 4. Live Sync Freshness
+- [ ] On the **live box**, verify `hermes-live-sync` cron is active and recently ran: check `hermes cron list | grep hermes-live-sync` for last run status.
+- [ ] On S3, check the age of `s3://hermes-vps-backups/live/state.db` — should be ≤15 min old: `aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com`.
+- [ ] Verify `gateway_state.json` and `channel_directory.json` are similarly fresh.
+
+### 5. SSH Key
+- [ ] Confirm the designated key is in `~/.ssh/authorized_keys` on the standby. Test: `ssh -i ~/.ssh/itpp-infra root@<standby-ip> "hostname"`.
+- [ ] Verify the same key is registered in the Hetzner Cloud project (SSH Keys tab).
+
+### 6. Failover Scripts on S3
+- [ ] `aws s3 ls s3://hermes-vps-backups/standby/ --endpoint-url https://s3.us-east-1.wasabisys.com` — should show all 6 files (DR-PLAN.md, server-inventory.md, both .sh scripts, .service, recovery-bundle).
+- [ ] The watchdog script on S3 should match the version deployed on the standby (compare checksum or date).
+
+### 7. Documentation Drift Check
+- [ ] Does `server-inventory.md` accurately describe the standby's power state (running vs off)?
+- [ ] Does `DR-PLAN.md` Scenario A match the actual failover procedure used?
+- [ ] Is the SSH key named correctly in the docs (`itpp-infra` vs old `wisp_rsa`)?
+
+### 8. Disk Space (CRITICAL — Jul 11, 2026)
+- [ ] `df -h /` — must be >20% free. **If <10% free, failover will FAIL.** The state.db alone is ~1.9 GB, sync logs accumulate endlessly, and the tarball extraction needs headroom. On a 38 GB CPX11, 10% is only 3.8 GB — tight but survivable. Below 2 GB, the watchdog script will crash mid-sync and the standby is dead weight.
+- [ ] Check sync log size: `ls -lh /var/log/hermes-standby-sync.log`. The sync script appends without rotation — logs can grow to 40+ MB in a week. Truncate if >500 MB: `> /var/log/hermes-standby-sync.log`.
+- [ ] Recovery commands if low on disk:
+  ```bash
+  apt-get clean && apt-get autoremove --purge -y
+  journalctl --vacuum-size=200M
+  > /var/log/hermes-standby-sync.log
+  docker system prune -a -f 2>/dev/null || true
+  df -h /  # verify >5 GB free
+  ```
+
+> Log each audit result in the session summary. Flag any drift between documented state (server-inventory.md / DR-PLAN.md) and live state as a finding.
+
+## Related Documents
+
+The following authoritative documents supersede some of this skill's detailed instructions. They were submitted by the user during the Jul 9, 2026 session and represent the current target state.
+
+| Document | Covers | Status |
+|---|---|---|
+| **Hermes DR Plan v2** | Two-layer health check, SSH fencing, S3 ownership lock, VACUUM snapshots with integrity gate, backward heartbeat, state machine diagram, app1-bu CPX21 upgrade | 🟡 Planned (not yet implemented) |
+| **Memory Auto-Consolidation DR** | Memory backup-before-prune guarantee, S3 restore procedures, safety invariants, size safety valve caveat | 🟢 Deployed and running |
+
+The v2 DR plan proposes upgrading app1-bu from CPX11 to CPX21 (for local Ollama) and introducing Tailscale-based cross-box communication. These changes have NOT yet been implemented — see the 16-item checklist at the end of the v2 document.
+
+When the user references "the DR plan," determine which version:
+- **v1 (this skill):** The currently deployed standby watchdog. Ping-only health check. Public IP SSH for fencing. S3-based 15-min sync.
+- **v2 (user-submitted doc):** The target architecture. HTTP health endpoint, SSH fence + S3 ownership lock, 10-min VACUUM snapshots, Tailscale tailnet for cross-box traffic, CPX21 upgrade for standby.
+
+- **Documentation drift between server-inventory.md, DR-PLAN.md, and live state.** The DR plan may say "offline" while the server is running, or the SSH key listed in the recovery bundle may differ from what's actually authorized. Verify all three agree during periodic audits.
+- **HETZNER_API_TOKEN as an env var can drift from the `.hetzner_token` file approach.** The recovery bundle references `/root/.hermes/scripts/.hetzner_token` but the token may only exist as `HETZNER_API_TOKEN` in the shell environment. The file can be deleted during cleanup without notice — check both sources during audit.
+- **Telegram split-brain:** Never run Hermes on both boxes simultaneously with the same bot token. The standby only starts Hermes if the primary is confirmed unreachable.
+- **Primary comes back after failover:** Power off the primary if it ever returns — two Hermes instances with the same token wrecks message delivery.
+- **SMTP from standby:** Works from Hetzner (same network as mail server). If standby is elsewhere, may need SMTP firewall rules.
+- **Watchdog script has credentials hardcoded** (Telegram token, SMTP password). Protect with `chmod 600`.
+- **S3 sync doesn't delete local files** — `aws s3 sync` only adds/updates, never removes.
+- **Hostname renaming:** After standby deployment, rename OS hostname to match DNS via `hostnamectl set-hostname`. See `references/standby-hostname-and-renaming.md`.
+- **Bulk rescue:** For batch SSH key injection across multiple Hetzner servers, see `references/bulk-hetzner-rescue.md`.
+- **CPX41 key injection fails via rescue.** The `enable_rescue` API with `ssh_keys` parameter works on CPX11/CPX21 but accepts and silently ignores keys on CPX41. You get `Permission denied` with the correct key. Workarounds: (1) omit `ssh_keys`, use the rescue root password to mount and inject; or (2) ask a user with existing access to add the key. The key WILL work on rebuild since it's registered in the project.
+- **Docker convention:** Docker services on the live box live under `~/docker/<service>/` with compose files. The auto-restore script currently only syncs `~/.hermes/` — `~/docker/` must be restored separately.
+- **app1.itpropartner.com (87.99.144.163)** has an intermittent SSH delay of 30-60s after boot even though port 80 responds immediately.
+- **After rescue-mode reboot, wait for SSH to respond** before attempting. Hostname briefly shows `rescue` — verify against OS hostname.
+- **Standby server should have Tailscale installed** to stay reachable for SSH if public SSH is ever locked down. Install via `curl -fsSL https://tailscale.com/install.sh | sh` and authenticate. See `tailscale-infrastructure-access` skill.
+- **Backup pipeline lives on the live box only.** The standby has no independent S3 sync. If running the standby as hot (always on), add a separate backup cron for its own state.
+- **Telegram conflict after moving profile from standby to live.** If a profile's gateway was running on the standby, the standby process holds the Telegram session lock. Resolution: (1) SSH into standby: `ssh -i ~/.ssh/itpp-infra root@<standby-ip> pkill -f 'profile <name>'`. (2) If it persists, hard reboot the standby via Hetzner API. (3) Restart the profile's gateway on the live box. (4) The conflict may recur for 1-2 cycles (~20-60s) as Telegram expires the old session — this is normal.
+
+## Verification
+
+After setup, the standby should:
+- Stay powered on with Hermes off
+- Log "Live box is REACHABLE" on boot
+- Crontab check silently exits if primary is healthy
+- If primary dies, standby detects within ~4 min and takes over
+
+## Pitfalls
+
+- **AWS CLI PATH in cron:** The standby sync script (`hermes-standby-sync.sh`) calls `aws s3 sync`. In cron's minimal PATH, `aws` may not be found even if it works interactively. The script already sources `/opt/awscli-venv/bin/activate`, but if this venv doesn't exist or the cron PATH changes, the sync silently breaks. Verify by running from cron's stripped environment: `env -i HOME=$HOME PATH=/usr/bin:/bin bash /root/.hermes/scripts/hermes-standby-sync.sh`.
+
+To test: temporarily power off the primary (or block it in the standby's firewall), confirm the watchdog fires, then clean up.
@@ -0,0 +1,51 @@
+# Bulk SSH Key Injection via Hetzner Rescue Mode
+
+When deploying the `itpp-infra` key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.
+
+## Process
+
+1. **Register the SSH key** in the Hetzner project (one-time per key, SKILL.md step 1)
+2. **Enable rescue mode + reboot on all targets** — batch via Python:
+   ```python
+   targets = [(SERVER_ID, "hostname", "IP"), ...]
+   for sid, name, ip in targets:
+       data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
+       req = urllib.request.Request(
+           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
+           data=data.encode(), headers=headers, method='POST')
+       urllib.request.urlopen(req, timeout=15)
+       req2 = urllib.request.Request(
+           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
+           data=b'{}', headers=headers, method='POST')
+       urllib.request.urlopen(req2, timeout=15)
+   ```
+3. **Wait 60-90s** — poll SSH port on each:
+   ```bash
+   for ip in "ip1" "ip2" ...; do
+       timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
+   done
+   ```
+4. **Inject key on all** — loop over each rescue host:
+   ```bash
+   PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
+   for ip in ...; do
+       ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
+           mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
+           echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
+           chmod 600 /mnt/root/.ssh/authorized_keys
+           umount /mnt
+       "
+   done
+   ```
+5. **Disable rescue + reboot all** (same pattern as step 2, but `disable_rescue` + `reboot`)
+6. **Wait 90s**, then verify SSH access with `hostname` command
+
+## Pitfalls
+
+- **Not all servers use `sda1`** — Hetzner rescue only has access to the raw block device. Check with `lsblk` on first SSH. On CPX11/CPX21 it's typically `sda1`; on systems with NVMe it could be `nvme0n1p1`. Fall back to `for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done`.
+- **Remount after first mount attempt** — If a partition is already auto-mounted from a previous rescue, the second `mount` fails. Check `mountpoint -q /mnt` before mounting.
+- **Boot order matters** — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
+- **`disable_rescue` BEFORE `reboot`** — Reboot with rescue still enabled just boots back into rescue.
+- **Server POST times vary** — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
+- **app1.itpropartner.com (87.99.144.163) has intermittent SSH delay** — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
+- **ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy** — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the `ssh_keys` parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue `root_password` from `enable_rescue` instead of key injection.
@@ -0,0 +1,25 @@
+# Failover Timing Update — July 8, 2026
+
+## Change
+Per Germaine's direction, the failover timing was updated:
+
+| Parameter | Before | After |
+|-----------|--------|-------|
+| Check interval | Every 10 min | Every 5 min |
+| Confirmation sleep | 60s × 4 cycles | 30s × 4 cycles |
+| Total confirmation window | ~3.5 min | ~2 min |
+| Max downtime before failover | ~13.5 min | ~7 min |
+
+## What was changed
+1. **Cron on app1-bu:** `*/10 * * * *` → `*/5 * * * *`
+2. **Watchdog script sleep:** `sleep 60` → `sleep 30` (line ~71)
+3. **Confirmation message:** Updated from "~3.5 minutes" to "~2.0 minutes" (lines ~68, ~96)
+
+## Rationale
+Faster detection = shorter failover. 7 min max vs 14 min max. User prioritized speed over cost.
+
+## DR Plan impact
+DR-PLAN.md and server-inventory.md should be updated to reflect:
+- Server is warm (always on), not cold (boot on demand)
+- Check interval is 5 min, not 10 min
+- Verification needs to check both the cron schedule AND the sleep value
@@ -0,0 +1,33 @@
+# Hermes DR Plan v2 — Pending Implementation Checklist
+
+**Submitted by Germaine:** July 9, 2026
+**Status:** Planned (not yet implemented)
+**Supersedes:** v1 ping-only watchdog
+
+## Key Architecture Changes
+
+- Two-layer health check (ping + HTTP health endpoint)
+- SSH fencing + S3 ownership lock (prevents split-brain)
+- Pre-failover integrity check (SQLite PRAGMA)
+- 10-min VACUUM snapshots with integrity gate
+- Backward heartbeat (Core monitors standby)
+- app1-bu upgrade CPX11 → CPX21 for local Ollama (+$6/mo)
+
+## 16-Item Checklist
+
+- [ ] Upgrade app1-bu from CPX11 to CPX21 (Hetzner API)
+- [ ] Install Ollama + llama3.2:3b on app1-bu
+- [ ] Deploy /healthz endpoint on Core (Tailscale-only, bearer token)
+- [ ] Deploy two-layer watchdog on app1-bu (Layer 1 + Layer 2)
+- [ ] Deploy SSH fencing script on app1-bu
+- [ ] Deploy S3 ownership lock with refresh
+- [ ] Deploy 10-min VACUUM snapshot cron on Core
+- [ ] Deploy pre-failover integrity check on app1-bu
+- [ ] Deploy backward heartbeat (Core → app1-bu over Tailscale SSH)
+- [ ] Update hermes-standby-deployment skill with v2 architecture
+- [ ] Whitelist cross-box IPs in fail2ban both directions
+- [ ] Verify: 4 consecutive pings → auto-takeover (server dead)
+- [ ] Verify: Hermes dead but server up → alert + fence → conditional takeover
+- [ ] Verify: state.db corruption → falls back to snapshot, not corrupt file
+- [ ] Verify: Core reboot with standby holding lock → Core stays dormant
+- [ ] Update dr-issue-log.md
@@ -0,0 +1,36 @@
+# Hermes Disaster Recovery Plan v2
+
+**Last updated:** July 9, 2026
+**Author:** Sho'Nuff + Network Services Team
+**Supersedes:** `hermes-standby-deployment` skill's failover logic
+
+---
+
+## Architecture
+
+```
+Core (netcup RS 2000 — 152.53.192.33)          app1-bu (Hetzner CPX21 — 5.161.114.8)
+┌──────────────────────────────────┐          ┌──────────────────────────────────┐
+│  Hermes Agent (active)           │          │  Hermes Gateway (dormant)        │
+│  Ollama + llama3.2:3b (fallback) │◄─Tailnet─►│  Ollama + llama3.2:3b (standby)  │
+│  state.db (SQLite, ~1.87 GB)     │          │  state.db (synced)               │
+│  Telegram gateway (active)       │          │  Telegram gateway (off)          │
+│                                  │          │                                  │
+│  S3 backup every 15 min ─────────┼─────────►│  S3 sync every 5 min ◄───────────┤
+│  Snapshot every 10 min ──────────┼─────────►│                                  │
+└──────────────────────────────────┘          └──────────────────────────────────┘
+         │                                              │
+         ▼                                              ▼
+  s3://hermes-vps-backups/live/
+  s3://hermes-vps-backups/snapshots/     (standby pulls snapshots on change)
+  s3://hermes-vps-backups/ACTIVE_OWNER   (standby writes on takeover only)
+```
+
+**Key changes from v1:**
+- app1-bu upgraded from CPX11 (2C/2G) to CPX21 (4C/8G, ~$14/mo) to run its own local Ollama
+- Both boxes have local Ollama (llama3.2:3b) — health checks never depend on external LLM providers
+- Traffic between boxes goes over Tailscale tailnet (not public IPs), avoiding the Hetzner↔netcup routing issue
+- No 60-second S3 writes from standby — all cross-box communication uses Tailscale directly
+- S3 is backup storage only, not a communication channel
+
+[Full document contents at /root/.hermes/cache/documents/doc_0e07de339e65_hermes-dr-plan-v2.md]
@@ -0,0 +1,59 @@
+# Standby Audit — July 8, 2026
+
+## Server: app1-bu.itpropartner.com (5.161.114.8)
+
+Hetzner CPX11 (2 vCPU / 2 GB RAM / 40 GB SSD) — warm standby for app1 (152.53.192.33, netcup).
+
+### Power State
+- Hetzner API: `running` (server is powered on, OS up)
+- Per DR plan: documented as "🟢 Dormant" but the intent in `hermes-standby-deployment` skill says "Stay powered on with Hermes off" — so running is **correct**, but server-inventory.md calls it "dormant" which is ambiguous.
+- **Note:** If electricity cost ($5/mo) is a concern, this could be a cold standby (boot on demand) with a ~60s slower failover.
+
+### Hermes Gateway
+- `systemctl is-active hermes-gateway` → `inactive` ✅
+
+### Standby Service
+- `systemctl is-active hermes-standby` → `active (exited)` ✅
+- Last boot: Jul 5, 2026
+- Uptime at audit: 3 days, 1:16
+- Restore log: `Live box is REACHABLE — staying dormant (clean exit)` ✅
+
+### Watchdog Cron
+- `crontab -l` → `*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh` ✅
+- Watchdog log: empty (no failures detected) ✅
+- Disk usage: 16G / 38G (43%)
+
+### SSH Access
+- Key: `itpp-infra` (Ed25519) ✅
+- Recent auth log shows SSH from 152.53.192.33 (live box) — heartbeat or previous manual check
+
+### S3 Artifacts
+- `s3://hermes-vps-backups/standby/` — all 6 files present ✅
+- `DR-PLAN.md` — 13 KB, present ✅
+- `hermes-standby-restore.sh` — 2.8 KB, matches deployed version ✅
+- `hermes-standby-watchdog.sh` — 4.3 KB, matches deployed version ✅
+- `hermes-standby.service` — 367 B, matches deployed version ✅
+- `server-inventory.md` — 8.1 KB, present ✅
+- `recovery-bundle-2026-07-05.md` — 280 KB, present ✅
+
+### Live Sync Freshness
+- `hermes-live-sync` cron runs every 15 min on live box ✅
+- `state.db` on S3: 2026-07-08T23:19:08Z (~5 min from audit time) ✅
+- `gateway_state.json`: 2026-07-08T23:19:06Z ✅
+- `channel_directory.json`: 2026-07-08T23:18:59Z ✅
+- Total S3 `live/` data: 32,151 files / 4.8 GiB
+
+### Credential Observations
+- `HETZNER_API_TOKEN` exists as a shell env var
+- `/root/.hermes/scripts/.hetzner_token` file is **missing** (deleted during prior cleanup?)
+- Recovery bundle references the file approach — potential doc/setup drift
+
+### Issues Found
+1. Minor: server-inventory.md says "dormant" (implies off) but server is running — warm standby is correct per skill, but docs ambiguous
+2. Minor: `.hetzner_token` file deleted; token only exists in env var — recovery bundle instructs restoring the file
+3. No critical issues
+
+### Next Audit Recommendation
+- Quarterly or after any DR-plan changes
+- Also verify the live-sync script hasn't drifted (exclusion patterns, credentials)
+- Check Hetzner snapshot cron (`hetzner-weekly-snapshots`) still runs
@@ -0,0 +1,29 @@
+# Standby Hostname & OS Renaming
+
+After deploying the standby watchdog script, also rename the standby's OS to match its DNS name.
+
+## On the standby box
+
+```bash
+hostnamectl set-hostname app1-bu.itpropartner.com
+```
+
+Verify with both `hostname` and `hostname -f`.
+
+## Also rename in the cloud provider API
+
+The cloud provider labels (Hetzner Cloud Console, API `server.name`) and the OS hostname are separate. Update both:
+
+```python
+import urllib.request, json
+headers = {'Authorization': f'Bearer {token}', 'Content-Type': 'application/json'}
+data = json.dumps({"name": "app1-bu.itpropartner.com"})
+req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}', data=data.encode(), headers=headers, method='PUT')
+urllib.request.urlopen(req, timeout=10)
+```
+
+## Why this matters
+
+- The DNS A record, the cloud provider name, and the OS hostname should all agree
+- Scripts that check `hostname` for routing decisions won't break
+- The DR documentation references the DNS name — matching OS hostname reduces confusion during failover
@@ -0,0 +1,75 @@
+# Tailscale Serve for Internal Services
+
+Standard way to expose Docker services on app1 to the tailnet without public exposure.
+
+## Setup
+
+```bash
+# Install Tailscale
+curl -fsSL https://tailscale.com/install.sh | sh
+
+# Authenticate — prints a URL to open in browser
+tailscale up
+```
+
+Once authenticated, the server gets a `100.x.x.x` Tailscale IP:
+
+```bash
+tailscale status
+```
+
+## Expose a Service via Tailscale Serve
+
+```bash
+# Expose port 8080 (Vaultwarden) as HTTPS via Tailscale
+tailscale serve --bg --https 443 --set-path / http://127.0.0.1:8080
+```
+
+This creates a URL like `https://<hostname>.tailc2f3b0.ts.net/` — only accessible within your tailnet, with a valid Tailscale TLS cert. No browser warnings.
+
+## Rename Tailscale hostname
+
+```bash
+tailscale set --hostname <name>
+```
+
+The hostname becomes part of the Tailscale Serve URL.
+
+## Switch URL without breaking the app
+
+If DNS naming changes after Tailscale Serve is configured:
+
+1. The service's DOMAIN env var in the Docker compose must match the Tailscale URL
+2. Update the compose file, restart the container
+3. The Bitwarden app on client devices needs the server URL updated to match
+
+## UFW: Tailscale traffic only
+
+```bash
+# Allow only tailscale interface for internal services
+ufw allow in on tailscale0 to any port 8080 proto tcp comment 'Service via Tailscale'
+
+# Block public access
+ufw delete allow 8080/tcp
+ufw reload
+```
+
+## Best Practices
+
+- Internal services do NOT need public DNS or Let's Encrypt — Tailscale Serve provides HTTPS automatically
+- The Tailscale IP (`100.x.x.x`) changes if the node re-registers — always use the Tailscale domain name for stable references
+- Tailscale must be running and authenticated before Serve will work
+- On the standby/backup, Tailscale is optional but recommended for SSH access if public SSH is ever locked down
+
+## Per-Device Setup
+
+Install the Tailscale app on each device:
+- **iPhone/iPad:** App Store → "Tailscale"
+- **Mac:** [tailscale.com/download-mac](https://tailscale.com/download-mac)
+- Login with the same identity provider as the server
+
+## Pitfalls
+
+- **HTTPS vs HTTP on iOS Bitwarden app:** The Bitwarden app for iOS sometimes has trouble with Tailscale Serve HTTPS URLs. Workaround is to clear the app's server config, force-close, and re-add. If the problem persists, the app may cache encryption parameters tied to the old server URL — delete and reinstall the app.
+- **Hostname with zeros/Os:** If the Tailscale hostname contains `0` (zero) or `O` (letter O), the iOS Bitwarden app may confuse them when displaying the URL. Avoid this by choosing a hostname without ambiguous characters.
+- **Server URL change invalidates login:** If the DOMAIN env var changes in Vaultwarden, users will get "username or password is incorrect" errors on existing client sessions even with the correct password. The client must be reconfigured with the new Server URL. The existing session on the old URL does not carry over — this is a client-side cache issue, not an actual password mismatch.