1.5 KiB
1.5 KiB
Hermes DR Plan v2 — Pending Implementation Checklist
Submitted by Germaine: July 9, 2026 Status: Planned (not yet implemented) Supersedes: v1 ping-only watchdog
Key Architecture Changes
- Two-layer health check (ping + HTTP health endpoint)
- SSH fencing + S3 ownership lock (prevents split-brain)
- Pre-failover integrity check (SQLite PRAGMA)
- 10-min VACUUM snapshots with integrity gate
- Backward heartbeat (Core monitors standby)
- app1-bu upgrade CPX11 → CPX21 for local Ollama (+$6/mo)
16-Item Checklist
- Upgrade app1-bu from CPX11 to CPX21 (Hetzner API)
- Install Ollama + llama3.2:3b on app1-bu
- Deploy /healthz endpoint on Core (Tailscale-only, bearer token)
- Deploy two-layer watchdog on app1-bu (Layer 1 + Layer 2)
- Deploy SSH fencing script on app1-bu
- Deploy S3 ownership lock with refresh
- Deploy 10-min VACUUM snapshot cron on Core
- Deploy pre-failover integrity check on app1-bu
- Deploy backward heartbeat (Core → app1-bu over Tailscale SSH)
- Update hermes-standby-deployment skill with v2 architecture
- Whitelist cross-box IPs in fail2ban both directions
- Verify: 4 consecutive pings → auto-takeover (server dead)
- Verify: Hermes dead but server up → alert + fence → conditional takeover
- Verify: state.db corruption → falls back to snapshot, not corrupt file
- Verify: Core reboot with standby holding lock → Core stays dormant
- Update dr-issue-log.md