34 lines
1.5 KiB
Markdown
34 lines
1.5 KiB
Markdown
# Hermes DR Plan v2 — Pending Implementation Checklist
|
|
|
|
**Submitted by Germaine:** July 9, 2026
|
|
**Status:** Planned (not yet implemented)
|
|
**Supersedes:** v1 ping-only watchdog
|
|
|
|
## Key Architecture Changes
|
|
|
|
- Two-layer health check (ping + HTTP health endpoint)
|
|
- SSH fencing + S3 ownership lock (prevents split-brain)
|
|
- Pre-failover integrity check (SQLite PRAGMA)
|
|
- 10-min VACUUM snapshots with integrity gate
|
|
- Backward heartbeat (Core monitors standby)
|
|
- app1-bu upgrade CPX11 → CPX21 for local Ollama (+$6/mo)
|
|
|
|
## 16-Item Checklist
|
|
|
|
- [ ] Upgrade app1-bu from CPX11 to CPX21 (Hetzner API)
|
|
- [ ] Install Ollama + llama3.2:3b on app1-bu
|
|
- [ ] Deploy /healthz endpoint on Core (Tailscale-only, bearer token)
|
|
- [ ] Deploy two-layer watchdog on app1-bu (Layer 1 + Layer 2)
|
|
- [ ] Deploy SSH fencing script on app1-bu
|
|
- [ ] Deploy S3 ownership lock with refresh
|
|
- [ ] Deploy 10-min VACUUM snapshot cron on Core
|
|
- [ ] Deploy pre-failover integrity check on app1-bu
|
|
- [ ] Deploy backward heartbeat (Core → app1-bu over Tailscale SSH)
|
|
- [ ] Update hermes-standby-deployment skill with v2 architecture
|
|
- [ ] Whitelist cross-box IPs in fail2ban both directions
|
|
- [ ] Verify: 4 consecutive pings → auto-takeover (server dead)
|
|
- [ ] Verify: Hermes dead but server up → alert + fence → conditional takeover
|
|
- [ ] Verify: state.db corruption → falls back to snapshot, not corrupt file
|
|
- [ ] Verify: Core reboot with standby holding lock → Core stays dormant
|
|
- [ ] Update dr-issue-log.md
|