hermes-skills/skills/devops/hermes-standby-deployment/references/standby-audit-2026-07-08.md

Standby Audit — July 8, 2026

Server: app1-bu.itpropartner.com (5.161.114.8)

Hetzner CPX11 (2 vCPU / 2 GB RAM / 40 GB SSD) — warm standby for app1 (152.53.192.33, netcup).

Power State

• Hetzner API: running (server is powered on, OS up)
• Per DR plan: documented as "🟢 Dormant" but the intent in hermes-standby-deployment skill says "Stay powered on with Hermes off" — so running is correct, but server-inventory.md calls it "dormant" which is ambiguous.
• Note: If electricity cost ($5/mo) is a concern, this could be a cold standby (boot on demand) with a ~60s slower failover.

Hermes Gateway

• systemctl is-active hermes-gateway → inactive ✅

Standby Service

• systemctl is-active hermes-standby → active (exited) ✅
• Last boot: Jul 5, 2026
• Uptime at audit: 3 days, 1:16
• Restore log: Live box is REACHABLE — staying dormant (clean exit) ✅

Watchdog Cron

• crontab -l → */10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh ✅
• Watchdog log: empty (no failures detected) ✅
• Disk usage: 16G / 38G (43%)

SSH Access

• Key: itpp-infra (Ed25519) ✅
• Recent auth log shows SSH from 152.53.192.33 (live box) — heartbeat or previous manual check

S3 Artifacts

• s3://hermes-vps-backups/standby/ — all 6 files present ✅
• DR-PLAN.md — 13 KB, present ✅
• hermes-standby-restore.sh — 2.8 KB, matches deployed version ✅
• hermes-standby-watchdog.sh — 4.3 KB, matches deployed version ✅
• hermes-standby.service — 367 B, matches deployed version ✅
• server-inventory.md — 8.1 KB, present ✅
• recovery-bundle-2026-07-05.md — 280 KB, present ✅

Live Sync Freshness

• hermes-live-sync cron runs every 15 min on live box ✅
• state.db on S3: 2026-07-08T23:19:08Z (~5 min from audit time) ✅
• gateway_state.json: 2026-07-08T23:19:06Z ✅
• channel_directory.json: 2026-07-08T23:18:59Z ✅
• Total S3 live/ data: 32,151 files / 4.8 GiB

Credential Observations

• HETZNER_API_TOKEN exists as a shell env var
• /root/.hermes/scripts/.hetzner_token file is missing (deleted during prior cleanup?)
• Recovery bundle references the file approach — potential doc/setup drift

Issues Found

1. Minor: server-inventory.md says "dormant" (implies off) but server is running — warm standby is correct per skill, but docs ambiguous
2. Minor: .hetzner_token file deleted; token only exists in env var — recovery bundle instructs restoring the file
3. No critical issues

Next Audit Recommendation

• Quarterly or after any DR-plan changes
• Also verify the live-sync script hasn't drifted (exclusion patterns, credentials)
• Check Hetzner snapshot cron (hetzner-weekly-snapshots) still runs