Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,59 @@
+# Standby Audit — July 8, 2026
+
+## Server: app1-bu.itpropartner.com (5.161.114.8)
+
+Hetzner CPX11 (2 vCPU / 2 GB RAM / 40 GB SSD) — warm standby for app1 (152.53.192.33, netcup).
+
+### Power State
+- Hetzner API: `running` (server is powered on, OS up)
+- Per DR plan: documented as "🟢 Dormant" but the intent in `hermes-standby-deployment` skill says "Stay powered on with Hermes off" — so running is **correct**, but server-inventory.md calls it "dormant" which is ambiguous.
+- **Note:** If electricity cost ($5/mo) is a concern, this could be a cold standby (boot on demand) with a ~60s slower failover.
+
+### Hermes Gateway
+- `systemctl is-active hermes-gateway` → `inactive` ✅
+
+### Standby Service
+- `systemctl is-active hermes-standby` → `active (exited)` ✅
+- Last boot: Jul 5, 2026
+- Uptime at audit: 3 days, 1:16
+- Restore log: `Live box is REACHABLE — staying dormant (clean exit)` ✅
+
+### Watchdog Cron
+- `crontab -l` → `*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh` ✅
+- Watchdog log: empty (no failures detected) ✅
+- Disk usage: 16G / 38G (43%)
+
+### SSH Access
+- Key: `itpp-infra` (Ed25519) ✅
+- Recent auth log shows SSH from 152.53.192.33 (live box) — heartbeat or previous manual check
+
+### S3 Artifacts
+- `s3://hermes-vps-backups/standby/` — all 6 files present ✅
+- `DR-PLAN.md` — 13 KB, present ✅
+- `hermes-standby-restore.sh` — 2.8 KB, matches deployed version ✅
+- `hermes-standby-watchdog.sh` — 4.3 KB, matches deployed version ✅
+- `hermes-standby.service` — 367 B, matches deployed version ✅
+- `server-inventory.md` — 8.1 KB, present ✅
+- `recovery-bundle-2026-07-05.md` — 280 KB, present ✅
+
+### Live Sync Freshness
+- `hermes-live-sync` cron runs every 15 min on live box ✅
+- `state.db` on S3: 2026-07-08T23:19:08Z (~5 min from audit time) ✅
+- `gateway_state.json`: 2026-07-08T23:19:06Z ✅
+- `channel_directory.json`: 2026-07-08T23:18:59Z ✅
+- Total S3 `live/` data: 32,151 files / 4.8 GiB
+
+### Credential Observations
+- `HETZNER_API_TOKEN` exists as a shell env var
+- `/root/.hermes/scripts/.hetzner_token` file is **missing** (deleted during prior cleanup?)
+- Recovery bundle references the file approach — potential doc/setup drift
+
+### Issues Found
+1. Minor: server-inventory.md says "dormant" (implies off) but server is running — warm standby is correct per skill, but docs ambiguous
+2. Minor: `.hetzner_token` file deleted; token only exists in env var — recovery bundle instructs restoring the file
+3. No critical issues
+
+### Next Audit Recommendation
+- Quarterly or after any DR-plan changes
+- Also verify the live-sync script hasn't drifted (exclusion patterns, credentials)
+- Check Hetzner snapshot cron (`hetzner-weekly-snapshots`) still runs