Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,67 @@
+# DR Audit Findings — Jul 11, 2026
+
+## Backup Pipeline Failure
+
+**Trigger:** Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.
+
+**Root cause:** `hermes-backup.sh` was missing `source /opt/awscli-venv/bin/activate`. The `aws` command is only available inside the virtualenv at `/opt/awscli-venv/`. In cron's minimal PATH (`/usr/bin:/bin`), `aws` is not found. The `root-essentials-backup.sh` script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.
+
+**Evidence from journalctl:**
+```
+Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found
+```
+
+**Fix applied (Jul 11, 07:30 UTC):**
+```bash
+# Added after "set -euo pipefail" in hermes-backup.sh:
+if [ -f /opt/awscli-venv/bin/activate ]; then
+    source /opt/awscli-venv/bin/activate
+fi
+```
+
+**Verification:** Manual backup ran successfully after fix — 541 MB tarball uploaded to `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz`.
+
+## Standby Server Disk Crisis
+
+**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.
+
+**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.
+
+**Required cleanup (not yet performed):**
+```bash
+ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
+  apt-get clean && apt-get autoremove --purge -y
+  journalctl --vacuum-size=200M
+  > /var/log/hermes-standby-sync.log
+  docker system prune -a -f 2>/dev/null || true
+  df -h /
+"
+```
+Target: at least 5 GB free.
+
+## Backup Integrity Verification Workflow
+
+The Jul 11 audit established a reliable pattern for verifying backup integrity:
+
+1. **List S3** to find the latest tarball
+2. **Download** it to `/tmp/` with `aws s3 cp`
+3. **Test** with `tar tzf` (not just `aws s3 ls`)
+4. **Report** file count and any corruption
+5. **Clean up** the local copy
+
+This catches silent failures that S3 listing alone misses. The audit watchdog (`backup-audit-check.sh`) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.
+
+## Restore Plan Template
+
+When an incident requires a restore plan, structure it with these sections:
+
+1. **Pre-Restore Assessment** — backup status table, standby status, system health, dependency inventory
+2. **Recovery Path A** — In-place fix (preferred when system is running)
+3. **Recovery Path B** — Failover to standby (when Core is unreachable)
+4. **Recovery Path C** — Full rebuild from backup tarball (when OS is corrupted)
+5. **Post-Restore Verification Checklist** — Hermes, web services, model access, email, backups, standby
+6. **Critical Fixes** — any permanent fixes needed to prevent recurrence
+7. **Team Contact & Escalation** — who to call and when
+8. **Rollback Plan** — how to undo the restore if it makes things worse
+
+See `/root/.hermes/references/restore-plan-2026-07-11.md` for the complete worked example.