--- name: disaster-recovery-audit description: "DR audit framework: S3 backup verification, warm standby failover testing, config/password inventory, issue tracking, and remediation. Used for periodic full-system audits and post-outage recovery verification." version: 2.0.0 author: Sho'Nuff tags: [devops, disaster-recovery, backup, audit, failover, s3, infrastructure] --- # Disaster Recovery Audit Standard procedure for auditing the Hermes infrastructure DR posture. Covers S3 backups, warm standby, config/password inventory, cron health, and issue tracking. This umbrella absorbs the content of `infrastructure-audit` (deleted 2026-07-08, was a duplicate of this skill). ## Audit Checklist Run these checks in order during any full DR audit: ### 2. Cron Job Health (updated Jul 10, 2026) - Use `cronjob(action='list')` or system crontab to get all scheduled jobs - Check `last_status` on each — flag any that are `error` - Specific jobs to verify: **Hermes cron scheduler:** - `hermes-live-sync` — every 15 min, should be OK - `hermes-system-config-sync` — every 15 min, should be OK (Caddyfile, systemd, scripts, SSH keys, env) - `hermes-docker-sync` — every 15 min, should be OK (Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama) - `service-health-check` — every 5 min, should be OK - `home-router-watchdog` — every 5 min, should be OK - `bounce-check` — every 60 min, should be OK - `home-router-daily-backup` — 6 AM daily, must exist (RESOLVED Jul 10 — paramiko installed, live test passed) - `hetzner-weekly-snapshots` — Monday 5 AM, must exist **System crontab (NOT in Hermes cron — blocked by gateway lifecycle guard):** - `hermes-full-backup` — 1:00 AM UTC daily via `/root/.hermes/scripts/hermes-backup.sh` - `root-essentials-backup` — 3:00 AM UTC daily via `/root/.hermes/scripts/root-essentials-backup.sh` - `backup-audit` — 2:00 AM UTC daily via `/root/.hermes/scripts/backup-audit-check.sh` **CRON BACKUP PITFALL — AWS CLI venv:** Backup scripts that call `aws s3 cp` will fail silently in cron with `aws: command not found` unless they activate the AWS CLI virtualenv first. The shell PATH in a cron context is minimal — even if `aws` works in your interactive session, it may not in cron. **Every backup script that uploads to S3 MUST include this block after `set -euo pipefail`:** ```bash # Activate AWS CLI from venv (required for cron — PATH is minimal) if [ -f /opt/awscli-venv/bin/activate ]; then source /opt/awscli-venv/bin/activate fi ``` - `hermes-backup.sh` was missing this line until Jul 11, 2026 — caused both nightly full and root-essentials backups to fail silently - `root-essentials-backup.sh` HAS this line but may still fail if the venv doesn't exist or AWS CLI wasn't installed there - Verify by running `bash -x /path/to/backup.sh 2>&1 | grep -i "aws: command"` in the audit - After patching any backup script, run it manually from cron's stripped environment to confirm: `env -i HOME=$HOME PATH=/usr/bin:/bin bash /path/to/backup.sh` **GATEWAY LIFECYCLE GUARD (#30719):** The full backup script (`hermes-backup.sh`) is blocked by the gateway lifecycle guard when run as a Hermes cron job. The guard prevents agent-driven SIGTERM-respawn loops. Symptom: cron job creation returns error `Blocked: cron job contains a gateway lifecycle command (restart/stop/kill)`. There is NO such command in the script — the guard triggers on script presence alone via the embedded restore.sh heredoc. FIX: The full backup MUST run from system crontab instead. Use `(crontab -l; echo "0 1 * * * /path/to/script.sh 2>&1 | logger -t jobname") | crontab -` to add. The audit watchdog runs at +1h to verify backup completion. Also applies to any agent-created cron job that shell-sources or wraps a script the guard considers risky. **Subagent model config drift:** Setting `delegation.model` via `hermes config set` does NOT affect the running gateway process. The gateway caches config at startup. Changes sit in config.yaml but won't be picked up by subagents until `hermes gateway restart` runs from outside the gateway. Do not debug subagent failures by re-applying config changes; restart the gateway instead. Also: Hermes cron jobs created under one model config will SKIP their run if the model config drifted later — fix by pinning via `cronjob action=update job_id=... provider=... model=...`. ### 2. S3 Bucket Integrity Check all 5 Wasabi S3 buckets for recent activity: | Bucket | Purpose | Sync Script | Key Check | |--------|---------|------------|-----------| | `hermes-vps-backups` | Hermes config/sessions/profiles | `hermes-live-sync.sh` (every 15 min) | Has `live/`, `standby/`, `hermes-full-backup/` | | `itpropartner-system-configs` | System configs (Caddy, systemd, SSH, scripts, env) | `hermes-system-config-sync.sh` (every 15 min) | Bucket created Jul 10, versioning enabled, IAM policy updated | | `itpropartner-docker-volumes` | Docker volumes (Docuseal, VW, Twenty, SearXNG, Ollama) | `hermes-docker-sync.sh` (every 15 min) | Bucket created Jul 10, versioning enabled, IAM policy updated | | `mikrotik-ccr-backups` | Router configs | `wisp-backup.py` (daily 6 AM) | Has recent uploads | | `itpropartner-backups` | Legacy files (mostly migrated) | None (no writer) | Known stale -- last write Jul 7 | **For each bucket, verify:** - Versioning enabled (via S3 API) - Most recent upload timestamp -- flag if >48h stale - Sub-paths exist: `live/`, `standby/`, `hermes-full-backup/` - For system-configs/docker-volumes: verify the sync script exists, is chmod 755, and its cron job is active - If a sync script targets a bucket that doesn't exist yet (itpropartner-system-configs, itpropartner-docker-volumes), flag as blocked -- buckets must be created in Wasabi Console first - Full backup tarball — flag if >48h stale - Expected file sizes (not 0 bytes, WAL/SHM artifacts excepted) **BACKUP INTEGRITY VERIFICATION (mandatory for every audit):** S3 listing alone is NOT sufficient — a tarball can exist on S3 and still be corrupt. For the full backup and root-essentials backup, download and test: ```bash # Download the latest tarball aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-$(date +%F).tar.gz \ /tmp/dr-audit-verify.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com # Test tar integrity tar tzf /tmp/dr-audit-verify.tar.gz > /dev/null 2>&1 && \ echo "VALID: $(tar tzf /tmp/dr-audit-verify.tar.gz | wc -l) files" || \ echo "CORRUPT — archive fails tar tzf" # Clean up rm /tmp/dr-audit-verify.tar.gz ``` Do NOT skip this step. The Jul 11, 2026 audit discovered that BOTH the nightly full backup AND root-essentials backup had failed silently — the S3 listing showed only stale Jul 10 files. The root cause was `aws: command not found` in cron (see Cron Job Health section above). The backup audit watchdog only checks S3 listing freshness, not integrity. ### S3 Command Pitfalls (Write-Only IAM) When transitioning AWS/Wasabi S3 backup scripts to use strict write-only IAM policies (e.g., `s3:PutObject` only, no `s3:ListBucket`), you **cannot use `aws s3 sync`**. - `aws s3 sync` fundamentally requires list permissions to compare the source against the destination. - **Fix:** Rewrite sync scripts to create local archives (e.g., `tar`) and perform blind uploads using `aws s3 cp`. - Scripts that read from S3 (e.g., `aws s3 ls` for audit checks or downloading for standby restores) must be separated into a different IAM user/role with read permissions, or moved to a pre-signed URL workflow. ### 3. Password & Config File Inventory | File | Expected | Permission | |------|----------|------------| | `/root/.hermes/config.yaml` | 600 | EXIST | | `/root/.hermes/.env` | 600 | EXIST | | `/etc/caddy/Caddyfile` | 640 | IN BACKUP | | `/root/.config/himalaya/g-germainebrown.pass` | 600 | EXIST | | `/root/.config/himalaya/shonuff.pass` | 600 | EXIST | | `/root/.config/himalaya/g-germainebrown-icloud-calendar.pass` | 600 | EXIST | | `/root/.aws/credentials` | 600 | EXIST | | `/root/.ssh/itpp-infra` | 600 | EXIST | | `/root/.hermes/migration-creds.txt` | 600 | IN BACKUP | | `/root/.hermes/references/dre-temp-passwords.txt` | 600 | EXIST | **Check each:** - Does the file exist? - Is the permission correct? (flags 644 → 600, 755 is fine for scripts) - Is it included in backup scripts? (check `hermes-backup.sh` and `hermes-live-sync.sh`) ### 4. Systemd Services List custom services and verify they're backed up to correct path: ```bash ls /etc/systemd/system/hermes*.service /etc/systemd/system/shark-game*.service /etc/systemd/system/*gateway*.service ``` **Service naming reality vs expected names:** - The main Hermes unit is `hermes.service` (NOT `hermes-agent.service`). The gateway runs embedded inside `hermes.service` as the main process. - `hermes-assistant.service` and `hermes-browser.service` exist as separate systemd units. - A user-level `hermes-gateway-anita.service` may exist at `~/.config/systemd/user/` for the Anita profile. - There is NO separate `hermes-gateway.service` systemd unit on the Core server -- the gateway runs inline. - If you run `systemctl status hermes-agent` or `hermes-gateway` and get "could not be found", check `hermes.service` instead. The process running is `hermes gateway`. - Running `hermes gateway status` returns the correct state showing gateway is running inside the main service. Backup script must collect from `/etc/systemd/system/` NOT `~/.config/systemd/user/`. ### 5. Warm Standby (app1-bu) SSH to the standby server via `itpp-infra` key and verify: | Check | Command | Expected | |-------|---------|----------| | Watchdog cron | `crontab -l \| grep standby` | `*/5 * * * *` | | Sync cron | `crontab -l \| grep sync` | `*/10 * * * *` | | Watchdog script | `cat` the script | 30s x 4 cycles = 2 min confirmation | | Sync script | `cat` the script | Exists, pulls from S3 every 10 min | | AWS CLI | `source /opt/awscli-venv/bin/activate && aws s3 ls ...` | Returns data | | Config freshness | `ls -la ~/.hermes/config.yaml` | Recent (today/tomorrow) | | Gateway status | `hermes gateway status` | Inactive/dead (dormant) | | Network to Core | `ping -c 2 152.53.192.33` | Successful (low ms) | | Disk space | `df -h /` | >20% free (CRITICAL if <10%: failover will fail) | ## Operational Truth and Restore Verification (Critical) A plan, command shown in chat, or agent statement is not an executed operation. During incidents and restores: - Never report a restart, reboot, DNS change, Caddy reload, provider test, failover, message delivery, disk cleanup, or restore as successful without tool output proving it. - Distinguish **requested**, **started**, **completed**, and **verified** states explicitly. - Verify a host reboot using boot ID or uptime before and after; an uninterrupted chat session is not evidence either way. - Verify DNS against the authoritative nameserver and at least two public resolvers. Compare with the local resolver because local caches can disagree with public DNS. - Verify the actual service name before operating it. On Core, the gateway is embedded in `hermes.service`; guessed units such as `hermes-gateway.service` are not evidence. - Verify failover on both sides: active Core state, standby gateway/process state, synchronized-state timestamp, disk headroom, and an end-to-end message. - A process listing proves only that a process exists. It does not prove Telegram delivery, provider authentication, or successful inference. - Subagent reports are leads. Read back files, query remote state, and verify external side effects before repeating their claims. - If verification cannot be performed, say `not verified`; do not fill the gap with expected output. ## Scope Discipline (Critical) When the user asks you to fix or audit a single server/app/bucket, DO ONLY THAT. Do not: - Touch the config of other servers you weren't asked about - Update credentials, API keys, or secrets in config files without explicit direction - Fix "related" problems you noticed along the way unless the user says "while you're at it" - Propagate changes to other profiles (Anita, Tony, etc.) unless specifically asked Every time I broke the session tonight, it was because I self-assigned extra scope. Fixing app1-bu? Fix app1-bu. Generating an API key? Generate it — don't wire it in. If you see a related problem, report it as a finding and ask if they want it fixed. Do not decide for them. This is not a suggestion — it was a repeated source of frustration this session. ## Issue Log Format The DR issue log lives at `/root/.hermes/references/dr-issue-log.md`. **Format rules (enforced Jul 8, 2026 after user called out unreadable pipe tables and garbled emoji on iPhone):** - **NO markdown pipe tables in the summary** -- use bullet lists grouped by status (`**Fixed** [OK]` then bullet items, `**Resolved** [INFO]`, `**Investigating** [PENDING]`) - **ASCII-only characters throughout** -- no emoji (`🔴` `🟡` `🟢` `✅` `🔄`), no em dashes (use ` -- `), no arrows (use `->`) - **Status labels:** `[OK]` for fixed, `[PENDING]` for investigating, `[HIGH]` `[MED]` `[INFO]` for severity - Each entry follows: Problem -> Root Cause -> Fix -> Verification - Apply ASCII-only rule to ALL reference files the user may read from a phone (Telegram file preview, email, open-in-browser) ```\n### DR-NNN: Short Title\n**Problem:** What happened or what was found\n**Root cause:** Why it happened\n**Fix:** What was done to resolve it\n**Status:** [OK] Fixed YYYY-MM-DD / [PENDING] Investigating\n**Verified by:** How we confirmed the fix works\n``` New entries are appended to the bottom. Existing entries are updated (status/verification) when a previously resolved issue is confirmed working on revisit. ## Failover Architecture **Design:** Warm standby. app1-bu runs 24/7 with Hermes inactive. Synchronizes state from S3 every 10 min. If Core goes down, watchdog confirms over 2 min (4 x 30s), then fails over. For a full infrastructure reference (IPs, ports, URLs, S3 buckets, API endpoints), see `references/network-and-service-endpoints.md`. For a **comprehensive infrastructure recovery manual** covering all 10 servers, every service, Docker containers, S3 backups, router networking, shared credentials, and step-by-step disaster scenarios, see `references/itpp-recovery-manual.md`. This was generated in July 2026 by synthesizing the server audit, app inventory, Docker compose files, Caddy config, systemd units, S3 bucket listings, and all deployment references into a single manual. For **gateway service lifecycle diagnostics** -- what to do when `hermes gateway restart` fails (linger, DBus, dual-process conflicts, stale systemd units, profile-specific gateways) -- see `references/gateway-lifecycle-troubleshooting.md`. For **backup pipeline failure patterns and integrity verification** — the Jul 11, 2026 audit that caught silent cron failures (`aws: command not found`) and the standby disk crisis — see `references/dr-audit-findings-2026-07-11.md`. ## When to update the recovery manual - A new service is deployed on Core -- add to Section 5 - A Hetzner server changes IP, hostname, or provider -- update Section 1 table - A Caddy domain or port mapping changes -- update Section 1 domain map and affected server section - A backup script or cron job is added/removed -- update Sections 8 and 13 - A server is decommissioned or provisioned -- update Section 1 and add/remove from Sections 2 or 7 - **User corrects the manual's scope** -- if the user says "this is too narrow" or "it's not only for X", broaden immediately. The manual must cover the ENTIRE infrastructure, not a single service. The Jul 9 session proved that scoping to one project (shark-game-recovery-manual.md) required near-immediate replacement with a full ITPP version. ``` ┌──────────────────────┐ ┌───────────────────────┐ │ Core (netcup) │ │ app1-bu (Hetzner) │ │ 152.53.192.33 │ │ 5.161.114.8 │ │ RS 2000 (8C/16G) │ │ CPX11 (2C/2G) │ │ Active Hermes │ │ Dormant Hermes │ │ Active gateway │ │ Active watchdog │ └────────┬─────────────┘ └──────────┬────────────┘ │ S3 sync (every 15 min) │ S3 sync (every 10 min) └──────────────┬───────────────┘ ▼ ┌──────────────────┐ │ Wasabi S3 │ │ hermes-vps- │ │ backups/live/ │ └──────────────────┘ ``` ### Failover Sequence 1. **Watchdog** (runs every 5 min via cron) pings Core (`152.53.192.33`) 2. If no ping response: check every **30s × 4 cycles** (~2 min total confirmation) 3. If still unreachable: sync latest state from S3 → start Hermes gateway → send **Telegram alert** + **email alert** 4. User communicates with standby instance ### Fallback Sequence (recovery) 1. Core comes back online 2. Shut down standby gateway (hermes gateway stop) or leave both running — user decides 3. If standby stays active, it keeps syncing. If powering off reduces confusion, do that. ### Recovery & Sync Timing | Component | Interval | Purpose | |-----------|----------|---------| | Live S3 sync (Core → S3) | Every 15 min | Pushes latest state from active server | | Standby sync (S3 → app1-bu) | Every 10 min | Pulls latest state to standby | | Watchdog health check | Every 5 min | Pings Core to detect failure | | Confirmation window | 4 × 30s = 2 min | Proves failure before failover | | Max total outage detection | ~7 min | Worst case: 5 min to next check + 2 min confirm | ## DR Issue Log Maintenance The DR issue log at `/root/.hermes/references/dr-issue-log.md` is the permanent record of all audit findings, fixes, root causes, and verification dates. ### Entry format **Format convention (CRITICAL — mobile-first rendering):** - NO markdown pipe/tables in the summary — unreadable on mobile - NO emoji or Unicode characters — plain ASCII labels only - Use `[HIGH]`, `[MED]`, `[INFO]`, `[OK]`, `[PENDING]` for severity/status - Use double hyphens (`--`) not em dashes - Summary is grouped bullet lists by status, see example below - Each entry has Problem -> Root Cause -> Fix -> Verification blocks - Separate entries with `---` Each entry MUST contain all 5 fields: ``` ### DR-NNN: Short Title **Problem:** What happened or what was found **Root cause:** Why it happened **Fix:** What was done to resolve it **Status:** [OK] Fixed YYYY-MM-DD / [PENDING] Investigating **Verified by:** How we confirmed the fix works ``` ### When to add entries - Any gap found during audit - Any untested failover path - Any configuration that was updated but not reflected in docs - Any server whose Hetzner type, tag, or purpose differs from DR plan - Any new server provisioned or decommissioned - Any cron job that shows a new error state - **Fault is common, not exceptional** — when multiple systems fail in the same audit, log each as a separate DR issue. Don't collapse the cluster into one. - **VPS resource threshold crossing** — when the automated VPS threshold checker (running every 15 min, checking 80/90/95% levels on RAM/disk/CPU across all servers) fires a new alert that requires action (e.g. ai.itpropartner.com disk at 92%). Log as a DR issue and investigate. ### When to update existing entries - A previously marked `[PENDING]` entry is now resolved - Verification step confirms the fix is still working ## Memory Consolidation Safety (Jul 9, 2026) The memory consolidation system (`hermes-consolidate.sh` + `hermes-consolidate.py`) runs every 10 min and prunes stale entries from MEMORY.md. It is a DR-critical component — if it over-prunes, identity and rules entries can be permanently lost from active memory. ### Architecture - **Backup-before-prune:** S3 upload is a hard gate — prune never runs without it - **Local retention:** 48h of pre-prune copies under `~/.hermes/memories/.consolidate-backups/` - **S3 history:** Timestamped copies at `s3://hermes-vps-backups/live/memories/history/` ### PROTECT entry types (these are NEVER pruned) The Python script guards entries containing: - Identity: `Sho'Nuff Brown`, `shonuff@`, `germainebrown.com` - Rules: `rule`, `never make up`, `fabrication`, `Germaine's #1 rule` - Infrastructure: `Core:`, `STANDING PRACTICE`, `app1-bu`, `netcup`, `admin-ai.itpropartner`, `Ops portal`, `Recovery manual`, `Ollama`, `RS 4000`, `Migration target` - Credentials: `credential`, `API`, `DR issue log`, `NETCUP_PASSWORD`, `LoveMyBoys`, `itpp-infra` - Email: `email`, `signature`, `format`, `shonuff.py` ### If memory gets over-pruned Restore from S3 backup — the JSON schema is `{backed_up_at, source, entry_count, char_count, entries}`. ### Known limitation The size safety valve drops oldest-appended entries when pattern-pruning doesn't reach 7,000 chars. Always verify after first consolidation run. If a critical entry was lost, restore from S3 and add it to the PROTECT regex. ## Per-Server DR Plans — v3 Consolidated (Jul 10, 2026) The v3 DR plan (`references/server-dr-plans-v3.md`) is the ACTIVE plan. It incorporates two expert DR reviews and replaces v1 (`references/server-dr-plans.md`). The v3 plan addresses every finding from the reviews. **DOCX versions** (formatted for sharing) are at: - `references/1-DR-Master-Plan-v2.docx` — Firm RTO/RPO, failover/failback design, split-brain prevention - `references/2-Backup-Security-Standard.docx` — Object Lock, write-only IAM, secondary destination, retention tiers - `references/3-Per-Server-Runbooks.docx` — Restore steps per host with bracketed fields - `references/4-DR-Testing-Schedule.docx` — Test cadence, log templates, sign-off table The v3 improvements: **v3 Improvements:** - **Failover timing corrected:** 30s checks (was 5 min), failover after 4 failures = 2 min detection, ~3-3.5 min actual RTO -> 5 min committed target. The old 5-min polling interval made a 2-min RTO mathematically impossible. - **Split-brain prevention:** S3 active-lock.json file, fencing, external quorum required before takeover - **Failback protocol:** 12-step sequence with rollback path — no auto-failback, no improvisation during incidents - **S3 single point of failure mitigated:** Secondary backup destination (separate provider/account) designed - **Backup immutability designed:** Object Lock + write-only IAM production credentials (not yet implemented) - **Per-server restore runbooks:** Exact commands, dependency maps, DNS records, validation checks, known failure modes - **RPO honesty by data type:** Core = 15 min (Hermes state) / 24h (Docker data), not a single number - **Testing schedule:** Daily automated checks -> monthly DB restore -> quarterly failover/failback -> biannual tabletop **Per-server RTO/RPO targets (v3):** | System | RTO | RPO | |---|---|---| | Core | 5 min | 15 min | | app1/2/3 | 60 min | 24h | | wphost02 | 2h | 24h | | fleettracker360 | 2h | 24h | | Legacy Hetzner | 4h | 24h | **Known gaps remaining (Jul 10, 2026):** - Docker volume backup automation not yet deployed - MariaDB dumps on fleettracker360 not automated - S3 secondary backup destination not provisioned - Write-only IAM credentials not yet created (policy designed, pending Wasabi Console) - 30-second watchdog deployed but needs load-test verification **Resolved this session (Jul 10):** - Object Lock enabled on all 5 buckets ✅ - S3 buckets itpropartner-system-configs + itpropartner-docker-volumes created, versioned, IAM updated ✅ - Daily full backup cron restored (system crontab at 1 AM UTC) ✅ - Daily backup audit watchdog deployed (2 AM UTC) ✅ - Home router backup paramiko error resolved (live test passed) ✅ - wphost02 full backup captured (656MB to S3) ✅ - AI server disk at 92% — documented as P0, deferred to next maintenance window **File locations:** - `references/server-dr-plans-v3.md` — ACTIVE (replaces all prior) - `references/server-dr-plans.md` — v1 (superseded) - `references/server-dr-plans-redacted.md` — for 3rd party review (IPs/credentials stripped) ## Cross-Reference with Ops Collector Data After completing sections 1-5, validate the ops portal's live data against the DR issue log: ```bash cat /var/www/ops/data/ops-status.json | python3 -m json.tool 2>/dev/null | grep -A5 '"s3_backups"' | head -20 cat /var/www/ops/data/ops-status.json | python3 -m json.tool 2>/dev/null | grep -B2 '"status": "error"' | head -20 ``` **Cross-reference checklist:** - [ ] DR issues marked [OK] Fixed -- does the ops collector confirm the fix is still working? - [ ] DR issues marked [PENDING] Investigating -- what does live data show now? Any new evidence? - [ ] Any new cron errors or stale buckets that aren't in the DR issue log yet? (open new entries) - [ ] S3 normalization buckets still missing? (itpropartner-system-configs, itpropartner-docker-volumes show NoSuchBucket) - [ ] Full backup fresh? (hermes-full-backups age_hours should be < 48) If the ops collector shows a status that contradicts the DR issue log (e.g., DR issue marked [OK] Fixed but the error persists in live data), re-open the issue rather than silently updating the log. If a new error appears that isn't logged, create a new DR entry.