Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,86 @@
+# DR Audit Process
+
+Reusable procedure for running a full disaster recovery audit. Follow these steps in order.
+
+## Trigger Conditions
+
+Run a full DR audit when:
+- First deployment or migration to a new server
+- Quarterly (every 3 months) for established infrastructure
+- After any backup pipeline change
+- When investigating reported data loss or stale backups
+- Before decommissioning any server
+
+## Step 1: S3 Bucket Audit
+
+Check all backup buckets for recent activity. Run in parallel via subagents.
+
+For each bucket, verify:
+- Bucket accessible (credentials work)
+- Versioning enabled
+- 10 most recent objects — timestamps, sizes
+- File counts and total size
+- Sub-path coverage (live/, standby/, full-backup/)
+- Staleness detection — any path with >48h gap
+
+## Step 2: Standby Server Audit
+
+If a warm/cold standby exists, check:
+- Server power state (on vs off — document if this contradicts the DR plan)
+- SSH accessibility and key validity
+- Recovery scripts present and valid (standby restore, watchdog)
+- Failover cron job active
+- Live sync data age (S3 should have today's objects)
+- Disk space (enough for S3 sync?)
+
+## Step 3: Config & Credentials Audit
+
+Check each file: exists, size >0, permissions correct, included in backup scope.
+
+Config files:
+- /root/.hermes/config.yaml
+- /etc/caddy/Caddyfile
+- /root/.hermes/.env
+- Systemd service files (/etc/systemd/system/*.service)
+
+Password files (should be chmod 600):
+- ~/.config/himalaya/*.pass
+- ~/.ssh/* (private keys = 600, public keys = 644)
+- ~/.aws/credentials
+- Any *cred*, *pass*, *token*, *secret* files outside profiles/
+
+## Step 4: Cron Job Audit
+
+- Retrieve full cron list via `cronjob(action='list')`
+- Flag any job with last_status = 'error'
+- Check that all expected backup schedules are present
+- Verify no jobs are silently failing (error status with no user notification)
+
+## Step 5: Document Findings
+
+Create entries in `/root/.hermes/references/dr-issue-log.md`. Each entry follows this format:
+
+```
+### DR-NNN: Short Issue Name
+**Problem:** What was actually happening (or not happening)
+**Root cause:** What caused it
+**Fix:** What was done to resolve it
+**Status:** ✅ Fixed & verified YYYY-MM-DD (or 🔄 Pending)
+**Verified by:** How you confirmed the fix worked
+```
+
+## Step 6: Fix and Verify
+
+- Apply fixes in order of severity (critical → medium → low)
+- After each fix, run the verification step
+- Update the DR issue log with verification results
+- If a fix can't be completed immediately, note what's blocking it
+
+## Step 7: Report
+
+Deliver a consolidated report to the user with three sections:
+- 🟢 Healthy — everything working
+- 🟡 Attention — minor issues documented
+- 🔴 Critical — issues that need immediate action
+
+Include the resolved/remaining counts and any recurring pattern worth noting.