DR Audit Process

Reusable procedure for running a full disaster recovery audit. Follow these steps in order.

Trigger Conditions

Run a full DR audit when:

First deployment or migration to a new server
Quarterly (every 3 months) for established infrastructure
After any backup pipeline change
When investigating reported data loss or stale backups
Before decommissioning any server

Step 1: S3 Bucket Audit

Check all backup buckets for recent activity. Run in parallel via subagents.

For each bucket, verify:

Bucket accessible (credentials work)
Versioning enabled
10 most recent objects — timestamps, sizes
File counts and total size
Sub-path coverage (live/, standby/, full-backup/)
Staleness detection — any path with >48h gap

Step 2: Standby Server Audit

If a warm/cold standby exists, check:

Server power state (on vs off — document if this contradicts the DR plan)
SSH accessibility and key validity
Recovery scripts present and valid (standby restore, watchdog)
Failover cron job active
Live sync data age (S3 should have today's objects)
Disk space (enough for S3 sync?)

Step 3: Config & Credentials Audit

Check each file: exists, size >0, permissions correct, included in backup scope.

Config files:

/root/.hermes/config.yaml
/etc/caddy/Caddyfile
/root/.hermes/.env
Systemd service files (/etc/systemd/system/*.service)

Password files (should be chmod 600):

~/.config/himalaya/*.pass
~/.ssh/* (private keys = 600, public keys = 644)
~/.aws/credentials
Any cred, pass, token, secret files outside profiles/

Step 4: Cron Job Audit

Retrieve full cron list via cronjob(action='list')
Flag any job with last_status = 'error'
Check that all expected backup schedules are present
Verify no jobs are silently failing (error status with no user notification)

Step 5: Document Findings

Create entries in /root/.hermes/references/dr-issue-log.md. Each entry follows this format:

### DR-NNN: Short Issue Name
**Problem:** What was actually happening (or not happening)
**Root cause:** What caused it
**Fix:** What was done to resolve it
**Status:** ✅ Fixed & verified YYYY-MM-DD (or 🔄 Pending)
**Verified by:** How you confirmed the fix worked

Step 6: Fix and Verify

Apply fixes in order of severity (critical → medium → low)
After each fix, run the verification step
Update the DR issue log with verification results
If a fix can't be completed immediately, note what's blocking it

Step 7: Report

Deliver a consolidated report to the user with three sections:

🟢 Healthy — everything working
🟡 Attention — minor issues documented
🔴 Critical — issues that need immediate action

Include the resolved/remaining counts and any recurring pattern worth noting.

2.8 KiB Raw Permalink Blame History