# DR Audit Process

Reusable procedure for running a full disaster recovery audit. Follow these steps in order.

## Trigger Conditions

Run a full DR audit when:
- First deployment or migration to a new server
- Quarterly (every 3 months) for established infrastructure
- After any backup pipeline change
- When investigating reported data loss or stale backups
- Before decommissioning any server

## Step 1: S3 Bucket Audit

Check all backup buckets for recent activity. Run in parallel via subagents.

For each bucket, verify:
- Bucket accessible (credentials work)
- Versioning enabled
- 10 most recent objects — timestamps, sizes
- File counts and total size
- Sub-path coverage (live/, standby/, full-backup/)
- Staleness detection — any path with >48h gap

## Step 2: Standby Server Audit

If a warm/cold standby exists, check:
- Server power state (on vs off — document if this contradicts the DR plan)
- SSH accessibility and key validity
- Recovery scripts present and valid (standby restore, watchdog)
- Failover cron job active
- Live sync data age (S3 should have today's objects)
- Disk space (enough for S3 sync?)

## Step 3: Config & Credentials Audit

Check each file: exists, size >0, permissions correct, included in backup scope.

Config files:
- /root/.hermes/config.yaml
- /etc/caddy/Caddyfile
- /root/.hermes/.env
- Systemd service files (/etc/systemd/system/*.service)

Password files (should be chmod 600):
- ~/.config/himalaya/*.pass
- ~/.ssh/* (private keys = 600, public keys = 644)
- ~/.aws/credentials
- Any *cred*, *pass*, *token*, *secret* files outside profiles/

## Step 4: Cron Job Audit

- Retrieve full cron list via `cronjob(action='list')`
- Flag any job with last_status = 'error'
- Check that all expected backup schedules are present
- Verify no jobs are silently failing (error status with no user notification)

## Step 5: Document Findings

Create entries in `/root/.hermes/references/dr-issue-log.md`. Each entry follows this format:

```
### DR-NNN: Short Issue Name
**Problem:** What was actually happening (or not happening)
**Root cause:** What caused it
**Fix:** What was done to resolve it
**Status:** ✅ Fixed & verified YYYY-MM-DD (or 🔄 Pending)
**Verified by:** How you confirmed the fix worked
```

## Step 6: Fix and Verify

- Apply fixes in order of severity (critical → medium → low)
- After each fix, run the verification step
- Update the DR issue log with verification results
- If a fix can't be completed immediately, note what's blocking it

## Step 7: Report

Deliver a consolidated report to the user with three sections:
- 🟢 Healthy — everything working
- 🟡 Attention — minor issues documented
- 🔴 Critical — issues that need immediate action

Include the resolved/remaining counts and any recurring pattern worth noting.