# DR Audit Findings — Jul 11, 2026

## Backup Pipeline Failure

**Trigger:** Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.

**Root cause:** `hermes-backup.sh` was missing `source /opt/awscli-venv/bin/activate`. The `aws` command is only available inside the virtualenv at `/opt/awscli-venv/`. In cron's minimal PATH (`/usr/bin:/bin`), `aws` is not found. The `root-essentials-backup.sh` script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.

**Evidence from journalctl:**
```
Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found
```

**Fix applied (Jul 11, 07:30 UTC):**
```bash
# Added after "set -euo pipefail" in hermes-backup.sh:
if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi
```

**Verification:** Manual backup ran successfully after fix — 541 MB tarball uploaded to `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz`.

## Standby Server Disk Crisis

**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.

**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.

**Required cleanup (not yet performed):**
```bash
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
  apt-get clean && apt-get autoremove --purge -y
  journalctl --vacuum-size=200M
  > /var/log/hermes-standby-sync.log
  docker system prune -a -f 2>/dev/null || true
  df -h /
"
```
Target: at least 5 GB free.

## Backup Integrity Verification Workflow

The Jul 11 audit established a reliable pattern for verifying backup integrity:

1. **List S3** to find the latest tarball
2. **Download** it to `/tmp/` with `aws s3 cp`
3. **Test** with `tar tzf` (not just `aws s3 ls`)
4. **Report** file count and any corruption
5. **Clean up** the local copy

This catches silent failures that S3 listing alone misses. The audit watchdog (`backup-audit-check.sh`) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.

## Restore Plan Template

When an incident requires a restore plan, structure it with these sections:

1. **Pre-Restore Assessment** — backup status table, standby status, system health, dependency inventory
2. **Recovery Path A** — In-place fix (preferred when system is running)
3. **Recovery Path B** — Failover to standby (when Core is unreachable)
4. **Recovery Path C** — Full rebuild from backup tarball (when OS is corrupted)
5. **Post-Restore Verification Checklist** — Hermes, web services, model access, email, backups, standby
6. **Critical Fixes** — any permanent fixes needed to prevent recurrence
7. **Team Contact & Escalation** — who to call and when
8. **Rollback Plan** — how to undo the restore if it makes things worse

See `/root/.hermes/references/restore-plan-2026-07-11.md` for the complete worked example.