67 lines
3.0 KiB
Markdown
67 lines
3.0 KiB
Markdown
# DR Audit Findings — Jul 11, 2026
|
|
|
|
## Backup Pipeline Failure
|
|
|
|
**Trigger:** Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.
|
|
|
|
**Root cause:** `hermes-backup.sh` was missing `source /opt/awscli-venv/bin/activate`. The `aws` command is only available inside the virtualenv at `/opt/awscli-venv/`. In cron's minimal PATH (`/usr/bin:/bin`), `aws` is not found. The `root-essentials-backup.sh` script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.
|
|
|
|
**Evidence from journalctl:**
|
|
```
|
|
Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found
|
|
```
|
|
|
|
**Fix applied (Jul 11, 07:30 UTC):**
|
|
```bash
|
|
# Added after "set -euo pipefail" in hermes-backup.sh:
|
|
if [ -f /opt/awscli-venv/bin/activate ]; then
|
|
source /opt/awscli-venv/bin/activate
|
|
fi
|
|
```
|
|
|
|
**Verification:** Manual backup ran successfully after fix — 541 MB tarball uploaded to `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz`.
|
|
|
|
## Standby Server Disk Crisis
|
|
|
|
**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.
|
|
|
|
**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.
|
|
|
|
**Required cleanup (not yet performed):**
|
|
```bash
|
|
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
|
|
apt-get clean && apt-get autoremove --purge -y
|
|
journalctl --vacuum-size=200M
|
|
> /var/log/hermes-standby-sync.log
|
|
docker system prune -a -f 2>/dev/null || true
|
|
df -h /
|
|
"
|
|
```
|
|
Target: at least 5 GB free.
|
|
|
|
## Backup Integrity Verification Workflow
|
|
|
|
The Jul 11 audit established a reliable pattern for verifying backup integrity:
|
|
|
|
1. **List S3** to find the latest tarball
|
|
2. **Download** it to `/tmp/` with `aws s3 cp`
|
|
3. **Test** with `tar tzf` (not just `aws s3 ls`)
|
|
4. **Report** file count and any corruption
|
|
5. **Clean up** the local copy
|
|
|
|
This catches silent failures that S3 listing alone misses. The audit watchdog (`backup-audit-check.sh`) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.
|
|
|
|
## Restore Plan Template
|
|
|
|
When an incident requires a restore plan, structure it with these sections:
|
|
|
|
1. **Pre-Restore Assessment** — backup status table, standby status, system health, dependency inventory
|
|
2. **Recovery Path A** — In-place fix (preferred when system is running)
|
|
3. **Recovery Path B** — Failover to standby (when Core is unreachable)
|
|
4. **Recovery Path C** — Full rebuild from backup tarball (when OS is corrupted)
|
|
5. **Post-Restore Verification Checklist** — Hermes, web services, model access, email, backups, standby
|
|
6. **Critical Fixes** — any permanent fixes needed to prevent recurrence
|
|
7. **Team Contact & Escalation** — who to call and when
|
|
8. **Rollback Plan** — how to undo the restore if it makes things worse
|
|
|
|
See `/root/.hermes/references/restore-plan-2026-07-11.md` for the complete worked example. |