hermes-skills/skills/devops/disaster-recovery-audit/references/dr-audit-findings-2026-07-11.md

DR Audit Findings — Jul 11, 2026

Backup Pipeline Failure

Trigger: Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.

Root cause: hermes-backup.sh was missing source /opt/awscli-venv/bin/activate. The aws command is only available inside the virtualenv at /opt/awscli-venv/. In cron's minimal PATH (/usr/bin:/bin), aws is not found. The root-essentials-backup.sh script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.

Evidence from journalctl:

Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found

Fix applied (Jul 11, 07:30 UTC):

# Added after "set -euo pipefail" in hermes-backup.sh:
if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi

Verification: Manual backup ran successfully after fix — 541 MB tarball uploaded to s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz.

Standby Server Disk Crisis

Finding: app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.

Impact: If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.

Required cleanup (not yet performed):

ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
  apt-get clean && apt-get autoremove --purge -y
  journalctl --vacuum-size=200M
  > /var/log/hermes-standby-sync.log
  docker system prune -a -f 2>/dev/null || true
  df -h /
"

Target: at least 5 GB free.

Backup Integrity Verification Workflow

The Jul 11 audit established a reliable pattern for verifying backup integrity:

1. List S3 to find the latest tarball
2. Download it to /tmp/ with aws s3 cp
3. Test with tar tzf (not just aws s3 ls)
4. Report file count and any corruption
5. Clean up the local copy

This catches silent failures that S3 listing alone misses. The audit watchdog (backup-audit-check.sh) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.

Restore Plan Template

When an incident requires a restore plan, structure it with these sections:

1. Pre-Restore Assessment — backup status table, standby status, system health, dependency inventory
2. Recovery Path A — In-place fix (preferred when system is running)
3. Recovery Path B — Failover to standby (when Core is unreachable)
4. Recovery Path C — Full rebuild from backup tarball (when OS is corrupted)
5. Post-Restore Verification Checklist — Hermes, web services, model access, email, backups, standby
6. Critical Fixes — any permanent fixes needed to prevent recurrence
7. Team Contact & Escalation — who to call and when
8. Rollback Plan — how to undo the restore if it makes things worse

See /root/.hermes/references/restore-plan-2026-07-11.md for the complete worked example.