hermes-recovery/references/restore-plan-2026-07-11.md

Restore Plan: Core Server Recovery to 6pm EST (Jul 10, 2026) Working Configuration

Generated: 2026-07-11 07:35 UTC (3:35 AM EST) Prepared by: Hermes Agent (Sho'Nuff) Status: READY — validated against live S3 backups and standby server

---

Executive Summary

The Core server (152.53.192.33) experienced a configuration/model issue around 6pm EST on July 10, 2026. The system is currently running but the daily full backup and root-essentials backup both FAILED overnight (July 11). The July 10 backups are intact and verified. The warm standby (app1-bu, 5.161.114.8) has critical disk space issues (97% used).

Recovery strategy: Two paths — (A) In-place fix on Core using verified backups, or (B) Failover to standby after clearing disk space.

---

Pre-Restore Assessment

Backup Status

Backup	Date	Size	Integrity	S3 Path
Full backup	Jul 10, 1:13 AM UTC	503 MB (2,005 files)	✅ VALID	s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz
Root-essentials	Jul 10, 10:38 AM UTC	42 MB	✅ VALID	s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz
Live sync (state.db)	Jul 11, 07:21 UTC	~1.9 GB	✅ Current	s3://hermes-vps-backups/live/state.db
Full backup Jul 11	FAILED	N/A	❌ aws: command not found	Not uploaded
Root-essentials Jul 11	FAILED	N/A	❌ Silent failure	Not uploaded

Root Cause of Backup Failures

hermes-backup.sh (line 215) calls aws s3 cp without activating the AWS CLI venv:

# MISSING from hermes-backup.sh:
source /opt/awscli-venv/bin/activate

The root-essentials-backup.sh has this line but the backup still failed silently (likely the same aws PATH issue or system-level tar path problems). Fix required before the restore plan can rely on future backups.

Warm Standby (app1-bu) Status

Check	Expected	Actual	Status
Online	Yes	Up 5d 13h	✅
Hermes dormant	Inactive	Inactive	✅
Watchdog cron	*/5 * * * *	*/5 * * * *	✅
Sync cron	*/10 * * * *	*/10 * * * *	✅
Last sync	Recent	Jul 11 11:30 UTC	✅
State.db age	Fresh	Jul 10 19:36 UTC	⚠️ Stale (pre-6pm)
Config age	Fresh	Jul 10 21:28 UTC	⚠️ Stale
Disk space	>20% free	3% free (1.2G/38G)	🔴 CRITICAL

Dependencies Verified

File	Path	Size	Perms	In Backup?
config.yaml	/root/.hermes/config.yaml	3,350 B	600	✅
.env	/root/.hermes/.env	849 B	600	✅
AWS creds	/root/.aws/credentials	116 B	600	✅
Caddyfile	/etc/caddy/Caddyfile	3,094 B	640	✅
SSH key	/root/.ssh/itpp-infra	399 B	600	✅
Hermes service	/etc/systemd/system/hermes.service	—	—	✅
Shark-game service	/etc/systemd/system/shark-game.service	—	—	✅
Ollama service	/etc/systemd/system/ollama.service	—	—	✅
MySQL tunnel	/etc/systemd/system/mysql-tunnel.service	—	—	✅

System Health (Current)

Metric	Value	Status
Disk	8% used (448G free)	✅
Memory	11 GB available	✅
Load	0.16	✅
Hermes gateway	Running (PID 1084)	✅
Anita gateway	Running (PID 935)	✅
All services	Active/running	✅

---

Recovery Path A: In-Place Restore from Backup (RECOMMENDED)

Use when Core is running but misconfigured. Restore specific config files from July 10 backups while keeping current state.db and live data.

Step A1: Restore config.yaml to pre-6pm state

# Download the root-essentials backup (already verified)
cd /tmp
aws s3 cp s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz \
  root-essentials-restore.tar.gz \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Extract just the config file
mkdir -p /tmp/restore-root
tar xzf root-essentials-restore.tar.gz -C /tmp/restore-root

# Back up current config before overwriting
cp /root/.hermes/config.yaml /root/.hermes/config.yaml.pre-restore-$(date +%F-%H%M)

# Restore config.yaml (the pre-6pm version)
cp /tmp/restore-root/.hermes/config.yaml /root/.hermes/config.yaml
chmod 600 /root/.hermes/config.yaml

# Also restore .env if needed
cp /tmp/restore-root/.hermes/.env /root/.hermes/.env
chmod 600 /root/.hermes/.env

# Restore SSH keys if needed
cp /tmp/restore-root/.ssh/* /root/.ssh/
chmod 600 /root/.ssh/*

# Restore AWS credentials if needed
cp /tmp/restore-root/.aws/credentials /root/.aws/credentials
chmod 600 /root/.aws/credentials

Step A2: Restart Hermes gateway to pick up config changes

hermes gateway restart

Wait 5-10 seconds, then verify:

hermes gateway status
# Expected: Gateway is running (PID: <new>)

Step A3: Verify services are operational

# Check Hermes responds
hermes terminal --command "echo online"

# Check Telegram connectivity
journalctl -u hermes -n 10 --no-pager | grep -i telegram

# Check all systemd services
systemctl is-active hermes hermes-assistant hermes-browser caddy ollama shark-game mysql-tunnel ops-portal

Step A4: Verify model config works

# Test model access via admin-ai
KEY=$(grep -A2 'admin-ai:' /root/.hermes/config.yaml | grep api_key | awk '{print $2}')
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $KEY" | python3 -c "import sys,json;print(len(json.load(sys.stdin).get('data',[])),'models available')"

---

Recovery Path B: Failover to Warm Standby (app1-bu)

Use if Core is unreachable or severely broken. Requires fixing standby disk space first.

Step B0: PREREQUISITE — Clear disk space on app1-bu (CRITICAL)

The standby has only 1.2 GB free on a 38 GB disk. Failover will fail without space.

ssh -i /root/.ssh/itpp-infra root@5.161.114.8

# Clean apt cache
apt-get clean
apt-get autoremove --purge -y

# Clean journal logs
journalctl --vacuum-size=200M

# Remove old sync logs if present
> /var/log/hermes-standby-sync.log

# Check what's using space
du -sh /* 2>/dev/null | sort -rh | head -10

# Docker cleanup if docker is installed
docker system prune -a -f 2>/dev/null || true
docker image prune -a -f 2>/dev/null || true

# Verify space recovered
df -h /
# Target: at least 5 GB free

Step B1: Sync latest state before failover

# From Core: force a live sync to S3
source /opt/awscli-venv/bin/activate
/root/.hermes/scripts/hermes-live-sync.sh

# From app1-bu: pull latest state
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 \
  "/root/.hermes/scripts/hermes-standby-sync.sh"

Step B2: Trigger failover

# On app1-bu: manually activate Hermes
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
  systemctl start hermes 2>/dev/null || \
  hermes gateway start
"

Step B3: Verify failover

# Check gateway status on standby
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "hermes gateway status"

# Communicate with standby via Telegram
# The watchdog should send Telegram + email alerts when it detects the failover

Step B4: Fencing — prevent split-brain

# On Core: shut down Hermes gateway to prevent dual-instance conflict
hermes gateway stop

# Or power down Core entirely if it's severely broken
# (Only with user confirmation per reboot-with-health-check protocol)

---

Recovery Path C: Full Rebuild from Full Backup

Use if Core OS is corrupted or server needs to be reprovisioned. Rebuilds everything from the July 10 full backup tarball.

Step C1: Provision replacement server (if needed)

If on netcup: use netcup SCP to provision a new RS 2000. If moving to Hetzner: provision CPX41 or similar (8C/16G minimum).

Step C2: Install base packages

apt-get update && apt-get install -y \
  python3 python3-pip python3-venv git curl wget \
  caddy docker.io docker-compose awscli sqlite3 \
  wireguard strongswan xl2tpd ollama

Step C3: Install Hermes

pip3 install hermes-agent

Step C4: Download and extract full backup

cd /tmp
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz \
  full-restore.tar.gz \
  --endpoint-url https://s3.us-east-1.wasabisys.com

tar xzf full-restore.tar.gz -C /tmp/restore

Step C5: Restore all components

# Hermes config + state
cp -r /tmp/restore/hermes-backup-2026-07-10/* /root/.hermes/ 2>/dev/null || true
cp /tmp/restore/hermes-backup-2026-07-10/config/config.yaml /root/.hermes/
cp /tmp/restore/hermes-backup-2026-07-10/config/.env /root/.hermes/

# SSH keys
cp /tmp/restore/hermes-backup-2026-07-10/ssh/* /root/.ssh/ 2>/dev/null || true

# AWS credentials
cp /tmp/restore/hermes-backup-2026-07-10/aws/* /root/.aws/ 2>/dev/null || true

# Mail passwords
cp -r /tmp/restore/hermes-backup-2026-07-10/himalaya/* /root/.config/himalaya/ 2>/dev/null || true

# Caddyfile
cp /tmp/restore/hermes-backup-2026-07-10/caddy/Caddyfile /etc/caddy/Caddyfile

# Systemd services
cp /tmp/restore/hermes-backup-2026-07-10/systemd/*.service /etc/systemd/system/

# Set permissions
chmod 600 /root/.hermes/config.yaml /root/.hermes/.env /root/.aws/credentials
chmod 600 /root/.ssh/*
chmod 640 /etc/caddy/Caddyfile

Step C6: Start services

systemctl daemon-reload
systemctl enable --now hermes caddy ollama shark-game mysql-tunnel ops-portal

Step C7: Run the embedded restore script

bash /tmp/restore/hermes-backup-2026-07-10/restore.sh

---

Post-Restore Verification Checklist

After ANY restore path, complete these checks:

1. Hermes Core

• hermes gateway status — Gateway is running
• hermes terminal --command "echo online" — Agent responds
• Telegram bot responds to /status

2. Web Services

• curl -s -o /dev/null -w "%{http_code}" https://core.itpropartner.com → 200
• curl -s -o /dev/null -w "%{http_code}" https://portal.debtrecoveryexperts.com → 200
• curl -s -o /dev/null -w "%{http_code}" https://shark.iamgmb.com → 200

3. Model Access

• curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $KEY" | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if 'data' in d else 'FAIL')" → OK

4. Email

• himalaya envelope list -s 5 — IMAP access works

5. Backups

• aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com | tail -3 — Recent backups exist
• aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com — Live sync is current

6. Standby

• ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "uptime" — Standby reachable
• ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "df -h / | tail -1" — Disk >10% free

---

Critical Fix: Repair Backup Script for Future Reliability

The daily backup script MUST be fixed to prevent recurrence:

# Add AWS CLI venv activation to hermes-backup.sh

Add after line 5 (set -euo pipefail):

# Activate AWS CLI from venv (added 2026-07-11 — DR FIX)
if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi

Similarly verify root-essentials-backup.sh works end-to-end:

# Test run root-essentials backup manually
bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1

Then update the DR issue log with this finding.

---

Team Contact & Escalation

Role	Name	Responsibility
CEO	Germaine	Final decision authority
Lead Sysadmin	Sho'Nuff (Hermes)	First responder, triage, fix
Operations	Anita	Comms, client notification

Escalation: If restore fails after 2 attempts → escalate to netcup support (Core) or Hetzner support (standby).

---

Rollback Plan

If restore makes things worse:

1. Restore config from pre-restore backup: cp /root/.hermes/config.yaml.pre-restore-* /root/.hermes/config.yaml
2. hermes gateway restart
3. Verify system returns to current state (which was operational but misconfigured)

---

Files Created/Modified During Assessment

File	Action
This restore plan	Created at /root/.hermes/references/restore-plan-2026-07-11.md
/tmp/root-verify.tar.gz	Downloaded root-essentials backup for integrity check (clean up after)
/tmp/full-verify.tar.gz	Downloaded full backup for integrity check (clean up after)
DR issue log	New entry needed for backup script AWS CLI PATH bug