hermes-recovery/references/restore-plan-2026-07-11.md

# Restore Plan: Core Server Recovery to 6pm EST (Jul 10, 2026) Working Configuration

> **Generated:** 2026-07-11 07:35 UTC (3:35 AM EST)
> **Prepared by:** Hermes Agent (Sho'Nuff)
> **Status:** READY — validated against live S3 backups and standby server

---

## Executive Summary

The Core server (152.53.192.33) experienced a configuration/model issue around 6pm EST on July 10, 2026. The system is currently running but the daily full backup and root-essentials backup both FAILED overnight (July 11). The July 10 backups are intact and verified. The warm standby (app1-bu, 5.161.114.8) has critical disk space issues (97% used).

**Recovery strategy:** Two paths — (A) In-place fix on Core using verified backups, or (B) Failover to standby after clearing disk space.

---

## Pre-Restore Assessment

### Backup Status

| Backup | Date | Size | Integrity | S3 Path |
|--------|------|------|-----------|---------|
| Full backup | Jul 10, 1:13 AM UTC | 503 MB (2,005 files) | ✅ VALID | `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz` |
| Root-essentials | Jul 10, 10:38 AM UTC | 42 MB | ✅ VALID | `s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz` |
| Live sync (state.db) | Jul 11, 07:21 UTC | ~1.9 GB | ✅ Current | `s3://hermes-vps-backups/live/state.db` |
| **Full backup Jul 11** | **FAILED** | N/A | ❌ `aws: command not found` | Not uploaded |
| **Root-essentials Jul 11** | **FAILED** | N/A | ❌ Silent failure | Not uploaded |

### Root Cause of Backup Failures

`hermes-backup.sh` (line 215) calls `aws s3 cp` without activating the AWS CLI venv:
```bash
# MISSING from hermes-backup.sh:
source /opt/awscli-venv/bin/activate
```
The `root-essentials-backup.sh` has this line but the backup still failed silently (likely the same `aws` PATH issue or system-level tar path problems). **Fix required before the restore plan can rely on future backups.**

### Warm Standby (app1-bu) Status

| Check | Expected | Actual | Status |
|-------|----------|--------|--------|
| Online | Yes | Up 5d 13h | ✅ |
| Hermes dormant | Inactive | Inactive | ✅ |
| Watchdog cron | `*/5 * * * *` | `*/5 * * * *` | ✅ |
| Sync cron | `*/10 * * * *` | `*/10 * * * *` | ✅ |
| Last sync | Recent | Jul 11 11:30 UTC | ✅ |
| State.db age | Fresh | Jul 10 19:36 UTC | ⚠️ Stale (pre-6pm) |
| Config age | Fresh | Jul 10 21:28 UTC | ⚠️ Stale |
| **Disk space** | >20% free | **3% free (1.2G/38G)** | 🔴 CRITICAL |

### Dependencies Verified

| File | Path | Size | Perms | In Backup? |
|------|------|------|-------|------------|
| config.yaml | /root/.hermes/config.yaml | 3,350 B | 600 | ✅ |
| .env | /root/.hermes/.env | 849 B | 600 | ✅ |
| AWS creds | /root/.aws/credentials | 116 B | 600 | ✅ |
| Caddyfile | /etc/caddy/Caddyfile | 3,094 B | 640 | ✅ |
| SSH key | /root/.ssh/itpp-infra | 399 B | 600 | ✅ |
| Hermes service | /etc/systemd/system/hermes.service | — | — | ✅ |
| Shark-game service | /etc/systemd/system/shark-game.service | — | — | ✅ |
| Ollama service | /etc/systemd/system/ollama.service | — | — | ✅ |
| MySQL tunnel | /etc/systemd/system/mysql-tunnel.service | — | — | ✅ |

### System Health (Current)

| Metric | Value | Status |
|--------|-------|--------|
| Disk | 8% used (448G free) | ✅ |
| Memory | 11 GB available | ✅ |
| Load | 0.16 | ✅ |
| Hermes gateway | Running (PID 1084) | ✅ |
| Anita gateway | Running (PID 935) | ✅ |
| All services | Active/running | ✅ |

---

## Recovery Path A: In-Place Restore from Backup (RECOMMENDED)

Use when Core is running but misconfigured. Restore specific config files from July 10 backups while keeping current state.db and live data.

### Step A1: Restore config.yaml to pre-6pm state

```bash
# Download the root-essentials backup (already verified)
cd /tmp
aws s3 cp s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz \
  root-essentials-restore.tar.gz \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Extract just the config file
mkdir -p /tmp/restore-root
tar xzf root-essentials-restore.tar.gz -C /tmp/restore-root

# Back up current config before overwriting
cp /root/.hermes/config.yaml /root/.hermes/config.yaml.pre-restore-$(date +%F-%H%M)

# Restore config.yaml (the pre-6pm version)
cp /tmp/restore-root/.hermes/config.yaml /root/.hermes/config.yaml
chmod 600 /root/.hermes/config.yaml

# Also restore .env if needed
cp /tmp/restore-root/.hermes/.env /root/.hermes/.env
chmod 600 /root/.hermes/.env

# Restore SSH keys if needed
cp /tmp/restore-root/.ssh/* /root/.ssh/
chmod 600 /root/.ssh/*

# Restore AWS credentials if needed
cp /tmp/restore-root/.aws/credentials /root/.aws/credentials
chmod 600 /root/.aws/credentials
```

### Step A2: Restart Hermes gateway to pick up config changes

```bash
hermes gateway restart
```

Wait 5-10 seconds, then verify:
```bash
hermes gateway status
# Expected: Gateway is running (PID: <new>)
```

### Step A3: Verify services are operational

```bash
# Check Hermes responds
hermes terminal --command "echo online"

# Check Telegram connectivity
journalctl -u hermes -n 10 --no-pager | grep -i telegram

# Check all systemd services
systemctl is-active hermes hermes-assistant hermes-browser caddy ollama shark-game mysql-tunnel ops-portal
```

### Step A4: Verify model config works

```bash
# Test model access via admin-ai
KEY=$(grep -A2 'admin-ai:' /root/.hermes/config.yaml | grep api_key | awk '{print $2}')
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $KEY" | python3 -c "import sys,json;print(len(json.load(sys.stdin).get('data',[])),'models available')"
```

---

## Recovery Path B: Failover to Warm Standby (app1-bu)

Use if Core is unreachable or severely broken. Requires fixing standby disk space first.

### Step B0: PREREQUISITE — Clear disk space on app1-bu (CRITICAL)

The standby has only 1.2 GB free on a 38 GB disk. Failover will fail without space.

```bash
ssh -i /root/.ssh/itpp-infra root@5.161.114.8

# Clean apt cache
apt-get clean
apt-get autoremove --purge -y

# Clean journal logs
journalctl --vacuum-size=200M

# Remove old sync logs if present
> /var/log/hermes-standby-sync.log

# Check what's using space
du -sh /* 2>/dev/null | sort -rh | head -10

# Docker cleanup if docker is installed
docker system prune -a -f 2>/dev/null || true
docker image prune -a -f 2>/dev/null || true

# Verify space recovered
df -h /
# Target: at least 5 GB free
```

### Step B1: Sync latest state before failover

```bash
# From Core: force a live sync to S3
source /opt/awscli-venv/bin/activate
/root/.hermes/scripts/hermes-live-sync.sh

# From app1-bu: pull latest state
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 \
  "/root/.hermes/scripts/hermes-standby-sync.sh"
```

### Step B2: Trigger failover

```bash
# On app1-bu: manually activate Hermes
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
  systemctl start hermes 2>/dev/null || \
  hermes gateway start
"
```

### Step B3: Verify failover

```bash
# Check gateway status on standby
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "hermes gateway status"

# Communicate with standby via Telegram
# The watchdog should send Telegram + email alerts when it detects the failover
```

### Step B4: Fencing — prevent split-brain

```bash
# On Core: shut down Hermes gateway to prevent dual-instance conflict
hermes gateway stop

# Or power down Core entirely if it's severely broken
# (Only with user confirmation per reboot-with-health-check protocol)
```

---

## Recovery Path C: Full Rebuild from Full Backup

Use if Core OS is corrupted or server needs to be reprovisioned. Rebuilds everything from the July 10 full backup tarball.

### Step C1: Provision replacement server (if needed)

If on netcup: use netcup SCP to provision a new RS 2000.
If moving to Hetzner: provision CPX41 or similar (8C/16G minimum).

### Step C2: Install base packages

```bash
apt-get update && apt-get install -y \
  python3 python3-pip python3-venv git curl wget \
  caddy docker.io docker-compose awscli sqlite3 \
  wireguard strongswan xl2tpd ollama
```

### Step C3: Install Hermes

```bash
pip3 install hermes-agent
```

### Step C4: Download and extract full backup

```bash
cd /tmp
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz \
  full-restore.tar.gz \
  --endpoint-url https://s3.us-east-1.wasabisys.com

tar xzf full-restore.tar.gz -C /tmp/restore
```

### Step C5: Restore all components

```bash
# Hermes config + state
cp -r /tmp/restore/hermes-backup-2026-07-10/* /root/.hermes/ 2>/dev/null || true
cp /tmp/restore/hermes-backup-2026-07-10/config/config.yaml /root/.hermes/
cp /tmp/restore/hermes-backup-2026-07-10/config/.env /root/.hermes/

# SSH keys
cp /tmp/restore/hermes-backup-2026-07-10/ssh/* /root/.ssh/ 2>/dev/null || true

# AWS credentials
cp /tmp/restore/hermes-backup-2026-07-10/aws/* /root/.aws/ 2>/dev/null || true

# Mail passwords
cp -r /tmp/restore/hermes-backup-2026-07-10/himalaya/* /root/.config/himalaya/ 2>/dev/null || true

# Caddyfile
cp /tmp/restore/hermes-backup-2026-07-10/caddy/Caddyfile /etc/caddy/Caddyfile

# Systemd services
cp /tmp/restore/hermes-backup-2026-07-10/systemd/*.service /etc/systemd/system/

# Set permissions
chmod 600 /root/.hermes/config.yaml /root/.hermes/.env /root/.aws/credentials
chmod 600 /root/.ssh/*
chmod 640 /etc/caddy/Caddyfile
```

### Step C6: Start services

```bash
systemctl daemon-reload
systemctl enable --now hermes caddy ollama shark-game mysql-tunnel ops-portal
```

### Step C7: Run the embedded restore script

```bash
bash /tmp/restore/hermes-backup-2026-07-10/restore.sh
```

---

## Post-Restore Verification Checklist

After ANY restore path, complete these checks:

### 1. Hermes Core
- [ ] `hermes gateway status` — Gateway is running
- [ ] `hermes terminal --command "echo online"` — Agent responds
- [ ] Telegram bot responds to `/status`

### 2. Web Services
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://core.itpropartner.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://portal.debtrecoveryexperts.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://shark.iamgmb.com` → 200

### 3. Model Access
- [ ] `curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $KEY" | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if 'data' in d else 'FAIL')"` → OK

### 4. Email
- [ ] `himalaya envelope list -s 5` — IMAP access works

### 5. Backups
- [ ] `aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com | tail -3` — Recent backups exist
- [ ] `aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com` — Live sync is current

### 6. Standby
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "uptime"` — Standby reachable
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "df -h / | tail -1"` — Disk >10% free

---

## Critical Fix: Repair Backup Script for Future Reliability

The daily backup script MUST be fixed to prevent recurrence:

```bash
# Add AWS CLI venv activation to hermes-backup.sh
```

Add after line 5 (`set -euo pipefail`):
```bash
# Activate AWS CLI from venv (added 2026-07-11 — DR FIX)
if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi
```

Similarly verify `root-essentials-backup.sh` works end-to-end:
```bash
# Test run root-essentials backup manually
bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1
```

Then update the DR issue log with this finding.

---

## Team Contact & Escalation

| Role | Name | Responsibility |
|------|------|---------------|
| CEO | Germaine | Final decision authority |
| Lead Sysadmin | Sho'Nuff (Hermes) | First responder, triage, fix |
| Operations | Anita | Comms, client notification |

**Escalation:** If restore fails after 2 attempts → escalate to netcup support (Core) or Hetzner support (standby).

---

## Rollback Plan

If restore makes things worse:
1. Restore config from pre-restore backup: `cp /root/.hermes/config.yaml.pre-restore-* /root/.hermes/config.yaml`
2. `hermes gateway restart`
3. Verify system returns to current state (which was operational but misconfigured)

---

## Files Created/Modified During Assessment

| File | Action |
|------|--------|
| This restore plan | Created at `/root/.hermes/references/restore-plan-2026-07-11.md` |
| `/tmp/root-verify.tar.gz` | Downloaded root-essentials backup for integrity check (clean up after) |
| `/tmp/full-verify.tar.gz` | Downloaded full backup for integrity check (clean up after) |
| DR issue log | New entry needed for backup script AWS CLI PATH bug |