ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

Browse Source
This commit is contained in:
root
2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
references/restore-plan-2026-07-11.md
+391
View File
@@ -0,0 +1,391 @@
# Restore Plan: Core Server Recovery to 6pm EST (Jul 10, 2026) Working Configuration
> **Generated:** 2026-07-11 07:35 UTC (3:35 AM EST)
> **Prepared by:** Hermes Agent (Sho'Nuff)
> **Status:** READY — validated against live S3 backups and standby server
---
## Executive Summary
The Core server (152.53.192.33) experienced a configuration/model issue around 6pm EST on July 10, 2026. The system is currently running but the daily full backup and root-essentials backup both FAILED overnight (July 11). The July 10 backups are intact and verified. The warm standby (app1-bu, 5.161.114.8) has critical disk space issues (97% used).
**Recovery strategy:** Two paths — (A) In-place fix on Core using verified backups, or (B) Failover to standby after clearing disk space.
---
## Pre-Restore Assessment
### Backup Status
| Backup | Date | Size | Integrity | S3 Path |
|--------|------|------|-----------|---------|
| Full backup | Jul 10, 1:13 AM UTC | 503 MB (2,005 files) | ✅ VALID | `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz` |
| Root-essentials | Jul 10, 10:38 AM UTC | 42 MB | ✅ VALID | `s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz` |
| Live sync (state.db) | Jul 11, 07:21 UTC | ~1.9 GB | ✅ Current | `s3://hermes-vps-backups/live/state.db` |
| **Full backup Jul 11** | **FAILED** | N/A | ❌ `aws: command not found` | Not uploaded |
| **Root-essentials Jul 11** | **FAILED** | N/A | ❌ Silent failure | Not uploaded |
### Root Cause of Backup Failures
`hermes-backup.sh` (line 215) calls `aws s3 cp` without activating the AWS CLI venv:
```bash
# MISSING from hermes-backup.sh:
source /opt/awscli-venv/bin/activate
```
The `root-essentials-backup.sh` has this line but the backup still failed silently (likely the same `aws` PATH issue or system-level tar path problems). **Fix required before the restore plan can rely on future backups.**
### Warm Standby (app1-bu) Status
| Check | Expected | Actual | Status |
|-------|----------|--------|--------|
| Online | Yes | Up 5d 13h | ✅ |
| Hermes dormant | Inactive | Inactive | ✅ |
| Watchdog cron | `*/5 * * * *` | `*/5 * * * *` | ✅ |
| Sync cron | `*/10 * * * *` | `*/10 * * * *` | ✅ |
| Last sync | Recent | Jul 11 11:30 UTC | ✅ |
| State.db age | Fresh | Jul 10 19:36 UTC | ⚠️ Stale (pre-6pm) |
| Config age | Fresh | Jul 10 21:28 UTC | ⚠️ Stale |
| **Disk space** | >20% free | **3% free (1.2G/38G)** | 🔴 CRITICAL |
### Dependencies Verified
| File | Path | Size | Perms | In Backup? |
|------|------|------|-------|------------|
| config.yaml | /root/.hermes/config.yaml | 3,350 B | 600 | ✅ |
| .env | /root/.hermes/.env | 849 B | 600 | ✅ |
| AWS creds | /root/.aws/credentials | 116 B | 600 | ✅ |
| Caddyfile | /etc/caddy/Caddyfile | 3,094 B | 640 | ✅ |
| SSH key | /root/.ssh/itpp-infra | 399 B | 600 | ✅ |
| Hermes service | /etc/systemd/system/hermes.service | — | — | ✅ |
| Shark-game service | /etc/systemd/system/shark-game.service | — | — | ✅ |
| Ollama service | /etc/systemd/system/ollama.service | — | — | ✅ |
| MySQL tunnel | /etc/systemd/system/mysql-tunnel.service | — | — | ✅ |
### System Health (Current)
| Metric | Value | Status |
|--------|-------|--------|
| Disk | 8% used (448G free) | ✅ |
| Memory | 11 GB available | ✅ |
| Load | 0.16 | ✅ |
| Hermes gateway | Running (PID 1084) | ✅ |
| Anita gateway | Running (PID 935) | ✅ |
| All services | Active/running | ✅ |
---
## Recovery Path A: In-Place Restore from Backup (RECOMMENDED)
Use when Core is running but misconfigured. Restore specific config files from July 10 backups while keeping current state.db and live data.
### Step A1: Restore config.yaml to pre-6pm state
```bash
# Download the root-essentials backup (already verified)
cd /tmp
aws s3 cp s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz \
root-essentials-restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
# Extract just the config file
mkdir -p /tmp/restore-root
tar xzf root-essentials-restore.tar.gz -C /tmp/restore-root
# Back up current config before overwriting
cp /root/.hermes/config.yaml /root/.hermes/config.yaml.pre-restore-$(date +%F-%H%M)
# Restore config.yaml (the pre-6pm version)
cp /tmp/restore-root/.hermes/config.yaml /root/.hermes/config.yaml
chmod 600 /root/.hermes/config.yaml
# Also restore .env if needed
cp /tmp/restore-root/.hermes/.env /root/.hermes/.env
chmod 600 /root/.hermes/.env
# Restore SSH keys if needed
cp /tmp/restore-root/.ssh/* /root/.ssh/
chmod 600 /root/.ssh/*
# Restore AWS credentials if needed
cp /tmp/restore-root/.aws/credentials /root/.aws/credentials
chmod 600 /root/.aws/credentials
```
### Step A2: Restart Hermes gateway to pick up config changes
```bash
hermes gateway restart
```
Wait 5-10 seconds, then verify:
```bash
hermes gateway status
# Expected: Gateway is running (PID: <new>)
```
### Step A3: Verify services are operational
```bash
# Check Hermes responds
hermes terminal --command "echo online"
# Check Telegram connectivity
journalctl -u hermes -n 10 --no-pager | grep -i telegram
# Check all systemd services
systemctl is-active hermes hermes-assistant hermes-browser caddy ollama shark-game mysql-tunnel ops-portal
```
### Step A4: Verify model config works
```bash
# Test model access via admin-ai
KEY=$(grep -A2 'admin-ai:' /root/.hermes/config.yaml | grep api_key | awk '{print $2}')
curl -s https://admin-ai.itpropartner.com/v1/models \
-H "Authorization: Bearer $KEY" | python3 -c "import sys,json;print(len(json.load(sys.stdin).get('data',[])),'models available')"
```
---
## Recovery Path B: Failover to Warm Standby (app1-bu)
Use if Core is unreachable or severely broken. Requires fixing standby disk space first.
### Step B0: PREREQUISITE — Clear disk space on app1-bu (CRITICAL)
The standby has only 1.2 GB free on a 38 GB disk. Failover will fail without space.
```bash
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
# Clean apt cache
apt-get clean
apt-get autoremove --purge -y
# Clean journal logs
journalctl --vacuum-size=200M
# Remove old sync logs if present
> /var/log/hermes-standby-sync.log
# Check what's using space
du -sh /* 2>/dev/null | sort -rh | head -10
# Docker cleanup if docker is installed
docker system prune -a -f 2>/dev/null || true
docker image prune -a -f 2>/dev/null || true
# Verify space recovered
df -h /
# Target: at least 5 GB free
```
### Step B1: Sync latest state before failover
```bash
# From Core: force a live sync to S3
source /opt/awscli-venv/bin/activate
/root/.hermes/scripts/hermes-live-sync.sh
# From app1-bu: pull latest state
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 \
"/root/.hermes/scripts/hermes-standby-sync.sh"
```
### Step B2: Trigger failover
```bash
# On app1-bu: manually activate Hermes
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
systemctl start hermes 2>/dev/null || \
hermes gateway start
"
```
### Step B3: Verify failover
```bash
# Check gateway status on standby
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "hermes gateway status"
# Communicate with standby via Telegram
# The watchdog should send Telegram + email alerts when it detects the failover
```
### Step B4: Fencing — prevent split-brain
```bash
# On Core: shut down Hermes gateway to prevent dual-instance conflict
hermes gateway stop
# Or power down Core entirely if it's severely broken
# (Only with user confirmation per reboot-with-health-check protocol)
```
---
## Recovery Path C: Full Rebuild from Full Backup
Use if Core OS is corrupted or server needs to be reprovisioned. Rebuilds everything from the July 10 full backup tarball.
### Step C1: Provision replacement server (if needed)
If on netcup: use netcup SCP to provision a new RS 2000.
If moving to Hetzner: provision CPX41 or similar (8C/16G minimum).
### Step C2: Install base packages
```bash
apt-get update && apt-get install -y \
python3 python3-pip python3-venv git curl wget \
caddy docker.io docker-compose awscli sqlite3 \
wireguard strongswan xl2tpd ollama
```
### Step C3: Install Hermes
```bash
pip3 install hermes-agent
```
### Step C4: Download and extract full backup
```bash
cd /tmp
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz \
full-restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
tar xzf full-restore.tar.gz -C /tmp/restore
```
### Step C5: Restore all components
```bash
# Hermes config + state
cp -r /tmp/restore/hermes-backup-2026-07-10/* /root/.hermes/ 2>/dev/null || true
cp /tmp/restore/hermes-backup-2026-07-10/config/config.yaml /root/.hermes/
cp /tmp/restore/hermes-backup-2026-07-10/config/.env /root/.hermes/
# SSH keys
cp /tmp/restore/hermes-backup-2026-07-10/ssh/* /root/.ssh/ 2>/dev/null || true
# AWS credentials
cp /tmp/restore/hermes-backup-2026-07-10/aws/* /root/.aws/ 2>/dev/null || true
# Mail passwords
cp -r /tmp/restore/hermes-backup-2026-07-10/himalaya/* /root/.config/himalaya/ 2>/dev/null || true
# Caddyfile
cp /tmp/restore/hermes-backup-2026-07-10/caddy/Caddyfile /etc/caddy/Caddyfile
# Systemd services
cp /tmp/restore/hermes-backup-2026-07-10/systemd/*.service /etc/systemd/system/
# Set permissions
chmod 600 /root/.hermes/config.yaml /root/.hermes/.env /root/.aws/credentials
chmod 600 /root/.ssh/*
chmod 640 /etc/caddy/Caddyfile
```
### Step C6: Start services
```bash
systemctl daemon-reload
systemctl enable --now hermes caddy ollama shark-game mysql-tunnel ops-portal
```
### Step C7: Run the embedded restore script
```bash
bash /tmp/restore/hermes-backup-2026-07-10/restore.sh
```
---
## Post-Restore Verification Checklist
After ANY restore path, complete these checks:
### 1. Hermes Core
- [ ] `hermes gateway status` — Gateway is running
- [ ] `hermes terminal --command "echo online"` — Agent responds
- [ ] Telegram bot responds to `/status`
### 2. Web Services
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://core.itpropartner.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://portal.debtrecoveryexperts.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://shark.iamgmb.com` → 200
### 3. Model Access
- [ ] `curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $KEY" | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if 'data' in d else 'FAIL')"` → OK
### 4. Email
- [ ] `himalaya envelope list -s 5` — IMAP access works
### 5. Backups
- [ ] `aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com | tail -3` — Recent backups exist
- [ ] `aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com` — Live sync is current
### 6. Standby
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "uptime"` — Standby reachable
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "df -h / | tail -1"` — Disk >10% free
---
## Critical Fix: Repair Backup Script for Future Reliability
The daily backup script MUST be fixed to prevent recurrence:
```bash
# Add AWS CLI venv activation to hermes-backup.sh
```
Add after line 5 (`set -euo pipefail`):
```bash
# Activate AWS CLI from venv (added 2026-07-11 — DR FIX)
if [ -f /opt/awscli-venv/bin/activate ]; then
source /opt/awscli-venv/bin/activate
fi
```
Similarly verify `root-essentials-backup.sh` works end-to-end:
```bash
# Test run root-essentials backup manually
bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1
```
Then update the DR issue log with this finding.
---
## Team Contact & Escalation
| Role | Name | Responsibility |
|------|------|---------------|
| CEO | Germaine | Final decision authority |
| Lead Sysadmin | Sho'Nuff (Hermes) | First responder, triage, fix |
| Operations | Anita | Comms, client notification |
**Escalation:** If restore fails after 2 attempts → escalate to netcup support (Core) or Hetzner support (standby).
---
## Rollback Plan
If restore makes things worse:
1. Restore config from pre-restore backup: `cp /root/.hermes/config.yaml.pre-restore-* /root/.hermes/config.yaml`
2. `hermes gateway restart`
3. Verify system returns to current state (which was operational but misconfigured)
---
## Files Created/Modified During Assessment
| File | Action |
|------|--------|
| This restore plan | Created at `/root/.hermes/references/restore-plan-2026-07-11.md` |
| `/tmp/root-verify.tar.gz` | Downloaded root-essentials backup for integrity check (clean up after) |
| `/tmp/full-verify.tar.gz` | Downloaded full backup for integrity check (clean up after) |
| DR issue log | New entry needed for backup script AWS CLI PATH bug |