ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-skills/skills/devops/hermes-standby-deployment/SKILL.md
T

17 KiB
Raw Permalink Blame History

name, description, version, author, license, platforms, metadata
name description version author license platforms metadata
hermes-standby-deployment Deploy a warm standby/failover Hermes box on another server — boot-time restore systemd service, periodic health check cron, Telegram + email alerts on failover. 1.2.0 ShoNuff MIT
linux
hermes
tags related_skills
dr
failover
standby
hetzner
s3
backup
hermes-backup
infrastructure-automation

Hermes Standby Deployment

Use this when setting up a secondary Hermes server that auto-failovers if the primary dies. The standby stays dormant (Hermes not running) until the primary is confirmed dead.

Architecture

Primary (netcup)                         Standby (Hetzner)
┌─────────────────────┐                 ┌──────────────────────────┐
│  Live Hermes        │      ping       │  Dormant — Hermes OFF    │
│  15-min sync → S3   │◄──────────────►│  systemd: check on boot  │
│  Telegram connected  │  152.53.192.33 │  cron: check every 10 min │
└─────────────────────┘                 └──────────────────────────┘
         │                                        │
         ▼                                        ▼
  s3://hermes-vps-backups/live/
         │
         └── If primary dies → standby syncs S3, starts Hermes, alerts via Telegram + email

Prerequisites

  • A Linux server (the standby) — Hetzner CPX11 (2C/2G) works, Ubuntu/Debian
  • The infrastructure SSH key (itpp-infra) registered in the Hetzner Cloud project
  • AWS CLI installed on the standby (pip3 install awscli)
  • Wasabi S3 credentials set up in ~/.aws/credentials
  • Hermes Agent installed (pip3 install hermes-agent)
  • Telegram bot token + chat ID

Files

All DR artifacts stored on S3 at s3://hermes-vps-backups/standby/:

File Purpose
DR-PLAN.md Full disaster recovery plan
server-inventory.md Full server inventory (all Hetzner + netcup boxes)
hermes-standby-restore.sh Boot-time failover — systemd runs this before Hermes would start
hermes-standby-watchdog.sh Periodic failover — system crontab every 10 min
hermes-standby.service systemd unit for boot restore
recovery-bundle-YYYY-MM-DD.md Full data dump including conversation history

SSH Key

The designated infrastructure key is itpp-infra (Ed25519, fingerprint e2:42:9c:e3:d6:ed:db:cd:b6:6a:f7:bc:7d:a4:19:92). It is deployed to all access-managed servers. On this installation it lives at /root/.ssh/itpp-infra and /root/.ssh/itpp-infra.pub. Key ID 114709791 in the Hetzner Cloud project.

Previous key wisp_rsa is superseded but kept locally as fallback.

Step-by-Step Deployment

1. Register SSH key in Hetzner project

Skip if itpp-infra is already registered — check via API:

import urllib.request, json
token = open('/root/.hermes/scripts/.hetzner_token').read().strip()
headers = {'Authorization': f'Bearer {token}'}
req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', headers=headers)
for k in json.loads(urllib.request.urlopen(req, timeout=10).read()).get('ssh_keys', []):
    print(f'{k["name"]} (ID {k["id"]})')

If not present, register:

with open('/root/.ssh/itpp-infra.pub') as f:
    pubkey = f.read().strip()
data = json.dumps({"name": "itpp-infra", "public_key": pubkey})
req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', data=data.encode(), headers={
    **headers, 'Content-Type': 'application/json'}, method='POST')
result = json.loads(urllib.request.urlopen(req, timeout=15).read())
key_id = result['ssh_key']['id']
print(f'Key registered: ID {key_id}')

Save the returned SSH key ID (e.g. 114709791) for use in rescue mode.

2. Enable rescue mode with SSH key, then power on

Note: Works reliably on CPX11 and CPX21. Does NOT work on CPX41 — see Pitfalls below.

data = json.dumps({"type": "linux64", "ssh_keys": [SSH_KEY_ID]})
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/enable_rescue', data=data.encode(), headers=headers, method='POST')
urllib.request.urlopen(req, timeout=15)
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/poweron', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req, timeout=15)

3. Inject SSH key via rescue

mount /dev/sda1 /mnt/root
mkdir -p /mnt/root/root/.ssh
echo '<public-key>' >> /mnt/root/root/.ssh/authorized_keys
chmod 600 /mnt/root/root/.ssh/authorized_keys

# If ippadmin user exists:
mkdir -p /mnt/root/home/ippadmin/.ssh
echo '<public-key>' >> /mnt/root/home/ippadmin/.ssh/authorized_keys
chown -R 1000:1000 /mnt/root/home/ippadmin/.ssh

4. Disable Hermes autostart + install standby service

systemctl disable hermes-gateway.service

wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
chmod +x /root/.hermes/scripts/hermes-standby-restore.sh

wget -O /root/.hermes/scripts/hermes-standby-watchdog.sh \
  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-watchdog.sh
chmod +x /root/.hermes/scripts/hermes-standby-watchdog.sh

wget -O /etc/systemd/system/hermes-standby.service \
  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby.service

systemctl daemon-reload
systemctl enable hermes-standby.service

5. Add watchdog crontab

echo "*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh" | crontab -

6. Wasabi credentials

mkdir -p ~/.aws
cat > ~/.aws/credentials << 'EOF'
[default]
aws_access_key_id = <ACCESS_KEY>
aws_secret_access_key = <SECRET_KEY>
EOF
cat > ~/.aws/config << 'EOF'
[default]
region = us-east-1
output = json
EOF

7. Telegram .env

cat > /root/.hermes/.env << 'EOF'
TELEGRAM_BOT_TOKEN=<TOKEN>
TELEGRAM_HOME_CHANNEL=<CHAT_ID>
TELEGRAM_ALLOWED_USERS=<CHAT_ID>
TERMINAL_TIMEOUT=60
TERMINAL_LIFETIME_SECONDS=300
EOF

8. Reboot normal

req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/disable_rescue', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req)
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/reboot', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req)

9. Verify

systemctl status hermes-standby   # "active (exited)" + "Live box is REACHABLE"
systemctl is-enabled hermes-gateway  # "disabled"
crontab -l  # shows watchdog script

Failover Logic

Boot-time (systemd)

  1. Wait for network
  2. Ping primary 3 times
  3. If primary reachable → exit (stay dormant)
  4. If primary dead → sync S3, start Hermes

Periodic (system crontab, every 5 min)

  1. If Hermes already running on standby → exit (already failed over)
  2. Ping primary 3 times immediately
  3. If all fail → confirm over 4 cycles at 30s intervals (~2 min)
  4. If any succeeds → abort (primary recovered)
  5. If all 4 cycles fail → send Telegram + email alert, sync S3, start Hermes

Update (July 8, 2026): Timing changed from every 10 min / 3.5 min confirmation to every 5 min / 2 min confirmation for faster failover detection. Update both the cron schedule (*/5 * * * *) and the watchdog script's sleep value (30s instead of 60s) when deploying new standby servers.

Periodic Audit Checklist

When auditing a deployed warm standby (e.g., DR readiness review), run through these checks:

1. Power State

  • Query Hetzner API: GET /v1/servers/{id} — note status field. The server should be running (warm standby: OS up, Hermes off) or off (cold standby: boots on demand). Confirm which state the DR plan expects.
  • SSH in and confirm Hermes gateway is inactive: systemctl is-active hermes-gateway should print inactive.
  • Check actual uptime: uptime. If the server has been up for weeks, confirm the intent is warm (always-on) vs cold (boot-on-demand).

2. Standby Systemd Service

  • systemctl is-active hermes-standby — should be active (exited).
  • Check the last restore log: tail -5 /var/log/hermes-standby-restore.log — should show "Live box is REACHABLE — staying dormant".
  • If the log is empty or shows errors, the restore script may need investigation.

3. Watchdog Cron

  • crontab -l — should contain */5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh (5 min interval, was */10 before Jul 8 2026 update).
  • Confirm the watchdog script's confirmation sleep is 30s (not 60s): grep 'sleep' /root/.hermes/scripts/hermes-standby-watchdog.sh.
  • Check the watchdog log: cat /var/log/hermes-standby-watchdog.log — should be empty if the live box has been healthy. Any entries are prior failure events.
  • Verify the watchdog script has chmod +x and readable credentials.

4. Live Sync Freshness

  • On the live box, verify hermes-live-sync cron is active and recently ran: check hermes cron list | grep hermes-live-sync for last run status.
  • On S3, check the age of s3://hermes-vps-backups/live/state.db — should be ≤15 min old: aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com.
  • Verify gateway_state.json and channel_directory.json are similarly fresh.

5. SSH Key

  • Confirm the designated key is in ~/.ssh/authorized_keys on the standby. Test: ssh -i ~/.ssh/itpp-infra root@<standby-ip> "hostname".
  • Verify the same key is registered in the Hetzner Cloud project (SSH Keys tab).

6. Failover Scripts on S3

  • aws s3 ls s3://hermes-vps-backups/standby/ --endpoint-url https://s3.us-east-1.wasabisys.com — should show all 6 files (DR-PLAN.md, server-inventory.md, both .sh scripts, .service, recovery-bundle).
  • The watchdog script on S3 should match the version deployed on the standby (compare checksum or date).

7. Documentation Drift Check

  • Does server-inventory.md accurately describe the standby's power state (running vs off)?
  • Does DR-PLAN.md Scenario A match the actual failover procedure used?
  • Is the SSH key named correctly in the docs (itpp-infra vs old wisp_rsa)?

8. Disk Space (CRITICAL — Jul 11, 2026)

  • df -h / — must be >20% free. If <10% free, failover will FAIL. The state.db alone is ~1.9 GB, sync logs accumulate endlessly, and the tarball extraction needs headroom. On a 38 GB CPX11, 10% is only 3.8 GB — tight but survivable. Below 2 GB, the watchdog script will crash mid-sync and the standby is dead weight.
  • Check sync log size: ls -lh /var/log/hermes-standby-sync.log. The sync script appends without rotation — logs can grow to 40+ MB in a week. Truncate if >500 MB: > /var/log/hermes-standby-sync.log.
  • Recovery commands if low on disk: 
    apt-get clean && apt-get autoremove --purge -y
journalctl --vacuum-size=200M
> /var/log/hermes-standby-sync.log
docker system prune -a -f 2>/dev/null || true
df -h /  # verify >5 GB free

Log each audit result in the session summary. Flag any drift between documented state (server-inventory.md / DR-PLAN.md) and live state as a finding.

Related Documents

The following authoritative documents supersede some of this skill's detailed instructions. They were submitted by the user during the Jul 9, 2026 session and represent the current target state.

Document Covers Status
Hermes DR Plan v2 Two-layer health check, SSH fencing, S3 ownership lock, VACUUM snapshots with integrity gate, backward heartbeat, state machine diagram, app1-bu CPX21 upgrade 🟡 Planned (not yet implemented)
Memory Auto-Consolidation DR Memory backup-before-prune guarantee, S3 restore procedures, safety invariants, size safety valve caveat 🟢 Deployed and running

The v2 DR plan proposes upgrading app1-bu from CPX11 to CPX21 (for local Ollama) and introducing Tailscale-based cross-box communication. These changes have NOT yet been implemented — see the 16-item checklist at the end of the v2 document.

When the user references "the DR plan," determine which version:

  • v1 (this skill): The currently deployed standby watchdog. Ping-only health check. Public IP SSH for fencing. S3-based 15-min sync.

  • v2 (user-submitted doc): The target architecture. HTTP health endpoint, SSH fence + S3 ownership lock, 10-min VACUUM snapshots, Tailscale tailnet for cross-box traffic, CPX21 upgrade for standby.

  • Documentation drift between server-inventory.md, DR-PLAN.md, and live state. The DR plan may say "offline" while the server is running, or the SSH key listed in the recovery bundle may differ from what's actually authorized. Verify all three agree during periodic audits.

  • HETZNER_API_TOKEN as an env var can drift from the .hetzner_token file approach. The recovery bundle references /root/.hermes/scripts/.hetzner_token but the token may only exist as HETZNER_API_TOKEN in the shell environment. The file can be deleted during cleanup without notice — check both sources during audit.

  • Telegram split-brain: Never run Hermes on both boxes simultaneously with the same bot token. The standby only starts Hermes if the primary is confirmed unreachable.

  • Primary comes back after failover: Power off the primary if it ever returns — two Hermes instances with the same token wrecks message delivery.

  • SMTP from standby: Works from Hetzner (same network as mail server). If standby is elsewhere, may need SMTP firewall rules.

  • Watchdog script has credentials hardcoded (Telegram token, SMTP password). Protect with chmod 600.

  • S3 sync doesn't delete local files — aws s3 sync only adds/updates, never removes.

  • Hostname renaming: After standby deployment, rename OS hostname to match DNS via hostnamectl set-hostname. See references/standby-hostname-and-renaming.md.

  • Bulk rescue: For batch SSH key injection across multiple Hetzner servers, see references/bulk-hetzner-rescue.md.

  • CPX41 key injection fails via rescue. The enable_rescue API with ssh_keys parameter works on CPX11/CPX21 but accepts and silently ignores keys on CPX41. You get Permission denied with the correct key. Workarounds: (1) omit ssh_keys, use the rescue root password to mount and inject; or (2) ask a user with existing access to add the key. The key WILL work on rebuild since it's registered in the project.

  • Docker convention: Docker services on the live box live under ~/docker/<service>/ with compose files. The auto-restore script currently only syncs ~/.hermes/ — ~/docker/ must be restored separately.

  • app1.itpropartner.com (87.99.144.163) has an intermittent SSH delay of 30-60s after boot even though port 80 responds immediately.

  • After rescue-mode reboot, wait for SSH to respond before attempting. Hostname briefly shows rescue — verify against OS hostname.

  • Standby server should have Tailscale installed to stay reachable for SSH if public SSH is ever locked down. Install via curl -fsSL https://tailscale.com/install.sh | sh and authenticate. See tailscale-infrastructure-access skill.

  • Backup pipeline lives on the live box only. The standby has no independent S3 sync. If running the standby as hot (always on), add a separate backup cron for its own state.

  • Telegram conflict after moving profile from standby to live. If a profile's gateway was running on the standby, the standby process holds the Telegram session lock. Resolution: (1) SSH into standby: ssh -i ~/.ssh/itpp-infra root@<standby-ip> pkill -f 'profile <name>'. (2) If it persists, hard reboot the standby via Hetzner API. (3) Restart the profile's gateway on the live box. (4) The conflict may recur for 1-2 cycles (~20-60s) as Telegram expires the old session — this is normal.

Verification

After setup, the standby should:

  • Stay powered on with Hermes off
  • Log "Live box is REACHABLE" on boot
  • Crontab check silently exits if primary is healthy
  • If primary dies, standby detects within ~4 min and takes over

Pitfalls

  • AWS CLI PATH in cron: The standby sync script (hermes-standby-sync.sh) calls aws s3 sync. In cron's minimal PATH, aws may not be found even if it works interactively. The script already sources /opt/awscli-venv/bin/activate, but if this venv doesn't exist or the cron PATH changes, the sync silently breaks. Verify by running from cron's stripped environment: env -i HOME=$HOME PATH=/usr/bin:/bin bash /root/.hermes/scripts/hermes-standby-sync.sh.

To test: temporarily power off the primary (or block it in the standby's firewall), confirm the watchdog fires, then clean up.

Reference in New Issue View Git Blame Copy Permalink