ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial skills documentation — 25 categories, all SKILL.md + references + scripts

Browse Source
This commit is contained in:
root
2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/hermes-standby-deployment/SKILL.md
+302
View File
@@ -0,0 +1,302 @@
---
name: hermes-standby-deployment
description: "Deploy a warm standby/failover Hermes box on another server — boot-time restore systemd service, periodic health check cron, Telegram + email alerts on failover."
version: 1.2.0
author: ShoNuff
license: MIT
platforms: [linux]
metadata:
hermes:
tags: [dr, failover, standby, hetzner, s3, backup]
related_skills: [hermes-backup, infrastructure-automation]
---
# Hermes Standby Deployment
Use this when setting up a secondary Hermes server that auto-failovers if the primary dies. The standby stays dormant (Hermes not running) until the primary is confirmed dead.
## Architecture
```
Primary (netcup) Standby (Hetzner)
┌─────────────────────┐ ┌──────────────────────────┐
│ Live Hermes │ ping │ Dormant — Hermes OFF │
│ 15-min sync → S3 │◄──────────────►│ systemd: check on boot │
│ Telegram connected │ 152.53.192.33 │ cron: check every 10 min │
└─────────────────────┘ └──────────────────────────┘
│ │
▼ ▼
s3://hermes-vps-backups/live/
│
└── If primary dies → standby syncs S3, starts Hermes, alerts via Telegram + email
```
## Prerequisites
- A Linux server (the standby) — Hetzner CPX11 (2C/2G) works, Ubuntu/Debian
- The infrastructure SSH key (`itpp-infra`) registered in the Hetzner Cloud project
- AWS CLI installed on the standby (`pip3 install awscli`)
- Wasabi S3 credentials set up in `~/.aws/credentials`
- Hermes Agent installed (`pip3 install hermes-agent`)
- Telegram bot token + chat ID
## Files
All DR artifacts stored on S3 at `s3://hermes-vps-backups/standby/`:
| File | Purpose |
|---|---|
| `DR-PLAN.md` | Full disaster recovery plan |
| `server-inventory.md` | Full server inventory (all Hetzner + netcup boxes) |
| `hermes-standby-restore.sh` | Boot-time failover — systemd runs this before Hermes would start |
| `hermes-standby-watchdog.sh` | Periodic failover — system crontab every 10 min |
| `hermes-standby.service` | systemd unit for boot restore |
| `recovery-bundle-YYYY-MM-DD.md` | Full data dump including conversation history |
## SSH Key
The designated infrastructure key is **`itpp-infra`** (Ed25519, fingerprint `e2:42:9c:e3:d6:ed:db:cd:b6:6a:f7:bc:7d:a4:19:92`). It is deployed to all access-managed servers. On this installation it lives at `/root/.ssh/itpp-infra` and `/root/.ssh/itpp-infra.pub`. Key ID `114709791` in the Hetzner Cloud project.
Previous key `wisp_rsa` is superseded but kept locally as fallback.
## Step-by-Step Deployment
### 1. Register SSH key in Hetzner project
Skip if `itpp-infra` is already registered — check via API:
```python
import urllib.request, json
token = open('/root/.hermes/scripts/.hetzner_token').read().strip()
headers = {'Authorization': f'Bearer {token}'}
req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', headers=headers)
for k in json.loads(urllib.request.urlopen(req, timeout=10).read()).get('ssh_keys', []):
print(f'{k["name"]} (ID {k["id"]})')
```
If not present, register:
```python
with open('/root/.ssh/itpp-infra.pub') as f:
pubkey = f.read().strip()
data = json.dumps({"name": "itpp-infra", "public_key": pubkey})
req = urllib.request.Request('https://api.hetzner.cloud/v1/ssh_keys', data=data.encode(), headers={
**headers, 'Content-Type': 'application/json'}, method='POST')
result = json.loads(urllib.request.urlopen(req, timeout=15).read())
key_id = result['ssh_key']['id']
print(f'Key registered: ID {key_id}')
```
Save the returned SSH key ID (e.g. `114709791`) for use in rescue mode.
### 2. Enable rescue mode with SSH key, then power on
**Note: Works reliably on CPX11 and CPX21. Does NOT work on CPX41 — see Pitfalls below.**
```python
data = json.dumps({"type": "linux64", "ssh_keys": [SSH_KEY_ID]})
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/enable_rescue', data=data.encode(), headers=headers, method='POST')
urllib.request.urlopen(req, timeout=15)
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/poweron', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req, timeout=15)
```
### 3. Inject SSH key via rescue
```bash
mount /dev/sda1 /mnt/root
mkdir -p /mnt/root/root/.ssh
echo '<public-key>' >> /mnt/root/root/.ssh/authorized_keys
chmod 600 /mnt/root/root/.ssh/authorized_keys
# If ippadmin user exists:
mkdir -p /mnt/root/home/ippadmin/.ssh
echo '<public-key>' >> /mnt/root/home/ippadmin/.ssh/authorized_keys
chown -R 1000:1000 /mnt/root/home/ippadmin/.ssh
```
### 4. Disable Hermes autostart + install standby service
```bash
systemctl disable hermes-gateway.service
wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
chmod +x /root/.hermes/scripts/hermes-standby-restore.sh
wget -O /root/.hermes/scripts/hermes-standby-watchdog.sh \
https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-watchdog.sh
chmod +x /root/.hermes/scripts/hermes-standby-watchdog.sh
wget -O /etc/systemd/system/hermes-standby.service \
https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby.service
systemctl daemon-reload
systemctl enable hermes-standby.service
```
### 5. Add watchdog crontab
```bash
echo "*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh" | crontab -
```
### 6. Wasabi credentials
```bash
mkdir -p ~/.aws
cat > ~/.aws/credentials << 'EOF'
[default]
aws_access_key_id = <ACCESS_KEY>
aws_secret_access_key = <SECRET_KEY>
EOF
cat > ~/.aws/config << 'EOF'
[default]
region = us-east-1
output = json
EOF
```
### 7. Telegram .env
```bash
cat > /root/.hermes/.env << 'EOF'
TELEGRAM_BOT_TOKEN=<TOKEN>
TELEGRAM_HOME_CHANNEL=<CHAT_ID>
TELEGRAM_ALLOWED_USERS=<CHAT_ID>
TERMINAL_TIMEOUT=60
TERMINAL_LIFETIME_SECONDS=300
EOF
```
### 8. Reboot normal
```bash
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/disable_rescue', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req)
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}/actions/reboot', data=b'{}', headers=headers, method='POST')
urllib.request.urlopen(req)
```
### 9. Verify
```bash
systemctl status hermes-standby # "active (exited)" + "Live box is REACHABLE"
systemctl is-enabled hermes-gateway # "disabled"
crontab -l # shows watchdog script
```
## Failover Logic
### Boot-time (systemd)
1. Wait for network
2. Ping primary 3 times
3. If primary reachable → exit (stay dormant)
4. If primary dead → sync S3, start Hermes
### Periodic (system crontab, every 5 min)
1. If Hermes already running on standby → exit (already failed over)
2. Ping primary 3 times immediately
3. If all fail → confirm over 4 cycles at 30s intervals (~2 min)
4. If any succeeds → abort (primary recovered)
5. If all 4 cycles fail → send Telegram + email alert, sync S3, start Hermes
**Update (July 8, 2026):** Timing changed from every 10 min / 3.5 min confirmation to every 5 min / 2 min confirmation for faster failover detection. Update both the cron schedule (`*/5 * * * *`) and the watchdog script's `sleep` value (30s instead of 60s) when deploying new standby servers.
## Periodic Audit Checklist
When auditing a deployed warm standby (e.g., DR readiness review), run through these checks:
### 1. Power State
- [ ] Query Hetzner API: `GET /v1/servers/{id}` — note `status` field. The server should be `running` (warm standby: OS up, Hermes off) or `off` (cold standby: boots on demand). Confirm which state the DR plan expects.
- [ ] SSH in and confirm Hermes gateway is **inactive**: `systemctl is-active hermes-gateway` should print `inactive`.
- [ ] Check actual uptime: `uptime`. If the server has been up for weeks, confirm the intent is warm (always-on) vs cold (boot-on-demand).
### 2. Standby Systemd Service
- [ ] `systemctl is-active hermes-standby` — should be `active (exited)`.
- [ ] Check the last restore log: `tail -5 /var/log/hermes-standby-restore.log` — should show `"Live box is REACHABLE — staying dormant"`.
- [ ] If the log is empty or shows errors, the restore script may need investigation.
### 3. Watchdog Cron
- [ ] `crontab -l` — should contain `*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh` (5 min interval, was */10 before Jul 8 2026 update).
- [ ] Confirm the watchdog script's confirmation sleep is 30s (not 60s): `grep 'sleep' /root/.hermes/scripts/hermes-standby-watchdog.sh`.
- [ ] Check the watchdog log: `cat /var/log/hermes-standby-watchdog.log` — should be empty if the live box has been healthy. Any entries are prior failure events.
- [ ] Verify the watchdog script has `chmod +x` and readable credentials.
### 4. Live Sync Freshness
- [ ] On the **live box**, verify `hermes-live-sync` cron is active and recently ran: check `hermes cron list | grep hermes-live-sync` for last run status.
- [ ] On S3, check the age of `s3://hermes-vps-backups/live/state.db` — should be ≤15 min old: `aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com`.
- [ ] Verify `gateway_state.json` and `channel_directory.json` are similarly fresh.
### 5. SSH Key
- [ ] Confirm the designated key is in `~/.ssh/authorized_keys` on the standby. Test: `ssh -i ~/.ssh/itpp-infra root@<standby-ip> "hostname"`.
- [ ] Verify the same key is registered in the Hetzner Cloud project (SSH Keys tab).
### 6. Failover Scripts on S3
- [ ] `aws s3 ls s3://hermes-vps-backups/standby/ --endpoint-url https://s3.us-east-1.wasabisys.com` — should show all 6 files (DR-PLAN.md, server-inventory.md, both .sh scripts, .service, recovery-bundle).
- [ ] The watchdog script on S3 should match the version deployed on the standby (compare checksum or date).
### 7. Documentation Drift Check
- [ ] Does `server-inventory.md` accurately describe the standby's power state (running vs off)?
- [ ] Does `DR-PLAN.md` Scenario A match the actual failover procedure used?
- [ ] Is the SSH key named correctly in the docs (`itpp-infra` vs old `wisp_rsa`)?
### 8. Disk Space (CRITICAL — Jul 11, 2026)
- [ ] `df -h /` — must be >20% free. **If <10% free, failover will FAIL.** The state.db alone is ~1.9 GB, sync logs accumulate endlessly, and the tarball extraction needs headroom. On a 38 GB CPX11, 10% is only 3.8 GB — tight but survivable. Below 2 GB, the watchdog script will crash mid-sync and the standby is dead weight.
- [ ] Check sync log size: `ls -lh /var/log/hermes-standby-sync.log`. The sync script appends without rotation — logs can grow to 40+ MB in a week. Truncate if >500 MB: `> /var/log/hermes-standby-sync.log`.
- [ ] Recovery commands if low on disk:
```bash
apt-get clean && apt-get autoremove --purge -y
journalctl --vacuum-size=200M
> /var/log/hermes-standby-sync.log
docker system prune -a -f 2>/dev/null || true
df -h / # verify >5 GB free
```
> Log each audit result in the session summary. Flag any drift between documented state (server-inventory.md / DR-PLAN.md) and live state as a finding.
## Related Documents
The following authoritative documents supersede some of this skill's detailed instructions. They were submitted by the user during the Jul 9, 2026 session and represent the current target state.
| Document | Covers | Status |
|---|---|---|
| **Hermes DR Plan v2** | Two-layer health check, SSH fencing, S3 ownership lock, VACUUM snapshots with integrity gate, backward heartbeat, state machine diagram, app1-bu CPX21 upgrade | 🟡 Planned (not yet implemented) |
| **Memory Auto-Consolidation DR** | Memory backup-before-prune guarantee, S3 restore procedures, safety invariants, size safety valve caveat | 🟢 Deployed and running |
The v2 DR plan proposes upgrading app1-bu from CPX11 to CPX21 (for local Ollama) and introducing Tailscale-based cross-box communication. These changes have NOT yet been implemented — see the 16-item checklist at the end of the v2 document.
When the user references "the DR plan," determine which version:
- **v1 (this skill):** The currently deployed standby watchdog. Ping-only health check. Public IP SSH for fencing. S3-based 15-min sync.
- **v2 (user-submitted doc):** The target architecture. HTTP health endpoint, SSH fence + S3 ownership lock, 10-min VACUUM snapshots, Tailscale tailnet for cross-box traffic, CPX21 upgrade for standby.
- **Documentation drift between server-inventory.md, DR-PLAN.md, and live state.** The DR plan may say "offline" while the server is running, or the SSH key listed in the recovery bundle may differ from what's actually authorized. Verify all three agree during periodic audits.
- **HETZNER_API_TOKEN as an env var can drift from the `.hetzner_token` file approach.** The recovery bundle references `/root/.hermes/scripts/.hetzner_token` but the token may only exist as `HETZNER_API_TOKEN` in the shell environment. The file can be deleted during cleanup without notice — check both sources during audit.
- **Telegram split-brain:** Never run Hermes on both boxes simultaneously with the same bot token. The standby only starts Hermes if the primary is confirmed unreachable.
- **Primary comes back after failover:** Power off the primary if it ever returns — two Hermes instances with the same token wrecks message delivery.
- **SMTP from standby:** Works from Hetzner (same network as mail server). If standby is elsewhere, may need SMTP firewall rules.
- **Watchdog script has credentials hardcoded** (Telegram token, SMTP password). Protect with `chmod 600`.
- **S3 sync doesn't delete local files** — `aws s3 sync` only adds/updates, never removes.
- **Hostname renaming:** After standby deployment, rename OS hostname to match DNS via `hostnamectl set-hostname`. See `references/standby-hostname-and-renaming.md`.
- **Bulk rescue:** For batch SSH key injection across multiple Hetzner servers, see `references/bulk-hetzner-rescue.md`.
- **CPX41 key injection fails via rescue.** The `enable_rescue` API with `ssh_keys` parameter works on CPX11/CPX21 but accepts and silently ignores keys on CPX41. You get `Permission denied` with the correct key. Workarounds: (1) omit `ssh_keys`, use the rescue root password to mount and inject; or (2) ask a user with existing access to add the key. The key WILL work on rebuild since it's registered in the project.
- **Docker convention:** Docker services on the live box live under `~/docker/<service>/` with compose files. The auto-restore script currently only syncs `~/.hermes/` — `~/docker/` must be restored separately.
- **app1.itpropartner.com (87.99.144.163)** has an intermittent SSH delay of 30-60s after boot even though port 80 responds immediately.
- **After rescue-mode reboot, wait for SSH to respond** before attempting. Hostname briefly shows `rescue` — verify against OS hostname.
- **Standby server should have Tailscale installed** to stay reachable for SSH if public SSH is ever locked down. Install via `curl -fsSL https://tailscale.com/install.sh | sh` and authenticate. See `tailscale-infrastructure-access` skill.
- **Backup pipeline lives on the live box only.** The standby has no independent S3 sync. If running the standby as hot (always on), add a separate backup cron for its own state.
- **Telegram conflict after moving profile from standby to live.** If a profile's gateway was running on the standby, the standby process holds the Telegram session lock. Resolution: (1) SSH into standby: `ssh -i ~/.ssh/itpp-infra root@<standby-ip> pkill -f 'profile <name>'`. (2) If it persists, hard reboot the standby via Hetzner API. (3) Restart the profile's gateway on the live box. (4) The conflict may recur for 1-2 cycles (~20-60s) as Telegram expires the old session — this is normal.
## Verification
After setup, the standby should:
- Stay powered on with Hermes off
- Log "Live box is REACHABLE" on boot
- Crontab check silently exits if primary is healthy
- If primary dies, standby detects within ~4 min and takes over
## Pitfalls
- **AWS CLI PATH in cron:** The standby sync script (`hermes-standby-sync.sh`) calls `aws s3 sync`. In cron's minimal PATH, `aws` may not be found even if it works interactively. The script already sources `/opt/awscli-venv/bin/activate`, but if this venv doesn't exist or the cron PATH changes, the sync silently breaks. Verify by running from cron's stripped environment: `env -i HOME=$HOME PATH=/usr/bin:/bin bash /root/.hermes/scripts/hermes-standby-sync.sh`.
To test: temporarily power off the primary (or block it in the standby's firewall), confirm the watchdog fires, then clean up.