ippadmin/hermes-skills

Fork 0

Files

T

root 1b4fcd4427 Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00

36 KiB

Raw Permalink Blame History

name, description, version, author, tags

name

description

version

author

Hermes Backup

Regularly backs up the entire Hermes environment so you can restore on a new server.

Pitfalls

Do not invent fallback keys. A top-level fallback_providers: [...] entry is not a valid Hermes setting and may be silently ignored. Current Hermes uses model.fallbacks for main-agent fallbacks and delegation.fallback for child-agent fallbacks. Treat the installed schema and hermes config check as authoritative. After editing, parse the YAML, run hermes config check, restart the gateway if required, and perform a real fallback inference test. See references/hermes-config-discipline.md.
Never use /tmp as staging directory — On most Linux systems, /tmp is a tmpfs (RAM-backed, usually ~1GB). Hermes backups include state.db and profiles/ which easily exceed this. Stage backups on root disk: BACKUP_DIR="/root/.hermes/.backups/hermes-backup-$(date +%F)".
Check /tmp usage before debugging — If the backup script exits with No space left on device but df -h / shows free space, run df -h /tmp. If it's a tmpfs at 100%, the fix is moving the staging dir off tmpfs, not clearing disk space.
The backup archive is created from the staging directory — If you move BACKUP_DIR off tmpfs, also move ARCHIVE off tmpfs to match. The tar czf runs from the staging dir's parent, so both paths need consistent disk backing.
Fix the config.yaml temp_dir too — The backup Python script (e.g. wisp-backup.py) likely has its own temp_dir in config.yaml. If you only change the bash wrapper's staging dir, the Python script still writes intermediate files to /tmp. Patch both.
tar cd path must match staging dir parent — If BACKUP_DIR moved off /tmp to ~/.hermes/.backups/, the tar czf command must cd "$(dirname "$BACKUP_DIR")" not cd /tmp. Otherwise tar produces Cannot stat: No such file.
Wasabi bucket names are globally unique — hermes-backups may be taken by another account. Use alternatives like hermes-vps-backups.
IAM policy needs PutBucketVersioning explicitly — Without it, put-bucket-versioning returns Access Denied even if other S3 permissions work.
Policy changes take 30-60s to propagate — If a CreateBucket or PutBucketVersioning call fails with AccessDenied immediately after policy update, retry after a short wait.
ListBuckets returns AccessDenied — this is expected — The IAM user's policy scopes to specific bucket ARNs, not s3:ListAllMyBuckets. Test by reading a specific bucket by name instead.
After migration, verify S3 access immediately — Run aws s3 ls s3://<bucket-name>/ --endpoint-url https://s3.us-east-1.wasabisys.com/. If you get InvalidAccessKeyId, the credentials file didn't survive the transfer. Check ~/.aws/credentials exists and is readable.
The cold spare / standby section was moved — the old "Failover to cold spare (~2 min recovery)" was replaced by the automatic warm standby restore below. The old manual procedure (curl ... install.sh && aws s3 cp latest-hermes.tar.gz && systemctl restart) is deprecated in favor of the systemd oneshot pattern.
Live sync and warm standby are independent layers — The 15-min live sync pushes TO S3. The warm standby restore pulls FROM S3. They don't depend on each other's scripts or configs.
Secret files consolidated into standalone files (.hetzner_token, .netcup_api_key) are fragile — They get left behind during migration, forgotten during key rotation, and break scripts silently. Best practice is to consolidate all secrets into ~/.hermes/.env (chmod 600) and remove the standalone files. See references/secret-management.md for the standard workflow.
Post-migration, check that standalone token files were properly merged into .env — grep -r "api_key\|token\|password\|secret" ~/.hermes/config.yaml (expect refs only, not values). If scripts reference deleted files, patch them to read from .env instead.
System-level systemd services are NOT backed up — The backup script only copies from ~/.config/systemd/user/hermes-gateway*.service (user-scoped). All services at /etc/systemd/system/ — hermes.service, hermes-assistant.service, hermes-browser.service, shark-game.service, mysql-tunnel.service, ollama.service — are silently missing from backups. A full restore on a new server loses every systemd unit except the Anita gateway. FIXED 2026-07-08: backup script now copies from /etc/systemd/system/hermes*.service plus mysql-tunnel, ollama, and shark-game. Restore script uses sudo to write to /etc/systemd/system/ and runs systemctl daemon-reload. If you add new systemd services, add cp /etc/systemd/system/<new-service>.service "$BACKUP_DIR/systemd/" to both the backup and embedded restore scripts.
Caddy reverse proxy config is NOT backed up — /etc/caddy/Caddyfile is the single point of failure for all web routing. A restore on a new server would have Hermes running but all Caddy-proxied services unreachable. FIXED 2026-07-08: backup script now copies /etc/caddy/Caddyfile to $BACKUP_DIR/caddy/, and the live-sync script pushes it to s3://hermes-vps-backups/live/caddy/Caddyfile. The embedded restore script restores to /etc/caddy/ with sudo.
Only wisp_rsa SSH keys are backed up — The backup script explicitly copies $HOME/.ssh/wisp_rsa and .pub but ignores itpp-infra, authorized_keys, and any other keys. The itpp-infra key is used by the mysql-tunnel systemd service — restore without it means crippled database access. Fix: extend the SSH key loop to include itpp-infra and authorized_keys.
Standalone credential files outside the profiles/ dir are not backed up — Files like /root/.hermes/config/migration-creds.txt, /root/.hermes/references/dre-temp-passwords.txt are invisible to the backup pipeline. The profiles/ dir is covered via rsync, but root-level credential files are only backed up if they match hardcoded names. FIXED 2026-07-08: migration-creds.txt now explicitly backed up to $BACKUP_DIR/config/migration-creds.txt and restored with chmod 600. Run find /root/.hermes -maxdepth 3 $ -name '*.env' -o -name '*cred*' -o -name '*pass*' -o -name '*secret*' -o -name '*token*' $ to audit remaining gaps and add them explicitly.
Private keys with wrong permissions (644 instead of 600) — /root/wg-server.key, /root/wg-ccr.key are WireGuard private keys world-readable. /root/.hermes/references/dre-temp-passwords.txt is 644 (passwords readable by all users). Find them with find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' -o -name '*secret*' ) -not -perm 600. Fix: chmod 600 on each.
Stale full backups are silent — The daily full backup cron can stop running (e.g. last ran 3+ days ago). No alert fires when a daily backup is missed. The live sync (every 15 min) continues running, masking the full backup's absence. Check recent full backup dates with aws s3 ls s3://$BUCKET/$DEST_DIR/ --endpoint-url $ENDPOINT | tail -5. FIXED 2026-07-10: Daily watchdog script at /root/.hermes/scripts/backup-audit-check.sh runs from system crontab at 2 AM UTC (1h after backup). Logs to syslog tag backup-audit. Full reference: references/backup-audit-watchdog.md.
~/.aws/config is not explicitly backed up — The backup script copies ~/.aws/credentials but not ~/.aws/config. While it only contains the region setting (43 bytes, 644 perms), it's still a gap if you're automating complete restore. Add cp "$HOME/.aws/config" "$BACKUP_DIR/aws/" 2>/dev/null || true.
All backup scripts MUST source the AWS CLI venv for cron — Cron has a minimal PATH. Without source /opt/awscli-venv/bin/activate, aws is not found and the upload step silently fails while the local archive is created successfully. This gives a false sense of security. hermes-backup.sh was missing this activation (fixed Jul 11, 2026). root-essentials-backup.sh and hermes-live-sync.sh had it correctly. Audit all scripts with grep -L 'awscli-venv' /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh.
root-essentials-backup.sh has a wrong Caddyfile path — The tar --append on line 30 uses -C /etc systemd/system/caddy/Caddyfile which resolves to /etc/systemd/system/caddy/Caddyfile (does not exist). Correct path: -C /etc caddy/Caddyfile → /etc/caddy/Caddyfile. The error is hidden by 2>/dev/null || true, so the tarball silently lacks the Caddyfile.
root-essentials-backup.sh uses /tmp for staging — The archive is written to /tmp/root-essentials-<date>.tar.gz before upload. On this installation /tmp has 7.5G free so it works, but the general guidance is to avoid tmpfs for backup staging. If the archive grows beyond tmpfs capacity, the script fails mid-backup.
root-essentials-backup.sh can silently fail the S3 upload while local tarballs succeed — The script has set -euo pipefail, so if the aws s3 cp upload step fails, the script exits immediately without cleaning up. This leaves stranded local tarballs in /tmp/root-essentials-*.tar.gz while S3 shows stale or missing files. The journal/syslog output (piped through logger) may show only the first echo line ("Creating backup...") with nothing after, because the upload failure prevents later echo lines from executing. The backup-audit watchdog only checks the full-backup bucket (hermes-vps-backups/hermes-full-backup/), not the root-backup prefix, so root-essentials failures go undetected. Diagnostic procedure when root-backup/ looks stale on S3:
1. ls -la /tmp/root-essentials-*.tar.gz — if recent tarballs exist locally, the upload step is the fault
2. journalctl -t root-essentials-backup --no-pager -n 10 — if output stops at "Creating backup...", the script died after tar but before or during upload
3. grep 'root-essentials-backup' /var/log/syslog — same check via syslog if journal isn't available
4. Run the script manually to see the actual error: bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1
Empty purpose-specific buckets are a silent gap — itpropartner-system-configs and itpropartner-docker-volumes exist on Wasabi (confirmed Jul 11, 2026) but contain 0 objects. The sync scripts (hermes-system-config-sync.sh, hermes-docker-sync.sh) are either not running or failing silently. Their output goes to cron's default mail spool, which may not be monitored. Verify with aws s3 ls s3://<bucket>/ --recursive --summarize --endpoint-url ....

Backup completeness audit

Run this after any backup script change to verify nothing critical is missing:

echo "=== Systemd services ==="
echo "User (backed up):"
ls ~/.config/systemd/user/hermes-gateway*.service 2>/dev/null || echo "(none)"
echo "System (NOT backed up):"
ls /etc/systemd/system/*.service 2>/dev/null || echo "(none)"
echo "=== Caddy config ==="
[ -f /etc/caddy/Caddyfile ] && echo "EXISTS (NOT backed up)" || echo "MISSING"
echo "=== SSH keys ==="
echo "Backed up:" && ls ~/.ssh/wisp_rsa* 2>/dev/null || echo "(none)"
echo "NOT backed up:" && for f in ~/.ssh/itpp-infra ~/.ssh/itpp-infra.pub ~/.ssh/authorized_keys; do
  [ -f "$f" ] && echo "  $f" || true; done
echo "=== WireGuard keys ==="
find /root -maxdepth 2 -name '*.key' 2>/dev/null || echo "(none)"
echo "=== Standalone creds outside backup scope ==="
find /root/.hermes -maxdepth 3 ( -name '*.env' -o -name '*cred*' -o -name '*pass*' \
  -o -name '*secret*' -o -name '*token*' ) -not -path '*/profiles/*' 2>/dev/null
echo "=== Permission audit (should all be 600) ==="
find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' \
  -o -name '*secret*' ) -not -perm 600 2>/dev/null || echo "(all correct)"
echo "=== Last full backup dates ==="
source /opt/awscli-venv/bin/activate 2>/dev/null && \
  aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com 2>&1 | tail -5

Post-fix verification (run after script changes)

After modifying the backup or live-sync scripts, verify the new items are actually picked up:

# 1. Dry-run the backup script's new sections
echo "=== systemd services to be backed up ==="
ls /etc/systemd/system/hermes*.service /etc/systemd/system/mysql-tunnel.service \
   /etc/systemd/system/ollama.service /etc/systemd/system/shark-game.service 2>/dev/null
echo "---"
echo "=== Caddyfile to be backed up ==="
stat /etc/caddy/Caddyfile 2>/dev/null && echo "EXISTS" || echo "MISSING"
echo "---"
echo "=== migration-creds.txt to be backed up ==="
stat /root/.hermes/migration-creds.txt 2>/dev/null && echo "EXISTS" || echo "MISSING"

# 2. Verify live-sync puts Caddyfile on S3
source /opt/awscli-venv/bin/activate
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com/

# 3. Verify the embedded restore.sh heredoc is correct
grep -n 'sudo mkdir -p /etc/systemd/system' /root/.hermes/scripts/hermes-backup.sh
grep -n 'sudo mkdir -p /etc/caddy' /root/.hermes/scripts/hermes-backup.sh
grep -n 'migration-creds.txt' /root/.hermes/scripts/hermes-backup.sh

# 4. Syntax check
bash -n /root/.hermes/scripts/hermes-backup.sh && echo "backup.sh: OK"
bash -n /root/.hermes/scripts/hermes-live-sync.sh && echo "live-sync.sh: OK"

# 5. Venv activation audit — ALL backup/sync scripts must source the venv for cron
echo "=== Scripts missing venv activation (will fail in cron) ==="
for script in /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh; do
    grep -q 'awscli-venv' "$script" 2>/dev/null || echo "  MISSING: $script"
done

SiteGround SFTP backup

SiteGround websites are backed up via SFTP to s3://hermes-vps-backups/siteground/<site>/. The connection uses an encrypted RSA key with passphrase. See references/siteground-backup-pattern.md for credentials, connection method, exclusion recommendations, and the database export strategy.

Pitfall — Fleet-wide SFTP backup times out: Attempting to backup all 21 SiteGround WordPress sites in a single run timed out at 600 seconds (Jul 10, 2026). SFTP over shared hosting is too slow for bulk migration. Two alternatives: (1) Back up individual sites in batches of 3-5, or (2) Use MainWP + WPvivid Pro (already configured, backs up 15 sites daily to Wasabi S3) as the primary migration path and rely on SFTP only for sites not in MainWP.

Pitfall — SFTP root is the site directory: SiteGround's SFTP user sees WordPress site directories in the root (/), not under /home/. Listing /home/ returns OSError: list failed. Always list / first to discover the actual site directories.

What's backed up

Item	Why
`config.yaml`, `.env`, `auth.json`, `migration-creds.txt`	All config + API keys + migration secrets
`state.db` + `sessions/`	Full conversation history (~1.1GB)
`skills/`	All installed/created skills
`scripts/`	Custom scripts (backup, feed digest, etc.)
`profiles/`	Anita's profile + any others
`cron/`	Job definitions
SSH keys	WISP backup keys
Mail passwords	Himalaya creds
AWS/Wasabi creds	For S3 access
Systemd services	All Hermes services from `/etc/systemd/system/`
Caddyfile	Reverse proxy config from `/etc/caddy/`

Included in backup

A restore.sh script is included — extract the tarball on a new server and run it.

Verified S3 endpoint

All buckets use:

--endpoint-url https://s3.us-east-1.wasabisys.com

Live buckets (created 2026-07-04, versioning ON, all 5 confirmed Jul 10)

Bucket	Purpose	Status
`hermes-vps-backups`	Hermes full backup (config, sessions, profiles, keys) + live sync + full archive tarballs + standby scripts	Active since Jul 4
`mikrotik-ccr-backups`	Router config exports (home gateway, wisp gateway)	Active since Jul 4
`itpropartner-system-configs`	System configs: Caddyfile, systemd services, ops scripts, SSH keys, env files	🟡 Bucket exists (confirmed Jul 11) but empty — sync script not running
`itpropartner-docker-volumes`	Docker volume data: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama	🟡 Bucket exists (confirmed Jul 11) but empty — sync script not running
`itpropartner-backups`	Legacy files (signature images, misc) — data mostly migrated to purpose-specific buckets	Active since Jul 4

Note: hermes-backups (simple name) is taken globally on Wasabi. Always try an alternative like hermes-vps-backups or append a qualifier. When adding new buckets to the Wasabi IAM policy, the existing hermes policy uses two Statement blocks (one for bucket-level actions like ListBucket/GetBucketVersioning, one for object-level actions like GetObject/PutObject/DeleteObject). Add the new bucket ARN (with and without /*) to BOTH blocks. Test with aws s3 ls, aws s3 cp (PUT), aws s3 cp ... - (GET/readback), and aws s3 rm (DELETE) — probe files must be cleaned up after.

Current cron status

Hermes live sync: Every 15 min via hermes-live-sync.sh (no_agent, silent) → s3://hermes-vps-backups/live/
Hermes system config sync: Every 15 min via hermes-system-config-sync.sh (no_agent, silent) → s3://itpropartner-system-configs/
Hermes docker volume sync: Every 15 min via hermes-docker-sync.sh (no_agent, silent) → s3://itpropartner-docker-volumes/
Hermes full backup: Daily at 1:00 AM UTC via system crontab (NOT Hermes cron — blocked by gateway lifecycle guard #30719). Script: hermes-backup.sh.
Full backup audit watchdog: Daily at 2:00 AM UTC via system crontab. Script: backup-audit-check.sh.
Root essentials backup: Daily at 3:00 AM UTC via system crontab (42MB tarball). Script: root-essentials-backup.sh. S3: s3://hermes-vps-backups/root-backup/. Includes /root configs, skills, scripts, references, profiles, SSH keys, systemd services, Caddyfile, and /var/www. Corruption check: downloads + tar tzf verify. Restore: extract + live sync for state.db = ~10 min recovery. Fastest recovery path.
Router backup: Daily at 6:00 AM UTC (script: run-wisp-backup.sh → wisp-backup/wisp-backup.py)
Hetzner snapshots: Weekly Monday 5:00 AM UTC (script: snapshot-hetzner.py)

Live sync (every 15m)

A lightweight aws s3 sync that mirrors the entire .hermes/ directory (minus bulky caches like audio_cache/, image_cache/, cache/, sandboxes/, kanban/, node/, bin/, logs/, *.lock) to s3://hermes-vps-backups/live/.

Design principles:

Silent on success — the no_agent cron script suppresses output when nothing changed. Only failures produce output.
aws s3 sync handles deltas — only changed files upload. Typically <5 seconds per run.
RPO ~15 minutes — worst case you lose 15 min of state.db changes.

Script (hermes-live-sync.sh):

#!/bin/bash
HERMES_HOME="${HERMES_HOME:-$HOME/.hermes}"
ENDPOINT="https://s3.us-east-1.wasabisys.com"
BUCKET="hermes-vps-backups"
DEST_PREFIX="live"

if [ -f /opt/awscli-venv/bin/activate ]; then
    source /opt/awscli-venv/bin/activate
fi

aws s3 sync "$HERMES_HOME" "s3://$BUCKET/$DEST_PREFIX/" \
    --endpoint-url "$ENDPOINT" \
    --exclude "audio_cache/*" \
    --exclude "image_cache/*" \
    --exclude "cache/*" \
    --exclude "sandboxes/*" \
    --exclude "kanban/*" \
    --exclude "node/*" \
    --exclude "bin/*" \
    --exclude "logs/*" \
    --exclude "*.lock" \
    --exclude ".skills_prompt_snapshot.json" \
    --exclude ".update_check" \
    2>&1 | grep -v "^$" || true

# NOTE: Docker volumes moved to hermes-docker-sync.sh (itpropartner-docker-volumes)
# NOTE: Caddyfile moved to hermes-system-config-sync.sh (itpropartner-system-configs)
exit 0

Purpose-specific sync scripts

The monolithic hermes-live-sync.sh was split into purpose-specific scripts — see references/purpose-specific-buckets.md for details on the normalization and the script contents.

`hermes-system-config-sync.sh` → `s3://itpropartner-system-configs/`

Exports: Caddyfile, systemd service files (hermes*, mysql-tunnel, ollama, shark-game), ops scripts, SSH keys (non-known_hosts), .env + AWS credentials.

`hermes-docker-sync.sh` → `s3://itpropartner-docker-volumes/`

Exports: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama volume data — each under its own prefix.

DB Dump Automation (KNOWN GAP — see server-dr-plans-v3.md)

MariaDB on wphost02 (Apex) and fleettracker360 are NOT backed up via scheduled dump to S3. This is a known gap flagged in the Jul 10 DR audit. The backup standard declares daily DB dumps with 90-day retention, but this is not yet deployed.

Docker Volume Backup (KNOWN GAP — see server-dr-plans-v3.md)

Docker volume backups declared in the provisioning standard but not yet automated. All Docker hosts need volume backup scripts pushed to S3 per-server buckets.

Memory Consolidation Backup (NEW — Jul 9, 2026)

Memory is auto-consolidated every 10 min with backup-before-prune guarantee:

Backup to: s3://hermes-vps-backups/live/memories/memory-backup.json
History: s3://hermes-vps-backups/live/memories/history/memory-backup-<UTC>.json
Local retention: 48h under ~/.hermes/memories/.consolidate-backups/
Restore procedure: See devops/disaster-recovery-audit skill's Memory Consolidation Safety section

Creating the cron jobs

hermes cron create \
  --name "hermes-live-sync" \
  --schedule "every 15m" \
  --script hermes-live-sync.sh \
  --no_agent \
  --deliver local

Failover to cold spare (automatic warm standby)

When a cold/spare server exists (e.g. an old primary that was migrated off), a systemd oneshot service syncs the latest live state from S3 before Hermes starts. No manual steps needed — boot it and it picks up from the last 15-min checkpoint.

The restore script (hermes-standby-restore.sh):

Waits for network (up to 30s)
Pings the live netcup box (152.53.192.33) first — if it responds, exits 0 without starting Hermes (split-brain guard)
Loads AWS CLI
Runs aws s3 sync from s3://hermes-vps-backups/live/ to ~/.hermes/ (same exclusion patterns)
Starts Hermes gateway

Split-brain prevention is mandatory. The standby box and live box share the same Telegram bot token. If both run Hermes simultaneously, messages get lost and cron jobs fire from both. The ping check happens BEFORE the S3 sync or Hermes start.

Two failover mechanisms exist — boot-time (systemd oneshot) and periodic (cron watchdog). Both check the live box before acting.

Boot-time (hermes-standby-restore.sh via systemd):

Runs once on boot, before Hermes starts
Pings live box once — if alive, exits 0 (stays dormant)
If dead, syncs S3 state and starts Hermes
Designed for cold-start: power on the standby and it picks up

Periodic watchdog (hermes-standby-watchdog.sh via system crontab):

Runs every 10 min (system crontab, not Hermes cron — Hermes isn't running on standby)
Pings live box 3 times immediately
If all fail, runs 4 confirmation cycles at 60s intervals (~3.5 min total)
If still dead → sends Telegram alert + email alert, syncs S3, starts Hermes
Telegram alert format: 🚨 Hermes Failover — app1 (netcup) is offline\n\nTimestamp: ...\nAction: S3 sync → Hermes gateway start
Email alert: same content via SMTP (mail.germainebrown.com:587) with password from /root/.config/himalaya/g-germainebrown.pass
This catches the case where the live box dies while running (not just on boot)

Pre-register the standby SSH key in your cloud provider's project. Without it, you can't deploy the script. On Hetzner: POST /v1/ssh_keys with your public key.

The systemd unit (hermes-standby.service):

[Unit]
Description=Hermes Standby Restore — pull latest state from S3 before Hermes starts
Before=hermes.service
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/root/.hermes/scripts/hermes-standby-restore.sh
RemainAfterExit=true
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Deploy on standby box

wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
  https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
chmod +x /root/.hermes/scripts/hermes-standby-restore.sh
cp /root/.hermes/scripts/hermes-standby.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable hermes-standby.service

Pre-requisites for fast failover

S3 access keys available externally (password manager, not on the server being restored)
AWS CLI installed and configured on the standby box
Cold spare server already provisioned and powered on
SSH key registered in your cloud provider's project before attempting standby deployment

Standby deployment without SSH access (Hetzner rescue mode)

If you have no SSH access to the standby box (no key deployed), use Hetzner rescue mode:

Register your SSH key in the Hetzner project via API or Cloud Console — rescue mode accepts numeric key IDs, not raw keys:

curl -s -X POST https://api.hetzner.cloud/v1/ssh_keys \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"my-key","public_key":"ssh-ed25519 AAAA..."}'

Enable rescue mode with that key ID (the numeric ID from the response, not a string):

curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/enable_rescue \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"linux64","ssh_keys":[114708824]}'

Power on — boots into rescue with your key authorized as root:

curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/poweron \
  -H "Authorization: Bearer $TOKEN" -d '{}'

SSH into rescue as root:
```
ssh -i ~/.ssh/your_key root@$SERVER_IP
```

Mount the main partition and inject your SSH key into the OS:

mkdir -p /mnt/root
mount /dev/sda1 /mnt/root
mkdir -p /mnt/root/root/.ssh
echo "$PUBKEY" >> /mnt/root/root/.ssh/authorized_keys
chmod 600 /mnt/root/root/.ssh/authorized_keys

Disable the old Hermes service on the mounted disk so it doesn't autostart:

mount -t proc none /mnt/root/proc   # needed for chroot systemctl
chroot /mnt/root systemctl disable hermes-gateway.service

Deploy your standby scripts, then disable rescue + reboot via API:

curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/disable_rescue \
  -H "Authorization: Bearer $TOKEN" -d '{}'
curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/reboot \
  -H "Authorization: Bearer $TOKEN" -d '{}'

Pitfalls:

Register the SSH key FIRST. enable_rescue needs a numeric ssh_key_id. If you get "SSH key not found" (404), the key string was passed instead of the integer ID, or the key isn't in the project yet.
Rescue mode also sets a root password — the enable_rescue response includes a root_password field. If SSH key auth fails, use root + that password.
After disable_rescue + reboot, wait ~40s for the normal OS to boot before attempting SSH.
Hetzner Cloud API hostname != OS hostname — Renaming via PUT /v1/servers/{id} {"name":"new-name"} only changes Hetzner's label. The OS still has the old /etc/hostname. Fix after first SSH: hostnamectl set-hostname new-name.fqdn. Verify with both hostname and hostname -f.
chroot systemctl still works without /proc but warns "This is not a supported mode of operation." The enable/disable commands succeed despite the warning. Mount /proc if you want clean output.
Check server API status before attempting SSH — GET /v1/servers/{id} returns a status field. If it's off, re-power-on and wait.

DR issue log

Track all audit findings, root causes, fixes, and verification in references/dr-issue-log.md. Every DR audit produces a dated entry.

DR issue log formatting rules

Germaine views these files on mobile. Pipe tables (| col | col |) render unreadably on small screens.

DR issue log format:

**Fixed** [OK]
- **DR-001** [HIGH] Short title -> what was done

**Resolved** [INFO]
- **DR-009** Short title -> what was done

**Investigating** [PENDING]
- **DR-006** [HIGH] Short title -> what was done

Individual issue entries use:

**Problem**
Description...

**Root Cause**
Why it happened...

**Fix**
What was applied...

**Verification**
- [OK] Evidence item 1
- [OK] Evidence item 2

Sub-status tags use [OK], [PENDING], [HIGH], [MED], [INFO] — never use emoji (they garble on some mobile viewers). Use -> not →. Use -- not —. Keep it pure ASCII so any text editor or terminal can display it cleanly. Format per issue: problem, root cause, fix, status, verified-by.

Periodic backup health audit

Run the S3 bucket audit procedure to verify all three backup buckets are healthy — versioning on, files being written, no silent pipeline failures.

The procedure covers:

Versioning status check on every bucket
File counts and total size
Staleness detection (objects from today vs >48h gap)
Live sync freshness verification (count today's objects in the live/ prefix)
Full backup archive age check
Common failure patterns with fixes

See references/backup-audit-procedure.md for the full quick-audit script and per-bucket methodology.

Weekly audit recommended. The MikroTik router backup pipeline once went 3 days without producing configs before being noticed — routine audits catch these before data loss becomes a risk.

Verify backup costs from invoices

To confirm backups are running within expected resource usage, extract Hetzner invoices:

apt-get install -y poppler-utils
pdftotext /path/to/invoice.pdf -

Hetzner invoice PDFs line items include: server type counts, unit prices, backup percentage (20% of instance price), and snapshot storage (GB-months). Also check Hetzner invoice line items: if backup charges appear but aren't on the right products, the backup config may need updating.

Setup

Create a Wasabi IAM user (subuser, not root). See infrastructure-automation skill's references/wasabi-iam-setup.md for the full policy and common pitfalls.
Create the Wasabi bucket (names are globally unique — if taken, try alternatives like hermes-vps-backups).
Edit /root/.hermes/scripts/hermes-backup.sh:
- BUCKET: set to the actual bucket name
- BACKUP_DIR and ARCHIVE: MUST use a root-disk path (e.g. ~/.hermes/.backups/), NOT /tmp (tmpfs is RAM-backed and fills up)
- The tar czf command runs from $(dirname "$BACKUP_DIR") — ensure the script's cd target matches the staging dir's actual parent, not /tmp
Ensure ~/.aws/credentials exists with the IAM user's access key + secret key, chmod 600
Test: Run bash /root/.hermes/scripts/hermes-backup-worker.sh or simulate with a dry aws s3 ls on each bucket (see Pitfalls section about ListBuckets). The first real backup fires at the next cron tick.
Create the live sync cron job (see Live sync section above)
If a cold spare exists, deploy the standby restore (see Failover section above)
Cron job handles daily runs automatically

Secret consolidation workflow

All Hermes secrets should live in a single ~/.hermes/.env file (chmod 600). Standalone secret files (scripts/.hetzner_token, scripts/.netcup_api_key, etc.) get lost during migration and forgotten during key rotation.

Standard consolidation steps:

Read existing .env and load into a dict
Read each standalone token file
Add them to the dict with descriptive names (HETZNER_API_TOKEN, NETCUP_API_KEY, ADMIN_AI_API_KEY, etc.)
Write the full dict back to .env
chmod 600 ~/.hermes/.env
rm the standalone files
Patch any scripts that referenced the old file paths to source from .env instead
Verify: stat -c '%a' ~/.hermes/.env → should show 600

Custom-provider credential rule: Do not assume an admin-ai key is still in config.yaml, .env, or auth.json. Inspect all three without exposing values. auth.json may retain only a fingerprint after the actual provider and secret were removed. Store the recoverable secret in .env using a descriptive variable such as ADMIN_AI_API_KEY, configure the provider through supported Hermes fields, and verify /models plus a real completion before calling it restored. If auxiliary tasks use a separate key field, verify that copy independently.

Post-migration verification

After restoring Hermes on a new server, verify these in order:

S3/Wasabi access

source /opt/awscli-venv/bin/activate
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
aws s3 ls s3://hermes-vps-backups/live/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | head -10
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5
aws s3 ls s3://itpropartner-system-configs/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://itpropartner-docker-volumes/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://itpropartner-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1

Pitfall — stale credentials: aws configure list can show credentials present but aws s3 ls returns InvalidAccessKeyId. This means the IAM key was rotated on the old server but ~/.aws/credentials on this server has the stale version. Get the current key from the password manager, update the file, and retest. The file is a simple 3-line INI — just aws_access_key_id and aws_secret_access_key under [default].

LLM provider availability

If the provider is a proxy (e.g., admin-ai routing to multiple backends), verify OpenRouter models are reachable:

curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"

Check for openrouter/* entries in the output. The proxy should expose 80+ models including OpenAI, Claude, Gemini, DeepSeek, etc.

Pitfall — mismatched API keys: The API key appears in config.yaml in two places — providers.admin-ai.api_key and auxiliary.vision.api_key. If the proxy rotated its key, both must be updated. The curl test uses the provider key (first hit), but the vision auxiliary provider may have the old key separately.

IMAP connectivity check

If email triage jobs exist, verify IMAP server reachable proactively:

python3 -c "
import ssl, socket
ctx = ssl.create_default_context()
s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
print(f'IMAP: {ss.version()}')
ss.close()
"

Also verify password files present: ls ~/.config/himalaya/*.pass 2>/dev/null

Heremes gateway health

hermes gateway status
journalctl -u hermes -n 10 --no-pager

Cron job restoration

hermes cron list

Should show all jobs. If empty, wait 30-60s for scheduler to discover them from state.db.

Restore on new server (full replacement)

# Install Hermes first
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

# Download latest backup
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-YYYY-MM-DD.tar.gz . \
    --endpoint-url https://s3.us-east-1.wasabisys.com

# Extract and restore
tar xzf hermes-full-backup-YYYY-MM-DD.tar.gz
cd hermes-backup-YYYY-MM-DD
bash restore.sh

# Then set up live sync and warm standby (see sections above)
# hermes cron create --name "hermes-live-sync" --schedule "every 15m" --script hermes-live-sync.sh --no_agent --deliver local

# Start gateways
hermes gateway restart
hermes profile use anita && anita gateway restart  # if Anita exists

36 KiB Raw Permalink Blame History