Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,287 @@
+---
+name: hermes-migration
+description: "Migrate a Hermes Agent installation between servers — data transfer, state verification, known pitfalls."
+version: 2.2.0
+author: Hermes Agent
+license: MIT
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [hermes, migration, server, deployment, vps, backup, restore]
+    related_skills: [hermes-agent]
+---
+
+# Hermes Migration
+
+Migrate a running Hermes Agent installation (config, sessions, skills, memory, cron, profiles) from one server to another. Covers the VPS-to-VPS migration that happens when upgrading hardware, switching providers (Hetzner → netcup, etc.), or reprovisioning.
+
+## Scope
+
+This skill handles the **data plane** migration of Hermes itself — config, state, skills, memory, cron, profiles. It does not cover:
+
+- DNS / SSH key / firewall reconfiguration on the new server
+- Reverse proxy (nginx/Caddy) transfer
+- OS-level package installation (pip, apt, etc.) — assume Hermes is already installed
+- Application-level migration (your code repos, databases, services)
+
+## The Migration Pattern
+
+The basic pattern is:
+
+1. **Archive origin** — tar.gz of `~/.hermes/`
+2. **Copy to destination** — scp/rsync
+3. **Extract on destination** — overwrite `~/.hermes/`
+4. **Verify** — check state DB, skills, cron, gateway
+
+## Cron Jobs — They Survive (corrected v2.0)
+
+**Cron jobs DO survive a raw `~/.hermes/` copy.** The cron scheduler stores job definitions inside `state.db`, which is part of the standard `~/.hermes/` tarball. Verified: 8 cron jobs (email triage, backups, router watchdog, tech digest, brother torment, Hetzner snapshots) migrated intact with a simple `tar czf ~/.hermes/` → `scp` → `tar xzf` transfer on Hermes v0.18.0.
+
+After extraction:
+1. **Wait for the ticker** — It runs on its own schedule (not immediately). The heartbeat file (`ticker_heartbeat`, `ticker_last_success`) will update within ~1 minute.
+2. **Verify with `hermes cron list`** — All jobs show up. No separate export/import needed.
+3. **Check job details** — `hermes cron update <job-id>` or inspect individual job flags to confirm schedule, skills, script, delivery, and no_agent mode are intact.
+
+**One caveat:** The ticker heartbeat and last-success timestamps are ephemeral — they reset after the move. The first scheduled run will show as a fresh tick. This is cosmetic, not a data loss.
+
+**Pitfall:** If the `hermes cron list` check returns empty jobs immediately after migration, wait 30-60s and retry. The scheduler needs time to discover existing jobs in the DB after startup.
+
+## Verification Checklist (run each after migration)
+
+### 1. State DB integrity
+
+```bash
+python3 -c "
+import sqlite3
+db = sqlite3.connect('/root/.hermes/state.db')
+c = db.cursor()
+c.execute('SELECT COUNT(*) FROM messages')
+print('Messages:', c.fetchone()[0])
+c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
+print('Sessions:', c.fetchone()[0])
+c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
+row = c.fetchone()
+import datetime
+print('Range:', datetime.datetime.fromtimestamp(row[0]).strftime('%Y-%m-%d'), '->', datetime.datetime.fromtimestamp(row[1]).strftime('%Y-%m-%d'))
+"
+```
+
+### 2. Skills integrity
+
+```bash
+ls -1 ~/.hermes/skills/ | wc -l
+hermes skills list      # should match source count
+```
+
+### 3. Cron jobs
+
+```bash
+hermes cron list        # if empty, jobs were lost — reconstruct
+```
+
+### 4. Memory store
+
+```bash
+python3 -c "
+import sqlite3
+db = sqlite3.connect('/root/.hermes/memory_store.db')
+c = db.cursor()
+tables = c.execute(\"SELECT name FROM sqlite_master WHERE type='table'\").fetchall()
+print('Tables:', [t[0] for t in tables])
+"
+```
+
+### 5. Gateway health
+
+```bash
+hermes gateway status
+# Check .env has correct platform tokens
+cat ~/.hermes/.env | head -10
+```
+
+### 6. S3 (Wasabi) bucket access
+
+The `aws` binary is in a virtualenv — always activate first:
+```bash
+source /opt/awscli-venv/bin/activate
+```
+
+Then test each bucket:
+```bash
+aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
+    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
+aws s3 ls s3://itpropartner-backups/ \
+    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
+aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
+    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5
+```
+
+**Note:** `ListBuckets` (listing all buckets) returns `AccessDenied` — expected. Test by name.
+
+**Pitfall — stale credentials that appear valid:** `aws configure list` can show credentials present, but `aws s3 ls s3://bucket-name/` returns `InvalidAccessKeyId`. This means the IAM user's key was rotated on the old server but `~/.aws/credentials` on the new server has the stale version. The file is a 3-line text file — if it migrated correctly it has the current key. If not, get the current key from the password manager and update the file.
+
+```bash
+# Verify credential file exists with non-zero size
+ls -la ~/.aws/credentials
+# Test against a known bucket — not ListBuckets
+aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
+```
+
+```bash
+hermes doctor
+hermes config check
+```
+
+### 7. LLM provider / proxy availability
+
+If the provider is a proxy that routes to multiple backends (e.g. admin-ai → OpenRouter), verify model access:
+
+```bash
+curl -s https://admin-ai.itpropartner.com/v1/models \
+  -H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"
+```
+
+Expected: 80+ models including `openrouter/*`, `openrouter/anthropic/claude-*`, `openrouter/openai/gpt-*`, etc. Key indicator: `openrouter/*` as a catch-all model.
+
+**Pitfall — stale API key on provider proxy:** If the models endpoint returns `Authentication Error, Invalid proxy server token passed`, the API key in `config.yaml` was rotated on the proxy side. Check with `grep 'api_key:' ~/.hermes/config.yaml | head -1`. The key appears in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. Both must match the current proxy key. Fix both, then `hermes gateway restart`.
+
+### 8. IMAP / email triage state files (critical)
+
+**Symptom:** After migration, the hourly IMAP triage cron fails with `RuntimeError: Context length exceeded (X tokens). Cannot compress further.`
+
+**Root cause:** `~/.hermes/email_triage/state.json` tracks processed UIDs. If this file is lost during migration, the triage script treats every inbox message as new — a large inbox (3000+) blows the context window on first run.
+
+**Fix:** See `references/email-triage-state-migration.md` for the full recovery procedure. Short version: copy the file from the old server or the standby, or manually initialize state by marking all but the last ~50 UIDs as processed.
+
+```bash
+# Check if old server or standby still has the file
+scp root@old-server-ip:/root/.hermes/email_triage/state.json /root/.hermes/email_triage/
+```
+
+**Prevention:** Add `email_triage/state.json` and `email_triage/actions.jsonl` to the pre-migration manifest.
+
+### 9. IMAP / email server check
+
+If email triage jobs exist, verify the IMAP server is reachable (the cron will surface failures, but it's faster to check proactively):
+
+```bash
+python3 -c "
+import ssl, socket
+ctx = ssl.create_default_context()
+s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
+ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
+print(f'IMAP reachable: {ss.version()}')
+ss.close()
+"
+```
+
+Also verify Himalaya password files are present:
+```bash
+ls ~/.config/himalaya/*.pass 2>/dev/null
+```
+
+### 11. Profile state completeness (Anita, other profiles)
+
+Each profile has its own `state.db` under `~/.hermes/profiles/<name>/state.db`. After migration, compare profile database sizes against the original server:
+
+```bash
+# On destination
+ls -la ~/.hermes/profiles/*/state.db
+
+# Compare sizes — a small state.db (e.g. 1.1 MB vs 1.9 MB) suggests partial data
+```
+
+**Symptom:** User reports conversation history is missing. Profile `state.db` is smaller on destination than it was on origin.
+
+**Root cause:** `~/.hermes/profiles/<name>/state.db` may not have fully copied during the initial `tar`/`scp` transfer. The 15-min S3 live backup (`s3://hermes-vps-backups/live/profiles/<name>/state.db`) may also carry the partial version since it captures the destination server's state.
+
+**Fix:** Pull from the standby server's copy:
+
+```bash
+# Check standby (app1-bu)
+ssh -i ~/.ssh/itpp-infra root@5.161.114.8 "ls -la ~/.hermes/profiles/<name>/state.db"
+
+# If larger, pull it
+scp -i ~/.ssh/itpp-infra root@5.161.114.8:/root/.hermes/profiles/<name>/state.db \
+  ~/.hermes/profiles/<name>/state.db
+```
+
+Then swap the files (standby is not running, so no gateway restart needed on standby):
+
+```bash
+mv ~/.hermes/profiles/<name>/state.db ~/.hermes/profiles/<name>/state.db.partial
+# The pulled file is already at ~/.hermes/profiles/<name>/state.db from scp above
+```
+
+Verify post-swap:
+
+```bash
+python3 -c "
+import sqlite3, datetime
+db = sqlite3.connect('/root/.hermes/profiles/anita/state.db')
+c = db.cursor()
+c.execute('SELECT COUNT(*) FROM messages')
+msgs = c.fetchone()[0]
+c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
+sessions = c.fetchone()[0]
+c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
+dr = c.fetchone()
+print(f'Messages: {msgs}, Sessions: {sessions}')
+print(f'Date range: {datetime.datetime.fromtimestamp(dr[0])} → {datetime.datetime.fromtimestamp(dr[1])}')
+"
+```
+
+**Prevention:** Add profile state.db to the pre-migration manifest. After initial migration and gateway verification, run the profile completeness check above. If the standby has the full data, pull immediately.
+
+**Full recovery procedure from standby:** `skill_view("hermes-migration", "references/profile-recovery-from-standby.md")`
+
+### 12. Cross-reference: hermes-backup skill
+
+After migration, the `hermes-backup` skill (category: devops) contains the canonical backup pipeline documentation, including:
+- The full `hermes-backup.sh` script pattern
+- Restore and cold-spare failover procedures
+- S3 bucket structure and IAM setup
+
+If this migration was triggered by a hardware change, run the backup skill's post-migration verification immediately. The first scheduled backup (typically 05:00 UTC) may fail if S3 credentials were not carried over.
+
+## Full Transfer Reference
+
+```bash
+# ON ORIGIN SERVER:
+tar czf ~/hermes-backup-$(date +%F).tar.gz -C /root .hermes/
+
+# COPY TO DESTINATION:
+scp ~/hermes-backup-*.tar.gz root@new-server:~/
+
+# ON DESTINATION SERVER:
+rm -rf ~/.hermes_bak 2>/dev/null
+mv ~/.hermes ~/.hermes_bak 2>/dev/null || true
+tar xzf ~/hermes-backup-*.tar.gz -C /root/
+
+# Restart gateway
+hermes gateway restart
+
+# Verify (see checklist above)
+```
+
+## Session History
+
+The state DB (`~/.hermes/state.db`, typically 500MB–2GB for active installations) carries:
+
+- All conversation messages with FTS5 search indexes
+- Session metadata (titles, timestamps, profiles)
+- Compression locks and state tracking
+
+This ALL transfers via the simple `~/.hermes/` copy above. No separate export/import needed for the DB itself.
+
+## Profiles
+
+Profiles live in `~/.hermes/profiles/<name>/`. Each has its own:
+
+- `config.yaml` and `.env`
+- `state.db` (separate session store)
+- `skills/` (profile-scoped skills)
+- `memories/` (profile-scoped memory files)
+- `cron/` (profile-scoped cron outputs)
+
+All transfer with the top-level `~/.hermes/` tarball.