ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-skills/skills/autonomous-ai-agents/hermes-migration/SKILL.md
T

11 KiB
Raw Permalink Blame History

name, description, version, author, license, platforms, metadata
name description version author license platforms metadata
hermes-migration Migrate a Hermes Agent installation between servers — data transfer, state verification, known pitfalls. 2.2.0 Hermes Agent MIT
linux
hermes
tags related_skills
hermes
migration
server
deployment
vps
backup
restore
hermes-agent

Hermes Migration

Migrate a running Hermes Agent installation (config, sessions, skills, memory, cron, profiles) from one server to another. Covers the VPS-to-VPS migration that happens when upgrading hardware, switching providers (Hetzner → netcup, etc.), or reprovisioning.

Scope

This skill handles the data plane migration of Hermes itself — config, state, skills, memory, cron, profiles. It does not cover:

  • DNS / SSH key / firewall reconfiguration on the new server
  • Reverse proxy (nginx/Caddy) transfer
  • OS-level package installation (pip, apt, etc.) — assume Hermes is already installed
  • Application-level migration (your code repos, databases, services)

The Migration Pattern

The basic pattern is:

  1. Archive origin — tar.gz of ~/.hermes/
  2. Copy to destination — scp/rsync
  3. Extract on destination — overwrite ~/.hermes/
  4. Verify — check state DB, skills, cron, gateway

Cron Jobs — They Survive (corrected v2.0)

Cron jobs DO survive a raw ~/.hermes/ copy. The cron scheduler stores job definitions inside state.db, which is part of the standard ~/.hermes/ tarball. Verified: 8 cron jobs (email triage, backups, router watchdog, tech digest, brother torment, Hetzner snapshots) migrated intact with a simple tar czf ~/.hermes/ → scp → tar xzf transfer on Hermes v0.18.0.

After extraction:

  1. Wait for the ticker — It runs on its own schedule (not immediately). The heartbeat file (ticker_heartbeat, ticker_last_success) will update within ~1 minute.
  2. Verify with hermes cron list — All jobs show up. No separate export/import needed.
  3. Check job details — hermes cron update <job-id> or inspect individual job flags to confirm schedule, skills, script, delivery, and no_agent mode are intact.

One caveat: The ticker heartbeat and last-success timestamps are ephemeral — they reset after the move. The first scheduled run will show as a fresh tick. This is cosmetic, not a data loss.

Pitfall: If the hermes cron list check returns empty jobs immediately after migration, wait 30-60s and retry. The scheduler needs time to discover existing jobs in the DB after startup.

Verification Checklist (run each after migration)

1. State DB integrity

python3 -c "
import sqlite3
db = sqlite3.connect('/root/.hermes/state.db')
c = db.cursor()
c.execute('SELECT COUNT(*) FROM messages')
print('Messages:', c.fetchone()[0])
c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
print('Sessions:', c.fetchone()[0])
c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
row = c.fetchone()
import datetime
print('Range:', datetime.datetime.fromtimestamp(row[0]).strftime('%Y-%m-%d'), '->', datetime.datetime.fromtimestamp(row[1]).strftime('%Y-%m-%d'))
"

2. Skills integrity

ls -1 ~/.hermes/skills/ | wc -l
hermes skills list      # should match source count

3. Cron jobs

hermes cron list        # if empty, jobs were lost — reconstruct

4. Memory store

python3 -c "
import sqlite3
db = sqlite3.connect('/root/.hermes/memory_store.db')
c = db.cursor()
tables = c.execute(\"SELECT name FROM sqlite_master WHERE type='table'\").fetchall()
print('Tables:', [t[0] for t in tables])
"

5. Gateway health

hermes gateway status
# Check .env has correct platform tokens
cat ~/.hermes/.env | head -10

6. S3 (Wasabi) bucket access

The aws binary is in a virtualenv — always activate first:

source /opt/awscli-venv/bin/activate

Then test each bucket:

aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
aws s3 ls s3://itpropartner-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5

Note: ListBuckets (listing all buckets) returns AccessDenied — expected. Test by name.

Pitfall — stale credentials that appear valid: aws configure list can show credentials present, but aws s3 ls s3://bucket-name/ returns InvalidAccessKeyId. This means the IAM user's key was rotated on the old server but ~/.aws/credentials on the new server has the stale version. The file is a 3-line text file — if it migrated correctly it has the current key. If not, get the current key from the password manager and update the file.

# Verify credential file exists with non-zero size
ls -la ~/.aws/credentials
# Test against a known bucket — not ListBuckets
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1

hermes doctor
hermes config check

7. LLM provider / proxy availability

If the provider is a proxy that routes to multiple backends (e.g. admin-ai → OpenRouter), verify model access:

curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"

Expected: 80+ models including openrouter/*, openrouter/anthropic/claude-*, openrouter/openai/gpt-*, etc. Key indicator: openrouter/* as a catch-all model.

Pitfall — stale API key on provider proxy: If the models endpoint returns Authentication Error, Invalid proxy server token passed, the API key in config.yaml was rotated on the proxy side. Check with grep 'api_key:' ~/.hermes/config.yaml | head -1. The key appears in config.yaml in two places — providers.admin-ai.api_key and auxiliary.vision.api_key. Both must match the current proxy key. Fix both, then hermes gateway restart.

8. IMAP / email triage state files (critical)

Symptom: After migration, the hourly IMAP triage cron fails with RuntimeError: Context length exceeded (X tokens). Cannot compress further.

Root cause: ~/.hermes/email_triage/state.json tracks processed UIDs. If this file is lost during migration, the triage script treats every inbox message as new — a large inbox (3000+) blows the context window on first run.

Fix: See references/email-triage-state-migration.md for the full recovery procedure. Short version: copy the file from the old server or the standby, or manually initialize state by marking all but the last ~50 UIDs as processed.

# Check if old server or standby still has the file
scp root@old-server-ip:/root/.hermes/email_triage/state.json /root/.hermes/email_triage/

Prevention: Add email_triage/state.json and email_triage/actions.jsonl to the pre-migration manifest.

9. IMAP / email server check

If email triage jobs exist, verify the IMAP server is reachable (the cron will surface failures, but it's faster to check proactively):

python3 -c "
import ssl, socket
ctx = ssl.create_default_context()
s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
print(f'IMAP reachable: {ss.version()}')
ss.close()
"

Also verify Himalaya password files are present:

ls ~/.config/himalaya/*.pass 2>/dev/null

11. Profile state completeness (Anita, other profiles)

Each profile has its own state.db under ~/.hermes/profiles/<name>/state.db. After migration, compare profile database sizes against the original server:

# On destination
ls -la ~/.hermes/profiles/*/state.db

# Compare sizes — a small state.db (e.g. 1.1 MB vs 1.9 MB) suggests partial data

Symptom: User reports conversation history is missing. Profile state.db is smaller on destination than it was on origin.

Root cause: ~/.hermes/profiles/<name>/state.db may not have fully copied during the initial tar/scp transfer. The 15-min S3 live backup (s3://hermes-vps-backups/live/profiles/<name>/state.db) may also carry the partial version since it captures the destination server's state.

Fix: Pull from the standby server's copy:

# Check standby (app1-bu)
ssh -i ~/.ssh/itpp-infra root@5.161.114.8 "ls -la ~/.hermes/profiles/<name>/state.db"

# If larger, pull it
scp -i ~/.ssh/itpp-infra root@5.161.114.8:/root/.hermes/profiles/<name>/state.db \
  ~/.hermes/profiles/<name>/state.db

Then swap the files (standby is not running, so no gateway restart needed on standby):

mv ~/.hermes/profiles/<name>/state.db ~/.hermes/profiles/<name>/state.db.partial
# The pulled file is already at ~/.hermes/profiles/<name>/state.db from scp above

Verify post-swap:

python3 -c "
import sqlite3, datetime
db = sqlite3.connect('/root/.hermes/profiles/anita/state.db')
c = db.cursor()
c.execute('SELECT COUNT(*) FROM messages')
msgs = c.fetchone()[0]
c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
sessions = c.fetchone()[0]
c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
dr = c.fetchone()
print(f'Messages: {msgs}, Sessions: {sessions}')
print(f'Date range: {datetime.datetime.fromtimestamp(dr[0])} → {datetime.datetime.fromtimestamp(dr[1])}')
"

Prevention: Add profile state.db to the pre-migration manifest. After initial migration and gateway verification, run the profile completeness check above. If the standby has the full data, pull immediately.

Full recovery procedure from standby: skill_view("hermes-migration", "references/profile-recovery-from-standby.md")

12. Cross-reference: hermes-backup skill

After migration, the hermes-backup skill (category: devops) contains the canonical backup pipeline documentation, including:

  • The full hermes-backup.sh script pattern
  • Restore and cold-spare failover procedures
  • S3 bucket structure and IAM setup

If this migration was triggered by a hardware change, run the backup skill's post-migration verification immediately. The first scheduled backup (typically 05:00 UTC) may fail if S3 credentials were not carried over.

Full Transfer Reference

# ON ORIGIN SERVER:
tar czf ~/hermes-backup-$(date +%F).tar.gz -C /root .hermes/

# COPY TO DESTINATION:
scp ~/hermes-backup-*.tar.gz root@new-server:~/

# ON DESTINATION SERVER:
rm -rf ~/.hermes_bak 2>/dev/null
mv ~/.hermes ~/.hermes_bak 2>/dev/null || true
tar xzf ~/hermes-backup-*.tar.gz -C /root/

# Restart gateway
hermes gateway restart

# Verify (see checklist above)

Session History

The state DB (~/.hermes/state.db, typically 500MB–2GB for active installations) carries:

  • All conversation messages with FTS5 search indexes
  • Session metadata (titles, timestamps, profiles)
  • Compression locks and state tracking

This ALL transfers via the simple ~/.hermes/ copy above. No separate export/import needed for the DB itself.

Profiles

Profiles live in ~/.hermes/profiles/<name>/. Each has its own:

  • config.yaml and .env
  • state.db (separate session store)
  • skills/ (profile-scoped skills)
  • memories/ (profile-scoped memory files)
  • cron/ (profile-scoped cron outputs)

All transfer with the top-level ~/.hermes/ tarball.

Reference in New Issue View Git Blame Copy Permalink