hermes-skills/skills/autonomous-ai-agents/hermes-migration/SKILL.md

---
name: hermes-migration
description: "Migrate a Hermes Agent installation between servers — data transfer, state verification, known pitfalls."
version: 2.2.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [hermes, migration, server, deployment, vps, backup, restore]
    related_skills: [hermes-agent]
---

# Hermes Migration

Migrate a running Hermes Agent installation (config, sessions, skills, memory, cron, profiles) from one server to another. Covers the VPS-to-VPS migration that happens when upgrading hardware, switching providers (Hetzner → netcup, etc.), or reprovisioning.

## Scope

This skill handles the **data plane** migration of Hermes itself — config, state, skills, memory, cron, profiles. It does not cover:

- DNS / SSH key / firewall reconfiguration on the new server
- Reverse proxy (nginx/Caddy) transfer
- OS-level package installation (pip, apt, etc.) — assume Hermes is already installed
- Application-level migration (your code repos, databases, services)

## The Migration Pattern

The basic pattern is:

1. **Archive origin** — tar.gz of `~/.hermes/`
2. **Copy to destination** — scp/rsync
3. **Extract on destination** — overwrite `~/.hermes/`
4. **Verify** — check state DB, skills, cron, gateway

## Cron Jobs — They Survive (corrected v2.0)

**Cron jobs DO survive a raw `~/.hermes/` copy.** The cron scheduler stores job definitions inside `state.db`, which is part of the standard `~/.hermes/` tarball. Verified: 8 cron jobs (email triage, backups, router watchdog, tech digest, brother torment, Hetzner snapshots) migrated intact with a simple `tar czf ~/.hermes/` → `scp` → `tar xzf` transfer on Hermes v0.18.0.

After extraction:
1. **Wait for the ticker** — It runs on its own schedule (not immediately). The heartbeat file (`ticker_heartbeat`, `ticker_last_success`) will update within ~1 minute.
2. **Verify with `hermes cron list`** — All jobs show up. No separate export/import needed.
3. **Check job details** — `hermes cron update <job-id>` or inspect individual job flags to confirm schedule, skills, script, delivery, and no_agent mode are intact.

**One caveat:** The ticker heartbeat and last-success timestamps are ephemeral — they reset after the move. The first scheduled run will show as a fresh tick. This is cosmetic, not a data loss.

**Pitfall:** If the `hermes cron list` check returns empty jobs immediately after migration, wait 30-60s and retry. The scheduler needs time to discover existing jobs in the DB after startup.

## Verification Checklist (run each after migration)

### 1. State DB integrity

```bash
python3 -c "
import sqlite3
db = sqlite3.connect('/root/.hermes/state.db')
c = db.cursor()
c.execute('SELECT COUNT(*) FROM messages')
print('Messages:', c.fetchone()[0])
c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
print('Sessions:', c.fetchone()[0])
c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
row = c.fetchone()
import datetime
print('Range:', datetime.datetime.fromtimestamp(row[0]).strftime('%Y-%m-%d'), '->', datetime.datetime.fromtimestamp(row[1]).strftime('%Y-%m-%d'))
"
```

### 2. Skills integrity

```bash
ls -1 ~/.hermes/skills/ | wc -l
hermes skills list      # should match source count
```

### 3. Cron jobs

```bash
hermes cron list        # if empty, jobs were lost — reconstruct
```

### 4. Memory store

```bash
python3 -c "
import sqlite3
db = sqlite3.connect('/root/.hermes/memory_store.db')
c = db.cursor()
tables = c.execute(\"SELECT name FROM sqlite_master WHERE type='table'\").fetchall()
print('Tables:', [t[0] for t in tables])
"
```

### 5. Gateway health

```bash
hermes gateway status
# Check .env has correct platform tokens
cat ~/.hermes/.env | head -10
```

### 6. S3 (Wasabi) bucket access

The `aws` binary is in a virtualenv — always activate first:
```bash
source /opt/awscli-venv/bin/activate
```

Then test each bucket:
```bash
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
aws s3 ls s3://itpropartner-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
    --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5
```

**Note:** `ListBuckets` (listing all buckets) returns `AccessDenied` — expected. Test by name.

**Pitfall — stale credentials that appear valid:** `aws configure list` can show credentials present, but `aws s3 ls s3://bucket-name/` returns `InvalidAccessKeyId`. This means the IAM user's key was rotated on the old server but `~/.aws/credentials` on the new server has the stale version. The file is a 3-line text file — if it migrated correctly it has the current key. If not, get the current key from the password manager and update the file.

```bash
# Verify credential file exists with non-zero size
ls -la ~/.aws/credentials
# Test against a known bucket — not ListBuckets
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
```

```bash
hermes doctor
hermes config check
```

### 7. LLM provider / proxy availability

If the provider is a proxy that routes to multiple backends (e.g. admin-ai → OpenRouter), verify model access:

```bash
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"
```

Expected: 80+ models including `openrouter/*`, `openrouter/anthropic/claude-*`, `openrouter/openai/gpt-*`, etc. Key indicator: `openrouter/*` as a catch-all model.

**Pitfall — stale API key on provider proxy:** If the models endpoint returns `Authentication Error, Invalid proxy server token passed`, the API key in `config.yaml` was rotated on the proxy side. Check with `grep 'api_key:' ~/.hermes/config.yaml | head -1`. The key appears in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. Both must match the current proxy key. Fix both, then `hermes gateway restart`.

### 8. IMAP / email triage state files (critical)

**Symptom:** After migration, the hourly IMAP triage cron fails with `RuntimeError: Context length exceeded (X tokens). Cannot compress further.`

**Root cause:** `~/.hermes/email_triage/state.json` tracks processed UIDs. If this file is lost during migration, the triage script treats every inbox message as new — a large inbox (3000+) blows the context window on first run.

**Fix:** See `references/email-triage-state-migration.md` for the full recovery procedure. Short version: copy the file from the old server or the standby, or manually initialize state by marking all but the last ~50 UIDs as processed.

```bash
# Check if old server or standby still has the file
scp root@old-server-ip:/root/.hermes/email_triage/state.json /root/.hermes/email_triage/
```

**Prevention:** Add `email_triage/state.json` and `email_triage/actions.jsonl` to the pre-migration manifest.

### 9. IMAP / email server check

If email triage jobs exist, verify the IMAP server is reachable (the cron will surface failures, but it's faster to check proactively):

```bash
python3 -c "
import ssl, socket
ctx = ssl.create_default_context()
s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
print(f'IMAP reachable: {ss.version()}')
ss.close()
"
```

Also verify Himalaya password files are present:
```bash
ls ~/.config/himalaya/*.pass 2>/dev/null
```

### 11. Profile state completeness (Anita, other profiles)

Each profile has its own `state.db` under `~/.hermes/profiles/<name>/state.db`. After migration, compare profile database sizes against the original server:

```bash
# On destination
ls -la ~/.hermes/profiles/*/state.db

# Compare sizes — a small state.db (e.g. 1.1 MB vs 1.9 MB) suggests partial data
```

**Symptom:** User reports conversation history is missing. Profile `state.db` is smaller on destination than it was on origin.

**Root cause:** `~/.hermes/profiles/<name>/state.db` may not have fully copied during the initial `tar`/`scp` transfer. The 15-min S3 live backup (`s3://hermes-vps-backups/live/profiles/<name>/state.db`) may also carry the partial version since it captures the destination server's state.

**Fix:** Pull from the standby server's copy:

```bash
# Check standby (app1-bu)
ssh -i ~/.ssh/itpp-infra root@5.161.114.8 "ls -la ~/.hermes/profiles/<name>/state.db"

# If larger, pull it
scp -i ~/.ssh/itpp-infra root@5.161.114.8:/root/.hermes/profiles/<name>/state.db \
  ~/.hermes/profiles/<name>/state.db
```

Then swap the files (standby is not running, so no gateway restart needed on standby):

```bash
mv ~/.hermes/profiles/<name>/state.db ~/.hermes/profiles/<name>/state.db.partial
# The pulled file is already at ~/.hermes/profiles/<name>/state.db from scp above
```

Verify post-swap:

```bash
python3 -c "
import sqlite3, datetime
db = sqlite3.connect('/root/.hermes/profiles/anita/state.db')
c = db.cursor()
c.execute('SELECT COUNT(*) FROM messages')
msgs = c.fetchone()[0]
c.execute('SELECT COUNT(DISTINCT session_id) FROM messages')
sessions = c.fetchone()[0]
c.execute('SELECT MIN(timestamp), MAX(timestamp) FROM messages')
dr = c.fetchone()
print(f'Messages: {msgs}, Sessions: {sessions}')
print(f'Date range: {datetime.datetime.fromtimestamp(dr[0])} → {datetime.datetime.fromtimestamp(dr[1])}')
"
```

**Prevention:** Add profile state.db to the pre-migration manifest. After initial migration and gateway verification, run the profile completeness check above. If the standby has the full data, pull immediately.

**Full recovery procedure from standby:** `skill_view("hermes-migration", "references/profile-recovery-from-standby.md")`

### 12. Cross-reference: hermes-backup skill

After migration, the `hermes-backup` skill (category: devops) contains the canonical backup pipeline documentation, including:
- The full `hermes-backup.sh` script pattern
- Restore and cold-spare failover procedures
- S3 bucket structure and IAM setup

If this migration was triggered by a hardware change, run the backup skill's post-migration verification immediately. The first scheduled backup (typically 05:00 UTC) may fail if S3 credentials were not carried over.

## Full Transfer Reference

```bash
# ON ORIGIN SERVER:
tar czf ~/hermes-backup-$(date +%F).tar.gz -C /root .hermes/

# COPY TO DESTINATION:
scp ~/hermes-backup-*.tar.gz root@new-server:~/

# ON DESTINATION SERVER:
rm -rf ~/.hermes_bak 2>/dev/null
mv ~/.hermes ~/.hermes_bak 2>/dev/null || true
tar xzf ~/hermes-backup-*.tar.gz -C /root/

# Restart gateway
hermes gateway restart

# Verify (see checklist above)
```

## Session History

The state DB (`~/.hermes/state.db`, typically 500MB–2GB for active installations) carries:

- All conversation messages with FTS5 search indexes
- Session metadata (titles, timestamps, profiles)
- Compression locks and state tracking

This ALL transfers via the simple `~/.hermes/` copy above. No separate export/import needed for the DB itself.

## Profiles

Profiles live in `~/.hermes/profiles/<name>/`. Each has its own:

- `config.yaml` and `.env`
- `state.db` (separate session store)
- `skills/` (profile-scoped skills)
- `memories/` (profile-scoped memory files)
- `cron/` (profile-scoped cron outputs)

All transfer with the top-level `~/.hermes/` tarball.