Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,552 @@
|
||||
---
|
||||
name: hermes-backup
|
||||
description: "Full Hermes backup to Wasabi S3 — config, sessions, profiles, keys, and restore script."
|
||||
version: 1.7.0
|
||||
author: ShoNuff
|
||||
tags: [hermes, backup, wasabi, s3, disaster-recovery, live-sync, warm-standby]
|
||||
---
|
||||
|
||||
# Hermes Backup
|
||||
|
||||
Regularly backs up the entire Hermes environment so you can restore on a new server.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Do not invent fallback keys.** A top-level `fallback_providers: [...]` entry is not a valid Hermes setting and may be silently ignored. Current Hermes uses `model.fallbacks` for main-agent fallbacks and `delegation.fallback` for child-agent fallbacks. Treat the installed schema and `hermes config check` as authoritative. After editing, parse the YAML, run `hermes config check`, restart the gateway if required, and perform a real fallback inference test. See `references/hermes-config-discipline.md`.
|
||||
- **Never use `/tmp` as staging directory** — On most Linux systems, `/tmp` is a **tmpfs** (RAM-backed, usually ~1GB). Hermes backups include state.db and profiles/ which easily exceed this. Stage backups on root disk: `BACKUP_DIR="/root/.hermes/.backups/hermes-backup-$(date +%F)"`.
|
||||
- **Check `/tmp` usage before debugging** — If the backup script exits with `No space left on device` but `df -h /` shows free space, run `df -h /tmp`. If it's a tmpfs at 100%, the fix is moving the staging dir off tmpfs, not clearing disk space.
|
||||
- **The backup archive is created from the staging directory** — If you move `BACKUP_DIR` off tmpfs, also move `ARCHIVE` off tmpfs to match. The `tar czf` runs from the staging dir's parent, so both paths need consistent disk backing.
|
||||
- **Fix the config.yaml temp_dir too** — The backup Python script (e.g. `wisp-backup.py`) likely has its own `temp_dir` in config.yaml. If you only change the bash wrapper's staging dir, the Python script still writes intermediate files to `/tmp`. Patch both.
|
||||
- **`tar` `cd` path must match staging dir parent** — If `BACKUP_DIR` moved off `/tmp` to `~/.hermes/.backups/`, the `tar czf` command must `cd "$(dirname "$BACKUP_DIR")"` not `cd /tmp`. Otherwise tar produces `Cannot stat: No such file`.
|
||||
- **Wasabi bucket names are globally unique** — `hermes-backups` may be taken by another account. Use alternatives like `hermes-vps-backups`.
|
||||
- **IAM policy needs `PutBucketVersioning` explicitly** — Without it, `put-bucket-versioning` returns `Access Denied` even if other S3 permissions work.
|
||||
- **Policy changes take 30-60s to propagate** — If a CreateBucket or PutBucketVersioning call fails with AccessDenied immediately after policy update, retry after a short wait.
|
||||
- **ListBuckets returns AccessDenied — this is expected** — The IAM user's policy scopes to specific bucket ARNs, not `s3:ListAllMyBuckets`. Test by reading a specific bucket by name instead.
|
||||
- **After migration, verify S3 access immediately** — Run `aws s3 ls s3://<bucket-name>/ --endpoint-url https://s3.us-east-1.wasabisys.com/`. If you get `InvalidAccessKeyId`, the credentials file didn't survive the transfer. Check `~/.aws/credentials` exists and is readable.
|
||||
- **The cold spare / standby section was moved** — the old "Failover to cold spare (~2 min recovery)" was replaced by the automatic warm standby restore below. The old manual procedure (`curl ... install.sh && aws s3 cp latest-hermes.tar.gz && systemctl restart`) is deprecated in favor of the systemd oneshot pattern.
|
||||
- **Live sync and warm standby are independent layers** — The 15-min live sync pushes TO S3. The warm standby restore pulls FROM S3. They don't depend on each other's scripts or configs.
|
||||
- **Secret files consolidated into standalone files (`.hetzner_token`, `.netcup_api_key`) are fragile** — They get left behind during migration, forgotten during key rotation, and break scripts silently. Best practice is to consolidate all secrets into `~/.hermes/.env` (chmod 600) and remove the standalone files. See `references/secret-management.md` for the standard workflow.
|
||||
- **Post-migration, check that standalone token files were properly merged into `.env`** — `grep -r "api_key\|token\|password\|secret" ~/.hermes/config.yaml` (expect refs only, not values). If scripts reference deleted files, patch them to read from `.env` instead.
|
||||
- **System-level systemd services are NOT backed up** — The backup script only copies from `~/.config/systemd/user/hermes-gateway*.service` (user-scoped). All services at `/etc/systemd/system/` — `hermes.service`, `hermes-assistant.service`, `hermes-browser.service`, `shark-game.service`, `mysql-tunnel.service`, `ollama.service` — are silently missing from backups. A full restore on a new server loses every systemd unit except the Anita gateway. **FIXED 2026-07-08:** backup script now copies from `/etc/systemd/system/hermes*.service` plus mysql-tunnel, ollama, and shark-game. Restore script uses `sudo` to write to `/etc/systemd/system/` and runs `systemctl daemon-reload`. If you add new systemd services, add `cp /etc/systemd/system/<new-service>.service "$BACKUP_DIR/systemd/"` to both the backup and embedded restore scripts.
|
||||
- **Caddy reverse proxy config is NOT backed up** — `/etc/caddy/Caddyfile` is the single point of failure for all web routing. A restore on a new server would have Hermes running but all Caddy-proxied services unreachable. **FIXED 2026-07-08:** backup script now copies `/etc/caddy/Caddyfile` to `$BACKUP_DIR/caddy/`, and the live-sync script pushes it to `s3://hermes-vps-backups/live/caddy/Caddyfile`. The embedded restore script restores to `/etc/caddy/` with `sudo`.
|
||||
- **Only wisp_rsa SSH keys are backed up** — The backup script explicitly copies `$HOME/.ssh/wisp_rsa` and `.pub` but ignores `itpp-infra`, `authorized_keys`, and any other keys. The itpp-infra key is used by the mysql-tunnel systemd service — restore without it means crippled database access. Fix: extend the SSH key loop to include `itpp-infra` and `authorized_keys`.
|
||||
- **Standalone credential files outside the profiles/ dir are not backed up** — Files like `/root/.hermes/config/migration-creds.txt`, `/root/.hermes/references/dre-temp-passwords.txt` are invisible to the backup pipeline. The profiles/ dir is covered via rsync, but root-level credential files are only backed up if they match hardcoded names. **FIXED 2026-07-08:** `migration-creds.txt` now explicitly backed up to `$BACKUP_DIR/config/migration-creds.txt` and restored with `chmod 600`. Run `find /root/.hermes -maxdepth 3 \( -name '*.env' -o -name '*cred*' -o -name '*pass*' -o -name '*secret*' -o -name '*token*' \)` to audit remaining gaps and add them explicitly.
|
||||
- **Private keys with wrong permissions (644 instead of 600)** — `/root/wg-server.key`, `/root/wg-ccr.key` are WireGuard private keys world-readable. `/root/.hermes/references/dre-temp-passwords.txt` is 644 (passwords readable by all users). Find them with `find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' -o -name '*secret*' ) -not -perm 600`. Fix: `chmod 600` on each.
|
||||
- **Stale full backups are silent** — The daily full backup cron can stop running (e.g. last ran 3+ days ago). No alert fires when a daily backup is missed. The live sync (every 15 min) continues running, masking the full backup's absence. Check recent full backup dates with `aws s3 ls s3://$BUCKET/$DEST_DIR/ --endpoint-url $ENDPOINT | tail -5`. **FIXED 2026-07-10:** Daily watchdog script at `/root/.hermes/scripts/backup-audit-check.sh` runs from system crontab at 2 AM UTC (1h after backup). Logs to syslog tag `backup-audit`. Full reference: `references/backup-audit-watchdog.md`.
|
||||
- **`~/.aws/config` is not explicitly backed up** — The backup script copies `~/.aws/credentials` but not `~/.aws/config`. While it only contains the region setting (43 bytes, 644 perms), it's still a gap if you're automating complete restore. Add `cp "$HOME/.aws/config" "$BACKUP_DIR/aws/" 2>/dev/null || true`.
|
||||
- **All backup scripts MUST source the AWS CLI venv for cron** — Cron has a minimal PATH. Without `source /opt/awscli-venv/bin/activate`, `aws` is not found and the upload step silently fails while the local archive is created successfully. This gives a false sense of security. `hermes-backup.sh` was missing this activation (fixed Jul 11, 2026). `root-essentials-backup.sh` and `hermes-live-sync.sh` had it correctly. Audit all scripts with `grep -L 'awscli-venv' /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh`.
|
||||
- **`root-essentials-backup.sh` has a wrong Caddyfile path** — The `tar --append` on line 30 uses `-C /etc systemd/system/caddy/Caddyfile` which resolves to `/etc/systemd/system/caddy/Caddyfile` (does not exist). Correct path: `-C /etc caddy/Caddyfile` → `/etc/caddy/Caddyfile`. The error is hidden by `2>/dev/null || true`, so the tarball silently lacks the Caddyfile.
|
||||
- **`root-essentials-backup.sh` uses `/tmp` for staging** — The archive is written to `/tmp/root-essentials-<date>.tar.gz` before upload. On this installation `/tmp` has 7.5G free so it works, but the general guidance is to avoid tmpfs for backup staging. If the archive grows beyond tmpfs capacity, the script fails mid-backup.
|
||||
- **`root-essentials-backup.sh` can silently fail the S3 upload while local tarballs succeed** — The script has `set -euo pipefail`, so if the `aws s3 cp` upload step fails, the script exits immediately without cleaning up. This leaves stranded local tarballs in `/tmp/root-essentials-*.tar.gz` while S3 shows stale or missing files. The journal/syslog output (piped through `logger`) may show only the first echo line ("Creating backup...") with nothing after, because the upload failure prevents later echo lines from executing. The backup-audit watchdog only checks the full-backup bucket (`hermes-vps-backups/hermes-full-backup/`), not the root-backup prefix, so root-essentials failures go undetected. **Diagnostic procedure when root-backup/ looks stale on S3:**
|
||||
1. `ls -la /tmp/root-essentials-*.tar.gz` — if recent tarballs exist locally, the upload step is the fault
|
||||
2. `journalctl -t root-essentials-backup --no-pager -n 10` — if output stops at "Creating backup...", the script died after tar but before or during upload
|
||||
3. `grep 'root-essentials-backup' /var/log/syslog` — same check via syslog if journal isn't available
|
||||
4. Run the script manually to see the actual error: `bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1`
|
||||
- **Empty purpose-specific buckets are a silent gap** — `itpropartner-system-configs` and `itpropartner-docker-volumes` exist on Wasabi (confirmed Jul 11, 2026) but contain 0 objects. The sync scripts (`hermes-system-config-sync.sh`, `hermes-docker-sync.sh`) are either not running or failing silently. Their output goes to cron's default mail spool, which may not be monitored. Verify with `aws s3 ls s3://<bucket>/ --recursive --summarize --endpoint-url ...`.
|
||||
|
||||
## Backup completeness audit
|
||||
|
||||
Run this after any backup script change to verify nothing critical is missing:
|
||||
|
||||
```bash
|
||||
echo "=== Systemd services ==="
|
||||
echo "User (backed up):"
|
||||
ls ~/.config/systemd/user/hermes-gateway*.service 2>/dev/null || echo "(none)"
|
||||
echo "System (NOT backed up):"
|
||||
ls /etc/systemd/system/*.service 2>/dev/null || echo "(none)"
|
||||
echo "=== Caddy config ==="
|
||||
[ -f /etc/caddy/Caddyfile ] && echo "EXISTS (NOT backed up)" || echo "MISSING"
|
||||
echo "=== SSH keys ==="
|
||||
echo "Backed up:" && ls ~/.ssh/wisp_rsa* 2>/dev/null || echo "(none)"
|
||||
echo "NOT backed up:" && for f in ~/.ssh/itpp-infra ~/.ssh/itpp-infra.pub ~/.ssh/authorized_keys; do
|
||||
[ -f "$f" ] && echo " $f" || true; done
|
||||
echo "=== WireGuard keys ==="
|
||||
find /root -maxdepth 2 -name '*.key' 2>/dev/null || echo "(none)"
|
||||
echo "=== Standalone creds outside backup scope ==="
|
||||
find /root/.hermes -maxdepth 3 ( -name '*.env' -o -name '*cred*' -o -name '*pass*' \
|
||||
-o -name '*secret*' -o -name '*token*' ) -not -path '*/profiles/*' 2>/dev/null
|
||||
echo "=== Permission audit (should all be 600) ==="
|
||||
find /root -maxdepth 3 ( -name '*.key' -o -name '*pass*' -o -name '*cred*' \
|
||||
-o -name '*secret*' ) -not -perm 600 2>/dev/null || echo "(all correct)"
|
||||
echo "=== Last full backup dates ==="
|
||||
source /opt/awscli-venv/bin/activate 2>/dev/null && \
|
||||
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com 2>&1 | tail -5
|
||||
```
|
||||
|
||||
## Post-fix verification (run after script changes)
|
||||
|
||||
After modifying the backup or live-sync scripts, verify the new items are actually picked up:
|
||||
|
||||
```bash
|
||||
# 1. Dry-run the backup script's new sections
|
||||
echo "=== systemd services to be backed up ==="
|
||||
ls /etc/systemd/system/hermes*.service /etc/systemd/system/mysql-tunnel.service \
|
||||
/etc/systemd/system/ollama.service /etc/systemd/system/shark-game.service 2>/dev/null
|
||||
echo "---"
|
||||
echo "=== Caddyfile to be backed up ==="
|
||||
stat /etc/caddy/Caddyfile 2>/dev/null && echo "EXISTS" || echo "MISSING"
|
||||
echo "---"
|
||||
echo "=== migration-creds.txt to be backed up ==="
|
||||
stat /root/.hermes/migration-creds.txt 2>/dev/null && echo "EXISTS" || echo "MISSING"
|
||||
|
||||
# 2. Verify live-sync puts Caddyfile on S3
|
||||
source /opt/awscli-venv/bin/activate
|
||||
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com/
|
||||
|
||||
# 3. Verify the embedded restore.sh heredoc is correct
|
||||
grep -n 'sudo mkdir -p /etc/systemd/system' /root/.hermes/scripts/hermes-backup.sh
|
||||
grep -n 'sudo mkdir -p /etc/caddy' /root/.hermes/scripts/hermes-backup.sh
|
||||
grep -n 'migration-creds.txt' /root/.hermes/scripts/hermes-backup.sh
|
||||
|
||||
# 4. Syntax check
|
||||
bash -n /root/.hermes/scripts/hermes-backup.sh && echo "backup.sh: OK"
|
||||
bash -n /root/.hermes/scripts/hermes-live-sync.sh && echo "live-sync.sh: OK"
|
||||
|
||||
# 5. Venv activation audit — ALL backup/sync scripts must source the venv for cron
|
||||
echo "=== Scripts missing venv activation (will fail in cron) ==="
|
||||
for script in /root/.hermes/scripts/*backup*.sh /root/.hermes/scripts/*sync*.sh; do
|
||||
grep -q 'awscli-venv' "$script" 2>/dev/null || echo " MISSING: $script"
|
||||
done
|
||||
```
|
||||
|
||||
## SiteGround SFTP backup
|
||||
|
||||
SiteGround websites are backed up via SFTP to `s3://hermes-vps-backups/siteground/<site>/`. The connection uses an encrypted RSA key with passphrase. See `references/siteground-backup-pattern.md` for credentials, connection method, exclusion recommendations, and the database export strategy.
|
||||
|
||||
**Pitfall — Fleet-wide SFTP backup times out:** Attempting to backup all 21 SiteGround WordPress sites in a single run timed out at 600 seconds (Jul 10, 2026). SFTP over shared hosting is too slow for bulk migration. Two alternatives: (1) Back up individual sites in batches of 3-5, or (2) Use MainWP + WPvivid Pro (already configured, backs up 15 sites daily to Wasabi S3) as the primary migration path and rely on SFTP only for sites not in MainWP.
|
||||
|
||||
**Pitfall — SFTP root is the site directory:** SiteGround's SFTP user sees WordPress site directories in the root (`/`), not under `/home/`. Listing `/home/` returns `OSError: list failed`. Always list `/` first to discover the actual site directories.
|
||||
|
||||
## What's backed up
|
||||
|
||||
| Item | Why |
|
||||
|------|-----|
|
||||
| `config.yaml`, `.env`, `auth.json`, `migration-creds.txt` | All config + API keys + migration secrets |
|
||||
| `state.db` + `sessions/` | Full conversation history (~1.1GB) |
|
||||
| `skills/` | All installed/created skills |
|
||||
| `scripts/` | Custom scripts (backup, feed digest, etc.) |
|
||||
| `profiles/` | Anita's profile + any others |
|
||||
| `cron/` | Job definitions |
|
||||
| SSH keys | WISP backup keys |
|
||||
| Mail passwords | Himalaya creds |
|
||||
| AWS/Wasabi creds | For S3 access |
|
||||
| Systemd services | All Hermes services from `/etc/systemd/system/` |
|
||||
| Caddyfile | Reverse proxy config from `/etc/caddy/` |
|
||||
|
||||
## Included in backup
|
||||
|
||||
A `restore.sh` script is included — extract the tarball on a new server and run it.
|
||||
|
||||
## Verified S3 endpoint
|
||||
|
||||
All buckets use:
|
||||
```
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
## Live buckets (created 2026-07-04, versioning ON, all 5 confirmed Jul 10)
|
||||
|
||||
| Bucket | Purpose | Status |
|
||||
|---|---|---|
|
||||
| `hermes-vps-backups` | Hermes full backup (config, sessions, profiles, keys) + live sync + full archive tarballs + standby scripts | Active since Jul 4 |
|
||||
| `mikrotik-ccr-backups` | Router config exports (home gateway, wisp gateway) | Active since Jul 4 |
|
||||
| `itpropartner-system-configs` | System configs: Caddyfile, systemd services, ops scripts, SSH keys, env files | 🟡 Bucket exists (confirmed Jul 11) but **empty** — sync script not running |
|
||||
| `itpropartner-docker-volumes` | Docker volume data: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama | 🟡 Bucket exists (confirmed Jul 11) but **empty** — sync script not running |
|
||||
| `itpropartner-backups` | Legacy files (signature images, misc) — data mostly migrated to purpose-specific buckets | Active since Jul 4 |
|
||||
|
||||
**Note:** `hermes-backups` (simple name) is taken globally on Wasabi. Always try an alternative like `hermes-vps-backups` or append a qualifier. When adding new buckets to the Wasabi IAM policy, the existing `hermes` policy uses two Statement blocks (one for bucket-level actions like ListBucket/GetBucketVersioning, one for object-level actions like GetObject/PutObject/DeleteObject). Add the new bucket ARN (with and without `/*`) to BOTH blocks. Test with `aws s3 ls`, `aws s3 cp` (PUT), `aws s3 cp ... -` (GET/readback), and `aws s3 rm` (DELETE) — probe files must be cleaned up after.
|
||||
|
||||
## Current cron status
|
||||
|
||||
- **Hermes live sync**: Every 15 min via `hermes-live-sync.sh` (no_agent, silent) → `s3://hermes-vps-backups/live/`
|
||||
- **Hermes system config sync**: Every 15 min via `hermes-system-config-sync.sh` (no_agent, silent) → `s3://itpropartner-system-configs/`
|
||||
- **Hermes docker volume sync**: Every 15 min via `hermes-docker-sync.sh` (no_agent, silent) → `s3://itpropartner-docker-volumes/`
|
||||
- **Hermes full backup**: Daily at 1:00 AM UTC via **system crontab** (NOT Hermes cron — blocked by gateway lifecycle guard #30719). Script: `hermes-backup.sh`.
|
||||
- **Full backup audit watchdog**: Daily at 2:00 AM UTC via system crontab. Script: `backup-audit-check.sh`.
|
||||
- **Root essentials backup**: Daily at 3:00 AM UTC via system crontab (42MB tarball). Script: `root-essentials-backup.sh`. S3: `s3://hermes-vps-backups/root-backup/`. Includes /root configs, skills, scripts, references, profiles, SSH keys, systemd services, Caddyfile, and /var/www. Corruption check: downloads + `tar tzf` verify. Restore: extract + live sync for state.db = ~10 min recovery. **Fastest recovery path.**
|
||||
- **Router backup**: Daily at 6:00 AM UTC (script: `run-wisp-backup.sh` → `wisp-backup/wisp-backup.py`)
|
||||
- **Hetzner snapshots**: Weekly Monday 5:00 AM UTC (script: `snapshot-hetzner.py`)
|
||||
|
||||
## Live sync (every 15m)
|
||||
|
||||
A lightweight `aws s3 sync` that mirrors the entire `.hermes/` directory (minus bulky caches like `audio_cache/`, `image_cache/`, `cache/`, `sandboxes/`, `kanban/`, `node/`, `bin/`, `logs/`, `*.lock`) to `s3://hermes-vps-backups/live/`.
|
||||
|
||||
**Design principles:**
|
||||
- **Silent on success** — the no_agent cron script suppresses output when nothing changed. Only failures produce output.
|
||||
- **`aws s3 sync` handles deltas** — only changed files upload. Typically <5 seconds per run.
|
||||
- **RPO ~15 minutes** — worst case you lose 15 min of state.db changes.
|
||||
|
||||
**Script** (`hermes-live-sync.sh`):
|
||||
```bash
|
||||
#!/bin/bash
|
||||
HERMES_HOME="${HERMES_HOME:-$HOME/.hermes}"
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
BUCKET="hermes-vps-backups"
|
||||
DEST_PREFIX="live"
|
||||
|
||||
if [ -f /opt/awscli-venv/bin/activate ]; then
|
||||
source /opt/awscli-venv/bin/activate
|
||||
fi
|
||||
|
||||
aws s3 sync "$HERMES_HOME" "s3://$BUCKET/$DEST_PREFIX/" \
|
||||
--endpoint-url "$ENDPOINT" \
|
||||
--exclude "audio_cache/*" \
|
||||
--exclude "image_cache/*" \
|
||||
--exclude "cache/*" \
|
||||
--exclude "sandboxes/*" \
|
||||
--exclude "kanban/*" \
|
||||
--exclude "node/*" \
|
||||
--exclude "bin/*" \
|
||||
--exclude "logs/*" \
|
||||
--exclude "*.lock" \
|
||||
--exclude ".skills_prompt_snapshot.json" \
|
||||
--exclude ".update_check" \
|
||||
2>&1 | grep -v "^$" || true
|
||||
|
||||
# NOTE: Docker volumes moved to hermes-docker-sync.sh (itpropartner-docker-volumes)
|
||||
# NOTE: Caddyfile moved to hermes-system-config-sync.sh (itpropartner-system-configs)
|
||||
exit 0
|
||||
```
|
||||
|
||||
## Purpose-specific sync scripts
|
||||
|
||||
The monolithic `hermes-live-sync.sh` was split into purpose-specific scripts — see `references/purpose-specific-buckets.md` for details on the normalization and the script contents.
|
||||
|
||||
### `hermes-system-config-sync.sh` → `s3://itpropartner-system-configs/`
|
||||
Exports: Caddyfile, systemd service files (hermes*, mysql-tunnel, ollama, shark-game), ops scripts, SSH keys (non-known_hosts), .env + AWS credentials.
|
||||
|
||||
### `hermes-docker-sync.sh` → `s3://itpropartner-docker-volumes/`
|
||||
Exports: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama volume data — each under its own prefix.
|
||||
|
||||
### DB Dump Automation (KNOWN GAP — see server-dr-plans-v3.md)
|
||||
MariaDB on wphost02 (Apex) and fleettracker360 are NOT backed up via scheduled dump to S3. This is a known gap flagged in the Jul 10 DR audit. The backup standard declares daily DB dumps with 90-day retention, but this is not yet deployed.
|
||||
|
||||
### Docker Volume Backup (KNOWN GAP — see server-dr-plans-v3.md)
|
||||
Docker volume backups declared in the provisioning standard but not yet automated. All Docker hosts need volume backup scripts pushed to S3 per-server buckets.
|
||||
|
||||
### Memory Consolidation Backup (NEW — Jul 9, 2026)
|
||||
Memory is auto-consolidated every 10 min with backup-before-prune guarantee:
|
||||
- Backup to: `s3://hermes-vps-backups/live/memories/memory-backup.json`
|
||||
- History: `s3://hermes-vps-backups/live/memories/history/memory-backup-<UTC>.json`
|
||||
- Local retention: 48h under `~/.hermes/memories/.consolidate-backups/`
|
||||
- Restore procedure: See `devops/disaster-recovery-audit` skill's Memory Consolidation Safety section
|
||||
|
||||
### Creating the cron jobs
|
||||
```bash
|
||||
hermes cron create \
|
||||
--name "hermes-live-sync" \
|
||||
--schedule "every 15m" \
|
||||
--script hermes-live-sync.sh \
|
||||
--no_agent \
|
||||
--deliver local
|
||||
```
|
||||
|
||||
## Failover to cold spare (automatic warm standby)
|
||||
|
||||
When a cold/spare server exists (e.g. an old primary that was migrated off), a systemd oneshot service syncs the latest live state from S3 before Hermes starts. No manual steps needed — boot it and it picks up from the last 15-min checkpoint.
|
||||
|
||||
**The restore script** (`hermes-standby-restore.sh`):
|
||||
1. Waits for network (up to 30s)
|
||||
2. Pings the **live netcup box** (152.53.192.33) first — **if it responds, exits 0 without starting Hermes** (split-brain guard)
|
||||
3. Loads AWS CLI
|
||||
4. Runs `aws s3 sync` from `s3://hermes-vps-backups/live/` to `~/.hermes/` (same exclusion patterns)
|
||||
5. Starts Hermes gateway
|
||||
|
||||
**Split-brain prevention is mandatory.** The standby box and live box share the same Telegram bot token. If both run Hermes simultaneously, messages get lost and cron jobs fire from both. The ping check happens BEFORE the S3 sync or Hermes start.
|
||||
|
||||
Two failover mechanisms exist — **boot-time** (systemd oneshot) and **periodic** (cron watchdog). Both check the live box before acting.
|
||||
|
||||
**Boot-time** (`hermes-standby-restore.sh` via systemd):
|
||||
- Runs once on boot, before Hermes starts
|
||||
- Pings live box once — if alive, exits 0 (stays dormant)
|
||||
- If dead, syncs S3 state and starts Hermes
|
||||
- Designed for cold-start: power on the standby and it picks up
|
||||
|
||||
**Periodic watchdog** (`hermes-standby-watchdog.sh` via system crontab):
|
||||
- Runs every 10 min (system crontab, not Hermes cron — Hermes isn't running on standby)
|
||||
- Pings live box 3 times immediately
|
||||
- If all fail, runs 4 confirmation cycles at 60s intervals (~3.5 min total)
|
||||
- If still dead → sends **Telegram alert** + **email alert**, syncs S3, starts Hermes
|
||||
- Telegram alert format: `🚨 Hermes Failover — app1 (netcup) is offline\n\nTimestamp: ...\nAction: S3 sync → Hermes gateway start`
|
||||
- Email alert: same content via SMTP (mail.germainebrown.com:587) with password from `/root/.config/himalaya/g-germainebrown.pass`
|
||||
- This catches the case where the live box dies *while running* (not just on boot)
|
||||
|
||||
**Pre-register the standby SSH key** in your cloud provider's project. Without it, you can't deploy the script. On Hetzner: `POST /v1/ssh_keys` with your public key.
|
||||
|
||||
**The systemd unit** (`hermes-standby.service`):
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Hermes Standby Restore — pull latest state from S3 before Hermes starts
|
||||
Before=hermes.service
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
ExecStart=/root/.hermes/scripts/hermes-standby-restore.sh
|
||||
RemainAfterExit=true
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Deploy on standby box
|
||||
```bash
|
||||
wget -O /root/.hermes/scripts/hermes-standby-restore.sh \
|
||||
https://s3.us-east-1.wasabisys.com/hermes-vps-backups/standby/hermes-standby-restore.sh
|
||||
chmod +x /root/.hermes/scripts/hermes-standby-restore.sh
|
||||
cp /root/.hermes/scripts/hermes-standby.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable hermes-standby.service
|
||||
```
|
||||
|
||||
### Pre-requisites for fast failover
|
||||
- S3 access keys available externally (password manager, not on the server being restored)
|
||||
- AWS CLI installed and configured on the standby box
|
||||
- Cold spare server already provisioned and powered on
|
||||
- SSH key registered in your cloud provider's project **before** attempting standby deployment
|
||||
|
||||
### Standby deployment without SSH access (Hetzner rescue mode)
|
||||
|
||||
If you have no SSH access to the standby box (no key deployed), use Hetzner rescue mode:
|
||||
|
||||
1. **Register your SSH key** in the Hetzner project via API or Cloud Console — rescue mode accepts numeric key IDs, not raw keys:
|
||||
```bash
|
||||
curl -s -X POST https://api.hetzner.cloud/v1/ssh_keys \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name":"my-key","public_key":"ssh-ed25519 AAAA..."}'
|
||||
```
|
||||
|
||||
2. **Enable rescue mode with that key ID** (the numeric ID from the response, not a string):
|
||||
```bash
|
||||
curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/enable_rescue \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"type":"linux64","ssh_keys":[114708824]}'
|
||||
```
|
||||
|
||||
3. **Power on** — boots into rescue with your key authorized as root:
|
||||
```bash
|
||||
curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/poweron \
|
||||
-H "Authorization: Bearer $TOKEN" -d '{}'
|
||||
```
|
||||
|
||||
4. **SSH into rescue** as root:
|
||||
```bash
|
||||
ssh -i ~/.ssh/your_key root@$SERVER_IP
|
||||
```
|
||||
|
||||
5. **Mount the main partition** and inject your SSH key into the OS:
|
||||
```bash
|
||||
mkdir -p /mnt/root
|
||||
mount /dev/sda1 /mnt/root
|
||||
mkdir -p /mnt/root/root/.ssh
|
||||
echo "$PUBKEY" >> /mnt/root/root/.ssh/authorized_keys
|
||||
chmod 600 /mnt/root/root/.ssh/authorized_keys
|
||||
```
|
||||
|
||||
6. **Disable the old Hermes service** on the mounted disk so it doesn't autostart:
|
||||
```bash
|
||||
mount -t proc none /mnt/root/proc # needed for chroot systemctl
|
||||
chroot /mnt/root systemctl disable hermes-gateway.service
|
||||
```
|
||||
|
||||
7. **Deploy your standby scripts**, then disable rescue + reboot via API:
|
||||
```bash
|
||||
curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/disable_rescue \
|
||||
-H "Authorization: Bearer $TOKEN" -d '{}'
|
||||
curl -s -X POST https://api.hetzner.cloud/v1/servers/$SERVER_ID/actions/reboot \
|
||||
-H "Authorization: Bearer $TOKEN" -d '{}'
|
||||
```
|
||||
|
||||
**Pitfalls:**
|
||||
- **Register the SSH key FIRST.** `enable_rescue` needs a numeric `ssh_key_id`. If you get `"SSH key not found"` (404), the key string was passed instead of the integer ID, or the key isn't in the project yet.
|
||||
- **Rescue mode also sets a root password** — the `enable_rescue` response includes a `root_password` field. If SSH key auth fails, use root + that password.
|
||||
- **After disable_rescue + reboot, wait ~40s** for the normal OS to boot before attempting SSH.
|
||||
- **Hetzner Cloud API hostname != OS hostname** — Renaming via `PUT /v1/servers/{id} {"name":"new-name"}` only changes Hetzner's label. The OS still has the old `/etc/hostname`. Fix after first SSH: `hostnamectl set-hostname new-name.fqdn`. Verify with both `hostname` and `hostname -f`.
|
||||
- **chroot systemctl still works without /proc** but warns "This is not a supported mode of operation." The `enable`/`disable` commands succeed despite the warning. Mount `/proc` if you want clean output.
|
||||
- **Check server API status before attempting SSH** — `GET /v1/servers/{id}` returns a `status` field. If it's `off`, re-power-on and wait.
|
||||
|
||||
## DR issue log
|
||||
|
||||
Track all audit findings, root causes, fixes, and verification in `references/dr-issue-log.md`. Every DR audit produces a dated entry.
|
||||
|
||||
### DR issue log formatting rules
|
||||
|
||||
Germaine views these files on mobile. Pipe tables (`| col | col |`) render unreadably on small screens.
|
||||
|
||||
**DR issue log format:**
|
||||
|
||||
```
|
||||
**Fixed** [OK]
|
||||
- **DR-001** [HIGH] Short title -> what was done
|
||||
|
||||
**Resolved** [INFO]
|
||||
- **DR-009** Short title -> what was done
|
||||
|
||||
**Investigating** [PENDING]
|
||||
- **DR-006** [HIGH] Short title -> what was done
|
||||
```
|
||||
|
||||
Individual issue entries use:
|
||||
```
|
||||
**Problem**
|
||||
Description...
|
||||
|
||||
**Root Cause**
|
||||
Why it happened...
|
||||
|
||||
**Fix**
|
||||
What was applied...
|
||||
|
||||
**Verification**
|
||||
- [OK] Evidence item 1
|
||||
- [OK] Evidence item 2
|
||||
```
|
||||
|
||||
Sub-status tags use `[OK]`, `[PENDING]`, `[HIGH]`, `[MED]`, `[INFO]` — never use emoji (they garble on some mobile viewers). Use `->` not `→`. Use `--` not `—`. Keep it pure ASCII so any text editor or terminal can display it cleanly. Format per issue: problem, root cause, fix, status, verified-by.
|
||||
|
||||
## Periodic backup health audit
|
||||
|
||||
Run the **S3 bucket audit procedure** to verify all three backup buckets are healthy — versioning on, files being written, no silent pipeline failures.
|
||||
|
||||
The procedure covers:
|
||||
- Versioning status check on every bucket
|
||||
- File counts and total size
|
||||
- Staleness detection (objects from today vs >48h gap)
|
||||
- Live sync freshness verification (count today's objects in the live/ prefix)
|
||||
- Full backup archive age check
|
||||
- Common failure patterns with fixes
|
||||
|
||||
See `references/backup-audit-procedure.md` for the full quick-audit script and per-bucket methodology.
|
||||
|
||||
**Weekly audit recommended.** The MikroTik router backup pipeline once went 3 days without producing configs before being noticed — routine audits catch these before data loss becomes a risk.
|
||||
|
||||
## Verify backup costs from invoices
|
||||
|
||||
To confirm backups are running within expected resource usage, extract Hetzner invoices:
|
||||
|
||||
```bash
|
||||
apt-get install -y poppler-utils
|
||||
pdftotext /path/to/invoice.pdf -
|
||||
```
|
||||
|
||||
Hetzner invoice PDFs line items include: server type counts, unit prices, backup percentage (20% of instance price), and snapshot storage (GB-months). Also check Hetzner invoice line items: if backup charges appear but aren't on the right products, the backup config may need updating.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Create a Wasabi IAM user (subuser, not root). See `infrastructure-automation` skill's `references/wasabi-iam-setup.md` for the full policy and common pitfalls.
|
||||
2. Create the Wasabi bucket (names are globally unique — if taken, try alternatives like `hermes-vps-backups`).
|
||||
3. Edit `/root/.hermes/scripts/hermes-backup.sh`:
|
||||
- `BUCKET`: set to the actual bucket name
|
||||
- `BACKUP_DIR` and `ARCHIVE`: MUST use a root-disk path (e.g. `~/.hermes/.backups/`), NOT `/tmp` (tmpfs is RAM-backed and fills up)
|
||||
- The `tar czf` command runs from `$(dirname "$BACKUP_DIR")` — ensure the script's `cd` target matches the staging dir's actual parent, not `/tmp`
|
||||
4. Ensure `~/.aws/credentials` exists with the IAM user's access key + secret key, `chmod 600`
|
||||
5. **Test:** Run `bash /root/.hermes/scripts/hermes-backup-worker.sh` or simulate with a dry `aws s3 ls` on each bucket (see Pitfalls section about ListBuckets). The first real backup fires at the next cron tick.
|
||||
6. Create the live sync cron job (see Live sync section above)
|
||||
7. If a cold spare exists, deploy the standby restore (see Failover section above)
|
||||
8. Cron job handles daily runs automatically
|
||||
|
||||
## Secret consolidation workflow
|
||||
|
||||
All Hermes secrets should live in a single `~/.hermes/.env` file (chmod 600). Standalone secret files (`scripts/.hetzner_token`, `scripts/.netcup_api_key`, etc.) get lost during migration and forgotten during key rotation.
|
||||
|
||||
**Standard consolidation steps:**
|
||||
|
||||
1. Read existing `.env` and load into a dict
|
||||
2. Read each standalone token file
|
||||
3. Add them to the dict with descriptive names (`HETZNER_API_TOKEN`, `NETCUP_API_KEY`, `ADMIN_AI_API_KEY`, etc.)
|
||||
4. Write the full dict back to `.env`
|
||||
5. `chmod 600 ~/.hermes/.env`
|
||||
6. `rm` the standalone files
|
||||
7. Patch any scripts that referenced the old file paths to source from `.env` instead
|
||||
8. Verify: `stat -c '%a' ~/.hermes/.env` → should show `600`
|
||||
|
||||
**Custom-provider credential rule:** Do not assume an admin-ai key is still in `config.yaml`, `.env`, or `auth.json`. Inspect all three without exposing values. `auth.json` may retain only a fingerprint after the actual provider and secret were removed. Store the recoverable secret in `.env` using a descriptive variable such as `ADMIN_AI_API_KEY`, configure the provider through supported Hermes fields, and verify `/models` plus a real completion before calling it restored. If auxiliary tasks use a separate key field, verify that copy independently.
|
||||
|
||||
## Post-migration verification
|
||||
|
||||
After restoring Hermes on a new server, verify these in order:
|
||||
|
||||
### S3/Wasabi access
|
||||
```bash
|
||||
source /opt/awscli-venv/bin/activate
|
||||
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | sort
|
||||
aws s3 ls s3://hermes-vps-backups/live/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | head -10
|
||||
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1 | tail -5
|
||||
aws s3 ls s3://itpropartner-system-configs/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
|
||||
aws s3 ls s3://itpropartner-docker-volumes/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
|
||||
aws s3 ls s3://itpropartner-backups/ \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com/ 2>&1
|
||||
```
|
||||
|
||||
**Pitfall — stale credentials:** `aws configure list` can show credentials present but `aws s3 ls` returns `InvalidAccessKeyId`. This means the IAM key was rotated on the old server but `~/.aws/credentials` on this server has the stale version. Get the current key from the password manager, update the file, and retest. The file is a simple 3-line INI — just `aws_access_key_id` and `aws_secret_access_key` under `[default]`.
|
||||
|
||||
### LLM provider availability
|
||||
If the provider is a proxy (e.g., admin-ai routing to multiple backends), verify OpenRouter models are reachable:
|
||||
```bash
|
||||
curl -s https://admin-ai.itpropartner.com/v1/models \
|
||||
-H "Authorization: Bearer $(grep 'api_key:' ~/.hermes/config.yaml | head -1 | awk '{print $2}')"
|
||||
```
|
||||
Check for `openrouter/*` entries in the output. The proxy should expose 80+ models including OpenAI, Claude, Gemini, DeepSeek, etc.
|
||||
|
||||
**Pitfall — mismatched API keys:** The API key appears in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. If the proxy rotated its key, both must be updated. The curl test uses the provider key (first hit), but the vision auxiliary provider may have the old key separately.
|
||||
|
||||
### IMAP connectivity check
|
||||
If email triage jobs exist, verify IMAP server reachable proactively:
|
||||
```bash
|
||||
python3 -c "
|
||||
import ssl, socket
|
||||
ctx = ssl.create_default_context()
|
||||
s = socket.create_connection(('mail.germainebrown.com', 993), timeout=10)
|
||||
ss = ctx.wrap_socket(s, server_hostname='mail.germainebrown.com')
|
||||
print(f'IMAP: {ss.version()}')
|
||||
ss.close()
|
||||
"
|
||||
```
|
||||
Also verify password files present: `ls ~/.config/himalaya/*.pass 2>/dev/null`
|
||||
|
||||
### Heremes gateway health
|
||||
```bash
|
||||
hermes gateway status
|
||||
journalctl -u hermes -n 10 --no-pager
|
||||
```
|
||||
|
||||
### Cron job restoration
|
||||
```bash
|
||||
hermes cron list
|
||||
```
|
||||
Should show all jobs. If empty, wait 30-60s for scheduler to discover them from state.db.
|
||||
|
||||
## Restore on new server (full replacement)
|
||||
|
||||
```bash
|
||||
# Install Hermes first
|
||||
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
|
||||
|
||||
# Download latest backup
|
||||
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-YYYY-MM-DD.tar.gz . \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
|
||||
# Extract and restore
|
||||
tar xzf hermes-full-backup-YYYY-MM-DD.tar.gz
|
||||
cd hermes-backup-YYYY-MM-DD
|
||||
bash restore.sh
|
||||
|
||||
# Then set up live sync and warm standby (see sections above)
|
||||
# hermes cron create --name "hermes-live-sync" --schedule "every 15m" --script hermes-live-sync.sh --no_agent --deliver local
|
||||
|
||||
# Start gateways
|
||||
hermes gateway restart
|
||||
hermes profile use anita && anita gateway restart # if Anita exists
|
||||
```
|
||||
@@ -0,0 +1,163 @@
|
||||
# S3 Backup Bucket Audit Procedure
|
||||
|
||||
A repeatable methodology for auditing all Wasabi S3 backup buckets — versioning status, file counts, total sizes, staleness checks, and fresh-object verification.
|
||||
|
||||
## Why audit
|
||||
|
||||
- Catch backup pipeline failures early (the MikroTik router backup pipeline stopped producing configs after Jul 5, 2026, and went unnoticed for 3 days)
|
||||
- Verify the live-sync checkpoint is actually producing objects from today
|
||||
- Confirm versioning wasn't accidentally disabled during a bucket config change
|
||||
- Know your total storage consumption for cost tracking
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# AWS CLI with Wasabi endpoint configured
|
||||
/opt/awscli-venv/bin/aws configure # ~/.aws/credentials must exist
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
```
|
||||
|
||||
## Quick audit script
|
||||
|
||||
Run this on all existing buckets at once:
|
||||
|
||||
```bash
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
|
||||
for BUCKET in hermes-vps-backups itpropartner-backups mikrotik-ccr-backups itpropartner-system-configs itpropartner-docker-volumes; do
|
||||
echo "=== $BUCKET ==="
|
||||
|
||||
# Versioning status
|
||||
echo "--- Versioning ---"
|
||||
/opt/awscli-venv/bin/aws s3api get-bucket-versioning --bucket "$BUCKET" --endpoint-url "$ENDPOINT" 2>&1
|
||||
|
||||
# Top-level layout and latest object
|
||||
echo "--- Top-level listing (last 5 modified) ---"
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive 2>&1 | sort -k1,2r | head -5
|
||||
|
||||
# Summary (count + size)
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive --summarize 2>&1 | tail -2
|
||||
|
||||
# Staleness: count objects from today
|
||||
TODAY=$(date +%F)
|
||||
TODAY_COUNT=$(/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive 2>&1 | grep -c "$TODAY" || true)
|
||||
echo "Objects from today ($TODAY): $TODAY_COUNT"
|
||||
echo ""
|
||||
done
|
||||
```
|
||||
|
||||
## Detailed per-bucket checks
|
||||
|
||||
### 1. Versioning
|
||||
|
||||
```bash
|
||||
aws s3api get-bucket-versioning --bucket <name> --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
Expected: `{"Status": "Enabled", "MFADelete": "Disabled"}`
|
||||
If `Status` is absent or `Suspended`, the bucket is no longer protected against accidental overwrites/deletes.
|
||||
|
||||
### 2. File count and total size
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive --human-readable --summarize
|
||||
```
|
||||
|
||||
The summary lines at the bottom give you:
|
||||
- `Total Objects: N`
|
||||
- `Total Size: X.X GiB/MiB`
|
||||
|
||||
### 3. Staleness check (per top-level prefix)
|
||||
|
||||
Check whether any objects exist from today:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date +%F)" | sort -r | head -5
|
||||
```
|
||||
|
||||
If zero objects from today, check yesterday:
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date -d yesterday +%F)" | sort -r | head -5
|
||||
```
|
||||
|
||||
**Staleness thresholds:**
|
||||
| Type | Max acceptable gap |
|
||||
|---|---|
|
||||
| Live Hermes sync | < 30 minutes (expected every 15 min) |
|
||||
| Full Hermes backup | < 48 hours (daily at 1 AM UTC) |
|
||||
| Router config backup | < 48 hours (daily at 6 AM UTC) |
|
||||
|
||||
### 4. Live sync freshness (deep check for `hermes-vps-backups`)
|
||||
|
||||
The most recent object timestamp tells you when the 15-min live sync last ran:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | sort -k1,2r | head -5
|
||||
```
|
||||
|
||||
Also count today's objects in the live prefix specifically:
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | grep -c "$(date +%F)"
|
||||
```
|
||||
|
||||
This should be a large number (hundreds to thousands) indicating the sync is actively pushing state changes.
|
||||
|
||||
### 5. Full backup archive check
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com --human-readable
|
||||
```
|
||||
|
||||
Check that a `.tar.gz` file exists from within the last 48 hours. Also check the manifest file is present beside each archive.
|
||||
|
||||
### 6. Caddyfile copy check
|
||||
|
||||
The live sync pushes `/etc/caddy/Caddyfile` to `s3://hermes-vps-backups/live/caddy/Caddyfile` every 15 min. Verify it's current:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### 7. Empty purpose-specific bucket check
|
||||
|
||||
The `itpropartner-system-configs` and `itpropartner-docker-volumes` buckets exist on Wasabi but may contain 0 objects if their sync scripts are not running. Check both:
|
||||
|
||||
```bash
|
||||
for BUCKET in itpropartner-system-configs itpropartner-docker-volumes; do
|
||||
echo "=== $BUCKET ==="
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive --summarize 2>&1 | tail -2
|
||||
done
|
||||
```
|
||||
|
||||
Expected: `Total Objects: > 0`. If 0 objects, the corresponding sync script (`hermes-system-config-sync.sh` or `hermes-docker-sync.sh`) is not running or failing silently.
|
||||
|
||||
## Known bucket inventory (as of Jul 11, 2026)
|
||||
|
||||
| Bucket | Purpose | Status |
|
||||
|---|---|---|
|
||||
| `hermes-vps-backups` | Hermes full backup + live sync + standby | ✅ Versioning ON, active objects |
|
||||
| `itpropartner-backups` | Portal/ITP backups | ✅ Versioning ON, minimal objects |
|
||||
| `mikrotik-ccr-backups` | Router config exports | ✅ Versioning ON, stale |
|
||||
| `itpropartner-system-configs` | System configs | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
|
||||
| `itpropartner-docker-volumes` | Docker volume data | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
|
||||
|
||||
## Common failure patterns
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| Zero objects from today on `live/` | Live sync cron job stopped or credentials rotated | Check `hermes cron list` for `hermes-live-sync` status; verify `~/.aws/credentials` |
|
||||
| Bucket listing returns objects but versioning says `Suspended` | Someone ran `put-bucket-versioning` with `Status=Suspended` | Re-enable: `aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled --endpoint-url <endpoint>` |
|
||||
| Full backup archive is >48h old | Daily cron job stopped running; live sync masks the gap | Check system crontab; run `bash /root/.hermes/scripts/hermes-backup.sh` manually; verify `BACKUP_DIR` not on `/tmp` |
|
||||
| Router config backups stopped | VPN tunnel down, SSH key rotated, or pipeline script error | SSH to gateway CCR and test manually; check the tower config YAML |
|
||||
| `AccessDenied` on `ListBuckets` | Expected — IAM policy scopes to specific bucket ARNs | Test by bucket name directly, not `aws s3 ls` (no arg) |
|
||||
| `InvalidAccessKeyId` | Stale credentials in `~/.aws/credentials` | Get current key from password manager and update the file |
|
||||
| Bucket exists but has 0 objects | Sync script not running or failing silently | Check system crontab for the sync job; run the script manually and check output; verify `aws` is in PATH (venv activation) |
|
||||
| `s3api get-bucket-versioning` returns empty | Versioning not yet enabled on bucket or `s3:GetBucketVersioning` missing from IAM policy | Enable with `put-bucket-versioning`; check IAM policy includes `s3:GetBucketVersioning` on the bucket ARN |
|
||||
| AWS CLI `command not found` in cron | Backup/sync script missing venv activation | Add `source /opt/awscli-venv/bin/activate` near the top of the script (after `set -euo pipefail`, before first `aws` call) |
|
||||
| Staleness >48h but watchdog didn't alert | Watchdog uses 36h threshold, so it reports OK for ~1.5 days after last successful backup | The watchdog is working as designed (grace period prevents false alarms). The real fix is preventing the backup from failing in the first place |
|
||||
|
||||
## Periodic audit cadence
|
||||
|
||||
- **Weekly**: Run the quick audit script on existing buckets
|
||||
- **Monthly**: Review full backup vs live sync size trends for cost tracking
|
||||
- **Post-migration**: Run the full audit procedure immediately after any server migration or credential rotation
|
||||
@@ -0,0 +1,52 @@
|
||||
# Backup Audit Watchdog
|
||||
|
||||
Since Jul 10, 2026: a system crontab watchdog checks daily whether the full backup is fresh.
|
||||
|
||||
## Pattern
|
||||
|
||||
A no_agent script runs at 2 AM UTC (1 hour after the scheduled daily full backup at 1 AM). It checks S3 for the latest backup archive and reports freshness.
|
||||
|
||||
## Script: `/root/.hermes/scripts/backup-audit-check.sh`
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Checks S3 for latest full backup. Reports success if <36h old.
|
||||
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
BUCKET="hermes-vps-backups"
|
||||
PREFIX="hermes-full-backup"
|
||||
|
||||
source /opt/awscli-venv/bin/activate 2>/dev/null || true
|
||||
|
||||
latest=$(aws s3 ls "s3://$BUCKET/$PREFIX/" --endpoint-url "$ENDPOINT" 2>&1 | \
|
||||
grep "\.tar\.gz$" | sort | tail -1)
|
||||
|
||||
if [ -z "$latest" ]; then
|
||||
echo "🔴 No full backup archives found on S3!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
date_str=$(echo "$latest" | awk '{print $1}')
|
||||
size_bytes=$(echo "$latest" | awk '{print $3}')
|
||||
file_name=$(echo "$latest" | awk '{print $4}')
|
||||
age_hours=$(( ( $(date +%s) - $(date -d "$date_str" +%s) ) / 3600 ))
|
||||
|
||||
if [ "$age_hours" -lt 36 ]; then
|
||||
echo "✅ BACKUP OK — $file_name ($size_bytes bytes, ${age_hours}h ago)"
|
||||
elif [ "$age_hours" -lt 60 ]; then
|
||||
echo "⚠️ BACKUP WARNING — $age_hours hours old: $file_name"
|
||||
else
|
||||
echo "🔴 BACKUP STALE — $age_hours hours ago: $file_name"
|
||||
fi
|
||||
```
|
||||
|
||||
## System crontab
|
||||
|
||||
```
|
||||
0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit
|
||||
```
|
||||
|
||||
Output goes to syslog with `backup-audit` tag. View with:
|
||||
```bash
|
||||
journalctl -t backup-audit
|
||||
```
|
||||
@@ -0,0 +1,66 @@
|
||||
# DR Issue Log
|
||||
|
||||
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-08 — Initial Full DR Audit
|
||||
|
||||
### DR-001: Caddyfile not included in backup scripts
|
||||
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
|
||||
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
|
||||
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** Post-fix script syntax check and line reference
|
||||
|
||||
### DR-002: Systemd service files backed up to wrong path
|
||||
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files were not backed up.
|
||||
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
|
||||
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** Post-fix script syntax check
|
||||
|
||||
### DR-003: migration-creds.txt not in backup scope
|
||||
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
|
||||
**Root cause:** Added after backup script was written, never included in scope.
|
||||
**Fix:** Added to hermes-backup.sh file list
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** Post-fix script syntax check
|
||||
|
||||
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
|
||||
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
|
||||
**Root cause:** Script created the file without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-005: migration-creds.txt at 644 permissions
|
||||
**Problem:** Same as DR-004 — migration-creds.txt was 644.
|
||||
**Root cause:** Written without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-006: Full backup stale (last ran Jul 5)
|
||||
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron has missed 3 days.
|
||||
**Root cause:** Script may have errored, cron may have stopped — [INVESTIGATING]
|
||||
**Fix:** [PENDING]
|
||||
**Status:** 🔄 Investigating
|
||||
|
||||
### DR-007: home-router-daily-backup cron error
|
||||
**Problem:** Cron job errored at 06:00 today — backup script failed.
|
||||
**Root cause:** [INVESTIGATING]
|
||||
**Fix:** [PENDING]
|
||||
**Status:** 🔄 Investigating
|
||||
|
||||
### DR-008: MikroTik CCR backup stale (last Jul 5)
|
||||
**Problem:** mikrotik-ccr-backups S3 bucket has no uploads since Jul 5.
|
||||
**Root cause:** [INVESTIGATING]
|
||||
**Fix:** [PENDING]
|
||||
**Status:** 🔄 Investigating
|
||||
|
||||
### DR-009: app1-bu powered on when DR plan says offline
|
||||
**Problem:** Warm standby server is running (3 days uptime) but DR plan says "offline, boots on demand."
|
||||
**Root cause:** Server was previously started and not powered back off.
|
||||
**Fix:** [PENDING — confirm with Germaine]
|
||||
**Status:** 🔄 Awaiting decision
|
||||
@@ -0,0 +1,135 @@
|
||||
# Hermes Configuration Discipline
|
||||
|
||||
Do not modify Hermes config.yaml by adding keys you guessed or assumed existed.
|
||||
Always verify before applying.
|
||||
|
||||
## The Rule
|
||||
|
||||
**You must verify every config key exists in official Hermes documentation before adding it to config.yaml.**
|
||||
|
||||
Hermes config keys are not arbitrary YAML — they are read by specific code paths in the Hermes runtime. A made-up key is silently ignored, not validated, and leaves you thinking a feature was configured when it wasn't.
|
||||
|
||||
## Verification Sources (in priority order)
|
||||
|
||||
1. **The `hermes config` CLI** — `hermes config set KEY VAL` validates the key path before applying. If `hermes config set` accepts it, it's a real key.
|
||||
2. **The official docs** — https://hermes-agent.nousresearch.com/docs/user-guide/configuration
|
||||
3. **The `hermes-agent` skill** — loaded with `skill_view(name='hermes-agent')`, it documents known config sections.
|
||||
4. **The source code** — `hermes_cli/config.py` contains DEFAULT_CONFIG which enumerates every recognized top-level key.
|
||||
5. **The internal help** — `hermes config --help`, `hermes config check`, `hermes config show`
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- **Do not assume patterns from other tools carry over.** Just because some other tool uses `fallback_providers` doesn't mean Hermes reads them.
|
||||
- **Do not write a key + value into config.yaml that you haven't seen in the docs, the skill, or the source.** Writing and testing is not the right order — verify first, then write.
|
||||
- **Do not patch config.yaml by hand with arbitrary YAML when `hermes config set` exists.** The CLI is the safe path. Use it.
|
||||
|
||||
## What config changes look like for Hermes
|
||||
|
||||
If the user wants automatic fallback to a local model, verify the installed Hermes schema first. Current Hermes uses `model.fallbacks` for the main agent and `delegation.fallback` for child agents. Do not substitute guessed names such as `fallback_providers`, `fallback_chain`, or `fallback_order`.
|
||||
|
||||
Example shape (verify against the installed version before applying):
|
||||
|
||||
```yaml
|
||||
model:
|
||||
fallbacks: '[{"model":"llama3.2:3b","provider":"ollama-local"}]'
|
||||
|
||||
delegation:
|
||||
fallback: '[{"model":"llama3.2:3b","provider":"ollama-local"}]'
|
||||
|
||||
providers:
|
||||
ollama-local:
|
||||
base_url: http://localhost:11434/v1
|
||||
```
|
||||
|
||||
### Verify a custom OpenAI-compatible provider
|
||||
|
||||
A provider is active only if it is represented in current `config.yaml` under `providers` or the active `model` block. A stale `auth.json` record containing only a label, URL, status, or secret fingerprint is metadata -- it does not prove the credential still exists or that Hermes can use it.
|
||||
|
||||
1. Parse `config.yaml`; inspect `model`, `providers`, `delegation`, and `auxiliary` without printing secrets.
|
||||
2. Check `.env` for the expected named variable; report only presence and length.
|
||||
3. Treat `auth.json` fingerprints as clues, not usable credentials.
|
||||
4. Call `<base_url>/models` with the actual credential and verify the requested model ID appears.
|
||||
5. Send a minimal chat completion using that exact model.
|
||||
6. Only after both calls succeed report the provider and model as functional.
|
||||
|
||||
### Verify Ollama as a fallback
|
||||
|
||||
1. Confirm the binary, service, and listener. Keep it local-only unless remote access was explicitly requested.
|
||||
2. Pull the exact configured model.
|
||||
3. Verify both `/api/tags` and OpenAI-compatible `/v1/models` list it.
|
||||
4. Send a deterministic `/v1/chat/completions` probe and verify returned content.
|
||||
5. Confirm `model.fallbacks` and, when required, `delegation.fallback` reference the same provider/model.
|
||||
|
||||
Installing a model is not complete until inference succeeds. A configured provider with an empty model list is not operational.
|
||||
|
||||
## Config change workflow
|
||||
|
||||
```text
|
||||
User request → check docs/skill/source for real key →
|
||||
if EXISTS → use `hermes config set` or edit via `hermes config edit`
|
||||
if NOT FOUND → tell user it doesn't exist, suggest alternatives
|
||||
```
|
||||
|
||||
## When the user corrects you
|
||||
|
||||
If the user calls out a made-up config key:
|
||||
- Acknowledge immediately — don't deflect
|
||||
- Remove the invalid key from config.yaml via the proper CLI or a direct edit
|
||||
- Document the lesson
|
||||
|
||||
## Known missing features (as of 2026-07-05)
|
||||
|
||||
| User wants | Hermes has | Workaround |
|
||||
|---|---|---|
|
||||
| Automatic provider failover | `fallback_providers` does NOT exist | Manual provider switch via CLI, or local ollama for maintenance tasks |
|
||||
| Multiple fallback providers | Not supported | N/A |
|
||||
| Provider priority list | Not supported | N/A |
|
||||
|
||||
## Web tools `use_gateway` pitfall
|
||||
|
||||
Hermes has two config knobs for web tools:
|
||||
|
||||
```yaml
|
||||
web:
|
||||
backend: firecrawl # Which web backend to use
|
||||
use_gateway: false # true = route through Nous Portal gateway
|
||||
```
|
||||
|
||||
**The trap:** When `web.use_gateway: true` (the default or a previously-set value), Hermes routes Firecrawl requests through **Nous Portal's gateway** — it ignores your `FIRECRAWL_API_KEY` from `.env` entirely. If you're not logged into Nous Portal (no OAuth session), `web_search` and `web_extract` fail with:
|
||||
|
||||
> "Web tools are not configured. Set FIRECRAWL_API_KEY for cloud Firecrawl..."
|
||||
|
||||
Even though the key IS set and valid. The fix is:
|
||||
|
||||
```bash
|
||||
hermes config set web.use_gateway false
|
||||
```
|
||||
|
||||
Then a new session (`/reset`) picks up the change. **Note:** `/reset` itself does NOT fix this — the problem is the config value, not the session state. Multiple resets without changing the config value will all fail the same way.
|
||||
|
||||
**Verification:** Test the key directly against Firecrawl's API before and after:
|
||||
|
||||
```bash
|
||||
source ~/.hermes/.env 2>/dev/null
|
||||
curl -s -X POST "https://api.firecrawl.dev/v1/search" \
|
||||
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query":"test","limit":1}'
|
||||
# Expected: HTTP 200, {"success": true, "data": [...]}
|
||||
```
|
||||
|
||||
**Key learning:** `web.use_gateway: true` is NOT a fallback or override layer — it *replaces* the local key. If you're self-hosting the Firecrawl key in `.env`, this must be `false`. Setting it back to `true` after switching to Nous Portal OAuth would be correct for that auth mode.
|
||||
|
||||
### Debugging flow for "web tools not configured"
|
||||
|
||||
```
|
||||
1. Check .env: grep FIRECRAWL_API_KEY ~/.hermes/.env
|
||||
→ If missing, add key. If present, proceed.
|
||||
2. Check config: grep -A2 '^web:' ~/.hermes/config.yaml
|
||||
→ If use_gateway: true, flip to false via:
|
||||
hermes config set web.use_gateway false
|
||||
→ Then /reset or start new session.
|
||||
3. Test key: direct curl to api.firecrawl.dev (see above)
|
||||
→ 200 = key valid. Non-200 = key expired/wrong.
|
||||
4. Verify tools: try web_search or web_extract
|
||||
```
|
||||
@@ -0,0 +1,62 @@
|
||||
# Purpose-Specific S3 Bucket Architecture
|
||||
|
||||
## Motivation
|
||||
|
||||
The original `hermes-live-sync.sh` was a monolithic script pushing everything — Hermes state, Docker volumes, Caddyfile, systemd configs — into a single `s3://hermes-vps-backups/live/` prefix. This made it hard to:
|
||||
|
||||
- Grant granular IAM access (Docker volumes accessible to scripts that only needed Hermes configs)
|
||||
- Replicate specific data categories to different regions
|
||||
- Audit which scripts own which files
|
||||
- Reason about data isolation during DR
|
||||
|
||||
## Normalization (2026-07-08)
|
||||
|
||||
Split into 5 buckets plus 1 legacy bucket:
|
||||
|
||||
| Bucket | Primary Script | Data |
|
||||
|---|---|---|
|
||||
| `hermes-vps-backups/live/` | `hermes-live-sync.sh` | Hermes state: config.yaml, .env, state.db, sessions/, skills/, profiles/, cron/, memory_store.db |
|
||||
| `hermes-vps-backups/hermes-full-backup/` | `hermes-backup.sh` | Daily full archive tarballs (~350MB compressed) |
|
||||
| `itpropartner-system-configs/` | `hermes-system-config-sync.sh` | System configs: Caddyfile, systemd services, ops scripts, SSH keys, .env + AWS creds |
|
||||
| `itpropartner-docker-volumes/` | `hermes-docker-sync.sh` | Docker volume data: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama |
|
||||
| `mikrotik-ccr-backups/` | `run-wisp-backup.sh` + `wisp-backup.py` | Router configs (home gateway, wisp gateway) + logs |
|
||||
| `itpropartner-backups/` | (legacy — manual) | Old signatures, test files, migrated router config copies |
|
||||
|
||||
## Exclusions
|
||||
|
||||
The `hermes-vps-backups/live/` sync excludes: audio_cache, image_cache, cache, sandboxes, kanban, node, bin, logs, *.lock, .skills_prompt_snapshot.json, .update_check.
|
||||
|
||||
The `itpropartner-system-configs/` scripts/ sync excludes: `.backups/`, `migration-backup/`, `*.pyc`, `__pycache__/`.
|
||||
|
||||
The `itpropartner-system-configs/` ssh/ sync excludes: known_hosts, authorized_keys, config.
|
||||
|
||||
## Script Locations
|
||||
|
||||
All scripts at `/root/.hermes/scripts/`:
|
||||
- `hermes-live-sync.sh` — Hermes state only
|
||||
- `hermes-system-config-sync.sh` — System configs
|
||||
- `hermes-docker-sync.sh` — Docker volumes
|
||||
|
||||
## Dashboard
|
||||
|
||||
The ops portal (`/var/www/ops/backups.html`) tracks all 6 bucket paths via `ops-data-collector.py` (every 5 min). The dashboard JS maps bucket keys to display names via `DISPLAY_NAMES`:
|
||||
|
||||
```js
|
||||
'hermes-vps-backups': 'Hermes VPS (Live State)',
|
||||
'hermes-full-backups': 'Hermes VPS (Full Archives)',
|
||||
'mikrotik-backups': 'MikroTik Router Configs',
|
||||
'itpropartner-system-configs': 'System Configs (Caddy, systemd, SSH)',
|
||||
'itpropartner-docker-volumes': 'Docker Volumes (Docuseal, VW, Twenty)',
|
||||
'itpropartner-backups': 'IT Pro Partner (Legacy)'
|
||||
```
|
||||
|
||||
## Blocker
|
||||
|
||||
The IAM user `Hermes-User` lacks `s3:CreateBucket`. The two new buckets (`itpropartner-system-configs`, `itpropartner-docker-volumes`) must be created in the Wasabi Console, then versioning enabled via:
|
||||
|
||||
```bash
|
||||
aws s3api put-bucket-versioning --bucket itpropartner-system-configs --versioning-configuration Status=Enabled --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
aws s3api put-bucket-versioning --bucket itpropartner-docker-volumes --versioning-configuration Status=Enabled --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
Until the buckets exist, the new sync scripts will silently fail with `NoSuchBucket`.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Secret Management — Hermes Environment
|
||||
|
||||
## Principle
|
||||
|
||||
All secrets in one file: `~/.hermes/.env`, chmod 600, backed up in S3.
|
||||
Standalone token files (`.hetzner_token`, `.netcup_api_key`, etc.) are fragile — they get left behind during migration, broken during key rotation, and scripts that reference them fail silently when they're missing.
|
||||
|
||||
## Current secret inventory
|
||||
|
||||
| Secret | File location | Permissions | Notes |
|
||||
|---|---|---|---|
|
||||
| admin-ai API key | `~/.hermes/config.yaml` (2 places) | Root-readable | Also in `.env` as backup |
|
||||
| Telegram bot tokens | `~/.hermes/.env` | 600 | Default + Anita profiles |
|
||||
| Hetzner API token | `~/.hermes/.env` | 600 | Was standalone file |
|
||||
| Netcup API key | `~/.hermes/.env` | 600 | Was standalone file |
|
||||
| SMTP/IMAP passwords | `~/.config/himalaya/*.pass` | 600 | Himalaya standard |
|
||||
| Wasabi S3 keys | `~/.aws/credentials` | 600 | AWS CLI standard |
|
||||
| SSH keys | `~/.ssh/` | 600 | Standard |
|
||||
|
||||
## Consolidation workflow
|
||||
|
||||
When you discover a standalone secret file:
|
||||
|
||||
1. Read it and the current `.env`:
|
||||
```
|
||||
env_path = ~/.hermes/.env
|
||||
standalone_path = ~/.hermes/scripts/.some_token
|
||||
```
|
||||
|
||||
2. Load `.env` into a dict, add the new variable, write back:
|
||||
```
|
||||
env_vars['HETZNER_API_TOKEN'] = token_value
|
||||
```
|
||||
|
||||
3. `chmod 600 ~/.hermes/.env`
|
||||
|
||||
4. Remove the standalone file:
|
||||
```bash
|
||||
rm ~/.hermes/scripts/.some_token
|
||||
```
|
||||
|
||||
5. Patch any scripts that referenced the old path to source from `.env`.
|
||||
|
||||
## Reading secrets in scripts
|
||||
|
||||
Scripts should source `.env` at the top:
|
||||
```bash
|
||||
set -a; source ~/.hermes/.env; set +a
|
||||
```
|
||||
|
||||
Python scripts should use:
|
||||
```python
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(os.path.expanduser("~/.hermes/.env"))
|
||||
token = os.environ["HETZNER_API_TOKEN"]
|
||||
```
|
||||
|
||||
## Special case: config.yaml inline keys
|
||||
|
||||
The admin-ai API key lives in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. Hermes reads these directly from the YAML config, not from env vars. The `.env` copy serves as a backup record for recovery. If the key rotates, update both config.yaml entries AND the `.env` backup.
|
||||
@@ -0,0 +1,96 @@
|
||||
# SiteGround SFTP Backup Pattern
|
||||
|
||||
Back up websites from SiteGround shared hosting to Wasabi S3 via SFTP. SiteGround does not expose an API — SFTP is the only programmatic access method.
|
||||
|
||||
## Credentials
|
||||
|
||||
- Host: `sftp.siteground.net`
|
||||
- Port: `18765` (non-standard)
|
||||
- Username: `sftp7068-6b4804d3`
|
||||
- Key: encrypted RSA private key with passphrase `LoveMyBoys73!`
|
||||
- The key file lives at `/root/.ssh/siteground.key` (chmod 600)
|
||||
|
||||
## Connection Method
|
||||
|
||||
SiteGround uses an **encrypted RSA private key with a passphrase**. Plain key-based auth without the passphrase will fail. Two approaches:
|
||||
|
||||
### A. Python + Paramiko (preferred for scripting)
|
||||
|
||||
```python
|
||||
import paramiko, io
|
||||
|
||||
key = paramiko.RSAKey.from_private_key(io.StringIO(key_data), password="<passphrase>")
|
||||
transport = paramiko.Transport(("sftp.siteground.net", 18765))
|
||||
transport.connect(username="sftp7068-6b4804d3", pkey=key)
|
||||
sftp = paramiko.SFTPClient.from_transport(transport)
|
||||
```
|
||||
|
||||
### B. sshpass + ssh (manual/scripting)
|
||||
|
||||
Not recommended for scripting due to passphrase-in-command exposure, but valid for manual testing:
|
||||
|
||||
```bash
|
||||
sshpass -p "<passphrase>" ssh -i /root/.ssh/siteground.key \
|
||||
-o StrictHostKeyChecking=no -o PubkeyAuthentication=yes \
|
||||
-p 18765 sftp7068-6b4804d3@sftp.siteground.net "ls -la"
|
||||
```
|
||||
|
||||
## Sites Present (Jul 2026)
|
||||
|
||||
21 sites — all WordPress, all under `/home/` on SiteGround:
|
||||
|
||||
815bistro.com, abortionstory.org, apextrackexperience.com, chiefenergyofficer.org, costanzospizzeria.com, elliscountyinternet.com, fixourclub.com, forefrontwireless.com, forefrontwireless.net, freedominhonesty.com, garyjavo.com, germainebrown.com, grandlakeclub.com, injenuity413.com, injenuitysolutions.com, itpropartner.com, katiewattsdesign.com, savannahevents.com, sotobunch.com, theabortionstory.org, voipsimplicity.com
|
||||
|
||||
## Backup to S3
|
||||
|
||||
Each site is a full directory tarball uploaded to `s3://hermes-vps-backups/siteground/<site>/`:
|
||||
|
||||
```python
|
||||
import tarfile, io
|
||||
|
||||
# Stream tar.gz to S3 to avoid filling /tmp on large sites
|
||||
buf = io.BytesIO()
|
||||
with tarfile.open(fileobj=buf, mode='w:gz') as tar:
|
||||
tar.add(f"/home/{site}", arcname=".")
|
||||
buf.seek(0)
|
||||
|
||||
# Upload directly
|
||||
s3_client.upload_fileobj(buf, "hermes-vps-backups", f"siteground/{site}/{site}-{date}.tar.gz")
|
||||
```
|
||||
|
||||
For very large sites (apextrackexperience.com, itpropartner.com, voipsimplicity.com), write to `/tmp` first then upload — streaming through memory may timeout on SFTP.
|
||||
|
||||
## Exclusion Recommendations
|
||||
|
||||
WordPress sites on SiteGround typically contain:
|
||||
- `wp-content/uploads/` — images, media (backup all)
|
||||
- `wp-content/plugins/` — can reinstall, but backup for config preservation
|
||||
- `wp-content/themes/` — custom themes, backup
|
||||
- `wp-content/cache/` — OK to skip
|
||||
- `error_log` — large debug logs, OK to skip
|
||||
|
||||
## Restore
|
||||
|
||||
1. Download tar.gz from S3
|
||||
2. Extract on new host: `tar xzf <site>-<date>.tar.gz`
|
||||
3. Set correct permissions: `chown -R www-data:www-data <site>/`
|
||||
4. Import database from phpMyAdmin export or MySQL dump
|
||||
5. Update `wp-config.php` with new DB credentials
|
||||
6. Point DNS to new server
|
||||
|
||||
## Database
|
||||
|
||||
SFTP provides **files only**. The MySQL database must be exported separately:
|
||||
- Via SiteGround's phpMyAdmin interface
|
||||
- Or: `mysqldump -h <host> -u <user> -p <dbname> > <site>-db-<date>.sql`
|
||||
- Upload the SQL dump to Core via SFTP/scp and store alongside the files
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. SFTP backup all sites to S3 (done Jul 9, 2026)
|
||||
2. Export each site's database from phpMyAdmin → upload to S3
|
||||
3. On new box, restore files + import DB
|
||||
4. Update wp-config.php credentials
|
||||
5. Point DNS → done
|
||||
|
||||
No downtime per site if DNS is the last step.
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/bin/bash
|
||||
# backup-audit-check.sh — Daily check: was yesterday's full backup successful?
|
||||
# Runs ~1h after scheduled backup (2 AM UTC, backup at 1 AM UTC).
|
||||
# Exits 0 = OK, exits 1 = stale/missing.
|
||||
# Logs to syslog tag 'backup-audit'.
|
||||
# System crontab entry: 0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
BUCKET="hermes-vps-backups"
|
||||
PREFIX="hermes-full-backup"
|
||||
|
||||
if [ -f /opt/awscli-venv/bin/activate ]; then
|
||||
source /opt/awscli-venv/bin/activate
|
||||
fi
|
||||
|
||||
latest=$(aws s3 ls "s3://$BUCKET/$PREFIX/" --endpoint-url "$ENDPOINT" 2>&1 | grep "\.tar\.gz$" | sort | tail -1)
|
||||
|
||||
if [ -z "$latest" ]; then
|
||||
echo "[BACKUP-AUDIT] No full backup archives found on S3!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
date_str=$(echo "$latest" | awk '{print $1}')
|
||||
size_bytes=$(echo "$latest" | awk '{print $3}')
|
||||
file_name=$(echo "$latest" | awk '{print $4}')
|
||||
|
||||
backup_epoch=$(date -d "$date_str" +%s 2>/dev/null || date -j -f "%Y-%m-%d" "$date_str" +%s 2>/dev/null)
|
||||
now_epoch=$(date +%s)
|
||||
age_hours=$(( (now_epoch - backup_epoch) / 3600 ))
|
||||
|
||||
if [ "$age_hours" -lt 36 ]; then
|
||||
echo "[BACKUP-AUDIT] OK — Last: $file_name ($date_str, ${size_bytes}B, ${age_hours}h ago)"
|
||||
exit 0
|
||||
elif [ "$age_hours" -lt 60 ]; then
|
||||
echo "[BACKUP-AUDIT] WARNING — Last backup $age_hours hours old: $file_name"
|
||||
exit 0
|
||||
else
|
||||
echo "[BACKUP-AUDIT] STALE — Last backup was $age_hours hours ago: $file_name"
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user