ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial skills documentation — 25 categories, all SKILL.md + references + scripts

Browse Source
This commit is contained in:
root
2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/hermes-backup/references/backup-audit-procedure.md
+163
View File
@@ -0,0 +1,163 @@
# S3 Backup Bucket Audit Procedure
A repeatable methodology for auditing all Wasabi S3 backup buckets — versioning status, file counts, total sizes, staleness checks, and fresh-object verification.
## Why audit
- Catch backup pipeline failures early (the MikroTik router backup pipeline stopped producing configs after Jul 5, 2026, and went unnoticed for 3 days)
- Verify the live-sync checkpoint is actually producing objects from today
- Confirm versioning wasn't accidentally disabled during a bucket config change
- Know your total storage consumption for cost tracking
## Prerequisites
```bash
# AWS CLI with Wasabi endpoint configured
/opt/awscli-venv/bin/aws configure # ~/.aws/credentials must exist
ENDPOINT="https://s3.us-east-1.wasabisys.com"
```
## Quick audit script
Run this on all existing buckets at once:
```bash
ENDPOINT="https://s3.us-east-1.wasabisys.com"
for BUCKET in hermes-vps-backups itpropartner-backups mikrotik-ccr-backups itpropartner-system-configs itpropartner-docker-volumes; do
echo "=== $BUCKET ==="
# Versioning status
echo "--- Versioning ---"
/opt/awscli-venv/bin/aws s3api get-bucket-versioning --bucket "$BUCKET" --endpoint-url "$ENDPOINT" 2>&1
# Top-level layout and latest object
echo "--- Top-level listing (last 5 modified) ---"
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive 2>&1 | sort -k1,2r | head -5
# Summary (count + size)
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive --summarize 2>&1 | tail -2
# Staleness: count objects from today
TODAY=$(date +%F)
TODAY_COUNT=$(/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive 2>&1 | grep -c "$TODAY" || true)
echo "Objects from today ($TODAY): $TODAY_COUNT"
echo ""
done
```
## Detailed per-bucket checks
### 1. Versioning
```bash
aws s3api get-bucket-versioning --bucket <name> --endpoint-url https://s3.us-east-1.wasabisys.com
```
Expected: `{"Status": "Enabled", "MFADelete": "Disabled"}`
If `Status` is absent or `Suspended`, the bucket is no longer protected against accidental overwrites/deletes.
### 2. File count and total size
```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive --human-readable --summarize
```
The summary lines at the bottom give you:
- `Total Objects: N`
- `Total Size: X.X GiB/MiB`
### 3. Staleness check (per top-level prefix)
Check whether any objects exist from today:
```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date +%F)" | sort -r | head -5
```
If zero objects from today, check yesterday:
```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date -d yesterday +%F)" | sort -r | head -5
```
**Staleness thresholds:**
| Type | Max acceptable gap |
|---|---|
| Live Hermes sync | < 30 minutes (expected every 15 min) |
| Full Hermes backup | < 48 hours (daily at 1 AM UTC) |
| Router config backup | < 48 hours (daily at 6 AM UTC) |
### 4. Live sync freshness (deep check for `hermes-vps-backups`)
The most recent object timestamp tells you when the 15-min live sync last ran:
```bash
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | sort -k1,2r | head -5
```
Also count today's objects in the live prefix specifically:
```bash
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | grep -c "$(date +%F)"
```
This should be a large number (hundreds to thousands) indicating the sync is actively pushing state changes.
### 5. Full backup archive check
```bash
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com --human-readable
```
Check that a `.tar.gz` file exists from within the last 48 hours. Also check the manifest file is present beside each archive.
### 6. Caddyfile copy check
The live sync pushes `/etc/caddy/Caddyfile` to `s3://hermes-vps-backups/live/caddy/Caddyfile` every 15 min. Verify it's current:
```bash
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com
```
### 7. Empty purpose-specific bucket check
The `itpropartner-system-configs` and `itpropartner-docker-volumes` buckets exist on Wasabi but may contain 0 objects if their sync scripts are not running. Check both:
```bash
for BUCKET in itpropartner-system-configs itpropartner-docker-volumes; do
echo "=== $BUCKET ==="
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive --summarize 2>&1 | tail -2
done
```
Expected: `Total Objects: > 0`. If 0 objects, the corresponding sync script (`hermes-system-config-sync.sh` or `hermes-docker-sync.sh`) is not running or failing silently.
## Known bucket inventory (as of Jul 11, 2026)
| Bucket | Purpose | Status |
|---|---|---|
| `hermes-vps-backups` | Hermes full backup + live sync + standby | ✅ Versioning ON, active objects |
| `itpropartner-backups` | Portal/ITP backups | ✅ Versioning ON, minimal objects |
| `mikrotik-ccr-backups` | Router config exports | ✅ Versioning ON, stale |
| `itpropartner-system-configs` | System configs | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
| `itpropartner-docker-volumes` | Docker volume data | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
## Common failure patterns
| Symptom | Likely cause | Fix |
|---|---|---|
| Zero objects from today on `live/` | Live sync cron job stopped or credentials rotated | Check `hermes cron list` for `hermes-live-sync` status; verify `~/.aws/credentials` |
| Bucket listing returns objects but versioning says `Suspended` | Someone ran `put-bucket-versioning` with `Status=Suspended` | Re-enable: `aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled --endpoint-url <endpoint>` |
| Full backup archive is >48h old | Daily cron job stopped running; live sync masks the gap | Check system crontab; run `bash /root/.hermes/scripts/hermes-backup.sh` manually; verify `BACKUP_DIR` not on `/tmp` |
| Router config backups stopped | VPN tunnel down, SSH key rotated, or pipeline script error | SSH to gateway CCR and test manually; check the tower config YAML |
| `AccessDenied` on `ListBuckets` | Expected — IAM policy scopes to specific bucket ARNs | Test by bucket name directly, not `aws s3 ls` (no arg) |
| `InvalidAccessKeyId` | Stale credentials in `~/.aws/credentials` | Get current key from password manager and update the file |
| Bucket exists but has 0 objects | Sync script not running or failing silently | Check system crontab for the sync job; run the script manually and check output; verify `aws` is in PATH (venv activation) |
| `s3api get-bucket-versioning` returns empty | Versioning not yet enabled on bucket or `s3:GetBucketVersioning` missing from IAM policy | Enable with `put-bucket-versioning`; check IAM policy includes `s3:GetBucketVersioning` on the bucket ARN |
| AWS CLI `command not found` in cron | Backup/sync script missing venv activation | Add `source /opt/awscli-venv/bin/activate` near the top of the script (after `set -euo pipefail`, before first `aws` call) |
| Staleness >48h but watchdog didn't alert | Watchdog uses 36h threshold, so it reports OK for ~1.5 days after last successful backup | The watchdog is working as designed (grace period prevents false alarms). The real fix is preventing the backup from failing in the first place |
## Periodic audit cadence
- **Weekly**: Run the quick audit script on existing buckets
- **Monthly**: Review full backup vs live sync size trends for cost tracking
- **Post-migration**: Run the full audit procedure immediately after any server migration or credential rotation
skills/devops/hermes-backup/references/backup-audit-watchdog.md
+52
View File
@@ -0,0 +1,52 @@
# Backup Audit Watchdog
Since Jul 10, 2026: a system crontab watchdog checks daily whether the full backup is fresh.
## Pattern
A no_agent script runs at 2 AM UTC (1 hour after the scheduled daily full backup at 1 AM). It checks S3 for the latest backup archive and reports freshness.
## Script: `/root/.hermes/scripts/backup-audit-check.sh`
```bash
#!/bin/bash
# Checks S3 for latest full backup. Reports success if <36h old.
ENDPOINT="https://s3.us-east-1.wasabisys.com"
BUCKET="hermes-vps-backups"
PREFIX="hermes-full-backup"
source /opt/awscli-venv/bin/activate 2>/dev/null || true
latest=$(aws s3 ls "s3://$BUCKET/$PREFIX/" --endpoint-url "$ENDPOINT" 2>&1 | \
grep "\.tar\.gz$" | sort | tail -1)
if [ -z "$latest" ]; then
echo "🔴 No full backup archives found on S3!"
exit 1
fi
date_str=$(echo "$latest" | awk '{print $1}')
size_bytes=$(echo "$latest" | awk '{print $3}')
file_name=$(echo "$latest" | awk '{print $4}')
age_hours=$(( ( $(date +%s) - $(date -d "$date_str" +%s) ) / 3600 ))
if [ "$age_hours" -lt 36 ]; then
echo "✅ BACKUP OK — $file_name ($size_bytes bytes, ${age_hours}h ago)"
elif [ "$age_hours" -lt 60 ]; then
echo "⚠️ BACKUP WARNING — $age_hours hours old: $file_name"
else
echo "🔴 BACKUP STALE — $age_hours hours ago: $file_name"
fi
```
## System crontab
```
0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit
```
Output goes to syslog with `backup-audit` tag. View with:
```bash
journalctl -t backup-audit
```
skills/devops/hermes-backup/references/dr-issue-log.md
+66
View File
@@ -0,0 +1,66 @@
# DR Issue Log
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
---
## 2026-07-08 — Initial Full DR Audit
### DR-001: Caddyfile not included in backup scripts
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh
**Status:** ✅ Fixed 2026-07-08
**Verified by:** Post-fix script syntax check and line reference
### DR-002: Systemd service files backed up to wrong path
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files were not backed up.
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service
**Status:** ✅ Fixed 2026-07-08
**Verified by:** Post-fix script syntax check
### DR-003: migration-creds.txt not in backup scope
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
**Root cause:** Added after backup script was written, never included in scope.
**Fix:** Added to hermes-backup.sh file list
**Status:** ✅ Fixed 2026-07-08
**Verified by:** Post-fix script syntax check
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
**Root cause:** Script created the file without explicit permission setting.
**Fix:** chmod 600
**Status:** ✅ Fixed 2026-07-08
**Verified by:** ls -la confirms 600
### DR-005: migration-creds.txt at 644 permissions
**Problem:** Same as DR-004 — migration-creds.txt was 644.
**Root cause:** Written without explicit permission setting.
**Fix:** chmod 600
**Status:** ✅ Fixed 2026-07-08
**Verified by:** ls -la confirms 600
### DR-006: Full backup stale (last ran Jul 5)
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron has missed 3 days.
**Root cause:** Script may have errored, cron may have stopped — [INVESTIGATING]
**Fix:** [PENDING]
**Status:** 🔄 Investigating
### DR-007: home-router-daily-backup cron error
**Problem:** Cron job errored at 06:00 today — backup script failed.
**Root cause:** [INVESTIGATING]
**Fix:** [PENDING]
**Status:** 🔄 Investigating
### DR-008: MikroTik CCR backup stale (last Jul 5)
**Problem:** mikrotik-ccr-backups S3 bucket has no uploads since Jul 5.
**Root cause:** [INVESTIGATING]
**Fix:** [PENDING]
**Status:** 🔄 Investigating
### DR-009: app1-bu powered on when DR plan says offline
**Problem:** Warm standby server is running (3 days uptime) but DR plan says "offline, boots on demand."
**Root cause:** Server was previously started and not powered back off.
**Fix:** [PENDING — confirm with Germaine]
**Status:** 🔄 Awaiting decision
skills/devops/hermes-backup/references/hermes-config-discipline.md
+135
View File
@@ -0,0 +1,135 @@
# Hermes Configuration Discipline
Do not modify Hermes config.yaml by adding keys you guessed or assumed existed.
Always verify before applying.
## The Rule
**You must verify every config key exists in official Hermes documentation before adding it to config.yaml.**
Hermes config keys are not arbitrary YAML — they are read by specific code paths in the Hermes runtime. A made-up key is silently ignored, not validated, and leaves you thinking a feature was configured when it wasn't.
## Verification Sources (in priority order)
1. **The `hermes config` CLI** — `hermes config set KEY VAL` validates the key path before applying. If `hermes config set` accepts it, it's a real key.
2. **The official docs** — https://hermes-agent.nousresearch.com/docs/user-guide/configuration
3. **The `hermes-agent` skill** — loaded with `skill_view(name='hermes-agent')`, it documents known config sections.
4. **The source code** — `hermes_cli/config.py` contains DEFAULT_CONFIG which enumerates every recognized top-level key.
5. **The internal help** — `hermes config --help`, `hermes config check`, `hermes config show`
## What NOT to do
- **Do not assume patterns from other tools carry over.** Just because some other tool uses `fallback_providers` doesn't mean Hermes reads them.
- **Do not write a key + value into config.yaml that you haven't seen in the docs, the skill, or the source.** Writing and testing is not the right order — verify first, then write.
- **Do not patch config.yaml by hand with arbitrary YAML when `hermes config set` exists.** The CLI is the safe path. Use it.
## What config changes look like for Hermes
If the user wants automatic fallback to a local model, verify the installed Hermes schema first. Current Hermes uses `model.fallbacks` for the main agent and `delegation.fallback` for child agents. Do not substitute guessed names such as `fallback_providers`, `fallback_chain`, or `fallback_order`.
Example shape (verify against the installed version before applying):
```yaml
model:
fallbacks: '[{"model":"llama3.2:3b","provider":"ollama-local"}]'
delegation:
fallback: '[{"model":"llama3.2:3b","provider":"ollama-local"}]'
providers:
ollama-local:
base_url: http://localhost:11434/v1
```
### Verify a custom OpenAI-compatible provider
A provider is active only if it is represented in current `config.yaml` under `providers` or the active `model` block. A stale `auth.json` record containing only a label, URL, status, or secret fingerprint is metadata -- it does not prove the credential still exists or that Hermes can use it.
1. Parse `config.yaml`; inspect `model`, `providers`, `delegation`, and `auxiliary` without printing secrets.
2. Check `.env` for the expected named variable; report only presence and length.
3. Treat `auth.json` fingerprints as clues, not usable credentials.
4. Call `<base_url>/models` with the actual credential and verify the requested model ID appears.
5. Send a minimal chat completion using that exact model.
6. Only after both calls succeed report the provider and model as functional.
### Verify Ollama as a fallback
1. Confirm the binary, service, and listener. Keep it local-only unless remote access was explicitly requested.
2. Pull the exact configured model.
3. Verify both `/api/tags` and OpenAI-compatible `/v1/models` list it.
4. Send a deterministic `/v1/chat/completions` probe and verify returned content.
5. Confirm `model.fallbacks` and, when required, `delegation.fallback` reference the same provider/model.
Installing a model is not complete until inference succeeds. A configured provider with an empty model list is not operational.
## Config change workflow
```text
User request → check docs/skill/source for real key →
if EXISTS → use `hermes config set` or edit via `hermes config edit`
if NOT FOUND → tell user it doesn't exist, suggest alternatives
```
## When the user corrects you
If the user calls out a made-up config key:
- Acknowledge immediately — don't deflect
- Remove the invalid key from config.yaml via the proper CLI or a direct edit
- Document the lesson
## Known missing features (as of 2026-07-05)
| User wants | Hermes has | Workaround |
|---|---|---|
| Automatic provider failover | `fallback_providers` does NOT exist | Manual provider switch via CLI, or local ollama for maintenance tasks |
| Multiple fallback providers | Not supported | N/A |
| Provider priority list | Not supported | N/A |
## Web tools `use_gateway` pitfall
Hermes has two config knobs for web tools:
```yaml
web:
backend: firecrawl # Which web backend to use
use_gateway: false # true = route through Nous Portal gateway
```
**The trap:** When `web.use_gateway: true` (the default or a previously-set value), Hermes routes Firecrawl requests through **Nous Portal's gateway** — it ignores your `FIRECRAWL_API_KEY` from `.env` entirely. If you're not logged into Nous Portal (no OAuth session), `web_search` and `web_extract` fail with:
> "Web tools are not configured. Set FIRECRAWL_API_KEY for cloud Firecrawl..."
Even though the key IS set and valid. The fix is:
```bash
hermes config set web.use_gateway false
```
Then a new session (`/reset`) picks up the change. **Note:** `/reset` itself does NOT fix this — the problem is the config value, not the session state. Multiple resets without changing the config value will all fail the same way.
**Verification:** Test the key directly against Firecrawl's API before and after:
```bash
source ~/.hermes/.env 2>/dev/null
curl -s -X POST "https://api.firecrawl.dev/v1/search" \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query":"test","limit":1}'
# Expected: HTTP 200, {"success": true, "data": [...]}
```
**Key learning:** `web.use_gateway: true` is NOT a fallback or override layer — it *replaces* the local key. If you're self-hosting the Firecrawl key in `.env`, this must be `false`. Setting it back to `true` after switching to Nous Portal OAuth would be correct for that auth mode.
### Debugging flow for "web tools not configured"
```
1. Check .env: grep FIRECRAWL_API_KEY ~/.hermes/.env
→ If missing, add key. If present, proceed.
2. Check config: grep -A2 '^web:' ~/.hermes/config.yaml
→ If use_gateway: true, flip to false via:
hermes config set web.use_gateway false
→ Then /reset or start new session.
3. Test key: direct curl to api.firecrawl.dev (see above)
→ 200 = key valid. Non-200 = key expired/wrong.
4. Verify tools: try web_search or web_extract
```
skills/devops/hermes-backup/references/purpose-specific-buckets.md
+62
View File
@@ -0,0 +1,62 @@
# Purpose-Specific S3 Bucket Architecture
## Motivation
The original `hermes-live-sync.sh` was a monolithic script pushing everything — Hermes state, Docker volumes, Caddyfile, systemd configs — into a single `s3://hermes-vps-backups/live/` prefix. This made it hard to:
- Grant granular IAM access (Docker volumes accessible to scripts that only needed Hermes configs)
- Replicate specific data categories to different regions
- Audit which scripts own which files
- Reason about data isolation during DR
## Normalization (2026-07-08)
Split into 5 buckets plus 1 legacy bucket:
| Bucket | Primary Script | Data |
|---|---|---|
| `hermes-vps-backups/live/` | `hermes-live-sync.sh` | Hermes state: config.yaml, .env, state.db, sessions/, skills/, profiles/, cron/, memory_store.db |
| `hermes-vps-backups/hermes-full-backup/` | `hermes-backup.sh` | Daily full archive tarballs (~350MB compressed) |
| `itpropartner-system-configs/` | `hermes-system-config-sync.sh` | System configs: Caddyfile, systemd services, ops scripts, SSH keys, .env + AWS creds |
| `itpropartner-docker-volumes/` | `hermes-docker-sync.sh` | Docker volume data: Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama |
| `mikrotik-ccr-backups/` | `run-wisp-backup.sh` + `wisp-backup.py` | Router configs (home gateway, wisp gateway) + logs |
| `itpropartner-backups/` | (legacy — manual) | Old signatures, test files, migrated router config copies |
## Exclusions
The `hermes-vps-backups/live/` sync excludes: audio_cache, image_cache, cache, sandboxes, kanban, node, bin, logs, *.lock, .skills_prompt_snapshot.json, .update_check.
The `itpropartner-system-configs/` scripts/ sync excludes: `.backups/`, `migration-backup/`, `*.pyc`, `__pycache__/`.
The `itpropartner-system-configs/` ssh/ sync excludes: known_hosts, authorized_keys, config.
## Script Locations
All scripts at `/root/.hermes/scripts/`:
- `hermes-live-sync.sh` — Hermes state only
- `hermes-system-config-sync.sh` — System configs
- `hermes-docker-sync.sh` — Docker volumes
## Dashboard
The ops portal (`/var/www/ops/backups.html`) tracks all 6 bucket paths via `ops-data-collector.py` (every 5 min). The dashboard JS maps bucket keys to display names via `DISPLAY_NAMES`:
```js
'hermes-vps-backups': 'Hermes VPS (Live State)',
'hermes-full-backups': 'Hermes VPS (Full Archives)',
'mikrotik-backups': 'MikroTik Router Configs',
'itpropartner-system-configs': 'System Configs (Caddy, systemd, SSH)',
'itpropartner-docker-volumes': 'Docker Volumes (Docuseal, VW, Twenty)',
'itpropartner-backups': 'IT Pro Partner (Legacy)'
```
## Blocker
The IAM user `Hermes-User` lacks `s3:CreateBucket`. The two new buckets (`itpropartner-system-configs`, `itpropartner-docker-volumes`) must be created in the Wasabi Console, then versioning enabled via:
```bash
aws s3api put-bucket-versioning --bucket itpropartner-system-configs --versioning-configuration Status=Enabled --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3api put-bucket-versioning --bucket itpropartner-docker-volumes --versioning-configuration Status=Enabled --endpoint-url https://s3.us-east-1.wasabisys.com
```
Until the buckets exist, the new sync scripts will silently fail with `NoSuchBucket`.
skills/devops/hermes-backup/references/secret-management.md
+61
View File
@@ -0,0 +1,61 @@
# Secret Management — Hermes Environment
## Principle
All secrets in one file: `~/.hermes/.env`, chmod 600, backed up in S3.
Standalone token files (`.hetzner_token`, `.netcup_api_key`, etc.) are fragile — they get left behind during migration, broken during key rotation, and scripts that reference them fail silently when they're missing.
## Current secret inventory
| Secret | File location | Permissions | Notes |
|---|---|---|---|
| admin-ai API key | `~/.hermes/config.yaml` (2 places) | Root-readable | Also in `.env` as backup |
| Telegram bot tokens | `~/.hermes/.env` | 600 | Default + Anita profiles |
| Hetzner API token | `~/.hermes/.env` | 600 | Was standalone file |
| Netcup API key | `~/.hermes/.env` | 600 | Was standalone file |
| SMTP/IMAP passwords | `~/.config/himalaya/*.pass` | 600 | Himalaya standard |
| Wasabi S3 keys | `~/.aws/credentials` | 600 | AWS CLI standard |
| SSH keys | `~/.ssh/` | 600 | Standard |
## Consolidation workflow
When you discover a standalone secret file:
1. Read it and the current `.env`:
```
env_path = ~/.hermes/.env
standalone_path = ~/.hermes/scripts/.some_token
```
2. Load `.env` into a dict, add the new variable, write back:
```
env_vars['HETZNER_API_TOKEN'] = token_value
```
3. `chmod 600 ~/.hermes/.env`
4. Remove the standalone file:
```bash
rm ~/.hermes/scripts/.some_token
```
5. Patch any scripts that referenced the old path to source from `.env`.
## Reading secrets in scripts
Scripts should source `.env` at the top:
```bash
set -a; source ~/.hermes/.env; set +a
```
Python scripts should use:
```python
import os
from dotenv import load_dotenv
load_dotenv(os.path.expanduser("~/.hermes/.env"))
token = os.environ["HETZNER_API_TOKEN"]
```
## Special case: config.yaml inline keys
The admin-ai API key lives in `config.yaml` in two places — `providers.admin-ai.api_key` and `auxiliary.vision.api_key`. Hermes reads these directly from the YAML config, not from env vars. The `.env` copy serves as a backup record for recovery. If the key rotates, update both config.yaml entries AND the `.env` backup.
skills/devops/hermes-backup/references/siteground-backup-pattern.md
+96
View File
@@ -0,0 +1,96 @@
# SiteGround SFTP Backup Pattern
Back up websites from SiteGround shared hosting to Wasabi S3 via SFTP. SiteGround does not expose an API — SFTP is the only programmatic access method.
## Credentials
- Host: `sftp.siteground.net`
- Port: `18765` (non-standard)
- Username: `sftp7068-6b4804d3`
- Key: encrypted RSA private key with passphrase `LoveMyBoys73!`
- The key file lives at `/root/.ssh/siteground.key` (chmod 600)
## Connection Method
SiteGround uses an **encrypted RSA private key with a passphrase**. Plain key-based auth without the passphrase will fail. Two approaches:
### A. Python + Paramiko (preferred for scripting)
```python
import paramiko, io
key = paramiko.RSAKey.from_private_key(io.StringIO(key_data), password="<passphrase>")
transport = paramiko.Transport(("sftp.siteground.net", 18765))
transport.connect(username="sftp7068-6b4804d3", pkey=key)
sftp = paramiko.SFTPClient.from_transport(transport)
```
### B. sshpass + ssh (manual/scripting)
Not recommended for scripting due to passphrase-in-command exposure, but valid for manual testing:
```bash
sshpass -p "<passphrase>" ssh -i /root/.ssh/siteground.key \
-o StrictHostKeyChecking=no -o PubkeyAuthentication=yes \
-p 18765 sftp7068-6b4804d3@sftp.siteground.net "ls -la"
```
## Sites Present (Jul 2026)
21 sites — all WordPress, all under `/home/` on SiteGround:
815bistro.com, abortionstory.org, apextrackexperience.com, chiefenergyofficer.org, costanzospizzeria.com, elliscountyinternet.com, fixourclub.com, forefrontwireless.com, forefrontwireless.net, freedominhonesty.com, garyjavo.com, germainebrown.com, grandlakeclub.com, injenuity413.com, injenuitysolutions.com, itpropartner.com, katiewattsdesign.com, savannahevents.com, sotobunch.com, theabortionstory.org, voipsimplicity.com
## Backup to S3
Each site is a full directory tarball uploaded to `s3://hermes-vps-backups/siteground/<site>/`:
```python
import tarfile, io
# Stream tar.gz to S3 to avoid filling /tmp on large sites
buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode='w:gz') as tar:
tar.add(f"/home/{site}", arcname=".")
buf.seek(0)
# Upload directly
s3_client.upload_fileobj(buf, "hermes-vps-backups", f"siteground/{site}/{site}-{date}.tar.gz")
```
For very large sites (apextrackexperience.com, itpropartner.com, voipsimplicity.com), write to `/tmp` first then upload — streaming through memory may timeout on SFTP.
## Exclusion Recommendations
WordPress sites on SiteGround typically contain:
- `wp-content/uploads/` — images, media (backup all)
- `wp-content/plugins/` — can reinstall, but backup for config preservation
- `wp-content/themes/` — custom themes, backup
- `wp-content/cache/` — OK to skip
- `error_log` — large debug logs, OK to skip
## Restore
1. Download tar.gz from S3
2. Extract on new host: `tar xzf <site>-<date>.tar.gz`
3. Set correct permissions: `chown -R www-data:www-data <site>/`
4. Import database from phpMyAdmin export or MySQL dump
5. Update `wp-config.php` with new DB credentials
6. Point DNS to new server
## Database
SFTP provides **files only**. The MySQL database must be exported separately:
- Via SiteGround's phpMyAdmin interface
- Or: `mysqldump -h <host> -u <user> -p <dbname> > <site>-db-<date>.sql`
- Upload the SQL dump to Core via SFTP/scp and store alongside the files
## Migration Strategy
1. SFTP backup all sites to S3 (done Jul 9, 2026)
2. Export each site's database from phpMyAdmin → upload to S3
3. On new box, restore files + import DB
4. Update wp-config.php credentials
5. Point DNS → done
No downtime per site if DNS is the last step.