Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,385 @@
|
||||
---
|
||||
name: disaster-recovery-audit
|
||||
description: "DR audit framework: S3 backup verification, warm standby failover testing, config/password inventory, issue tracking, and remediation. Used for periodic full-system audits and post-outage recovery verification."
|
||||
version: 2.0.0
|
||||
author: Sho'Nuff
|
||||
tags: [devops, disaster-recovery, backup, audit, failover, s3, infrastructure]
|
||||
---
|
||||
|
||||
# Disaster Recovery Audit
|
||||
|
||||
Standard procedure for auditing the Hermes infrastructure DR posture. Covers S3 backups, warm standby, config/password inventory, cron health, and issue tracking. This umbrella absorbs the content of `infrastructure-audit` (deleted 2026-07-08, was a duplicate of this skill).
|
||||
|
||||
## Audit Checklist
|
||||
|
||||
Run these checks in order during any full DR audit:
|
||||
|
||||
### 2. Cron Job Health (updated Jul 10, 2026)
|
||||
- Use `cronjob(action='list')` or system crontab to get all scheduled jobs
|
||||
- Check `last_status` on each — flag any that are `error`
|
||||
- Specific jobs to verify:
|
||||
|
||||
**Hermes cron scheduler:**
|
||||
- `hermes-live-sync` — every 15 min, should be OK
|
||||
- `hermes-system-config-sync` — every 15 min, should be OK (Caddyfile, systemd, scripts, SSH keys, env)
|
||||
- `hermes-docker-sync` — every 15 min, should be OK (Docuseal, Vaultwarden, TwentyCRM, SearXNG, Ollama)
|
||||
- `service-health-check` — every 5 min, should be OK
|
||||
- `home-router-watchdog` — every 5 min, should be OK
|
||||
- `bounce-check` — every 60 min, should be OK
|
||||
- `home-router-daily-backup` — 6 AM daily, must exist (RESOLVED Jul 10 — paramiko installed, live test passed)
|
||||
- `hetzner-weekly-snapshots` — Monday 5 AM, must exist
|
||||
|
||||
**System crontab (NOT in Hermes cron — blocked by gateway lifecycle guard):**
|
||||
- `hermes-full-backup` — 1:00 AM UTC daily via `/root/.hermes/scripts/hermes-backup.sh`
|
||||
- `root-essentials-backup` — 3:00 AM UTC daily via `/root/.hermes/scripts/root-essentials-backup.sh`
|
||||
- `backup-audit` — 2:00 AM UTC daily via `/root/.hermes/scripts/backup-audit-check.sh`
|
||||
|
||||
**CRON BACKUP PITFALL — AWS CLI venv:** Backup scripts that call `aws s3 cp` will fail silently in cron with `aws: command not found` unless they activate the AWS CLI virtualenv first. The shell PATH in a cron context is minimal — even if `aws` works in your interactive session, it may not in cron. **Every backup script that uploads to S3 MUST include this block after `set -euo pipefail`:**
|
||||
|
||||
```bash
|
||||
# Activate AWS CLI from venv (required for cron — PATH is minimal)
|
||||
if [ -f /opt/awscli-venv/bin/activate ]; then
|
||||
source /opt/awscli-venv/bin/activate
|
||||
fi
|
||||
```
|
||||
|
||||
- `hermes-backup.sh` was missing this line until Jul 11, 2026 — caused both nightly full and root-essentials backups to fail silently
|
||||
- `root-essentials-backup.sh` HAS this line but may still fail if the venv doesn't exist or AWS CLI wasn't installed there
|
||||
- Verify by running `bash -x /path/to/backup.sh 2>&1 | grep -i "aws: command"` in the audit
|
||||
- After patching any backup script, run it manually from cron's stripped environment to confirm: `env -i HOME=$HOME PATH=/usr/bin:/bin bash /path/to/backup.sh`
|
||||
|
||||
**GATEWAY LIFECYCLE GUARD (#30719):** The full backup script (`hermes-backup.sh`) is blocked by the gateway lifecycle guard when run as a Hermes cron job. The guard prevents agent-driven SIGTERM-respawn loops. Symptom: cron job creation returns error `Blocked: cron job contains a gateway lifecycle command (restart/stop/kill)`. There is NO such command in the script — the guard triggers on script presence alone via the embedded restore.sh heredoc. FIX: The full backup MUST run from system crontab instead. Use `(crontab -l; echo "0 1 * * * /path/to/script.sh 2>&1 | logger -t jobname") | crontab -` to add. The audit watchdog runs at +1h to verify backup completion. Also applies to any agent-created cron job that shell-sources or wraps a script the guard considers risky.
|
||||
|
||||
**Subagent model config drift:** Setting `delegation.model` via `hermes config set` does NOT affect the running gateway process. The gateway caches config at startup. Changes sit in config.yaml but won't be picked up by subagents until `hermes gateway restart` runs from outside the gateway. Do not debug subagent failures by re-applying config changes; restart the gateway instead. Also: Hermes cron jobs created under one model config will SKIP their run if the model config drifted later — fix by pinning via `cronjob action=update job_id=... provider=... model=...`.
|
||||
|
||||
### 2. S3 Bucket Integrity
|
||||
Check all 5 Wasabi S3 buckets for recent activity:
|
||||
|
||||
| Bucket | Purpose | Sync Script | Key Check |
|
||||
|--------|---------|------------|-----------|
|
||||
| `hermes-vps-backups` | Hermes config/sessions/profiles | `hermes-live-sync.sh` (every 15 min) | Has `live/`, `standby/`, `hermes-full-backup/` |
|
||||
| `itpropartner-system-configs` | System configs (Caddy, systemd, SSH, scripts, env) | `hermes-system-config-sync.sh` (every 15 min) | Bucket created Jul 10, versioning enabled, IAM policy updated |
|
||||
| `itpropartner-docker-volumes` | Docker volumes (Docuseal, VW, Twenty, SearXNG, Ollama) | `hermes-docker-sync.sh` (every 15 min) | Bucket created Jul 10, versioning enabled, IAM policy updated |
|
||||
| `mikrotik-ccr-backups` | Router configs | `wisp-backup.py` (daily 6 AM) | Has recent uploads |
|
||||
| `itpropartner-backups` | Legacy files (mostly migrated) | None (no writer) | Known stale -- last write Jul 7 |
|
||||
|
||||
**For each bucket, verify:**
|
||||
- Versioning enabled (via S3 API)
|
||||
- Most recent upload timestamp -- flag if >48h stale
|
||||
- Sub-paths exist: `live/`, `standby/`, `hermes-full-backup/`
|
||||
- For system-configs/docker-volumes: verify the sync script exists, is chmod 755, and its cron job is active
|
||||
- If a sync script targets a bucket that doesn't exist yet (itpropartner-system-configs, itpropartner-docker-volumes), flag as blocked -- buckets must be created in Wasabi Console first
|
||||
- Full backup tarball — flag if >48h stale
|
||||
- Expected file sizes (not 0 bytes, WAL/SHM artifacts excepted)
|
||||
|
||||
**BACKUP INTEGRITY VERIFICATION (mandatory for every audit):** S3 listing alone is NOT sufficient — a tarball can exist on S3 and still be corrupt. For the full backup and root-essentials backup, download and test:
|
||||
|
||||
```bash
|
||||
# Download the latest tarball
|
||||
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-$(date +%F).tar.gz \
|
||||
/tmp/dr-audit-verify.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
|
||||
# Test tar integrity
|
||||
tar tzf /tmp/dr-audit-verify.tar.gz > /dev/null 2>&1 && \
|
||||
echo "VALID: $(tar tzf /tmp/dr-audit-verify.tar.gz | wc -l) files" || \
|
||||
echo "CORRUPT — archive fails tar tzf"
|
||||
|
||||
# Clean up
|
||||
rm /tmp/dr-audit-verify.tar.gz
|
||||
```
|
||||
|
||||
Do NOT skip this step. The Jul 11, 2026 audit discovered that BOTH the nightly full backup AND root-essentials backup had failed silently — the S3 listing showed only stale Jul 10 files. The root cause was `aws: command not found` in cron (see Cron Job Health section above). The backup audit watchdog only checks S3 listing freshness, not integrity.
|
||||
|
||||
### S3 Command Pitfalls (Write-Only IAM)
|
||||
When transitioning AWS/Wasabi S3 backup scripts to use strict write-only IAM policies (e.g., `s3:PutObject` only, no `s3:ListBucket`), you **cannot use `aws s3 sync`**.
|
||||
- `aws s3 sync` fundamentally requires list permissions to compare the source against the destination.
|
||||
- **Fix:** Rewrite sync scripts to create local archives (e.g., `tar`) and perform blind uploads using `aws s3 cp`.
|
||||
- Scripts that read from S3 (e.g., `aws s3 ls` for audit checks or downloading for standby restores) must be separated into a different IAM user/role with read permissions, or moved to a pre-signed URL workflow.
|
||||
|
||||
### 3. Password & Config File Inventory
|
||||
|
||||
| File | Expected | Permission |
|
||||
|------|----------|------------|
|
||||
| `/root/.hermes/config.yaml` | 600 | EXIST |
|
||||
| `/root/.hermes/.env` | 600 | EXIST |
|
||||
| `/etc/caddy/Caddyfile` | 640 | IN BACKUP |
|
||||
| `/root/.config/himalaya/g-germainebrown.pass` | 600 | EXIST |
|
||||
| `/root/.config/himalaya/shonuff.pass` | 600 | EXIST |
|
||||
| `/root/.config/himalaya/g-germainebrown-icloud-calendar.pass` | 600 | EXIST |
|
||||
| `/root/.aws/credentials` | 600 | EXIST |
|
||||
| `/root/.ssh/itpp-infra` | 600 | EXIST |
|
||||
| `/root/.hermes/migration-creds.txt` | 600 | IN BACKUP |
|
||||
| `/root/.hermes/references/dre-temp-passwords.txt` | 600 | EXIST |
|
||||
|
||||
**Check each:**
|
||||
- Does the file exist?
|
||||
- Is the permission correct? (flags 644 → 600, 755 is fine for scripts)
|
||||
- Is it included in backup scripts? (check `hermes-backup.sh` and `hermes-live-sync.sh`)
|
||||
|
||||
### 4. Systemd Services
|
||||
|
||||
List custom services and verify they're backed up to correct path:
|
||||
|
||||
```bash
|
||||
ls /etc/systemd/system/hermes*.service /etc/systemd/system/shark-game*.service /etc/systemd/system/*gateway*.service
|
||||
```
|
||||
|
||||
**Service naming reality vs expected names:**
|
||||
- The main Hermes unit is `hermes.service` (NOT `hermes-agent.service`). The gateway runs embedded inside `hermes.service` as the main process.
|
||||
- `hermes-assistant.service` and `hermes-browser.service` exist as separate systemd units.
|
||||
- A user-level `hermes-gateway-anita.service` may exist at `~/.config/systemd/user/` for the Anita profile.
|
||||
- There is NO separate `hermes-gateway.service` systemd unit on the Core server -- the gateway runs inline.
|
||||
- If you run `systemctl status hermes-agent` or `hermes-gateway` and get "could not be found", check `hermes.service` instead. The process running is `hermes gateway`.
|
||||
- Running `hermes gateway status` returns the correct state showing gateway is running inside the main service.
|
||||
|
||||
Backup script must collect from `/etc/systemd/system/` NOT `~/.config/systemd/user/`.
|
||||
|
||||
### 5. Warm Standby (app1-bu)
|
||||
|
||||
SSH to the standby server via `itpp-infra` key and verify:
|
||||
|
||||
| Check | Command | Expected |
|
||||
|-------|---------|----------|
|
||||
| Watchdog cron | `crontab -l \| grep standby` | `*/5 * * * *` |
|
||||
| Sync cron | `crontab -l \| grep sync` | `*/10 * * * *` |
|
||||
| Watchdog script | `cat` the script | 30s x 4 cycles = 2 min confirmation |
|
||||
| Sync script | `cat` the script | Exists, pulls from S3 every 10 min |
|
||||
| AWS CLI | `source /opt/awscli-venv/bin/activate && aws s3 ls ...` | Returns data |
|
||||
| Config freshness | `ls -la ~/.hermes/config.yaml` | Recent (today/tomorrow) |
|
||||
| Gateway status | `hermes gateway status` | Inactive/dead (dormant) |
|
||||
| Network to Core | `ping -c 2 152.53.192.33` | Successful (low ms) |
|
||||
| Disk space | `df -h /` | >20% free (CRITICAL if <10%: failover will fail) |
|
||||
|
||||
## Operational Truth and Restore Verification (Critical)
|
||||
|
||||
A plan, command shown in chat, or agent statement is not an executed operation. During incidents and restores:
|
||||
|
||||
- Never report a restart, reboot, DNS change, Caddy reload, provider test, failover, message delivery, disk cleanup, or restore as successful without tool output proving it.
|
||||
- Distinguish **requested**, **started**, **completed**, and **verified** states explicitly.
|
||||
- Verify a host reboot using boot ID or uptime before and after; an uninterrupted chat session is not evidence either way.
|
||||
- Verify DNS against the authoritative nameserver and at least two public resolvers. Compare with the local resolver because local caches can disagree with public DNS.
|
||||
- Verify the actual service name before operating it. On Core, the gateway is embedded in `hermes.service`; guessed units such as `hermes-gateway.service` are not evidence.
|
||||
- Verify failover on both sides: active Core state, standby gateway/process state, synchronized-state timestamp, disk headroom, and an end-to-end message.
|
||||
- A process listing proves only that a process exists. It does not prove Telegram delivery, provider authentication, or successful inference.
|
||||
- Subagent reports are leads. Read back files, query remote state, and verify external side effects before repeating their claims.
|
||||
- If verification cannot be performed, say `not verified`; do not fill the gap with expected output.
|
||||
|
||||
## Scope Discipline (Critical)
|
||||
|
||||
When the user asks you to fix or audit a single server/app/bucket, DO ONLY THAT. Do not:
|
||||
- Touch the config of other servers you weren't asked about
|
||||
- Update credentials, API keys, or secrets in config files without explicit direction
|
||||
- Fix "related" problems you noticed along the way unless the user says "while you're at it"
|
||||
- Propagate changes to other profiles (Anita, Tony, etc.) unless specifically asked
|
||||
|
||||
Every time I broke the session tonight, it was because I self-assigned extra scope. Fixing app1-bu? Fix app1-bu. Generating an API key? Generate it — don't wire it in. If you see a related problem, report it as a finding and ask if they want it fixed. Do not decide for them.
|
||||
|
||||
This is not a suggestion — it was a repeated source of frustration this session.
|
||||
|
||||
## Issue Log Format
|
||||
|
||||
The DR issue log lives at `/root/.hermes/references/dr-issue-log.md`.
|
||||
|
||||
**Format rules (enforced Jul 8, 2026 after user called out unreadable pipe tables and garbled emoji on iPhone):**
|
||||
- **NO markdown pipe tables in the summary** -- use bullet lists grouped by status (`**Fixed** [OK]` then bullet items, `**Resolved** [INFO]`, `**Investigating** [PENDING]`)
|
||||
- **ASCII-only characters throughout** -- no emoji (`🔴` `🟡` `🟢` `✅` `🔄`), no em dashes (use ` -- `), no arrows (use `->`)
|
||||
- **Status labels:** `[OK]` for fixed, `[PENDING]` for investigating, `[HIGH]` `[MED]` `[INFO]` for severity
|
||||
- Each entry follows: Problem -> Root Cause -> Fix -> Verification
|
||||
- Apply ASCII-only rule to ALL reference files the user may read from a phone (Telegram file preview, email, open-in-browser)
|
||||
|
||||
```\n### DR-NNN: Short Title\n**Problem:** What happened or what was found\n**Root cause:** Why it happened\n**Fix:** What was done to resolve it\n**Status:** [OK] Fixed YYYY-MM-DD / [PENDING] Investigating\n**Verified by:** How we confirmed the fix works\n```
|
||||
|
||||
New entries are appended to the bottom. Existing entries are updated (status/verification) when a previously resolved issue is confirmed working on revisit.
|
||||
|
||||
## Failover Architecture
|
||||
|
||||
**Design:** Warm standby. app1-bu runs 24/7 with Hermes inactive. Synchronizes state from S3 every 10 min. If Core goes down, watchdog confirms over 2 min (4 x 30s), then fails over.
|
||||
|
||||
For a full infrastructure reference (IPs, ports, URLs, S3 buckets, API endpoints), see `references/network-and-service-endpoints.md`.
|
||||
|
||||
For a **comprehensive infrastructure recovery manual** covering all 10 servers, every service, Docker containers, S3 backups, router networking, shared credentials, and step-by-step disaster scenarios, see `references/itpp-recovery-manual.md`. This was generated in July 2026 by synthesizing the server audit, app inventory, Docker compose files, Caddy config, systemd units, S3 bucket listings, and all deployment references into a single manual.
|
||||
|
||||
For **gateway service lifecycle diagnostics** -- what to do when `hermes gateway restart` fails (linger, DBus, dual-process conflicts, stale systemd units, profile-specific gateways) -- see `references/gateway-lifecycle-troubleshooting.md`.
|
||||
|
||||
For **backup pipeline failure patterns and integrity verification** — the Jul 11, 2026 audit that caught silent cron failures (`aws: command not found`) and the standby disk crisis — see `references/dr-audit-findings-2026-07-11.md`.
|
||||
## When to update the recovery manual
|
||||
|
||||
- A new service is deployed on Core -- add to Section 5
|
||||
- A Hetzner server changes IP, hostname, or provider -- update Section 1 table
|
||||
- A Caddy domain or port mapping changes -- update Section 1 domain map and affected server section
|
||||
- A backup script or cron job is added/removed -- update Sections 8 and 13
|
||||
- A server is decommissioned or provisioned -- update Section 1 and add/remove from Sections 2 or 7
|
||||
- **User corrects the manual's scope** -- if the user says "this is too narrow" or "it's not only for X", broaden immediately. The manual must cover the ENTIRE infrastructure, not a single service. The Jul 9 session proved that scoping to one project (shark-game-recovery-manual.md) required near-immediate replacement with a full ITPP version.
|
||||
|
||||
```
|
||||
┌──────────────────────┐ ┌───────────────────────┐
|
||||
│ Core (netcup) │ │ app1-bu (Hetzner) │
|
||||
│ 152.53.192.33 │ │ 5.161.114.8 │
|
||||
│ RS 2000 (8C/16G) │ │ CPX11 (2C/2G) │
|
||||
│ Active Hermes │ │ Dormant Hermes │
|
||||
│ Active gateway │ │ Active watchdog │
|
||||
└────────┬─────────────┘ └──────────┬────────────┘
|
||||
│ S3 sync (every 15 min) │ S3 sync (every 10 min)
|
||||
└──────────────┬───────────────┘
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Wasabi S3 │
|
||||
│ hermes-vps- │
|
||||
│ backups/live/ │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
### Failover Sequence
|
||||
1. **Watchdog** (runs every 5 min via cron) pings Core (`152.53.192.33`)
|
||||
2. If no ping response: check every **30s × 4 cycles** (~2 min total confirmation)
|
||||
3. If still unreachable: sync latest state from S3 → start Hermes gateway → send **Telegram alert** + **email alert**
|
||||
4. User communicates with standby instance
|
||||
|
||||
### Fallback Sequence (recovery)
|
||||
1. Core comes back online
|
||||
2. Shut down standby gateway (hermes gateway stop) or leave both running — user decides
|
||||
3. If standby stays active, it keeps syncing. If powering off reduces confusion, do that.
|
||||
|
||||
### Recovery & Sync Timing
|
||||
|
||||
| Component | Interval | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| Live S3 sync (Core → S3) | Every 15 min | Pushes latest state from active server |
|
||||
| Standby sync (S3 → app1-bu) | Every 10 min | Pulls latest state to standby |
|
||||
| Watchdog health check | Every 5 min | Pings Core to detect failure |
|
||||
| Confirmation window | 4 × 30s = 2 min | Proves failure before failover |
|
||||
| Max total outage detection | ~7 min | Worst case: 5 min to next check + 2 min confirm |
|
||||
|
||||
## DR Issue Log Maintenance
|
||||
|
||||
The DR issue log at `/root/.hermes/references/dr-issue-log.md` is the permanent record of all audit findings, fixes, root causes, and verification dates.
|
||||
|
||||
### Entry format
|
||||
|
||||
**Format convention (CRITICAL — mobile-first rendering):**
|
||||
- NO markdown pipe/tables in the summary — unreadable on mobile
|
||||
- NO emoji or Unicode characters — plain ASCII labels only
|
||||
- Use `[HIGH]`, `[MED]`, `[INFO]`, `[OK]`, `[PENDING]` for severity/status
|
||||
- Use double hyphens (`--`) not em dashes
|
||||
- Summary is grouped bullet lists by status, see example below
|
||||
- Each entry has Problem -> Root Cause -> Fix -> Verification blocks
|
||||
- Separate entries with `---`
|
||||
|
||||
Each entry MUST contain all 5 fields:
|
||||
|
||||
```
|
||||
### DR-NNN: Short Title
|
||||
**Problem:** What happened or what was found
|
||||
**Root cause:** Why it happened
|
||||
**Fix:** What was done to resolve it
|
||||
**Status:** [OK] Fixed YYYY-MM-DD / [PENDING] Investigating
|
||||
**Verified by:** How we confirmed the fix works
|
||||
```
|
||||
|
||||
### When to add entries
|
||||
- Any gap found during audit
|
||||
- Any untested failover path
|
||||
- Any configuration that was updated but not reflected in docs
|
||||
- Any server whose Hetzner type, tag, or purpose differs from DR plan
|
||||
- Any new server provisioned or decommissioned
|
||||
- Any cron job that shows a new error state
|
||||
- **Fault is common, not exceptional** — when multiple systems fail in the same audit, log each as a separate DR issue. Don't collapse the cluster into one.
|
||||
- **VPS resource threshold crossing** — when the automated VPS threshold checker (running every 15 min, checking 80/90/95% levels on RAM/disk/CPU across all servers) fires a new alert that requires action (e.g. ai.itpropartner.com disk at 92%). Log as a DR issue and investigate.
|
||||
|
||||
### When to update existing entries
|
||||
- A previously marked `[PENDING]` entry is now resolved
|
||||
- Verification step confirms the fix is still working
|
||||
|
||||
## Memory Consolidation Safety (Jul 9, 2026)
|
||||
|
||||
The memory consolidation system (`hermes-consolidate.sh` + `hermes-consolidate.py`) runs every 10 min and prunes stale entries from MEMORY.md. It is a DR-critical component — if it over-prunes, identity and rules entries can be permanently lost from active memory.
|
||||
|
||||
### Architecture
|
||||
- **Backup-before-prune:** S3 upload is a hard gate — prune never runs without it
|
||||
- **Local retention:** 48h of pre-prune copies under `~/.hermes/memories/.consolidate-backups/`
|
||||
- **S3 history:** Timestamped copies at `s3://hermes-vps-backups/live/memories/history/`
|
||||
|
||||
### PROTECT entry types (these are NEVER pruned)
|
||||
The Python script guards entries containing:
|
||||
- Identity: `Sho'Nuff Brown`, `shonuff@`, `germainebrown.com`
|
||||
- Rules: `rule`, `never make up`, `fabrication`, `Germaine's #1 rule`
|
||||
- Infrastructure: `Core:`, `STANDING PRACTICE`, `app1-bu`, `netcup`, `admin-ai.itpropartner`, `Ops portal`, `Recovery manual`, `Ollama`, `RS 4000`, `Migration target`
|
||||
- Credentials: `credential`, `API`, `DR issue log`, `NETCUP_PASSWORD`, `LoveMyBoys`, `itpp-infra`
|
||||
- Email: `email`, `signature`, `format`, `shonuff.py`
|
||||
|
||||
### If memory gets over-pruned
|
||||
Restore from S3 backup — the JSON schema is `{backed_up_at, source, entry_count, char_count, entries}`.
|
||||
|
||||
### Known limitation
|
||||
The size safety valve drops oldest-appended entries when pattern-pruning doesn't reach 7,000 chars. Always verify after first consolidation run. If a critical entry was lost, restore from S3 and add it to the PROTECT regex.
|
||||
|
||||
## Per-Server DR Plans — v3 Consolidated (Jul 10, 2026)
|
||||
|
||||
The v3 DR plan (`references/server-dr-plans-v3.md`) is the ACTIVE plan. It incorporates two expert DR reviews and replaces v1 (`references/server-dr-plans.md`). The v3 plan addresses every finding from the reviews.
|
||||
|
||||
**DOCX versions** (formatted for sharing) are at:
|
||||
- `references/1-DR-Master-Plan-v2.docx` — Firm RTO/RPO, failover/failback design, split-brain prevention
|
||||
- `references/2-Backup-Security-Standard.docx` — Object Lock, write-only IAM, secondary destination, retention tiers
|
||||
- `references/3-Per-Server-Runbooks.docx` — Restore steps per host with bracketed fields
|
||||
- `references/4-DR-Testing-Schedule.docx` — Test cadence, log templates, sign-off table
|
||||
|
||||
The v3 improvements:
|
||||
|
||||
**v3 Improvements:**
|
||||
- **Failover timing corrected:** 30s checks (was 5 min), failover after 4 failures = 2 min detection, ~3-3.5 min actual RTO -> 5 min committed target. The old 5-min polling interval made a 2-min RTO mathematically impossible.
|
||||
- **Split-brain prevention:** S3 active-lock.json file, fencing, external quorum required before takeover
|
||||
- **Failback protocol:** 12-step sequence with rollback path — no auto-failback, no improvisation during incidents
|
||||
- **S3 single point of failure mitigated:** Secondary backup destination (separate provider/account) designed
|
||||
- **Backup immutability designed:** Object Lock + write-only IAM production credentials (not yet implemented)
|
||||
- **Per-server restore runbooks:** Exact commands, dependency maps, DNS records, validation checks, known failure modes
|
||||
- **RPO honesty by data type:** Core = 15 min (Hermes state) / 24h (Docker data), not a single number
|
||||
- **Testing schedule:** Daily automated checks -> monthly DB restore -> quarterly failover/failback -> biannual tabletop
|
||||
|
||||
**Per-server RTO/RPO targets (v3):**
|
||||
|
||||
| System | RTO | RPO |
|
||||
|---|---|---|
|
||||
| Core | 5 min | 15 min |
|
||||
| app1/2/3 | 60 min | 24h |
|
||||
| wphost02 | 2h | 24h |
|
||||
| fleettracker360 | 2h | 24h |
|
||||
| Legacy Hetzner | 4h | 24h |
|
||||
|
||||
**Known gaps remaining (Jul 10, 2026):**
|
||||
- Docker volume backup automation not yet deployed
|
||||
- MariaDB dumps on fleettracker360 not automated
|
||||
- S3 secondary backup destination not provisioned
|
||||
- Write-only IAM credentials not yet created (policy designed, pending Wasabi Console)
|
||||
- 30-second watchdog deployed but needs load-test verification
|
||||
|
||||
**Resolved this session (Jul 10):**
|
||||
- Object Lock enabled on all 5 buckets ✅
|
||||
- S3 buckets itpropartner-system-configs + itpropartner-docker-volumes created, versioned, IAM updated ✅
|
||||
- Daily full backup cron restored (system crontab at 1 AM UTC) ✅
|
||||
- Daily backup audit watchdog deployed (2 AM UTC) ✅
|
||||
- Home router backup paramiko error resolved (live test passed) ✅
|
||||
- wphost02 full backup captured (656MB to S3) ✅
|
||||
- AI server disk at 92% — documented as P0, deferred to next maintenance window
|
||||
|
||||
**File locations:**
|
||||
- `references/server-dr-plans-v3.md` — ACTIVE (replaces all prior)
|
||||
- `references/server-dr-plans.md` — v1 (superseded)
|
||||
- `references/server-dr-plans-redacted.md` — for 3rd party review (IPs/credentials stripped)
|
||||
|
||||
## Cross-Reference with Ops Collector Data
|
||||
|
||||
After completing sections 1-5, validate the ops portal's live data against the DR issue log:
|
||||
|
||||
```bash
|
||||
cat /var/www/ops/data/ops-status.json | python3 -m json.tool 2>/dev/null | grep -A5 '"s3_backups"' | head -20
|
||||
cat /var/www/ops/data/ops-status.json | python3 -m json.tool 2>/dev/null | grep -B2 '"status": "error"' | head -20
|
||||
```
|
||||
|
||||
**Cross-reference checklist:**
|
||||
- [ ] DR issues marked [OK] Fixed -- does the ops collector confirm the fix is still working?
|
||||
- [ ] DR issues marked [PENDING] Investigating -- what does live data show now? Any new evidence?
|
||||
- [ ] Any new cron errors or stale buckets that aren't in the DR issue log yet? (open new entries)
|
||||
- [ ] S3 normalization buckets still missing? (itpropartner-system-configs, itpropartner-docker-volumes show NoSuchBucket)
|
||||
- [ ] Full backup fresh? (hermes-full-backups age_hours should be < 48)
|
||||
|
||||
If the ops collector shows a status that contradicts the DR issue log (e.g., DR issue marked [OK] Fixed but the error persists in live data), re-open the issue rather than silently updating the log. If a new error appears that isn't logged, create a new DR entry.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Apex WPForms Mail Debugging
|
||||
|
||||
Quick reference for diagnosing and fixing Apex Track Experience WPForms email delivery failures.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Value |
|
||||
|-----------|-------|
|
||||
| Server | wphost02 (5.161.62.38, Hetzner CPX21) |
|
||||
| DB | MySQL via RunCloud |
|
||||
| SMTP | c1113726.sgvps.net:2525, STARTTLS |
|
||||
| SMTP user | contact@apextrackexperience.com |
|
||||
| SMTP pass | apex.track!! |
|
||||
| Plugin | WP Mail SMTP (Lite) |
|
||||
| Forms | NASA registration (270), Waiver (268) |
|
||||
| Last known SMTP creds | Username: `contact@apextrackexperience.com`, Password: `apex.track!!` |
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
### 1. PHP Serialized Password Mismatch (most common cause)
|
||||
|
||||
WP Mail SMTP stores the SMTP password in the `wp_options` table as `wp_mail_smtp` in PHP serialized format. If the declared string length doesn't match the actual string, the plugin silently fails to authenticate.
|
||||
|
||||
**Symptoms:** Form submissions appear successful in the UI, but no email is sent. The debug events table shows `event_type = 0` with no useful error text.
|
||||
|
||||
**Check command:**
|
||||
```sql
|
||||
SELECT JSON_EXTRACT(option_value, '$.smtp.pass') FROM wp_options WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
The JSON_EXTRACT result shows the raw serialized PHP value. A broken entry looks like:
|
||||
```
|
||||
s:72:\"apex.track!!\"
|
||||
-- length 72 declared, but 'apex.track!!' is only 13 characters
|
||||
```
|
||||
|
||||
**Fix command:**
|
||||
```sql
|
||||
UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\"apex.track!!\"', 's:13:\"apex.track!!\"') WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
### 2. WPForms sender_address
|
||||
|
||||
Both forms had `contact@apextrackexperience.com, g@germainebrown.com` as the `sender_address`. This is invalid — `From:` headers require a single address. An SMTP server receiving two comma-separated addresses may reject the `MAIL FROM` command.
|
||||
|
||||
**Check command (form 270 and 268):**
|
||||
```sql
|
||||
SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications') AS notifs FROM wp_posts WHERE ID IN (270, 268);
|
||||
```
|
||||
|
||||
**Fix:** Set `sender_address` to `contact@apextrackexperience.com` only. The notification recipient `g@germainebrown.com` is configured separately as an email notification destination, not the sender.
|
||||
|
||||
### 3. SMTP Network Reachability
|
||||
|
||||
From wphost02:
|
||||
```bash
|
||||
# Test connection (not sending mail)
|
||||
timeout 5 bash -c 'echo | openssl s_client -connect c1113726.sgvps.net:2525 -starttls smtp'
|
||||
```
|
||||
|
||||
The `CONNECTED` line should appear in the output.
|
||||
|
||||
### 4. Debug Events Table
|
||||
|
||||
WP Mail SMTP logs email delivery attempts to `wp_wpmailsmtp_debug_events`:
|
||||
```sql
|
||||
SELECT id, subject, status, date_sent, error_text FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 5;
|
||||
```
|
||||
|
||||
- `event_type = 0` — email sent (or attempted)
|
||||
- `event_type = 1` — email failed
|
||||
- Check `created_at >= NOW() - INTERVAL 10 MINUTE` for recent attempts
|
||||
|
||||
## Watchdog
|
||||
|
||||
A cron job on Core monitors Apex mail every 5 minutes:
|
||||
- `/root/.hermes/scripts/apex-mail-watchdog.sh`
|
||||
- Tests SMTP LOGIN (does NOT send a test email — only verifies credentials)
|
||||
- Checks debug events for recent failures
|
||||
- Silent on success, alerts Germaine on failure
|
||||
- Do NOT use `tee -a` in the log function — it causes every log line to be delivered as a cron message
|
||||
|
||||
## Watchdog Blindness (SSH-dependency failure mode)
|
||||
|
||||
The watchdog runs FROM Core and SSHes to wphost02 (`root@5.161.62.38`, key `/root/.ssh/itpp-infra`, port 22) to test SMTP and read the debug table. Both steps require that SSH hop. **If SSH is refused/unreachable, the watchdog keeps "running" every 5 min but tests NOTHING** — the log fills with:
|
||||
|
||||
```
|
||||
SMTP FAILED: ssh: connect to host 5.161.62.38 port 22: Connection refused
|
||||
```
|
||||
|
||||
This is a monitoring blind spot, not a mail failure: the watchdog cannot distinguish "SMTP broken" from "I can't reach the box." When auditing Apex mail, ALWAYS confirm the SSH hop works before trusting a "FAIL" verdict:
|
||||
|
||||
```bash
|
||||
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=5 root@5.161.62.38 'echo reachable'
|
||||
tail -5 /var/log/apex-mail-watchdog.log # look for "connect to host ... Connection refused"
|
||||
```
|
||||
|
||||
If SSH is refused: the box may be down, rebooting, mid-migration, or its SSH port/IP changed. **During the Hetzner->netcup migration this is expected churn** — wphost02 (Apex/WordPress) is slated to move to app3, which changes IP and possibly port. When a WordPress/mail host migrates, update `WPHOST` and any port in `apex-mail-watchdog.sh` and re-verify the hop. Do not report "Apex mail is down" from a `Connection refused` line alone.
|
||||
|
||||
## Watching the Watchdog
|
||||
|
||||
```bash
|
||||
# Check the watchdog cron
|
||||
cronjob action=list | grep apex
|
||||
|
||||
# View the watchdog log
|
||||
cat /var/log/apex-mail-watchdog.log
|
||||
|
||||
# The ops portal at ops.itpropartner.com/cron.html shows the job status live
|
||||
```
|
||||
@@ -0,0 +1,69 @@
|
||||
# Apex WPForms Mail Fix (Post-Outage)
|
||||
|
||||
When Apex Track Experience form notifications (register/waiver) stop sending, follow this checklist:
|
||||
|
||||
## 1. Check Debug Events
|
||||
|
||||
```sql
|
||||
SELECT id, subject, status, date_sent, error_text
|
||||
FROM wp_wpmailsmtp_debug_events
|
||||
ORDER BY id DESC LIMIT 10;
|
||||
```
|
||||
|
||||
## 2. Check SMTP Password Serialization
|
||||
|
||||
The WP Mail SMTP plugin stores passwords in PHP serialized format in wp_options:
|
||||
|
||||
```sql
|
||||
SELECT option_value FROM wp_options WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
Look for: `s:72:"apex.track!!"` - the length prefix (72) must match actual password length (13).
|
||||
|
||||
**Fix:**
|
||||
```sql
|
||||
UPDATE wp_options
|
||||
SET option_value = REPLACE(option_value,
|
||||
's:72:\\"apex.track!!\\"',
|
||||
's:13:\\"apex.track!!\\"')
|
||||
WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
## 3. Check Form Sender Address
|
||||
|
||||
WPForms notification sender_address should be a single email, not comma-separated.
|
||||
|
||||
```sql
|
||||
SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications."1".sender_address') as sender
|
||||
FROM wp_posts WHERE ID IN (270, 268) AND post_type = 'wpforms';
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```sql
|
||||
UPDATE wp_posts SET post_content = REPLACE(
|
||||
post_content,
|
||||
'contact@apextrackexperience.com, g@germainebrown.com',
|
||||
'contact@apextrackexperience.com'
|
||||
) WHERE ID IN (270, 268);
|
||||
```
|
||||
|
||||
## 4. Re-send Missed Notifications
|
||||
|
||||
```sql
|
||||
SELECT entry_id, form_id, fields FROM wp_wpforms_entries
|
||||
WHERE DATE(date) = CURDATE() AND form_id IN (268, 270, 272);
|
||||
```
|
||||
|
||||
Re-send via SMTP Python script using the Apex SMTP server at c1113726.sgvps.net:2525.
|
||||
|
||||
## 5. Verify SMTP Connectivity
|
||||
|
||||
Test from the server:
|
||||
```python
|
||||
import smtplib, ssl
|
||||
ctx = ssl.create_default_context()
|
||||
with smtplib.SMTP('c1113726.sgvps.net', 2525, timeout=15) as s:
|
||||
s.starttls(context=ctx)
|
||||
s.login('contact@apextrackexperience.com', 'apex.track!!')
|
||||
print('OK')
|
||||
```
|
||||
@@ -0,0 +1,41 @@
|
||||
## Apex WPForms SMTP Serialization Bug
|
||||
|
||||
**Root cause:** WP Mail SMTP stores passwords in PHP serialized format within the `wp_options` table. The serialized string has a length prefix (e.g., `s:13:` for 13 characters). If the declared length doesn't match the actual password string length, the plugin cannot deserialize the password and **ALL form email delivery fails silently**.
|
||||
|
||||
**Affected configuration:** `option_name = 'wp_mail_smtp'`, stored in `wp_options` table.
|
||||
|
||||
**The bug:** The password `apex.track!!` is 13 characters. The stored value had `s:72` instead of `s:13` — likely caused by a pre-save sanitization or paste issue. WP Mail SMTP read `s:72` and expected a 72-character string, found fewer characters, and failed with a PHP warning that never surfaced to the UI.
|
||||
|
||||
**Diagnosis:**
|
||||
1. Check the stored password: `SELECT option_value FROM wp_options WHERE option_name = 'wp_mail_smtp'`
|
||||
2. Search for `smtp.pass` in the serialized data
|
||||
3. Count the actual password length and compare to the serialized `s:N` prefix
|
||||
|
||||
**Fix (verified Jul 8, 2026):**
|
||||
```sql
|
||||
UPDATE wp_options
|
||||
SET option_value = REPLACE(
|
||||
option_value,
|
||||
's:72:"apex.track!!"',
|
||||
's:13:"apex.track!!"'
|
||||
)
|
||||
WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
**Also fix sender_address on forms:**
|
||||
Both Form 270 (NASA Top Speed) and Form 268 (Waiver) had comma-separated email addresses in the `sender_address` notification field. This produces an invalid `From:` email header. Set `sender_address` to a single email address (e.g., `contact@apextrackexperience.com`).
|
||||
|
||||
**Verification:**
|
||||
1. Send a test email from the WP Mail SMTP settings page
|
||||
2. Check debug events table: `SELECT * FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 10`
|
||||
3. Submit a test WPForms entry and verify the notification arrives
|
||||
4. Enable the 5-min watchdog script
|
||||
|
||||
**Resending missed notifications:**
|
||||
After fixing the serialization, previously failed form entries won't auto-resend. Check `wp_wpforms_entries` for entries with no corresponding email notification. Manually re-send via SMTP Python script.
|
||||
|
||||
**Watchdog script** at `/root/.hermes/scripts/apex-mail-watchdog.sh` runs every 5 minutes via no_agent cron. It:
|
||||
- Tests SMTP LOGIN authenticity (no test email sent — just `s.login()` then `s.quit()`)
|
||||
- Queries `wp_wpmailsmtp_debug_events` for recent failures
|
||||
- Silent on success, alerts on failure
|
||||
- **CRITICAL:** `log()` function uses `>> "$LOG"` not `tee -a "$LOG"` to prevent every log line from being delivered as a cron message
|
||||
@@ -0,0 +1,67 @@
|
||||
# DR Audit Findings — Jul 11, 2026
|
||||
|
||||
## Backup Pipeline Failure
|
||||
|
||||
**Trigger:** Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.
|
||||
|
||||
**Root cause:** `hermes-backup.sh` was missing `source /opt/awscli-venv/bin/activate`. The `aws` command is only available inside the virtualenv at `/opt/awscli-venv/`. In cron's minimal PATH (`/usr/bin:/bin`), `aws` is not found. The `root-essentials-backup.sh` script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.
|
||||
|
||||
**Evidence from journalctl:**
|
||||
```
|
||||
Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found
|
||||
```
|
||||
|
||||
**Fix applied (Jul 11, 07:30 UTC):**
|
||||
```bash
|
||||
# Added after "set -euo pipefail" in hermes-backup.sh:
|
||||
if [ -f /opt/awscli-venv/bin/activate ]; then
|
||||
source /opt/awscli-venv/bin/activate
|
||||
fi
|
||||
```
|
||||
|
||||
**Verification:** Manual backup ran successfully after fix — 541 MB tarball uploaded to `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz`.
|
||||
|
||||
## Standby Server Disk Crisis
|
||||
|
||||
**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.
|
||||
|
||||
**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.
|
||||
|
||||
**Required cleanup (not yet performed):**
|
||||
```bash
|
||||
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
|
||||
apt-get clean && apt-get autoremove --purge -y
|
||||
journalctl --vacuum-size=200M
|
||||
> /var/log/hermes-standby-sync.log
|
||||
docker system prune -a -f 2>/dev/null || true
|
||||
df -h /
|
||||
"
|
||||
```
|
||||
Target: at least 5 GB free.
|
||||
|
||||
## Backup Integrity Verification Workflow
|
||||
|
||||
The Jul 11 audit established a reliable pattern for verifying backup integrity:
|
||||
|
||||
1. **List S3** to find the latest tarball
|
||||
2. **Download** it to `/tmp/` with `aws s3 cp`
|
||||
3. **Test** with `tar tzf` (not just `aws s3 ls`)
|
||||
4. **Report** file count and any corruption
|
||||
5. **Clean up** the local copy
|
||||
|
||||
This catches silent failures that S3 listing alone misses. The audit watchdog (`backup-audit-check.sh`) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.
|
||||
|
||||
## Restore Plan Template
|
||||
|
||||
When an incident requires a restore plan, structure it with these sections:
|
||||
|
||||
1. **Pre-Restore Assessment** — backup status table, standby status, system health, dependency inventory
|
||||
2. **Recovery Path A** — In-place fix (preferred when system is running)
|
||||
3. **Recovery Path B** — Failover to standby (when Core is unreachable)
|
||||
4. **Recovery Path C** — Full rebuild from backup tarball (when OS is corrupted)
|
||||
5. **Post-Restore Verification Checklist** — Hermes, web services, model access, email, backups, standby
|
||||
6. **Critical Fixes** — any permanent fixes needed to prevent recurrence
|
||||
7. **Team Contact & Escalation** — who to call and when
|
||||
8. **Rollback Plan** — how to undo the restore if it makes things worse
|
||||
|
||||
See `/root/.hermes/references/restore-plan-2026-07-11.md` for the complete worked example.
|
||||
@@ -0,0 +1,82 @@
|
||||
# DR Issue Log — Hermes Infrastructure
|
||||
|
||||
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-08 — Initial Full DR Audit
|
||||
|
||||
### DR-001: Caddyfile not included in backup scripts
|
||||
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
|
||||
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
|
||||
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates).
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed all paths in script
|
||||
|
||||
### DR-002: Systemd service files backed up to wrong path
|
||||
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up.
|
||||
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
|
||||
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed paths in script
|
||||
|
||||
### DR-003: migration-creds.txt not in backup scope
|
||||
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
|
||||
**Root cause:** Added after backup script was written, never included in scope.
|
||||
**Fix:** Added to hermes-backup.sh file list with chmod 600 restore.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed file referenced
|
||||
|
||||
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
|
||||
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
|
||||
**Root cause:** Script created the file without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-005: migration-creds.txt at 644 permissions
|
||||
**Problem:** Same issue as DR-004 — migration-creds.txt was 644.
|
||||
**Root cause:** Written without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-006: Full backup had no cron job (last ran Jul 5)
|
||||
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
|
||||
**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
|
||||
**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Cron job created and scheduled for next 5 AM ET
|
||||
|
||||
### DR-007: home-router-daily-backup cron error
|
||||
**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed.
|
||||
**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP.
|
||||
**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||||
|
||||
### DR-008: MikroTik CCR backup stale (last Jul 5)
|
||||
**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5.
|
||||
**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24.
|
||||
**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan).
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||||
|
||||
### DR-009: app1-bu warm vs cold docs mismatch
|
||||
**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime).
|
||||
**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant).
|
||||
**Fix:** Updated DR plan to reflect warm standby design. Server stays running.
|
||||
**Status:** ✅ Resolved — not a bug, docs were wrong
|
||||
|
||||
### DR-010: Failover timing change (per Germaine's direction)
|
||||
**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles).
|
||||
**Rationale:** Faster detection, shorter failover.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu.
|
||||
|
||||
### DR-011: AWS CLI missing on app1-bu standby
|
||||
**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3.
|
||||
**Root cause:** python3-venv package not installed on standby server.
|
||||
**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.
|
||||
@@ -0,0 +1,43 @@
|
||||
# DR Plan v3 — Response to Third-Party Reviewers
|
||||
|
||||
## Two reviewers, two passes. All findings addressed.
|
||||
|
||||
### Reviewer 1 (Technical DR Specialist)
|
||||
|
||||
| Finding | Status | Addressed In |
|
||||
|---|---|---|
|
||||
| Failover timing mismatch (5min check can't detect 2min outage) | Fixed | Corrected to 30s checks, 4 failures to trigger = 2min detection, ~3-3.5min actual RTO, 5min committed target |
|
||||
| Backup standard says daily but not happening | Flagged | Section 3.1 backup matrix shows live vs gap status; P0 items for Docker volume + DB dump automation |
|
||||
| AI server 92% disk critical | Flagged | P0 action item #1 |
|
||||
| Memory consolidation data loss risk | Mitigated | S3 backup-before-prune + PROTECT regex expansion + restore procedure documented |
|
||||
| Tony's Hermes undocumented | Flagged | P1 action item #7 |
|
||||
|
||||
### Reviewer 2 (Systems Management SME)
|
||||
|
||||
| Finding | Status | Addressed In |
|
||||
|---|---|---|
|
||||
| S3 single point of failure | Mitigated | Two-destination design (Wasabi us-east-1 primary + Wasabi us-west-2 or Backblaze B2 secondary) |
|
||||
| Backup frequency vs RPO mismatch | Fixed | Per-data-type RPO: 15min (Hermes state), 24h (Docker/data), 24h backups |
|
||||
| Restore procedures not detailed | Fixed | Section 6 — per-server runbooks with exact commands |
|
||||
| Failback under-defined | Fixed | Section 4.4 — 12-step protocol with rollback, no auto-failback |
|
||||
| No backup restore testing | Fixed | Section 7 — testing schedule with cadence, log template, sign-off table |
|
||||
| Retention too short (48h snapshots) | Flagged | P1 item #9 — extend to 30 days |
|
||||
| Backup security not described | Fixed | Section 3.2 — Object Lock, write-only IAM, credential separation |
|
||||
| Application backup details vague | Partial | Section 2.3 — data paths defined per app |
|
||||
| Secrets recovery not documented | Fixed | Section 1 — shared dependencies table with credential locations |
|
||||
| Provider/account outage missing | Fixed | Section 4.5 — netcup account failure scenario |
|
||||
|
||||
## Still Pending (Not Yet Implemented, Only Designed)
|
||||
|
||||
| Item | Status | Required For Sign-Off |
|
||||
|---|---|---|
|
||||
| Object Lock enabled on buckets | Designed, not done | Yes |
|
||||
| Write-only IAM credentials created | Designed, not done | Yes |
|
||||
| S3 secondary backup destination provisioned | Designed, not done | Yes |
|
||||
| Docker volume backup automation deployed | Flagged, not done | Yes |
|
||||
| MariaDB dump cron deployed (wphost02, fleettracker360) | Flagged, not done | Yes |
|
||||
| 30-second watchdog deployed on app1-bu | Designed, not done | Yes |
|
||||
| Tony's Hermes documented + backed up | Flagged, not done | No (non-critical) |
|
||||
| Quarterly failover/failback test performed | Scheduled, not done | Yes |
|
||||
| One database restore test performed | Scheduled, not done | Yes |
|
||||
| One Docker volume restore test performed | Scheduled, not done | Yes |
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
# Gateway Lifecycle Troubleshooting
|
||||
|
||||
Diagnostic flow when `hermes gateway restart` doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).
|
||||
|
||||
## Symptom: `hermes gateway restart` fails with "linger is not enabled"
|
||||
|
||||
```
|
||||
Cannot restart gateway as a service -- linger is not enabled.
|
||||
The gateway user service requires linger to function on headless servers.
|
||||
Run: sudo loginctl enable-linger root
|
||||
```
|
||||
|
||||
On containers, LXC, or VPS environments without a DBus session bus, this will fail with:
|
||||
|
||||
```
|
||||
Failed to connect to system scope bus via local transport: Connection refused
|
||||
```
|
||||
|
||||
On these systems the gateway may run via a **system-level** `hermes.service` unit (at `/etc/systemd/system/`), not a user-level one. The user-level service guard (`linger`) is a red herring -- the system-level unit uses `User=root` and the global `EnvironmentFile`, bypassing the DBus session issue entirely.
|
||||
|
||||
## Diagnostic Steps
|
||||
|
||||
### 1. Find what's running
|
||||
|
||||
```bash
|
||||
# Check system-level service
|
||||
systemctl cat hermes.service # shows the full unit
|
||||
systemctl status hermes.service # active, failed, auto-restart loop?
|
||||
|
||||
# Check for manually-started gateway processes
|
||||
ps aux | grep "hermes.*gateway" | grep -v grep
|
||||
```
|
||||
|
||||
This reveals the critical fork: is the gateway running under systemd or as an orphan process?
|
||||
|
||||
### 2. Categorize what you find
|
||||
|
||||
| Situation | What to do |
|
||||
|-----------|-----------|
|
||||
| Gateway running under systemd (`hermes.service` active) | `systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level |
|
||||
| Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`) | `hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly |
|
||||
| `hermes.service` in `auto-restart` loop (exit-code 1) | Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process. |
|
||||
| No gateway at all | `systemctl start hermes.service` or `hermes gateway run` |
|
||||
|
||||
### 3. The common cause: dual processes
|
||||
|
||||
The gateway can be started in two ways:
|
||||
- **Manually:** `hermes gateway run` (or `hermes gateway run --replace`)
|
||||
- **As a systemd service:** `hermes.service` at `/etc/systemd/system/`
|
||||
|
||||
If someone starts it manually (e.g. `hermes gateway run` in a tmux session), systemd's `hermes.service` will fail with "Gateway already running" and loop forever. The fix:
|
||||
|
||||
```bash
|
||||
hermes gateway stop # kills the manual process
|
||||
# systemd auto-restarts within 5s -- check:
|
||||
sleep 6 && systemctl is-active hermes.service
|
||||
```
|
||||
|
||||
### 4. Stale systemd unit warning
|
||||
|
||||
After restarting, check logs for:
|
||||
|
||||
```
|
||||
Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.
|
||||
```
|
||||
|
||||
This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:
|
||||
|
||||
```bash
|
||||
hermes gateway install --force
|
||||
```
|
||||
|
||||
This rewrites the unit with `TimeoutStopSec = drain_timeout + 30s`.
|
||||
|
||||
### 5. User-level vs system-level units
|
||||
|
||||
This environment has BOTH:
|
||||
- **System-level:** `/etc/systemd/system/hermes.service` -- runs the main Hermes gateway (default profile). `User=root`, uses `EnvironmentFile=/root/.hermes/.env`. This is the primary unit.
|
||||
- **User-level:** `~/.config/systemd/user/hermes-gateway-anita.service` -- runs the Anita profile gateway. Requires `loginctl enable-linger` (on this VPS, the system bus is unavailable so user-level units can't start).
|
||||
|
||||
Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.
|
||||
|
||||
### 6. Profile gateways
|
||||
|
||||
The Anita profile runs as a separate systemd service. Check its status:
|
||||
|
||||
```bash
|
||||
cat /proc/<pid>/cmdline | tr '\0' ' ' # tells you which profile is running
|
||||
ps aux | grep "profile anita" # Anita's gateway process
|
||||
```
|
||||
|
||||
Multiple gateway processes (one per profile) can run simultaneously without conflict.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# See the unit definition
|
||||
systemctl cat hermes.service
|
||||
|
||||
# Check gateway status
|
||||
hermes gateway status
|
||||
|
||||
# Kill manual process, let systemd take over
|
||||
hermes gateway stop
|
||||
|
||||
# Regenerate the systemd unit with correct timeouts
|
||||
hermes gateway install --force
|
||||
|
||||
# Check latest logs
|
||||
journalctl -u hermes.service --no-pager -n 15
|
||||
|
||||
# Check for any gateway-related processes
|
||||
ps aux | grep "hermes.*gateway" | grep -v grep
|
||||
# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
|
||||
# The first is manual, the second is often tmux/systemd-launched
|
||||
```
|
||||
@@ -0,0 +1,49 @@
|
||||
# Hermes Memory Auto-Consolidation
|
||||
|
||||
**Deployed:** July 9, 2026
|
||||
**Schedule:** Every 10 min (Hermes cron, no_agent mode)
|
||||
**Scripts:** `/root/.hermes/scripts/hermes-consolidate.sh` + `hermes-consolidate.py`
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Every 10 min:
|
||||
1. Dump MEMORY.md entries to JSON
|
||||
2. Upload to S3 (backup-before-prune guarantee — aborts if S3 fails)
|
||||
3. Run pruner (pattern prune + size safety valve)
|
||||
4. Prune fact store (entries older than 7 days, never-retrieved, trust < 0.5)
|
||||
5. Keep local pre-prune copy in ~/.hermes/memories/.consolidate-backups/ (48h retention)
|
||||
```
|
||||
|
||||
## Safety Guarantees
|
||||
|
||||
- **Backup-before-prune:** S3 upload is a hard gate; prune never runs without it
|
||||
- **Never writes empty file:** pruner aborts if kept set would be empty
|
||||
- **Atomic write:** `os.replace()` on temp file — no partial-write risk
|
||||
- **Protected entries:** regex PROTECT prevents identity, rules, credentials, and key infrastructure facts from EVER being pruned (pattern or size valve)
|
||||
- **Idempotent:** if already under target, backs up and exits without modifying
|
||||
|
||||
## PROTECT List (entries never pruned)
|
||||
|
||||
As of Jul 9 2026: includes "Core:", "STANDING PRACTICE", "app1-bu", "netcup|admin-ai.itpropartner", "Ops portal", "Recovery manual", "Ollama", "RS 4000", "Migration target", plus all credential/identity/formatting rules.
|
||||
|
||||
## Restore Procedure
|
||||
|
||||
```bash
|
||||
# From S3:
|
||||
source /opt/awscli-venv/bin/activate
|
||||
aws s3 cp s3://hermes-vps-backups/live/memories/memory-backup.json /tmp/mem.json \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
|
||||
# Rebuild MEMORY.md:
|
||||
python3 -c "
|
||||
import json
|
||||
d = json.load(open('/tmp/mem.json'))
|
||||
open('/root/.hermes/memories/MEMORY.md','w').write(('\n§\n'.join(d['entries'])) + '\n')
|
||||
print('restored', d['entry_count'], 'entries')
|
||||
"
|
||||
```
|
||||
|
||||
## Known Caveat
|
||||
|
||||
The size safety valve drops oldest-appended entries when pattern-pruning alone doesn't reach 7,000 chars. This can remove useful entries at the top of the file if they don't match a PROTECT pattern. All data preserved in S3.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Server Pricing Comparison (Jul 2026)
|
||||
|
||||
## netcup Root Server G12 (current)
|
||||
Monthly pricing (incl. 0% VAT). Dedicated AMD EPYC 9645 cores.
|
||||
|
||||
| Plan | Cores | RAM | NVMe | EUR/mo | USD/mo |
|
||||
|------|-------|-----|------|--------|--------|
|
||||
| RS 1000 | 4 | 8 GB | 256 GB | €10.74 | ~$12.24 |
|
||||
| RS 2000 ← Core | 8 | 16 GB | 512 GB | €18.00 | ~$20.52 |
|
||||
| RS 4000 | 12 | 32 GB | 1 TB | €33.54 | ~$38.24 |
|
||||
|
||||
## Hetzner Cloud (current — inflated pricing)
|
||||
|
||||
| Tier | vCPU | RAM | Disk | EUR/mo | USD/mo |
|
||||
|------|------|-----|------|--------|--------|
|
||||
| CPX11 | 2 | 2 GB | 40 GB | ~€11 | ~$13 |
|
||||
| CPX21 | 3 | 4 GB | 80 GB | ~€32 | ~$36 |
|
||||
| CPX32 | 4 | 8 GB | 160 GB | ~€35 | ~$40 |
|
||||
| CPX42 | 8 | 16 GB | 320 GB | ~€69 | ~$79 |
|
||||
|
||||
## Key Insight
|
||||
Hetzner's pricing was raised significantly for new/recreated servers as of Jul 2026. The API returns inflated rates. Core's equivalent compute (8C/16G) at Hetzner would be CPX42 at ~$79/mo vs **€18/mo (~$20.52)** on netcup RS 2000 — a ~74% savings.
|
||||
|
||||
**Strategy:** Consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers. Freed-up existing Hetzner boxes can be reused for standby duty without incurring the inflated pricing.
|
||||
|
||||
## Exchange Rate
|
||||
€1 ≈ $1.14 USD (as of Jul 8, 2026)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,24 @@
|
||||
# Hermes Memory Auto-Consolidation — DR / Restore Procedure
|
||||
|
||||
**Component:** `hermes-consolidate.sh` + `hermes-consolidate.py`
|
||||
**Schedule:** every 10 min (Hermes cron, `no_agent`)
|
||||
**Last reviewed:** July 9, 2026 — Network Services Team (Backup Specialist)
|
||||
|
||||
---
|
||||
|
||||
## What it does
|
||||
|
||||
Every 10 minutes:
|
||||
1. Dumps MEMORY.md to JSON
|
||||
2. Uploads backup to S3 FIRST (aborts prune on failure)
|
||||
3. Prunes stale entries
|
||||
4. Keeps local timestamped copies for 48h
|
||||
|
||||
## Safety guarantees
|
||||
- Backup-before-prune: upload to S3 is a hard gate
|
||||
- Never writes empty file
|
||||
- Atomic write via os.replace()
|
||||
- Protected entries never pruned
|
||||
|
||||
## Restore procedures
|
||||
See full document at /root/.hermes/cache/documents/doc_498756c88128_memory-consolidation-dr-sanitized.md
|
||||
@@ -0,0 +1,43 @@
|
||||
# Formatting rules for user-facing reference files
|
||||
|
||||
These rules apply to all `.md` files the user may view on mobile (Telegram preview, file open, email attachment).
|
||||
|
||||
## ASCII-only status indicators
|
||||
|
||||
Do NOT use emoji for status indicators in files the user reads. Common mobile viewers misinterpret UTF-8 emoji sequences as Latin-1 characters, producing garbled text like `🔴` instead of `:red_circle:`.
|
||||
|
||||
Use these replacements instead:
|
||||
|
||||
- :red_circle: -> [HIGH]
|
||||
- :yellow_circle: -> [MED] or [WARN]
|
||||
- :green_circle: -> [OK] or [LOW]
|
||||
- :check_mark_button: -> [OK]
|
||||
- :check_mark: -> [OK]
|
||||
- :cross_mark: -> [FAIL]
|
||||
- :globe_with_meridians: -> [INFO]
|
||||
- :cyclone: -> [PENDING]
|
||||
- :right_arrow: -> ->
|
||||
- em dash -- -> --
|
||||
- times sign x -> x
|
||||
|
||||
## No pipe tables in summaries
|
||||
|
||||
Markdown pipe tables (`| col | col |`) are unreadable on small phone screens. Use bullet lists or comma-separated inline format instead:
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
| ID | Issue | Status |
|
||||
|----|-------|--------|
|
||||
| 1 | Caddyfile | Fixed |
|
||||
```
|
||||
|
||||
**Good:**
|
||||
```
|
||||
Fixed:
|
||||
- 1 Caddyfile -> Fixed
|
||||
- 2 Backups -> Fixed
|
||||
```
|
||||
|
||||
## Keep it pure ASCII
|
||||
|
||||
Any non-ASCII character is a risk. Stick to `[0-9A-Za-z]`, hyphens, underscores, periods, and standard punctuation. Smart quotes, long dashes, mathematical symbols, and Unicode arrows all break on some mobile renderers.
|
||||
@@ -0,0 +1,37 @@
|
||||
# MSP Documentation Suite
|
||||
|
||||
Generated Jul 10, 2026. Standardized document structure for the IT Pro Partner MSP.
|
||||
|
||||
## Core Infrastructure Docs
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| DR Plan v3 | /root/.hermes/references/server-dr-plans-v3.md | RTO/RPO targets, failover architecture, provider diversity |
|
||||
| DR Plan Public | /root/.hermes/references/server-dr-plans-public.md | Redacted for Reddit/Facebook |
|
||||
| Server Inventory | /root/.hermes/references/server-inventory.md | All 9 servers: specs, IPs, roles, costs, status |
|
||||
| Application Inventory | /root/.hermes/references/application-inventory.md | Every service: ports, dependencies, backup methods |
|
||||
| Server Provisioning Standard | /root/.hermes/references/server-provisioning-standard-public.md | Step-by-step build checklist |
|
||||
| Network Diagram | /root/.hermes/references/network-diagram.md | Topology, ports, VPNs, DNS zones |
|
||||
|
||||
## Security & Recovery
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| Backup Policy | /root/.hermes/references/backup-policy.md | What/when/how, retention, RPO, verification |
|
||||
| Incident Response Plan | /root/.hermes/references/incident-response-plan.md | Severity levels, team, comms, escalation |
|
||||
| Restore Runbooks | /root/.hermes/references/restore-runbooks.md | Per-server numbered disaster recovery steps |
|
||||
| DR Issue Log | /root/.hermes/references/dr-issue-log.md | Permanent record of all findings and fixes |
|
||||
|
||||
## Operations
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| CloudPanel vs Enhance vs Coolify | /root/runcloud_replacements_report.md | WordPress panel selection research |
|
||||
| Model Price Research | /root/.hermes/references/model-price-research.md | LLM pricing comparison across providers |
|
||||
|
||||
## Gaps Still Needed
|
||||
|
||||
- Client Onboarding Checklist
|
||||
- Service Catalog (what ITPP offers, pricing tiers)
|
||||
- Access Control Policy
|
||||
- Change Management Log
|
||||
@@ -0,0 +1,80 @@
|
||||
# Netcup SCP REST API Access
|
||||
|
||||
## Authentication
|
||||
|
||||
The netcup SCP API at `servercontrolpanel.de` uses **Keycloak OIDC** — NOT API key headers. The CCP API key from `customercontrolpanel.de` Master Data does **not** work with the SCP REST API directly.
|
||||
|
||||
### Get a token
|
||||
|
||||
```bash
|
||||
TOKEN=$(curl -s "https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token" \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=password&client_id=scp&username=${NETCUP_CUSTOMER}&password=${NETCUP_PASSWORD}&scope=openid" \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token')")
|
||||
```
|
||||
|
||||
| Parameter | Value | Source |
|
||||
|-----------|-------|--------|
|
||||
| Token endpoint | `https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token` | From `keycloak.js` in SCP UI |
|
||||
| Client ID | `scp` | From `keycloak.js` |
|
||||
| Grant type | `password` | Resource owner password grant |
|
||||
| Username | CCP customer number (numeric, e.g. `389212`) | From CCP Master Data |
|
||||
| Password | CCP account password | User-provided |
|
||||
| Scope | `openid` | Required for ID token |
|
||||
| Token lifetime | 300 seconds (5 min) | From token response `expires_in` |
|
||||
| Refresh | 1800 seconds (30 min) | From token response `refresh_expires_in` |
|
||||
|
||||
### API base URL
|
||||
|
||||
The SCP has three API paths:
|
||||
|
||||
| Path | Purpose | Status |
|
||||
|------|---------|--------|
|
||||
| `https://www.servercontrolpanel.de/scp-core/api/v1/` | Server management CRUD | ✅ Works (returns JSON) |
|
||||
| `https://www.servercontrolpanel.de/scp-agent/api/v1/` | Agent/service operations | ⚠️ Returns 403 Forbidden |
|
||||
| `https://www.servercontrolpanel.de/scp-ui/api/v1/` | UI data endpoints | ❌ Returns HTML login page, not JSON |
|
||||
|
||||
**Always use `scp-core` for server operations.**
|
||||
|
||||
## Available endpoints
|
||||
|
||||
### List servers
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/servers" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
Returns: `[{"id": 890903, "name": "v2202607377162478911", "template": {"name": "RS 2000 G12"}}]`
|
||||
|
||||
### Single server details
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/servers/890903" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
|
||||
Returns: IP addresses, hostname, architecture, template name, disk available space, RAM.
|
||||
|
||||
### Templates / Images (provisioning)
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/templates" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
**Note:** May return `{"code": "error.notfound", "message": "Resource not found"}` for non-reseller accounts. This is a known limitation — provisioning via API may require reseller status with netcup.
|
||||
|
||||
## Credentials
|
||||
|
||||
Stored in `/root/.hermes/.env`:
|
||||
```
|
||||
NETCUP_CUSTOMER=389212
|
||||
NETCUP_PASSWORD=<CCP password>
|
||||
NETCUP_KEY=<API key from Master Data page>
|
||||
NETCUP_API_KEY=<same as NETCUP_KEY>
|
||||
```
|
||||
|
||||
The API key from the Master Data page is NOT used by the REST API. The password grant flows use the customer number + CCP password directly. The API key exists for legacy SOAP/JSON-RPC endpoints.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- **Token expires every 5 minutes** — need to re-authenticate or implement refresh flow for long-running operations
|
||||
- **Provisioning may be restricted** — `/templates` and `/images` return 404 for non-reseller accounts
|
||||
- **This is the SCP (Server Control Panel), not CCP** — the CCP (`customercontrolpanel.de`) has a separate API surface for billing/invoicing
|
||||
- **Discovered Jul 8 2026** — after multiple failed attempts with API key auth that returned 404/401, the working auth flow was found by inspecting `keycloak.js` in the SCP UI JavaScript bundle
|
||||
@@ -0,0 +1,70 @@
|
||||
# IT Pro Partner — Network & Service Endpoints
|
||||
|
||||
## DNS & Hosting
|
||||
|
||||
| Domain | Nameservers | Hosting |
|
||||
|--------|------------|---------|
|
||||
| itpropartner.com | SiteGround (ns1/ns2.siteground.net) | SiteGround |
|
||||
| iamgmb.com | Cloudflare | MXroute (email) |
|
||||
| germainebrown.com | Cloudflare | MXroute (email) |
|
||||
| debtrecoveryexperts.com | Cloudflare | SiteGround/self-hosted |
|
||||
|
||||
## Core (netcup RS 2000 G12)
|
||||
|
||||
| Service | Port | URL / Notes |
|
||||
|---------|------|-------------|
|
||||
| Hermes gateway | — | Telegram bot |
|
||||
| Hermes assistant (PWA) | 8082 | app.itpropartner.com |
|
||||
| Shark game API | 8083 | shark.iamgmb.com |
|
||||
| Chromium CDP | 9222 | local browser automation |
|
||||
| SMTP relay | 2525 | mail.germainebrown.com (STARTTLS) |
|
||||
| Caddy | 443 | Reverse proxy for all sites |
|
||||
| Tailscale | — | vaultwarden.tailc2f3b0.ts.net |
|
||||
|
||||
## Hetzner Servers
|
||||
|
||||
| Name | Type | IP | Purpose |
|
||||
|------|------|----|---------|
|
||||
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
|
||||
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby |
|
||||
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
|
||||
| unms | CPX21 | 5.161.225.131 | UISP/UNMS |
|
||||
| unifi | CPX21 | 178.156.131.57 | UniFi controller |
|
||||
| hudu | CPX21 | 178.156.130.130 | IT documentation |
|
||||
| ai | CPX41 | 178.156.167.181 | admin-ai LLM proxy |
|
||||
| fleet | CPX11 | 178.156.149.32 | Fleet tracker |
|
||||
| docker | CPX11 | 178.156.168.35 | Docker host |
|
||||
| app1 (old) | CPX11 | 87.99.144.163 | Legacy |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| API | Auth | Base URL |
|
||||
|-----|------|----------|
|
||||
| Hetzner Cloud | Bearer token | https://api.hetzner.cloud/v1 |
|
||||
| Cloudflare | Bearer token | https://api.cloudflare.com/client/v4 |
|
||||
| admin-ai | Bearer token | https://admin-ai.itpropartner.com/v1 |
|
||||
| netcup SCP | Keycloak OIDC | https://www.servercontrolpanel.de/scp-core/api/v1 |
|
||||
| Firecrawl | Bearer token | https://api.firecrawl.com/v1 |
|
||||
| Wasabi S3 | AWS4-HMAC-SHA256 | s3.wasabisys.com |
|
||||
|
||||
## S3 Backup Buckets
|
||||
|
||||
| Bucket | Prefix | Purpose |
|
||||
|--------|--------|---------|
|
||||
| hermes-vps-backups | live/ | Active Hermes state (15 min sync) |
|
||||
| hermes-vps-backups | standby/ | Standby recovery bundle |
|
||||
| hermes-vps-backups | hermes-full-backup/ | Daily 5 AM full tarball |
|
||||
| itpropartner-backups | — | Portal files & public data |
|
||||
| mikrotik-ccr-backups | wisp-backups/ | Router configs |
|
||||
|
||||
## External Services
|
||||
|
||||
| Service | Login URL | Notes |
|
||||
|---------|-----------|-------|
|
||||
| netcup CCP | customercontrolpanel.de | Account management |
|
||||
| netcup SCP | servercontrolpanel.de | Server management |
|
||||
| Hetzner Cloud | console.hetzner.cloud | Hetzner project |
|
||||
| Cloudflare | dash.cloudflare.com | DNS, Access, Registrar |
|
||||
| MXroute | mxlogin.com | Email hosting |
|
||||
| SiteGround | siteground.com | WP hosting |
|
||||
| Wasabi | console.wasabisys.com | S3-compatible storage |
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
# Push Notification Pitfalls (iOS PWA)
|
||||
|
||||
Captured from the Jul 9, 2026 shark game push notification debugging session.
|
||||
|
||||
## iOS Safari does NOT support PushManager in browser tabs
|
||||
|
||||
On iPhones, `PushManager` is only available when the web app is opened from the Home Screen as a standalone PWA (no URL bar, no browser chrome). Safari's browser tab does NOT expose it.
|
||||
|
||||
**Symptom:** `FAIL: No PushManager` when tapping "Enable Notifications" in Safari.
|
||||
|
||||
**Fix:** User must:
|
||||
1. Tap Share icon -> "Add to Home Screen"
|
||||
2. Open from home screen icon (fullscreen mode)
|
||||
3. Push subscription now works via Apple Push Service (APNs endpoint: `web.push.apple.com`)
|
||||
|
||||
## Subscription to APNs requires the subscribe-noauth endpoint
|
||||
|
||||
The standard `POST /api/push/subscribe` requires JWT authentication. For test pages without login, create a separate `POST /api/push/subscribe-noauth` endpoint that stores subscriptions with `user_id: NULL` to avoid FK constraint failures.
|
||||
|
||||
## Database schema: user_id must be nullable
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS push_subscriptions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
user_id INTEGER, -- NULL for no-auth subscriptions; FK to users.id otherwise
|
||||
endpoint TEXT NOT NULL UNIQUE,
|
||||
p256dh TEXT NOT NULL,
|
||||
auth TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
```
|
||||
|
||||
## Common backend errors
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `FOREIGN KEY constraint failed` | user_id=0 but no user.id=0 exists | Use NULL or valid FK |
|
||||
| `{"sent":0}` | All push send attempts failed silently | Change `except: pass` to `except: print(f"Push failed: {e}")` |
|
||||
| `name 'json' is not defined` | `import json` missing at top of server.py | Add `import json` to imports |
|
||||
| `ModuleNotFoundError: pywebpush` | Installed in wrong venv | Install in the systemd service's venv, not system Python |
|
||||
@@ -0,0 +1,31 @@
|
||||
# IT Pro Partner — Per-Server Disaster Recovery Plan (REDACTED)
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team
|
||||
**Date:** July 9, 2026
|
||||
**Version:** 1.0 — Redacted for 3rd Party Review
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
11 servers total. 1 primary (dedicated EPYC), 10 Hetzner cloud VPSes.
|
||||
1 warm standby. Shared dependencies: S3-compatible storage, Cloudflare DNS, SSH key auth.
|
||||
|
||||
## DR Classification
|
||||
|
||||
| Tier | Count | RTO | RPO |
|
||||
|------|-------|-----|-----|
|
||||
| Critical | 2 | 5 min - 2 hours | 15 min - 24 hours |
|
||||
| Important | 4 | 4 hours | 24 hours |
|
||||
| Standard | 4 | 8 hours | 24 hours |
|
||||
| Standby | 1 | 2 min | 15 min |
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. AI/LLM server disk at 92% — cleanup needed before migration
|
||||
2. Docker volume backups not automated to S3
|
||||
3. Database dumps not scheduled for most servers
|
||||
4. Hermes warm standby functional via S3 sync + WireGuard tunnel
|
||||
|
||||
## Full version
|
||||
Full DR plans with IPs, credentials, and detailed restore procedures are stored internally at /root/.hermes/references/server-dr-plans.md and available on request.
|
||||
@@ -0,0 +1,292 @@
|
||||
# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
|
||||
**Date:** July 10, 2026
|
||||
**Status:** LIVE — replaces v1.0
|
||||
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. Architecture & Shared Dependencies
|
||||
2. Server Standards
|
||||
3. Backup Strategy
|
||||
4. Failover / Failback
|
||||
5. Monitoring & Alerting
|
||||
6. Restore Runbooks
|
||||
7. Testing & Validation
|
||||
8. Security
|
||||
9. Immediate Action Items
|
||||
10. Appendix: Third-Party Review Responses
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture & Shared Dependencies
|
||||
|
||||
```
|
||||
Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
|
||||
|
|
||||
+-----------+-----------+
|
||||
| | |
|
||||
SSH/WG Tailscale Management
|
||||
(itpp- (Core <-> VPN (WireGuard
|
||||
infra) app1-bu) to home router)
|
||||
| | |
|
||||
+-----------+-----------+
|
||||
|
|
||||
+-----------+-----------+
|
||||
| |
|
||||
S3 Primary S3 Secondary
|
||||
(Wasabi us-east-1) (Wasabi us-west-2
|
||||
versioning ON or Backblaze B2)
|
||||
object lock ON object lock ON
|
||||
```
|
||||
|
||||
### Shared Dependencies
|
||||
|
||||
| Dependency | Primary | Backup / Fallback | Credentials Location |
|
||||
|---|---|---|---|
|
||||
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
|
||||
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
|
||||
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
|
||||
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
|
||||
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
|
||||
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
|
||||
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
|
||||
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
|
||||
|
||||
---
|
||||
|
||||
## 2. Server Standards
|
||||
|
||||
### 2.1 Tier Definitions
|
||||
|
||||
| Tier | Spec | OS | Use | Cost |
|
||||
|---|---|---|---|---|
|
||||
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
|
||||
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
|
||||
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
|
||||
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |
|
||||
|
||||
### 2.2 Base Install -- Verified Checklist
|
||||
|
||||
```bash
|
||||
# S1: hostname + timezone
|
||||
hostnamectl set-hostname <name>
|
||||
timedatectl set-timezone America/New_York
|
||||
|
||||
# S2: system update
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# S3: security baseline
|
||||
apt install -y fail2ban ufw
|
||||
ufw default deny incoming; ufw default allow outgoing
|
||||
ufw allow ssh; ufw --force enable
|
||||
|
||||
# S4: monitoring
|
||||
apt install -y prometheus-node-exporter
|
||||
|
||||
# S5: standard user + key
|
||||
adduser ippadmin && usermod -aG sudo ippadmin
|
||||
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
|
||||
|
||||
# S6: harden SSH
|
||||
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
systemctl restart sshd
|
||||
|
||||
# S7: Docker (if needed)
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin
|
||||
```
|
||||
|
||||
### 2.3 Application Data Paths
|
||||
|
||||
Every server must document:
|
||||
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
|
||||
- What can be rebuilt vs what must be restored
|
||||
- Secrets location
|
||||
- Caddy/nginx configs
|
||||
- Cron jobs
|
||||
- External API dependencies
|
||||
|
||||
---
|
||||
|
||||
## 3. Backup Strategy
|
||||
|
||||
### 3.1 Backup Matrix
|
||||
|
||||
| Data Type | Frequency | Target | Retention | Status |
|
||||
|---|---|---|---|---|
|
||||
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
|
||||
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
|
||||
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE |
|
||||
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE |
|
||||
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
|
||||
| System configs | Daily | S3 primary (<server>/) | 90 days | 🔲 NOT AUTOMATED |
|
||||
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Application data | Per-app | S3 primary (<server>/) | 90 days | 🔲 PARTIAL |
|
||||
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |
|
||||
|
||||
### 3.2 Backup Security
|
||||
|
||||
- **Versioning:** ON (all 3 buckets)
|
||||
- **Object Lock:** NOT enabled
|
||||
- **Delete protection:** NOT configured
|
||||
- **Backup credentials:** Admin-level (can delete) -- NOT write-only
|
||||
- **Encryption:** Server-side only (Wasabi default)
|
||||
|
||||
---
|
||||
|
||||
## 4. Failover / Failback
|
||||
|
||||
### 4.1 Warm Standby Timing (Corrected)
|
||||
|
||||
**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)
|
||||
|
||||
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO
|
||||
|
||||
### 4.2 Health Check -- Two Layers
|
||||
|
||||
**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
|
||||
**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama
|
||||
|
||||
### 4.3 Split-Brain Prevention
|
||||
|
||||
S3 active-lock.json mechanism:
|
||||
|
||||
```json
|
||||
{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}
|
||||
```
|
||||
|
||||
Core writes heartbeat every 30s while healthy. Failover only proceeds after:
|
||||
1. Core unreachable for 4 consecutive checks (app1-bu local)
|
||||
2. External monitor independently confirms Core down
|
||||
3. active-lock.json heartbeat stale (>90s)
|
||||
4. app1-bu writes itself as lock_holder
|
||||
5. Gateway starts
|
||||
6. Operator alerted
|
||||
|
||||
If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.
|
||||
|
||||
### 4.4 Failback Protocol
|
||||
|
||||
**No auto-failback.** Sequence:
|
||||
1. Confirm app1-bu is active node
|
||||
2. Keep Core stopped -- no auto-start
|
||||
3. Restore Core OS + stack
|
||||
4. Sync latest state from app1-bu/S3 into Core
|
||||
5. Validate sync timestamp
|
||||
6. Start Core in passive mode
|
||||
7. Stop app1-bu gateway
|
||||
8. Update active-lock.json: "active_node": "core"
|
||||
9. Start Core gateway
|
||||
10. 5 consecutive health checks pass
|
||||
11. Release standby lock
|
||||
12. Monitor 30 min before closing
|
||||
|
||||
**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.
|
||||
|
||||
### 4.5 Provider/Account Outage
|
||||
|
||||
If entire netcup account is unavailable:
|
||||
1. app1-bu (Hetzner) takes over
|
||||
2. DNS records pointed to app1-bu IP
|
||||
3. No auto-failback -- human decision
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-Server RTO/RPO Targets
|
||||
|
||||
| System | RTO | RPO | Notes |
|
||||
|---|---|---|---|
|
||||
| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) |
|
||||
| app1-bu (standby) | N/A | <=15 min | Recovery path for Core |
|
||||
| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose |
|
||||
| wphost02 (WordPress) | 2h | 24h | DB restore from dump |
|
||||
| fleettracker360 | 2h | 24h | DB restore from dump |
|
||||
| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Immediate Action Items
|
||||
|
||||
### P0 -- This Week
|
||||
|
||||
| # | Action | Details |
|
||||
|---|---|---|
|
||||
| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs |
|
||||
| 2 | Create S3 write-only IAM user | PutObject only, no delete/list |
|
||||
| 3 | Enable S3 Object Lock | 7-day minimum retention |
|
||||
| 4 | Deploy Docker volume backup | All Docker hosts |
|
||||
| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) |
|
||||
| 6 | Deploy 30s watchdog | Currently documented, need deployment |
|
||||
|
||||
### P1 -- Next 2 Weeks
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 7 | Document Tony's Hermes config + backup to S3 |
|
||||
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) |
|
||||
| 9 | Extend snapshot retention from 48h to 30 days |
|
||||
| 10 | Add backup success/failure monitoring to ops portal |
|
||||
|
||||
### P2 -- Next 30 Days
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 11 | Full DR simulation for Core failover |
|
||||
| 12 | Monthly restore testing schedule begins |
|
||||
| 13 | Deploy Prometheus + Grafana stack |
|
||||
| 14 | Add disk threshold alerts for all servers |
|
||||
| 15 | Begin netcup migration |
|
||||
|
||||
---
|
||||
|
||||
## 7. Restore Runbooks (Summary)
|
||||
|
||||
Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:
|
||||
- Purpose + dependencies
|
||||
- DNS records, firewall ports, backup path
|
||||
- Required secrets
|
||||
- Step-by-step restore commands
|
||||
- DB restore with pre-import dump verification
|
||||
- Docker volume restore
|
||||
- Health checks
|
||||
- Known failure modes
|
||||
|
||||
Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.
|
||||
|
||||
---
|
||||
|
||||
## 8. Third-Party Review Responses
|
||||
|
||||
### Reviewer 1 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks |
|
||||
| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan |
|
||||
| AI server 92% disk critical | Section 6 -- P0 action item #1 |
|
||||
| Memory consolidation data loss risk | Separate docs per priority-tagging proposal |
|
||||
| Tony's Hermes undocumented | Section 6 -- P1 action item #7 |
|
||||
|
||||
### Reviewer 2 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan |
|
||||
| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type |
|
||||
| Restore procedures not detailed | Section 7 -- Full runbooks reference |
|
||||
| Failback under-defined | Section 4.4 -- Protocol |
|
||||
| No backup restore testing | Separate testing schedule doc |
|
||||
| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 |
|
||||
| Backup security not described | Section 3.2 |
|
||||
| Application backup details vague | Section 2.3 |
|
||||
| Secrets recovery not documented | Section 1 (shared deps table) |
|
||||
| Provider/account outage missing | Section 4.5 |
|
||||
|
||||
---
|
||||
|
||||
*This plan is ACTIVE. Review quarterly or after any infrastructure change.*
|
||||
@@ -0,0 +1,38 @@
|
||||
# IT Pro Partner — Per-Server Disaster Recovery Plan
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team
|
||||
**Date:** July 9, 2026
|
||||
**Version:** 1.0
|
||||
**Stored at:** /root/.hermes/references/server-dr-plans.md (primary), s3://hermes-vps-backups/live/config/server-dr-plans.md (S3 backup)
|
||||
|
||||
**Redacted version:** /root/.hermes/references/server-dr-plans-redacted.md (for 3rd party review)
|
||||
|
||||
---
|
||||
|
||||
## Server Inventory
|
||||
|
||||
| # | Server | IP | Provider | Type | Status | Role |
|
||||
|---|--------|-----|----------|------|--------|------|
|
||||
| 1 | core | 152.53.192.33 | netcup | RS 2000 G12 | 🟢 LIVE | Primary ops |
|
||||
| 2 | wphost02 | 5.161.62.38 | Hetzner | CPX21 | 🟢 LIVE | WordPress/RunCloud |
|
||||
| 3 | ai | 178.156.167.181 | Hetzner | CPX41 | 🟡 LIVE (92% disk) | LiteLLM + Ollama |
|
||||
| 4 | unms | 5.161.225.131 | Hetzner | CPX21 | 🟢 LIVE | UISP/UNMS |
|
||||
| 5 | unifi | 178.156.131.57 | Hetzner | CPX21 | 🟢 LIVE | UniFi Controller |
|
||||
| 6 | hudu | 178.156.130.130 | Hetzner | CPX21 | 🟢 LIVE | IT documentation |
|
||||
| 7 | fleettracker360 | 178.156.149.32 | Hetzner | CPX11 | 🟢 LIVE | Fleet tracking |
|
||||
| 8 | docker | 178.156.168.35 | Hetzner | CPX11 | 🟢 LIVE | Utility Docker |
|
||||
| 9 | n8n | 87.99.144.163 | Hetzner | CPX11 | 🟢 LIVE | Automation |
|
||||
| 10 | tony-vps | 87.99.159.142 | Hetzner | CPX21 | 🟢 LIVE | Tony's Hermes |
|
||||
| 11 | app1-bu | 5.161.114.8 | Hetzner | CPX11 | 🟢 STANDBY | Warm standby |
|
||||
|
||||
## Full DR plans
|
||||
See the full document at /root/.hermes/references/server-dr-plans.md for:
|
||||
- Per-server: services, Docker containers, backup paths, restore procedures
|
||||
- DR classification matrix (RTO/RPO/Complexity per server)
|
||||
- Shared dependency map (SSH key, S3, SMTP, DNS, Cloudflare)
|
||||
- ai.itpropartner.com pre-reboot checklist (🔴 critical — 92% disk)
|
||||
- app1-bu failover procedure
|
||||
- Recovery bundle locations
|
||||
|
||||
## Redacted version
|
||||
See /root/.hermes/references/server-dr-plans-redacted.md for the sanitized version suitable for external review.
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
# hermes-standby-sync.sh — Periodic S3 sync for warm standby
|
||||
# Pulls latest state from S3 every 10 min to keep app1-bu fresh.
|
||||
# Runs in parallel with the watchdog. If failover happens,
|
||||
# the latest data is already local.
|
||||
set -euo pipefail
|
||||
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
BUCKET="hermes-vps-backups"
|
||||
SRC_PREFIX="live"
|
||||
HERMES_HOME="${HOME}/.hermes"
|
||||
LOG="/var/log/hermes-standby-sync.log"
|
||||
LIVE_HOST="152.53.192.33"
|
||||
|
||||
source /opt/awscli-venv/bin/activate
|
||||
|
||||
log() {
|
||||
echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] $*" >> "${LOG}"
|
||||
}
|
||||
|
||||
# Don't sync if live box is unreachable (failover in progress)
|
||||
if ! ping -c 1 -W 2 "${LIVE_HOST}" >/dev/null 2>&1; then
|
||||
log "Live box unreachable — skipping sync"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "Syncing from s3://${BUCKET}/${SRC_PREFIX}/..."
|
||||
aws s3 sync "s3://${BUCKET}/${SRC_PREFIX}/" "${HERMES_HOME}/" \
|
||||
--endpoint-url "${ENDPOINT}" \
|
||||
--exclude "audio_cache/*" \
|
||||
--exclude "image_cache/*" \
|
||||
--exclude "cache/*" \
|
||||
--exclude "sandboxes/*" \
|
||||
--exclude "kanban/*" \
|
||||
--exclude "node/*" \
|
||||
--exclude "bin/*" \
|
||||
--exclude "logs/*" \
|
||||
--exclude "*.lock" \
|
||||
--exclude "state.db-shm" \
|
||||
--exclude "state.db-wal" \
|
||||
--no-progress 2>&1 >> "${LOG}"
|
||||
|
||||
log "Sync complete"
|
||||
Reference in New Issue
Block a user