From f57f67c66126f4423cf2c96153e345cd290d5139 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 28 Aug 2026 09:42:36 -0400 Subject: [PATCH] docs: mark wphost02 decommissioned (2026-08-28); app1-bu is sole Hetzner box --- README.md | 6 ++++-- api-master-list.md | 2 +- backup-plan.md | 17 +++-------------- docs/infrastructure/key-inventory.md | 6 +++--- hermes-skills/systems-networks-team.md | 2 +- sites/app3.md | 2 +- 6 files changed, 13 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 5b01bc3..c8bfe69 100644 --- a/README.md +++ b/README.md @@ -88,9 +88,11 @@ ### Legacy / Decommissioned - **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1) - **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted** -- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3** +- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **DECOMMISSIONED (2026-08-28)** — deleted from Hetzner account; sites migrated to app3 - **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17 +> **Hetzner Cloud (current):** As of 2026-08-28, the Hetzner Cloud API returns exactly **one** server — **app1-bu / core-bu** (`5.161.225.131`, CPX21, warm standby for Core). All other Hetzner boxes (old-ai, wphost02) are decommissioned/deleted. + --- ## Model Fallback Chain @@ -171,7 +173,7 @@ Direct API keys for all providers. Claude Sonnet 5 for primary quality, then dir | app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP configs | | app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi | | app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress | -| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL | +| ~~wphost02-backup~~ | ~~Daily 5 AM~~ | ~~s3://hermes-vps-backups/wphost02-backup/~~ | **REMOVED — wphost02 decommissioned 2026-08-28** | | warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness | --- diff --git a/api-master-list.md b/api-master-list.md index a7d782c..cc8d0b2 100644 --- a/api-master-list.md +++ b/api-master-list.md @@ -100,7 +100,7 @@ | SyncroMSP | SyncroMSP | RMM/PSA, client asset management | | Bitdefender GravityZone | Bitdefender | Endpoint security, client AV | | Cloudflare | Cloudflare | DNS zones, domains, records | -| Hetzner Cloud | Hetzner | app1-bu standby server, wphost02 | +| Hetzner Cloud | Hetzner | app1-bu standby server (sole online Hetzner box as of 2026-08-28; wphost02 decommissioned) | | netcup | netcup | Core/app1/app2/app3 servers | | Wasabi S3 | Wasabi | All backups (hermes-vps-backups bucket, app backups) | | Firecrawl | Firecrawl | Web extraction (Super Search fallback) | diff --git a/backup-plan.md b/backup-plan.md index d0e1bc3..e8407f6 100644 --- a/backup-plan.md +++ b/backup-plan.md @@ -63,11 +63,9 @@ | Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 | | modelortho.com | `modelortho-backup.sh` — htdocs tar.gz + nginx configs | `s3://hermes-vps-backups/app3/modelortho/` | 4:30 AM | 2026-08-08 | -### wphost02 (5.161.62.38) — Hetzner CPX21 +### wphost02 (5.161.62.38) — Hetzner CPX21 — **DECOMMISSIONED (2026-08-28)** -| Service | Method | Destination | Schedule | Last Verified | -|---------|--------|-------------|----------|---------------| -| WordPress (7 sites) | `ssh → /root/backup.sh` — per-site tar.gz + all DBs | `s3://hermes-vps-backups/wphost02-backup/` | 5:00 AM | 2026-07-28 | +wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above). ### Home Router @@ -101,7 +99,6 @@ | 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` | | 4:15 AM | **RAGFlow (App2)** | Hermes cron | `ragflow-backup.sh` | | 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` | -| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` | | 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` | | 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` | | 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` | @@ -133,7 +130,6 @@ | `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM | | `dawarich-backup.sh` | Dawarich PostgreSQL dump + compose + env (SSH to App2) | Hermes cron 4:00 AM | | `ragflow-backup.sh` | RAGFlow MySQL dump + compose + env (SSH to App2) | Hermes cron 4:15 AM | -| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM | | `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM | | `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM | | `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM | @@ -147,12 +143,6 @@ |--------|---------|------| | `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM | -### On wphost02 (`/root/`) - -| Script | Purpose | Runs | -|--------|---------|------| -| `backup.sh` | All 7 WordPress sites + all MySQL DBs | Triggered via SSH from Core | - --- ## S3 Bucket Structure @@ -192,7 +182,6 @@ hermes-vps-backups/ ├── gitea/daily/ — Gitea repos + DB ├── unms-backups/live/ — UNMS auto-backups ├── unifi-backups/ — UniFi controller backups -├── wphost02-backup/ — wphost02 per-site + DB ├── volumes/ — Docker volume dumps ├── caddy/ — (unused) ├── assets/ — Static assets @@ -266,4 +255,4 @@ All the following services were migrated from Core to App1 in a single session: - Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/` - Provider diversity: netcup outage won't kill both Core and standby - **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services -- **24 backup scripts** on Core, 1 on App3, 1 on wphost02 +- **Backup scripts:** 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed) diff --git a/docs/infrastructure/key-inventory.md b/docs/infrastructure/key-inventory.md index 03cbccc..5d5c777 100644 --- a/docs/infrastructure/key-inventory.md +++ b/docs/infrastructure/key-inventory.md @@ -11,7 +11,7 @@ | Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To | |----------|------|------|-----------------------|---------|-------------| -| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) | +| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, app1-bu, home router). wphost02 DECOMMISSIONED (2026-08-28), removed from scope. | | **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) | | **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core | | **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS | @@ -148,7 +148,7 @@ All stored in `/root/.hermes/.env` and Hudu API assets (layout 49). | Database | Host | User | Password Location | Purpose | |----------|------|------|-------------------|---------| -| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress | +| **MySQL (apex track)** | app3:3306 | `apextrackexperience_1781549652` | RunCloud-era credential — target host wphost02 DECOMMISSIONED (2026-08-28); site now on app3/CloudPanel with a different credential scheme | Apex Track Experience WordPress (STALE — see comprehensive-audit-summary-2026-08-09.md) | | **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting | | **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB | @@ -217,7 +217,7 @@ The following credentials are known to exist but were not found in the standard | **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. | | **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. | | **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). | -| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. | +| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on app3 (wphost02 DECOMMISSIONED 2026-08-28). | | **app1-bu** | 5.161.225.131 | Hetzner CPX21 — accessed via itpp-infra SSH key only. | | **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. | | **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). | diff --git a/hermes-skills/systems-networks-team.md b/hermes-skills/systems-networks-team.md index 219fe56..675ad6c 100644 --- a/hermes-skills/systems-networks-team.md +++ b/hermes-skills/systems-networks-team.md @@ -47,7 +47,7 @@ Always load these before beginning a sys/net task: | app2 | 152.53.39.202 | Infrastructure server | | app3 | 152.53.241.111 | Web hosting + backup | | core-bu | 5.161.225.131 | Warm standby | -| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) | +| wphost02 | 5.161.62.38 | DECOMMISSIONED (2026-08-28) — deleted from Hetzner account | ## Key Credentials diff --git a/sites/app3.md b/sites/app3.md index 590c5ea..615fb42 100644 --- a/sites/app3.md +++ b/sites/app3.md @@ -45,5 +45,5 @@ ## Pending -- Migrate remaining WordPress sites from wphost02 to app3 +- ~~Migrate remaining WordPress sites from wphost02 to app3~~ — moot: wphost02 DECOMMISSIONED (2026-08-28), deleted from Hetzner account - Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)