ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: mark wphost02 decommissioned (2026-08-28); app1-bu is sole Hetzner box

Browse Source
This commit is contained in:
root
2026-08-28 09:42:36 -04:00
parent f5175f1ce0
commit f57f67c661
6 changed files with 13 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -2
View File
@@ -88,9 +88,11 @@
### Legacy / Decommissioned
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **DECOMMISSIONED (2026-08-28)** — deleted from Hetzner account; sites migrated to app3
- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
> **Hetzner Cloud (current):** As of 2026-08-28, the Hetzner Cloud API returns exactly **one** server — **app1-bu / core-bu** (`5.161.225.131`, CPX21, warm standby for Core). All other Hetzner boxes (old-ai, wphost02) are decommissioned/deleted.
---
## Model Fallback Chain
@@ -171,7 +173,7 @@ Direct API keys for all providers. Claude Sonnet 5 for primary quality, then dir
| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP configs |
| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
| ~~wphost02-backup~~ | ~~Daily 5 AM~~ | ~~s3://hermes-vps-backups/wphost02-backup/~~ | **REMOVED — wphost02 decommissioned 2026-08-28** |
| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
---
api-master-list.md
+1 -1
View File
@@ -100,7 +100,7 @@
| SyncroMSP | SyncroMSP | RMM/PSA, client asset management |
| Bitdefender GravityZone | Bitdefender | Endpoint security, client AV |
| Cloudflare | Cloudflare | DNS zones, domains, records |
| Hetzner Cloud | Hetzner | app1-bu standby server, wphost02 |
| Hetzner Cloud | Hetzner | app1-bu standby server (sole online Hetzner box as of 2026-08-28; wphost02 decommissioned) |
| netcup | netcup | Core/app1/app2/app3 servers |
| Wasabi S3 | Wasabi | All backups (hermes-vps-backups bucket, app backups) |
| Firecrawl | Firecrawl | Web extraction (Super Search fallback) |
backup-plan.md
+3 -14
View File
@@ -63,11 +63,9 @@
| Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
| modelortho.com | `modelortho-backup.sh` — htdocs tar.gz + nginx configs | `s3://hermes-vps-backups/app3/modelortho/` | 4:30 AM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21
### wphost02 (5.161.62.38) — Hetzner CPX21 — **DECOMMISSIONED (2026-08-28)**
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| WordPress (7 sites) | `ssh → /root/backup.sh` — per-site tar.gz + all DBs | `s3://hermes-vps-backups/wphost02-backup/` | 5:00 AM | 2026-07-28 |
wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
### Home Router
@@ -101,7 +99,6 @@
| 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
| 4:15 AM | **RAGFlow (App2)** | Hermes cron | `ragflow-backup.sh` |
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
| 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
| 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
| 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
@@ -133,7 +130,6 @@
| `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
| `dawarich-backup.sh` | Dawarich PostgreSQL dump + compose + env (SSH to App2) | Hermes cron 4:00 AM |
| `ragflow-backup.sh` | RAGFlow MySQL dump + compose + env (SSH to App2) | Hermes cron 4:15 AM |
| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
| `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
| `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
| `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM |
@@ -147,12 +143,6 @@
|--------|---------|------|
| `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
### On wphost02 (`/root/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `backup.sh` | All 7 WordPress sites + all MySQL DBs | Triggered via SSH from Core |
---
## S3 Bucket Structure
@@ -192,7 +182,6 @@ hermes-vps-backups/
├── gitea/daily/ — Gitea repos + DB
├── unms-backups/live/ — UNMS auto-backups
├── unifi-backups/ — UniFi controller backups
├── wphost02-backup/ — wphost02 per-site + DB
├── volumes/ — Docker volume dumps
├── caddy/ — (unused)
├── assets/ — Static assets
@@ -266,4 +255,4 @@ All the following services were migrated from Core to App1 in a single session:
- Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/`
- Provider diversity: netcup outage won't kill both Core and standby
- **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
- **24 backup scripts** on Core, 1 on App3, 1 on wphost02
- **Backup scripts:** 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)
docs/infrastructure/key-inventory.md
+3 -3
View File
@@ -11,7 +11,7 @@
| Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To |
|----------|------|------|-----------------------|---------|-------------|
| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) |
| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, app1-bu, home router). wphost02 DECOMMISSIONED (2026-08-28), removed from scope. |
| **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) |
| **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core |
| **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS |
@@ -148,7 +148,7 @@ All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
| Database | Host | User | Password Location | Purpose |
|----------|------|------|-------------------|---------|
| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
| **MySQL (apex track)** | app3:3306 | `apextrackexperience_1781549652` | RunCloud-era credential — target host wphost02 DECOMMISSIONED (2026-08-28); site now on app3/CloudPanel with a different credential scheme | Apex Track Experience WordPress (STALE — see comprehensive-audit-summary-2026-08-09.md) |
| **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
@@ -217,7 +217,7 @@ The following credentials are known to exist but were not found in the standard
| **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
| **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
| **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on app3 (wphost02 DECOMMISSIONED 2026-08-28). |
| **app1-bu** | 5.161.225.131 | Hetzner CPX21 — accessed via itpp-infra SSH key only. |
| **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. |
| **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). |
hermes-skills/systems-networks-team.md
+1 -1
View File
@@ -47,7 +47,7 @@ Always load these before beginning a sys/net task:
| app2 | 152.53.39.202 | Infrastructure server |
| app3 | 152.53.241.111 | Web hosting + backup |
| core-bu | 5.161.225.131 | Warm standby |
| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) |
| wphost02 | 5.161.62.38 | DECOMMISSIONED (2026-08-28) — deleted from Hetzner account |
## Key Credentials
sites/app3.md
+1 -1
View File
@@ -45,5 +45,5 @@
## Pending
- Migrate remaining WordPress sites from wphost02 to app3
- ~~Migrate remaining WordPress sites from wphost02 to app3~~ — moot: wphost02 DECOMMISSIONED (2026-08-28), deleted from Hetzner account
- Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)