docs: add ARCHITECTURE.md for ops-portal and backup-restore

docs/backup-restore/ARCHITECTURE.md

@@ -0,0 +1,158 @@
+# Backup-Restore — Architecture
+
+## Topology
+
+```
+                         INTERNET
+                            |
+                    [Caddy on Core]
+                   my.itpropartner.com
+                            |
+            +---------------+---------------+
+            |               |               |
+     /backups/*      /api/restore    /api/backup
+     /api/log        /api/download   /api/delete
+            |               |               |
+            +-------+-------+-------+-------+
+                            |
+                    app3 (152.53.241.111)
+                     netcup RS 4000
+                            |
+                    [Flask :8090]
+                   /opt/backup-restore/
+                            |
+        +-------------------+-------------------+
+        |                   |                   |
+  snapshot.sh         app.py (UI+API)     snapshots/
+  (cron 1AM,1PM)      Jinja templates     /opt/backup-restore/
+        |                   |              snapshots/<domain>/
+        v                   v                   |
+  [tar files]          [render HTML]     +------+------+
+  [mysqldump]          [REST API]        |      |      |
+        |                   |          .tar.gz  .sql  note.txt
+        v                   v
+  /opt/backup-restore/  [Browser]
+  snapshots/<domain>/
+  <timestamp>/
+```
+
+## Data Flow — Manual Backup
+
+```
+Browser (user clicks "Backup Now")
+  |
+  |-- POST /api/backup {"domain":"x.com","note":"pre-deploy"}
+  |       |
+  |       v
+  |   Caddy → app3:8090
+  |       |
+  |       v
+  |   Flask api_backup()
+  |       |
+  |       |-- Parse nginx config → find htdocs path
+  |       |-- tar -czf files.tar.gz (timeout 300s)
+  |       |-- Parse wp-config.php → find DB_NAME
+  |       |-- mysqldump → database.sql (timeout 300s)
+  |       |-- Save note.txt, size.txt
+  |       |-- Return {"ok":true, "snapshot":"<timestamp>"}
+  |       |
+  |       v
+  |   snapshots/x.com/2026-07-20_163208/
+  |       files.tar.gz (16MB)
+  |       database.sql (74KB)
+  |       note.txt ("pre-deploy")
+  |       size.txt
+  |
+  v
+Browser reloads → new snapshot in list
+```
+
+## Data Flow — Restore
+
+```
+Browser (user clicks Restore on a snapshot)
+  |
+  |-- POST /api/restore {"domain":"x.com","snapshot":"2026-07-20_130001"}
+  |       |
+  |       v
+  |   Caddy → app3:8090 (flush_interval -1, 300s timeouts)
+  |       |
+  |       v
+  |   Flask api_restore()
+  |       |
+  |       |-- Find snapshot path
+  |       |-- tar -xzf files.tar.gz → htdocs (timeout 300s)
+  |       |-- mysql < database.sql → WordPress DB (timeout 300s)
+  |       |-- chown -R site-user:site-user
+  |       |-- Log to restore.log: "TS|x.com|snap_id|OK"
+  |       |-- Return {"ok":true, "msg":"x.com restored to <snap>"}
+  |       |
+  |       v
+  |   Site is restored
+  |
+  v
+Browser shows success toast → Restore History updates
+```
+
+## Components
+
+### 1. Flask App (`/opt/backup-restore/app/app.py`)
+- Single-file Flask application, port 8090
+- Jinja2 templating for backup dashboard (render_template_string)
+- 6 API endpoints (backup, restore, delete, download, log, index)
+- All HTML/CSS/JS inline in a single Python triple-quoted string
+- No auth — accessible via Caddy-only routing
+- Systemd: `backup-restore.service`
+
+### 2. Snapshot Engine (`/opt/backup-restore/snapshot.sh`)
+- Bash script, runs at 1 AM and 1 PM via cron
+- Iterates all WordPress sites in `/etc/nginx/sites-enabled/`
+- Creates: files.tar.gz (document root), database.sql (MySQL dump)
+- Auto-cleanup: deletes snapshots older than 30 days
+- Log: `/opt/backup-restore/logs/snapshots.log`
+
+### 3. Snapshot Storage (`/opt/backup-restore/snapshots/`)
+- Structure: `/<domain>/<YYYY-MM-DD_HHMMSS>/`
+- 9 WordPress domains, 10 snapshots each (10 days retention shown)
+- Average snapshot size: 16MB files + 74KB database
+- Total: ~1.4GB for full snapshot set
+
+### 4. Restore Log (`/opt/backup-restore/logs/restore.log`)
+- Pipe-delimited format: `timestamp|domain|snapshot_id|status`
+- Written by api_restore() on every restore attempt
+- Read by /api/log → displayed in Restore History table
+- Last 50 entries retained
+
+### 5. Caddy Proxy (on Core)
+- `handle /api/backup` → app3:8090
+- `handle /api/restore` → app3:8090 (flush_interval -1, 300s read/write timeouts)
+- `handle /api/download/*` → app3:8090
+- `handle /api/log` → app3:8090
+- `handle_path /backups/*` → app3:8090 (300s timeouts for long restores)
+- Domain: my.itpropartner.com
+
+## 9 Hosted WordPress Sites
+
+All served by CloudPanel on app3, backed up by this system:
+
+| Domain | htdocs Path | DB Pattern |
+|---|---|---|
+| apextrackexperience.com | /home/apx/htdocs/apextrackexperience.com | wp-config DB_NAME |
+| boxpilotlogistics.com | /home/boxpilotlogistics/htdocs/boxpilotlogistics.com | wp-config DB_NAME |
+| debtrecoveryexperts.com | /home/debtrecoveryexperts/... | wp-config DB_NAME |
+| iamgmb.com | /home/iamgmb/... | wp-config DB_NAME |
+| katiewattdesign.com | /home/katiewattdesign/htdocs/katiewattdesign.com | wp-config DB_NAME |
+| katiewattsdesign.com | /home/katiewattsdesign/... | wp-config DB_NAME |
+| mainwp.itpropartner.com | /home/mainwp/... | wp-config DB_NAME |
+| vigilanttac.com | /home/vigilanttac/... | wp-config DB_NAME |
+| voipsimplicity.com | /home/voipsimplicity/... | wp-config DB_NAME |
+
+## Key Design Decisions
+
+1. **Single-file Flask app:** No package structure needed — the app has 6 endpoints and one HTML template. Keeping it in one file makes deployment trivial (scp + systemctl restart).
+
+2. **Caddy on Core as single entry point:** app3 isn't exposed to the internet directly. All access goes through Core's Caddy with proper timeouts. The restore operation takes 30-45s and Caddy's default proxy timeout was killing connections mid-operation.
+
+3. **Tar + mysqldump over rsync:** Snapshots are point-in-time archives, not incremental backups. Each snapshot is self-contained (files.tar.gz + database.sql). Restore is a single operation with no dependency chain.
+
+4. **No auth on backup API:** The endpoints have no authentication. Access is controlled by Caddy routing — only requests through my.itpropartner.com reach the app. Internal network only.

docs/ops-portal/ARCHITECTURE.md

@@ -0,0 +1,112 @@
+# Ops Portal — Architecture
+
+## Topology
+
+```
+                         INTERNET
+                            |
+                      [Caddy :443]
+                            |
+                    Core (152.53.192.33)
+                            |
+            +---------------+---------------+
+            |               |               |
+     /api/* :8090    /data/* :files     /static/*
+            |               |               |
+     [FastAPI app]    ops-status.json   [HTML/CSS/JS]
+     server.py       /var/www/ops/     /opt/ops-portal/
+            |            /data/           static/
+            |
+    +-------+-------+-------+-------+
+    |       |       |       |       |
+  S3 API  UISP   Wazuh  Bitdef  systemd
+ (Wasabi) (FFW) (app1) (Cloud) (Core)
+```
+
+## Data Flow
+
+```
+[Collector]                     [Dashboard]
+     |                               |
+     |-- python3 ops-data-           |
+     |   collector.py                |
+     |                               |
+     v                               |
+  S3 buckets ----+                   |
+  UISP API ------+---> ops-status    |
+  Wazuh API -----+     .json ------> GET /api/status
+  Bitdefender ---+                   |
+  systemd -------+                   |
+  cron jobs -----+                   |
+                                     v
+                              [Browser renders
+                               health grid,
+                               widgets, alerts]
+```
+
+## Components
+
+### 1. Collector (`/root/.hermes/scripts/ops-data-collector.py`)
+- Runs every 5 min via cron
+- Gathers: S3 backup status (6 buckets), UISP devices (90), Wazuh agents/alerts, Bitdefender endpoints, systemd services, cron jobs, server health, disk/memory/CPU
+- Timeout: 90s (was 20s — too short for 94K-file S3 bucket)
+- Output: `/var/www/ops/data/ops-status.json`
+
+### 2. Backend (`/opt/ops-portal/server.py`)
+- FastAPI on port 8090
+- 7 API endpoints (health, status, servers, servers/health, audit-log, ft360/status)
+- JWT auth from `/root/.hermes/.env` (ADMIN_USERNAME, ADMIN_PASSWORD, JWT_SECRET)
+- Critical service restart protection (hermes, caddy, ops-portal blocked)
+- Systemd: `ops-portal.service`
+
+### 3. Frontend (`/opt/ops-portal/static/`)
+- 11 HTML pages with shared ops.css, app.js, utils.js
+- Auth: login overlay → localStorage JWT → all API calls Bearer
+- Auto-refresh: 60s interval + tab visibility API
+- Mobile: hamburger toggle with .nav-links.open CSS
+- Cache-busting: all assets versioned with timestamps
+
+### 4. Proxy (Caddy on Core)
+- `/` and `/*.html` → static file server from `/opt/ops-portal/static/`
+- `/api/*` → reverse_proxy to 127.0.0.1:8090
+- `/data/*` → file server from `/var/www/ops/data/`
+- Domain: ops.itpropartner.com
+
+## Cross-Service Dependencies
+
+| Dependency | Server | Purpose | Fallback |
+|---|---|---|---|
+| Wasabi S3 | External | Backup bucket status | Shows "Issues" |
+| UISP API | unms.forefrontwireless.com | Device/site count | Shows 0 devices |
+| Wazuh | app1 (152.53.36.131) | Agent count, alerts | Shows "Offline" |
+| Bitdefender | External API | Endpoint monitoring | Shows "Offline" |
+| Traccar | app2 (152.53.39.202) | FleetTracker data | Dedicated endpoint |
+| Core systemd | Local | Service health, disk, memory | N/A (local) |
+
+## Auth Flow
+
+```
+Browser                    Server
+  |                          |
+  |-- POST /api/auth/login ->|
+  |   {username, password}   |
+  |                          |-- Validate against ADMIN_USERNAME/ADMIN_PASSWORD
+  |                          |-- Generate JWT with JWT_SECRET
+  |<- {access_token} --------|
+  |                          |
+  |-- GET /api/status ------->|
+  |   Authorization: Bearer   |
+  |                          |-- Verify JWT
+  |                          |-- Read ops-status.json
+  |<- {full dashboard} ------|
+```
+
+## Key Design Decisions
+
+1. **Collector pattern over direct API calls:** Dashboard fetches one JSON blob rather than 6 separate APIs. Single point of failure but fast rendering and offline-capable (shows last-cached data).
+
+2. **Python/FastAPI over Node:** Already have Python toolchain on Core. FastAPI is lightweight, async-native, and the ops portal is read-heavy with minimal write paths.
+
+3. **Static HTML + vanilla JS over React/Vue:** 11-page dashboard with no SPA routing. Auth via localStorage JWT. Zero build step, zero dependencies beyond ops.css.
+
+4. **JWT over session cookies:** Cross-page auth without server-side session state. Token survives page navigations and ops-portal restarts (persistent JWT_SECRET in .env).