diff --git a/docs/backup-restore/ARCHITECTURE.md b/docs/backup-restore/ARCHITECTURE.md new file mode 100644 index 0000000..a318fea --- /dev/null +++ b/docs/backup-restore/ARCHITECTURE.md @@ -0,0 +1,158 @@ +# Backup-Restore — Architecture + +## Topology + +``` + INTERNET + | + [Caddy on Core] + my.itpropartner.com + | + +---------------+---------------+ + | | | + /backups/* /api/restore /api/backup + /api/log /api/download /api/delete + | | | + +-------+-------+-------+-------+ + | + app3 (152.53.241.111) + netcup RS 4000 + | + [Flask :8090] + /opt/backup-restore/ + | + +-------------------+-------------------+ + | | | + snapshot.sh app.py (UI+API) snapshots/ + (cron 1AM,1PM) Jinja templates /opt/backup-restore/ + | | snapshots// + v v | + [tar files] [render HTML] +------+------+ + [mysqldump] [REST API] | | | + | | .tar.gz .sql note.txt + v v + /opt/backup-restore/ [Browser] + snapshots// + / +``` + +## Data Flow — Manual Backup + +``` +Browser (user clicks "Backup Now") + | + |-- POST /api/backup {"domain":"x.com","note":"pre-deploy"} + | | + | v + | Caddy → app3:8090 + | | + | v + | Flask api_backup() + | | + | |-- Parse nginx config → find htdocs path + | |-- tar -czf files.tar.gz (timeout 300s) + | |-- Parse wp-config.php → find DB_NAME + | |-- mysqldump → database.sql (timeout 300s) + | |-- Save note.txt, size.txt + | |-- Return {"ok":true, "snapshot":""} + | | + | v + | snapshots/x.com/2026-07-20_163208/ + | files.tar.gz (16MB) + | database.sql (74KB) + | note.txt ("pre-deploy") + | size.txt + | + v +Browser reloads → new snapshot in list +``` + +## Data Flow — Restore + +``` +Browser (user clicks Restore on a snapshot) + | + |-- POST /api/restore {"domain":"x.com","snapshot":"2026-07-20_130001"} + | | + | v + | Caddy → app3:8090 (flush_interval -1, 300s timeouts) + | | + | v + | Flask api_restore() + | | + | |-- Find snapshot path + | |-- tar -xzf files.tar.gz → htdocs (timeout 300s) + | |-- mysql < database.sql → WordPress DB (timeout 300s) + | |-- chown -R site-user:site-user + | |-- Log to restore.log: "TS|x.com|snap_id|OK" + | |-- Return {"ok":true, "msg":"x.com restored to "} + | | + | v + | Site is restored + | + v +Browser shows success toast → Restore History updates +``` + +## Components + +### 1. Flask App (`/opt/backup-restore/app/app.py`) +- Single-file Flask application, port 8090 +- Jinja2 templating for backup dashboard (render_template_string) +- 6 API endpoints (backup, restore, delete, download, log, index) +- All HTML/CSS/JS inline in a single Python triple-quoted string +- No auth — accessible via Caddy-only routing +- Systemd: `backup-restore.service` + +### 2. Snapshot Engine (`/opt/backup-restore/snapshot.sh`) +- Bash script, runs at 1 AM and 1 PM via cron +- Iterates all WordPress sites in `/etc/nginx/sites-enabled/` +- Creates: files.tar.gz (document root), database.sql (MySQL dump) +- Auto-cleanup: deletes snapshots older than 30 days +- Log: `/opt/backup-restore/logs/snapshots.log` + +### 3. Snapshot Storage (`/opt/backup-restore/snapshots/`) +- Structure: `///` +- 9 WordPress domains, 10 snapshots each (10 days retention shown) +- Average snapshot size: 16MB files + 74KB database +- Total: ~1.4GB for full snapshot set + +### 4. Restore Log (`/opt/backup-restore/logs/restore.log`) +- Pipe-delimited format: `timestamp|domain|snapshot_id|status` +- Written by api_restore() on every restore attempt +- Read by /api/log → displayed in Restore History table +- Last 50 entries retained + +### 5. Caddy Proxy (on Core) +- `handle /api/backup` → app3:8090 +- `handle /api/restore` → app3:8090 (flush_interval -1, 300s read/write timeouts) +- `handle /api/download/*` → app3:8090 +- `handle /api/log` → app3:8090 +- `handle_path /backups/*` → app3:8090 (300s timeouts for long restores) +- Domain: my.itpropartner.com + +## 9 Hosted WordPress Sites + +All served by CloudPanel on app3, backed up by this system: + +| Domain | htdocs Path | DB Pattern | +|---|---|---| +| apextrackexperience.com | /home/apx/htdocs/apextrackexperience.com | wp-config DB_NAME | +| boxpilotlogistics.com | /home/boxpilotlogistics/htdocs/boxpilotlogistics.com | wp-config DB_NAME | +| debtrecoveryexperts.com | /home/debtrecoveryexperts/... | wp-config DB_NAME | +| iamgmb.com | /home/iamgmb/... | wp-config DB_NAME | +| katiewattdesign.com | /home/katiewattdesign/htdocs/katiewattdesign.com | wp-config DB_NAME | +| katiewattsdesign.com | /home/katiewattsdesign/... | wp-config DB_NAME | +| mainwp.itpropartner.com | /home/mainwp/... | wp-config DB_NAME | +| vigilanttac.com | /home/vigilanttac/... | wp-config DB_NAME | +| voipsimplicity.com | /home/voipsimplicity/... | wp-config DB_NAME | + +## Key Design Decisions + +1. **Single-file Flask app:** No package structure needed — the app has 6 endpoints and one HTML template. Keeping it in one file makes deployment trivial (scp + systemctl restart). + +2. **Caddy on Core as single entry point:** app3 isn't exposed to the internet directly. All access goes through Core's Caddy with proper timeouts. The restore operation takes 30-45s and Caddy's default proxy timeout was killing connections mid-operation. + +3. **Tar + mysqldump over rsync:** Snapshots are point-in-time archives, not incremental backups. Each snapshot is self-contained (files.tar.gz + database.sql). Restore is a single operation with no dependency chain. + +4. **No auth on backup API:** The endpoints have no authentication. Access is controlled by Caddy routing — only requests through my.itpropartner.com reach the app. Internal network only. diff --git a/docs/ops-portal/ARCHITECTURE.md b/docs/ops-portal/ARCHITECTURE.md new file mode 100644 index 0000000..c786d6c --- /dev/null +++ b/docs/ops-portal/ARCHITECTURE.md @@ -0,0 +1,112 @@ +# Ops Portal — Architecture + +## Topology + +``` + INTERNET + | + [Caddy :443] + | + Core (152.53.192.33) + | + +---------------+---------------+ + | | | + /api/* :8090 /data/* :files /static/* + | | | + [FastAPI app] ops-status.json [HTML/CSS/JS] + server.py /var/www/ops/ /opt/ops-portal/ + | /data/ static/ + | + +-------+-------+-------+-------+ + | | | | | + S3 API UISP Wazuh Bitdef systemd + (Wasabi) (FFW) (app1) (Cloud) (Core) +``` + +## Data Flow + +``` +[Collector] [Dashboard] + | | + |-- python3 ops-data- | + | collector.py | + | | + v | + S3 buckets ----+ | + UISP API ------+---> ops-status | + Wazuh API -----+ .json ------> GET /api/status + Bitdefender ---+ | + systemd -------+ | + cron jobs -----+ | + v + [Browser renders + health grid, + widgets, alerts] +``` + +## Components + +### 1. Collector (`/root/.hermes/scripts/ops-data-collector.py`) +- Runs every 5 min via cron +- Gathers: S3 backup status (6 buckets), UISP devices (90), Wazuh agents/alerts, Bitdefender endpoints, systemd services, cron jobs, server health, disk/memory/CPU +- Timeout: 90s (was 20s — too short for 94K-file S3 bucket) +- Output: `/var/www/ops/data/ops-status.json` + +### 2. Backend (`/opt/ops-portal/server.py`) +- FastAPI on port 8090 +- 7 API endpoints (health, status, servers, servers/health, audit-log, ft360/status) +- JWT auth from `/root/.hermes/.env` (ADMIN_USERNAME, ADMIN_PASSWORD, JWT_SECRET) +- Critical service restart protection (hermes, caddy, ops-portal blocked) +- Systemd: `ops-portal.service` + +### 3. Frontend (`/opt/ops-portal/static/`) +- 11 HTML pages with shared ops.css, app.js, utils.js +- Auth: login overlay → localStorage JWT → all API calls Bearer +- Auto-refresh: 60s interval + tab visibility API +- Mobile: hamburger toggle with .nav-links.open CSS +- Cache-busting: all assets versioned with timestamps + +### 4. Proxy (Caddy on Core) +- `/` and `/*.html` → static file server from `/opt/ops-portal/static/` +- `/api/*` → reverse_proxy to 127.0.0.1:8090 +- `/data/*` → file server from `/var/www/ops/data/` +- Domain: ops.itpropartner.com + +## Cross-Service Dependencies + +| Dependency | Server | Purpose | Fallback | +|---|---|---|---| +| Wasabi S3 | External | Backup bucket status | Shows "Issues" | +| UISP API | unms.forefrontwireless.com | Device/site count | Shows 0 devices | +| Wazuh | app1 (152.53.36.131) | Agent count, alerts | Shows "Offline" | +| Bitdefender | External API | Endpoint monitoring | Shows "Offline" | +| Traccar | app2 (152.53.39.202) | FleetTracker data | Dedicated endpoint | +| Core systemd | Local | Service health, disk, memory | N/A (local) | + +## Auth Flow + +``` +Browser Server + | | + |-- POST /api/auth/login ->| + | {username, password} | + | |-- Validate against ADMIN_USERNAME/ADMIN_PASSWORD + | |-- Generate JWT with JWT_SECRET + |<- {access_token} --------| + | | + |-- GET /api/status ------->| + | Authorization: Bearer | + | |-- Verify JWT + | |-- Read ops-status.json + |<- {full dashboard} ------| +``` + +## Key Design Decisions + +1. **Collector pattern over direct API calls:** Dashboard fetches one JSON blob rather than 6 separate APIs. Single point of failure but fast rendering and offline-capable (shows last-cached data). + +2. **Python/FastAPI over Node:** Already have Python toolchain on Core. FastAPI is lightweight, async-native, and the ops portal is read-heavy with minimal write paths. + +3. **Static HTML + vanilla JS over React/Vue:** 11-page dashboard with no SPA routing. Auth via localStorage JWT. Zero build step, zero dependencies beyond ops.css. + +4. **JWT over session cookies:** Cross-page auth without server-side session state. Token survives page navigations and ops-portal restarts (persistent JWT_SECRET in .env).