From 2d3228e263ea753db977d02af57373087b5d0ef3 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 20 Jul 2026 17:10:04 -0400 Subject: [PATCH] docs: ops portal and backup-restore documentation (2026-07-20) --- docs/backup-restore/CHANGELOG.md | 30 +++++++++++++++++ docs/backup-restore/README.md | 50 +++++++++++++++++++++++++++++ docs/ops-portal/CHANGELOG.md | 39 ++++++++++++++++++++++ docs/ops-portal/README.md | 55 ++++++++++++++++++++++++++++++++ docs/project-log.md | 30 +++++++++++++++++ 5 files changed, 204 insertions(+) create mode 100644 docs/backup-restore/CHANGELOG.md create mode 100644 docs/backup-restore/README.md create mode 100644 docs/ops-portal/CHANGELOG.md create mode 100644 docs/ops-portal/README.md create mode 100644 docs/project-log.md diff --git a/docs/backup-restore/CHANGELOG.md b/docs/backup-restore/CHANGELOG.md new file mode 100644 index 0000000..07ba660 --- /dev/null +++ b/docs/backup-restore/CHANGELOG.md @@ -0,0 +1,30 @@ +# Backup-Restore Changelog + +## 2026-07-20 — Restore logging and manual backup + +### Added +- **Backup Now button:** Green "+ Backup Now" button on backup page +- **Domain dropdown:** Select element with all 9 hosted domains +- **Note field:** Optional "why" note saved as note.txt in snapshot +- **Restore History section:** Auto-expanded table at bottom — Domain, Snapshot, Date/Time, Status +- **Restore logging:** Every restore writes to `/opt/backup-restore/logs/restore.log` +- **Status formatting:** Green OK / red FAILED with centered status column + +### Fixed +- **Restore timing out:** Caddy flush_interval added + 300s transport timeouts +- **Route ordering:** `/api/restore` decorator was stacked on backup function → reconnected to restore function +- **API routing:** `/api/restore`, `/api/backup`, `/api/log` not proxied → added to Caddy config +- **Mobile toggle:** Inline `display:none` on site tables removed → CSS class toggle now works +- **Mobile touch:** role="button", tabindex="0", Enter key support added to card headers +- **Auto-expand first domain:** Removed — all domains now start collapsed +- **Restore History auto-expanded:** tbl-log has class="show", arrow is ▼ + +### Changed +- "Backup Log History" → "Restore History" +- Config page scripts directory now shows content when clicked + +## 2026-07-17 — Initial deployment +- Flask app deployed on app3 as systemd service +- Snapshot script scheduled (1 AM, 1 PM) +- 9 WordPress sites configured for backup +- Caddy proxy from Core via my.itpropartner.com diff --git a/docs/backup-restore/README.md b/docs/backup-restore/README.md new file mode 100644 index 0000000..d71d530 --- /dev/null +++ b/docs/backup-restore/README.md @@ -0,0 +1,50 @@ +# Backup-Restore — my.itpropartner.com/backups/ + +## Architecture +- **Server:** app3 (152.53.241.111, netcup RS 4000) +- **Backend:** Flask Python app at `/opt/backup-restore/app/app.py` (port 8090) +- **Proxy:** Caddy on Core → reverse_proxy to 152.53.241.111:8090 with 300s timeouts +- **Snapshots:** `/opt/backup-restore/snapshots///` +- **Scheduled:** `0 1,13 * * * /opt/backup-restore/snapshot.sh` — 1 AM and 1 PM daily +- **Systemd:** `backup-restore.service` +- **Retention:** 30 days (auto-cleanup) + +## Sites Backed Up (9 domains) +apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com, iamgmb.com, katiewattdesign.com, katiewattsdesign.com, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com + +## Snapshot Contents +Each snapshot directory contains: +- `files.tar.gz` — WordPress document root tarball +- `database.sql` — MySQL dump +- `size.txt` — Total backup size in bytes +- `note.txt` — Optional manual backup note + +## API Endpoints +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| GET | /backups/ | No | Backup dashboard page | +| POST | /api/backup | No | Trigger manual backup | +| POST | /api/restore | No | Restore a snapshot | +| POST | /api/delete | No | Delete a snapshot | +| GET | /api/download// | No | Download snapshot archive | +| GET | /api/log | No | Restore history | + +## Caddy Routes (on Core) +``` +handle /api/backup → app3:8090 +handle /api/restore → app3:8090 (flush_interval -1, 300s timeouts) +handle /api/download/* → app3:8090 +handle /api/log → app3:8090 +handle_path /backups/* → app3:8090 (300s timeouts) +``` + +## Recovery +``` +systemctl restart backup-restore +# Manual snapshot: +/opt/backup-restore/snapshot.sh +# Manual restore via curl: +curl -X POST https://my.itpropartner.com/api/restore \ + -H "Content-Type: application/json" \ + -d '{"domain":"katiewattdesign.com","snapshot":"2026-07-20_130001"}' +``` diff --git a/docs/ops-portal/CHANGELOG.md b/docs/ops-portal/CHANGELOG.md new file mode 100644 index 0000000..bebb4ff --- /dev/null +++ b/docs/ops-portal/CHANGELOG.md @@ -0,0 +1,39 @@ +# Ops Portal Changelog + +## 2026-07-20 — Major audit and fix session + +### Fixed +- **`/api/health` returning 404:** Caddy `handle_path` was stripping the path → changed to `handle`, port corrected to 8090 +- **`/api/servers` returning null:** Server list now returns all 5 servers with live ping health +- **Server IPs stale:** app1-bu updated to 5.161.225.131, legacy entries removed +- **Page titles inconsistent:** All 11 pages standardized to "X — IT Pro Partner Ops" format +- **Missing nav icons:** All 11 nav items now have SVG icons +- **FleetTracker360 missing from nav:** Added to navigation with car icon +- **Backups page no data:** `s3_buckets` → `s3_backups` key fix +- **FleetTracker360 page no nav:** Added ops.css, app.js, utils.js includes +- **Network page dark sidebar:** Replaced with standard top nav bar +- **Cache-busting broken:** All JS/CSS references now versioned with timestamps +- **Mobile nav broken:** `.nav-links.open` CSS rule missing → hamburger menu now toggles properly on iOS/Android +- **Auth guard race condition:** IIFE scripts replaced with DOMContentLoaded event listeners — pages now load data when user is authenticated +- **Cost page broken:** Missing `loadData` function → defined and wired +- **Dependency diagram 404:** File copied to static dir, link corrected +- **Logs page mangled title:** Triple-nested `` tags from sed accident → cleaned +- **Config page scripts directory:** Now populates directory listing when clicked +- **Services page:** Server column added showing "Core (152.53.192.33)" +- **Dashboard auto-refresh on tab focus:** Visibility API handler added +- **Critical service protection:** hermes, caddy, ops-portal restarts blocked via API + +### Removed +- Duplicate server entries: "app1 (AI Stack)" and "Docker Box (legacy)" +- Server count: 7 → 5 clean entries + +### Changed +- Admin credentials: germaine/itpp2026! → ippadmin/LoveMyBoys.1520! +- JWT_SECRET made persistent in /root/.hermes/.env to survive restarts +- Collector timeout: 20s → 90s to handle 94K-file S3 bucket scanning + +## Jul 17, 2026 — Initial deployment +- Ops portal deployed on Core as FastAPI app +- Caddy reverse proxy configured +- 10 HTML pages created +- Ops collector built for S3, system health, server status diff --git a/docs/ops-portal/README.md b/docs/ops-portal/README.md new file mode 100644 index 0000000..43b378d --- /dev/null +++ b/docs/ops-portal/README.md @@ -0,0 +1,55 @@ +# Ops Portal — ops.itpropartner.com + +## Architecture +- **Server:** Core (152.53.192.33, netcup RS 2000) +- **Backend:** FastAPI at `/opt/ops-portal/server.py` (port 8090) +- **Proxy:** Caddy → reverse_proxy to 127.0.0.1:8090 +- **Static files:** `/opt/ops-portal/static/` — 11 HTML pages, ops.css, app.js, utils.js +- **Auth:** JWT via `POST /api/auth/login`, token in localStorage +- **Data:** `/var/www/ops/data/ops-status.json` (5-min collector refresh) +- **Collector:** `/root/.hermes/scripts/ops-data-collector.py` — Wazuh, Bitdefender, S3, UISP, system health +- **Systemd:** `ops-portal.service`, env from `/root/.hermes/.env` +- **Credentials:** ippadmin / LoveMyBoys.1520! + +## Pages (11 total) +| Page | Path | Description | +|------|------|-------------| +| Dashboard | / | System health, widgets, audit log | +| Services | /services.html | Systemd service control, audit log, server column | +| Servers | /servers.html | 5 servers with ping health | +| Network | /network.html | UISP data (44 sites, 90 devices), DNS zones | +| Backups | /backups.html | S3 bucket status (6 buckets) | +| FleetTracker | /fleettracker360.html | Traccar device tracking | +| Cron Jobs | /cron.html | Hermes cron jobs with expandable scripts | +| Config | /config.html | Active configs, /root/.hermes/scripts/ listing | +| Logs | /logs.html | Aggregated log viewer | +| Audit | /audit.html | Full audit trail | +| Costs | /cost.html | API cost tracking by model | + +## API Endpoints +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| POST | /api/auth/login | No | JWT authentication | +| GET | /api/health | No | Health check, DB status | +| GET | /api/status | JWT | Full dashboard data (17 sections) | +| GET | /api/servers | JWT | Server list with IPs | +| GET | /api/servers/health | JWT | Ping health (all 7 LIVE) | +| GET | /api/audit-log?limit=N | JWT | Audit trail entries | +| GET | /api/ft360/status | JWT | FleetTracker device data | + +## Critical Services (API restart blocked) +hermes, hermes-assistant, hermes-browser, caddy, ops-portal, mysql-tunnel + +## Dashboard Widgets +- System Health — Core metrics (jobs, disk, memory, S3, APIs) +- Wazuh Security — agent count, alerts +- Bitdefender GravityZone — 9 managed endpoints +- Alerts and Notifications — DR issues, backup failures, cron errors +- Quick Actions — Restart Ops Portal + +## Recovery +``` +systemctl restart ops-portal +systemctl reload caddy +python3 /root/.hermes/scripts/ops-data-collector.py +``` diff --git a/docs/project-log.md b/docs/project-log.md new file mode 100644 index 0000000..82308ad --- /dev/null +++ b/docs/project-log.md @@ -0,0 +1,30 @@ +# Project Log — All Completed Projects + +## 2026-07-20 + +### Ops Portal Audit and Overhaul +- Full audit of all 11 pages, 7 API endpoints, and 5 dashboard widgets +- Fixed 15 bugs: auth guards, cache-busting, mobile nav, page titles, missing icons, data keys +- Added 3 new widgets: Wazuh Security, Bitdefender GravityZone, Alerts and Notifications +- Standardized credentials: ippadmin / LoveMyBoys.1520! +- Added critical service protection (hermes/caddy/ops-portal restart blocked via API) +- Server list cleaned up (7→5), dependency diagram fixed, config page scripts listing + +### Backup-Restore Enhancements +- Added manual backup with domain dropdown and note field +- Added restore history logging with formatted 4-column table +- Fixed Caddy routing and timeouts (restore was returning 404 via proxy) +- Fixed mobile toggle on domain expansion cards +- 9 WordPress sites under daily backup (1 AM and 1 PM) + +### Docs Written +- `/root/projects/ops-portal/README.md` + `CHANGELOG.md` +- `/root/projects/backup-restore/README.md` + `CHANGELOG.md` + +--- + +## 2026-07-17 — Backup-Restore Initial Deployment +- Flask backup/restore app deployed on app3 (152.53.241.111) +- Daily snapshots scheduled at 1 AM and 1 PM +- Caddy reverse proxy from my.itpropartner.com +- 9 WordPress sites configured