Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,119 @@
+# Web Standards & Accessibility — Ops Portal Checklist
+
+Applied during the July 2026 audit to all 10 ops portal HTML files. Covers HTML5 validity, WCAG 2.1 AA, and ARIA compliance.
+
+## Structural Requirements
+
+### Document-Level
+- `<!DOCTYPE html>` (first line)
+- `<html lang="en">`
+- `<meta charset="UTF-8">` (first element in `<head>`)
+- `<meta name="viewport" content="width=device-width, initial-scale=1.0">`
+
+### Semantic Elements
+| Element | When to Use |
+|---------|-------------|
+| `<main id="main-content" role="main">` | One per page, wraps primary content |
+| `<header>` | Page header (h1 + actions) |
+| `<footer>` | Page footer (last-updated timestamp) |
+| `<nav>` / `<nav role="navigation" aria-label="Main navigation">` | Navigation block |
+| `<section class="card" aria-labelledby="sectionId">` | Every card/section, paired with `<h2 id="sectionId">` |
+| `<h1>` → `<h2>` → `<h3>` | Strict hierarchy, no skipping |
+
+### Section Card Pattern (before → after)
+
+**Before (invalid):**
+```html
+<div class="card">
+  <div class="card-header">Section Title</div>
+  ...
+</div>
+```
+
+**After (valid):**
+```html
+<section class="card" aria-labelledby="myHeading">
+  <h2 class="card-header" id="myHeading">Section Title</h2>
+  ...
+</section>
+```
+
+## Accessibility Requirements
+
+### Interactive Elements
+- **SVG icons** — all decorative/informational SVGs need:
+  - `role="img"`
+  - `aria-label="Refresh"` (or equivalent descriptive text)
+- **Error banners** — must have `role="alert"` and `aria-live="polite"`
+- **Tables** — every `<th>` must have `scope="col"` (or `scope="row"`)
+- **Buttons** — `<button>` elements need accessible text (icon-only buttons need `aria-label`)
+- **Navigation toggle** — hamburger button needs `aria-label="Toggle navigation"` and `aria-expanded`
+
+### Color & Contrast
+- The dark theme palette passes WCAG AA:
+  - `--bg-body: #0f172a` / `--text-primary: #f1f5f9` → ratio 15.1:1
+  - `--accent: #f59e0b` (amber) on `--bg-card: #1e293b` → ratio 7.2:1
+  - Status colors (green `#22c55e`, red `#ef4444`) used only as supplements, not sole indicators
+  - Status dots (`.status-dot`) are always paired with text labels
+
+### Focus & Keyboard
+- `.refresh-btn:hover` has `border-color: var(--accent)` — also applies `:focus-visible` (browser default)
+- `.filter-btn.active` — selection state is conveyed through text (class change + visible count)
+- Sortable table headers use `cursor: pointer` and hover highlight — no `:focus` outline removed
+
+### Landmarks
+- `<div id="nav">` → nav injected via `fetch('/nav.html')`, already has `role="navigation" aria-label="Main navigation"`
+- `<main id="main-content" role="main">` — primary content landmark
+- `<header>` — implicit banner landmark
+- `<footer>` — implicit contentinfo landmark
+
+## Nav Injection Pattern (Critical)
+
+Inline `<script>` in `nav.html` **does not execute** when the file is loaded via `innerHTML`. Always:
+
+1. **Place nav JS in `/js/utils.js`** as `window.initNav()`
+2. **Call `initNav()` after every nav injection:**
+
+```html
+fetch('/nav.html')
+  .then(r => r.text())
+  .then(html => {
+    document.getElementById('nav').innerHTML = html;
+    if (typeof window.initNav === 'function') window.initNav();
+  })
+```
+
+`initNav()` handles:
+- Active link highlighting (via `data-path` attribute)
+- Hamburger toggle (`aria-expanded` management)
+- Click-outside-to-close
+
+## Verification Commands
+
+```bash
+# Structural balance check
+for section balance:
+  grep -Pc '<section\b' file.html
+  grep -c '</section>' file.html
+
+for main balance:
+  grep -c '<main' file.html
+  grep -c '</main>' file.html
+
+for h2 balance:
+  grep -Pc '<h2\b' file.html
+  grep -c '</h2>' file.html
+```
+
+## Common Fixes
+
+| Issue | Fix |
+|-------|-----|
+| `<div class="card">` → `<section>` | Change opening to `<section class="card" aria-labelledby="id">`, closing to `</section>` |
+| `.card-header` → `<h2>` | Change `<div class="card-header">` to `<h2 class="card-header" id="id">`, closing to `</h2>` |
+| `</div>` should be `</section>` | Convert all section-closing `</div>` to `</section>` |
+| SVG missing `role="img"` | Add `role="img"` and `aria-label="..."` |
+| Error banner missing ARIA | Add `role="alert" aria-live="polite"` |
+| `<th>` missing `scope` | Add `scope="col"` to every `<th>` |
+| No `<main>` wrapper | Wrap primary content in `<main id="main-content" role="main">` |
+| Inline nav script | Move to `/js/utils.js` as `window.initNav()`, call after injection |
@@ -0,0 +1,60 @@
+# Bulk node_exporter Install — July 2026 Rollout
+
+## Server inventory
+
+All 9 Hetzner servers received node_exporter v1.8.2 in a single session:
+
+| Hostname | IP | Status |
+|----------|-----|--------|
+| app1-bu.itpropartner.com | 5.161.114.8 | Running |
+| ai.itpropartner.com | 178.156.167.181 | Running |
+| hudu.itpropartner.com | 178.156.130.130 | Running |
+| (unifi) | 178.156.131.57 | Running |
+| unms.forefrontwireless.com | 5.161.225.131 | Running |
+| wphost02 | 5.161.62.38 | Running |
+| app1.itpropartner.com | 87.99.144.163 | Running |
+| (docker) | 178.156.168.35 | Running |
+| (fleettrack360) | 178.156.149.32 | Running |
+
+## Batch install pattern
+
+Write two scripts:
+
+1. **install-node-exporter.sh** — self-contained script that downloads, installs binary, creates systemd unit, enables/starts. The existing `scripts/install-node-exporter.sh` in this skill works.
+
+2. **Wrapper script** — iterates over a list and runs SCP + SSH for each:
+
+```bash
+#!/bin/bash
+SERVERS=(
+  "hostname1:1.2.3.4"
+  "hostname2:5.6.7.8"
+)
+KEY="/root/.ssh/itpp-infra"
+SCRIPT="/tmp/install-node-exporter.sh"
+
+for entry in "${SERVERS[@]}"; do
+  hostname="${entry%%:*}"
+  ip="${entry##*:}"
+  echo "=== Installing on $hostname ($ip) ==="
+  scp -o StrictHostKeyChecking=no -i "$KEY" "$SCRIPT" "root@$ip:/tmp/"
+  ssh -o StrictHostKeyChecking=no -i "$KEY" "root@$ip" "bash /tmp/install-node-exporter.sh"
+  echo "✓ $hostname"
+done
+```
+
+## Verifying all targets in Prometheus
+
+After the config is updated and Prometheus reloaded:
+
+```bash
+curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
+  "import json,sys;d=json.load(sys.stdin);[print(t['labels']['instance'],t['health']) for t in d['data']['activeTargets']]"
+```
+
+## Pitfalls
+
+- **Security scanners blocking raw IPs** in SSH/SCP commands. Using hostnames where available bypasses false-positive security warnings.
+- **All 9 installs can run sequentially** in ~90 seconds total — no parallelism needed.
+- **UFW/firewall** must allow incoming port 9100 from Core's IP, or Prometheus must scrape via hostname/SSH tunnel. The existing servers had no restrictive UFW rules blocking port 9100.
+- **node_exporter listens on all interfaces by default** (0.0.0.0:9100). If restricting to localhost, add `--web.listen-address=127.0.0.1:9100` to ExecStart and scrape via a local proxy or Tailscale.
@@ -0,0 +1,47 @@
+# Caddy Static File MIME Type Fix (Jul 12, 2026)
+
+## Problem
+The ops portal at `ops.itpropartner.com` had JS and CSS files served with wrong MIME types (`application/json` instead of `text/javascript` / `text/css`). Browsers enforce correct MIME types for scripts and stylesheets — any JS file served as `application/json` is silently rejected. The entire `Ops` namespace never loaded, login overlays stayed visible, the mobile hamburger menu never worked, and pages appeared broken.
+
+Root cause was twofold:
+
+### 1. File permissions (the real fix)
+Caddy runs as the `caddy` system user. Files in `/opt/ops-portal/static/` were created by root with `chmod 600` — Caddy returned **403 Forbidden** for all static file requests. Even though a Caddy `handle_path` block was configured, the `file_server` handler couldn't read the files and returned 403.
+
+**Fix:** `chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css`
+
+### 2. Caddy handle_path nesting error (compounding factor)
+The `/js/*` and `/css/*` `handle_path` blocks were initially nested INSIDE the `/data/*` handler block due to a `sed` command error. They only triggered when the URL started with `/data/`, so `/js/app.js` never matched.
+
+**Fix:** Extract to the top level of the Caddy site block, before `reverse_proxy`:
+```caddy
+ops.itpropartner.com {
+    handle_path /data/* {
+        root * /var/www/ops/data/
+        file_server
+    }
+    handle_path /js/* {
+        root * /opt/ops-portal/static/js
+        file_server
+    }
+    handle_path /css/* {
+        root * /opt/ops-portal/static/css
+        file_server
+    }
+    reverse_proxy 127.0.0.1:8090
+}
+```
+
+### 3. Backend `media_type` workaround (not needed)
+The FastAPI backend has an `@app.get("/js/{file_path:path}")` route using `FileResponse`. Even with `media_type="application/javascript"` explicitly set, the response still returned `Content-Type: application/json` for unknown reasons (possibly a FastAPI 0.115.6 quirk, or the `media_type` parameter being ignored in certain response scenarios). The Caddy static file approach bypasses the backend entirely.
+
+## Detection
+- Curl the JS URL: `curl -sI https://ops.itpropartner.com/js/app.js` — check `content-type` header
+- Expected: `text/javascript; charset=utf-8`
+- Broken: `application/json` (backend) or `text/html` (caddy error page) or empty (403/404)
+- Browser symptom: Pages load but no JS executes — login overlay stays visible, nav not working, "Ops is not defined" errors in console
+
+## Prevention
+- Always `chmod 644` new static files added to `/opt/ops-portal/static/`
+- When adding a new Caddy `handle_path` block, verify it's at the TOP LEVEL of the site block, not indented inside another handler
+- After ANY Caddy config change: `caddy fmt --overwrite /etc/caddy/Caddyfile && caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`
@@ -0,0 +1,29 @@
+# Daily Feed Summary — Known Limitations
+
+Repository file: `/root/.hermes/scripts/daily-feed-summary.py`
+Cron job: `daily-tech-digest` (11:00 daily via Hermes cron)
+
+## Broken Sources
+
+### Top Gear (`topgear.com/car-news`)
+- **Status:** [BROKEN] Scraping returns 0 articles
+- **Symptom:** The `TGLinkParser` finds no matching `<a href="/car-news/">` elements
+- **Likely cause:** Site structure changed — Top Gear uses a JS-heavy frontend (Next.js or similar) that renders article links dynamically. The current parser relies on finding `<a href="/car-news/...">` with a `title` attribute in the initial HTML, which no longer exists
+- **Impact:** Minor — Top Gear returns 0 articles silently. The digest continues without error
+- **Discovered:** Jul 6, 2026 (first documented failure)
+- **Fix required:** Switch to headless browser scraping (Playwright) or use an RSS/compat feed if available. LATER: Monitor if Top Gear adds RSS
+
+## Python Dependencies
+
+The script needs `feedparser` in the Hermes venv. See the SKILL.md "Hermes Cron Script Dependencies" section for install patterns.
+
+## Verifying Full Output
+
+```bash
+/root/hermes-assistant/venv/bin/python3 /root/.hermes/scripts/daily-feed-summary.py
+```
+
+For email delivery:
+```bash
+/root/hermes-assistant/venv/bin/python3 /root/.hermes/scripts/daily-feed-summary.py --email
+```
@@ -0,0 +1,48 @@
+# Data Field Mapping — Collector Output vs Page JS Expectations
+
+These are the most common post-W3C-audit failure points. Every time a page's JS is
+rewritten, re-verify these six paths against the ACTUAL JSON output.
+
+## Canonical Mapping
+
+| JS reads | Collector provides | Type |
+|----------|-------------------|------|
+| `data.cron_jobs` | `cron_jobs` | Array of {name, schedule, lastRun, status, script} |
+| `data.services` | `services` | Dict {service_name: status_string} |
+| `data.s3_backups` | `s3_backups` | Dict {bucket_name: {status, last_upload, age_hours}} |
+| `data.api_checks` | `api_checks` | Dict {check_name: status_string_or_emoji} |
+| `data.overall.disk_used_pct` | `overall.disk_used_pct` | Integer (0-100) |
+| `data.overall.memory_used_pct` | `overall.memory_used_pct` | Integer (0-100) |
+| `data.cloudflare_zones` | `cloudflare_zones` | Array of {name, status} |
+| `data.netcup_server` | `netcup_server` | Dict {id, template, ip} |
+| `data.hetzner_servers` | `hetzner_servers` | Array of {name, type, status, ip, id} |
+| `data.syncromsp` | `syncromsp` | Dict {status, customers, tickets, customer_list} |
+| `job.lastRun` | `lastRun` | String (ISO 8601 with TZ offset) — NOT `last_run` |
+
+## Common wrong expectations (DO NOT use)
+- `scheduled_jobs` — use `cron_jobs`
+- `api_health` — use `api_checks` (dict, not array)
+- `disk_used_pct` at top level — it's inside `overall`
+- `memory_used_pct` at top level — it's inside `overall`
+- `job.last_run` (snake_case) — use `job.lastRun` (camelCase)
+- `services` as flat array — it's a dict, iterate `Object.keys(services)`
+- `s3_backups` as array — it's a dict {bucket_name: content}
+- `api_checks` as array — it's a dict {check_name: status_string}
+
+## Root cause
+The W3C subagent writes new markup and JS from the task brief's schema description,
+not from the actual JSON file on disk. Always read the live file before writing JS.
+
+## Verification command
+```bash
+python3 -c "import json; d=json.load(open('/var/www/ops/data/ops-status.json')); cj=d['cron_jobs'][0]; print('lastRun:', cj.get('lastRun','MISSING')); print('last_run:', cj.get('last_run','check')); print('services type:', type(d['services']).__name__)"
+```
+
+## Post-audit checklist
+After ANY page markup change, before declaring complete:
+1. Run the verification command above
+2. Check all JS field references match the actual JSON output
+3. curl the page for HTTP 200
+4. curl the data endpoint for valid JSON
+5. Verify nav injection works (check for initNav in both the page and utils.js)
+6. Verify files have 644 perms (Caddy returns 403 on 600 files)
@@ -0,0 +1,222 @@
+# FastAPI Ops Portal Backend Pattern
+
+Reference architecture for the FastAPI backend built Jul 9, 2026 at `/opt/ops-portal/server.py`.
+
+## Route ordering (CRITICAL)
+
+FastAPI processes routes in definition order. The route list must be:
+
+1. **API routes first** (`/api/auth/login`, `/api/health`, `/api/status`, ...)
+2. **Metrics** (`/metrics`)
+3. **Static assets** (`/css/{path}`, `/js/{path}`)
+4. **Page routes** (`/`, `/{page}.html`)
+
+If a catch-all `/{page}.html` comes before `/api/*`, ALL API calls return "Page not found".
+
+## Static file serving pattern
+
+```python
+# Do NOT use app.mount("/", StaticFiles(...)) — this swallows API routes.
+# Instead, serve explicitly:
+
+STATIC_DIR = Path(__file__).parent / "static"
+
+@app.get("/css/{file_path:path}")
+async def serve_css(file_path: str):
+    fpath = STATIC_DIR / "css" / file_path
+    if fpath.exists() and fpath.is_file():
+        return FileResponse(str(fpath))
+    raise HTTPException(status_code=404, detail="Not found")
+
+@app.get("/js/{file_path:path}")
+async def serve_js(file_path: str):
+    fpath = STATIC_DIR / "js" / file_path
+    if fpath.exists() and fpath.is_file():
+        return FileResponse(str(fpath))
+    raise HTTPException(status_code=404, detail="Not found")
+
+@app.get("/")
+async def serve_index():
+    index_path = STATIC_DIR / "index.html"
+    if index_path.exists():
+        return FileResponse(str(index_path))
+    return JSONResponse({"status": "ok", "message": "Ops Portal API ready"})
+
+@app.get("/{page}.html")
+async def serve_html_page(page: str):
+    fpath = STATIC_DIR / f"{page}.html"
+    if fpath.exists():
+        return FileResponse(str(fpath))
+    raise HTTPException(status_code=404, detail="Page not found")
+```
+
+## JWT auth pattern
+
+```python
+from jose import jwt, JWTError
+from fastapi import Depends, HTTPException, Request as FastAPIRequest
+from datetime import datetime, timezone, timedelta
+
+ADMIN_USERNAME = "germaine"
+ADMIN_PASSWORD = "..."  # from .env
+JWT_SECRET = "..."      # from .env
+JWT_ALGORITHM = "HS256"
+JWT_EXPIRE_HOURS = 24
+
+def create_jwt(username: str) -> str:
+    payload = {
+        "sub": username,
+        "iat": datetime.now(timezone.utc),
+        "exp": datetime.now(timezone.utc) + timedelta(hours=JWT_EXPIRE_HOURS),
+        "jti": secrets.token_hex(16),
+    }
+    return jwt.encode(payload, JWT_SECRET, algorithm=JWT_ALGORITHM)
+
+async def get_current_user(request: FastAPIRequest) -> str:
+    auth = request.headers.get("Authorization", "")
+    if not auth.startswith("Bearer "):
+        raise HTTPException(status_code=401, detail="Missing or invalid Authorization header")
+    token = auth[7:]
+    payload = jwt.decode(token, JWT_SECRET, algorithms=[JWT_ALGORITHM])
+    return payload["sub"]
+
+# Usage:
+@app.get("/api/status")
+async def get_status(user: str = Depends(get_current_user)):
+    ...
+```
+
+## Audit log pattern (SQLite)
+
+```python
+import sqlite3
+
+DB_PATH = "/opt/ops-portal/ops.db"
+
+def get_db():
+    conn = sqlite3.connect(DB_PATH)
+    conn.row_factory = sqlite3.Row
+    conn.execute("PRAGMA journal_mode=WAL")
+    return conn
+
+def init_db():
+    conn = get_db()
+    conn.executescript("""
+        CREATE TABLE IF NOT EXISTS audit_log (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            timestamp TEXT NOT NULL DEFAULT (datetime('now')),
+            user TEXT NOT NULL,
+            action TEXT NOT NULL,
+            target TEXT,
+            result TEXT NOT NULL,
+            detail TEXT
+        );
+        CREATE TABLE IF NOT EXISTS api_tokens (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            user TEXT NOT NULL,
+            token_hash TEXT NOT NULL UNIQUE,
+            created TEXT NOT NULL DEFAULT (datetime('now')),
+            last_used TEXT
+        );
+    """)
+    conn.commit()
+    conn.close()
+
+def log_audit(user: str, action: str, target, result: str, detail=None):
+    conn = get_db()
+    conn.execute(
+        "INSERT INTO audit_log (user, action, target, result, detail) VALUES (?, ?, ?, ?, ?)",
+        (user, action, target, result, detail)
+    )
+    conn.commit()
+    conn.close()
+```
+
+## Service control pattern (systemd)
+
+```python
+import subprocess
+from fastapi import HTTPException
+
+@app.post("/api/services/{name}/restart")
+async def restart_service(name: str, user: str = Depends(get_current_user)):
+    svc = name if name.endswith(".service") else f"{name}.service"
+    try:
+        result = subprocess.run(
+            ["systemctl", "restart", svc],
+            capture_output=True, text=True, timeout=30
+        )
+        success = result.returncode == 0
+        log_audit(user, "restart_service", name, "ok" if success else "error", result.stderr[:500])
+        if success:
+            return {"status": "ok", "message": f"Service {name} restarted"}
+        return JSONResponse(status_code=500, content={"status": "error", "message": result.stderr[:500]})
+    except subprocess.TimeoutExpired:
+        raise HTTPException(status_code=504, detail="Service restart timed out")
+```
+
+## Prometheus counters
+
+```python
+from prometheus_client import Counter, generate_latest, CONTENT_TYPE_LATEST
+from fastapi.responses import JSONResponse
+
+HTTP_REQUESTS = Counter("ops_portal_http_requests_total", "Total HTTP requests", ["method", "endpoint"])
+SERVICE_ACTIONS = Counter("ops_portal_service_actions_total", "Service actions", ["action", "service"])
+BACKUP_TRIGGERS = Counter("ops_portal_backup_triggers_total", "Backup triggers", ["bucket"])
+
+@app.get("/metrics")
+async def metrics():
+    return JSONResponse(
+        content=generate_latest().decode(),
+        media_type=CONTENT_TYPE_LATEST
+    )
+```
+
+## .env file layout
+
+```
+# /opt/ops-portal/.env — chmod 600
+ADMIN_USERNAME=germaine
+ADMIN_PASSWORD=...
+JWT_SECRET=...
+CLOUDFLARE_API_TOKEN=...
+CLOUDFLARE_EMAIL=...
+HETZNER_API_TOKEN=...
+```
+
+## Systemd service
+
+```ini
+[Unit]
+Description=ITPP Ops Portal Backend
+After=network.target
+
+[Service]
+Type=simple
+User=root
+WorkingDirectory=/opt/ops-portal
+ExecStart=/opt/awscli-venv/bin/uvicorn server:app --host 127.0.0.1 --port 8090
+Restart=always
+RestartSec=5
+StandardOutput=journal
+StandardError=journal
+EnvironmentFile=/opt/ops-portal/.env
+
+[Install]
+WantedBy=multi-user.target
+```
+
+## Caddy routing
+
+```caddy
+ops.itpropartner.com {
+    reverse_proxy 127.0.0.1:8090
+    encode gzip
+    log {
+        output file /var/log/caddy/ops.log
+    }
+}
+```
+
+The entire Caddy block is just a reverse proxy — no static file serving, no API matchers. The FastAPI backend handles everything.
@@ -0,0 +1,182 @@
+# Frontend Auth & Action Pattern — Ops Portal
+
+Created: 2026-07-09
+Applies to: `/opt/ops-portal/static/` pages (index.html, servers.html, backups.html, services.html)
+
+## Architecture
+
+Each page is a self-contained HTML file served by FastAPI with:
+1. Inline login overlay (no separate login page)
+2. Inline nav (not fetched via nav.html partial — auth-aware logout button is tightly coupled)
+3. Single JS module: `/js/app.js` (exposed as `window.Ops`)
+
+## JS Module (`/js/app.js`)
+
+### Auth
+
+```javascript
+// Login: stores JWT in localStorage['ops_token']
+await Ops.login('username', 'password');
+
+// Check auth state
+if (Ops.isAuthenticated()) { ... }
+
+// Logout — clears storage + page reload
+Ops.logout();
+```
+
+### API Client
+
+```javascript
+// All calls auto-attach Authorization header from stored token
+var data = await Ops.apiFetch('/api/status');
+var servers = await Ops.getServers();
+var audit = await Ops.getAuditLog(10);
+
+// Service actions (all POST with JWT)
+await Ops.restartService('caddy');
+await Ops.stopService('hermes');
+await Ops.startService('ops-portal');
+
+// Backup trigger
+await Ops.triggerBackup('hermes-vps-backups');
+
+// Server reboot (by Hetzner server ID)
+await Ops.rebootServer(51140464);
+```
+
+**401 handling:** `apiFetch()` automatically clears localStorage and calls `window.location.reload()` on 401 responses, showing the login overlay again.
+
+### UI Components
+
+```javascript
+// Toast notification (auto-dismiss 5s)
+Ops.toast('Server reboot initiated', 'success');  // 'success' | 'error' | 'info'
+
+// Confirmation dialog (returns Promise<boolean>)
+var ok = await Ops.confirmDialog(
+  'Reboot Server',
+  'Are you sure you want to reboot wphost02?',
+  'Reboot',
+  'Cancel'
+);
+
+// Button loading state
+Ops.spinning(document.getElementById('refreshBtn'), true);
+Ops.spinning(document.getElementById('refreshBtn'), false);
+
+// Nav initialization (hamburger toggle + active link + logout)
+Ops.initNav();
+
+// Formatting utilities
+Ops.fmtTime('2026-07-09T12:22:28-04:00');  // "Jul 9, 12:22 PM"
+Ops.fmtAge('...');                          // "3h 12m ago"
+Ops.badgeHtml('ok');                        // '<span class="badge badge-ok">✓ ok</span>'
+Ops.dotColor('running');                    // 'dot-green'
+Ops.escapeHtml('<script>');                 // '&lt;script&gt;'
+Ops.valOrDash(null);                        // '—'
+Ops.updateLastUpdated();                    // Sets #lastUpdated textContent
+```
+
+### Toast container (lazy-created)
+
+If `#toastContainer` doesn't exist, `Ops.toast()` creates it as a `<div>` appended to `document.body`. Position: `fixed; top: 20px; right: 20px; z-index: 10001`. Styled via `ops.css` `.toast`, `.toast-success`, `.toast-error`, `.toast-info`.
+
+## Page Template Pattern
+
+```html
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
+  <meta name="theme-color" content="#0f1628">
+  <title>Page — Operations Dashboard</title>
+  <link rel="stylesheet" href="/css/ops.css">
+</head>
+<body>
+
+<!-- Login Overlay -->
+<div class="login-overlay show" id="loginOverlay">
+  <div class="login-card">
+    <!-- logo, title, error div, form with #loginUser + #loginPass -->
+  </div>
+</div>
+
+<!-- Navigation (inline, not fetched) -->
+<nav class="ops-nav" id="opsNav">
+  <!-- brand, nav-links with SVG icons, logout button -->
+</nav>
+
+<div class="page-wrapper">
+  <header class="page-header">
+    <h1>Page Title</h1>
+    <div class="header-actions">
+      <span class="last-updated" id="lastUpdated">—</span>
+      <button class="refresh-btn" onclick="loadPageData()">Refresh</button>
+    </div>
+  </header>
+  <!-- card sections with data -->
+</div>
+
+<script src="/js/app.js"></script>
+<script>
+(function() {
+  'use strict';
+
+  // Auth guard
+  if (!Ops.isAuthenticated()) {
+    document.getElementById('loginOverlay').classList.add('show');
+  } else {
+    document.getElementById('logoutBtn').classList.add('show');
+    loadPageData();
+  }
+
+  // Login handler
+  window.handleLogin = async function(e) { ... await Ops.login(...) ... };
+
+  // Data loading
+  window.loadPageData = async function() { ... };
+
+  // Action handlers with confirmation
+  window.handleAction = async function(...) {
+    if (await Ops.confirmDialog(...)) { ... await Ops.apiCall(...) ... Ops.toast(...) }
+  };
+
+  Ops.initNav();
+  // Auto-refresh
+  setInterval(function() {
+    if (Ops.isAuthenticated()) loadPageData();
+  }, 60000);
+})();
+</script>
+</body>
+</html>
+```
+
+## Files created Jul 9, 2026
+
+| File | Size | Sections |
+|------|------|----------|
+| `/opt/ops-portal/static/index.html` | 19.3 KB | Health grid, Quick actions (restart/trigger), Servers summary, Cron jobs, Audit log |
+| `/opt/ops-portal/static/servers.html` | 15.0 KB | Quick specs (vCPUs/RAM), Sortable server table with click-to-expand details, Reboot button per server |
+| `/opt/ops-portal/static/backups.html` | 15.1 KB | Summary bar, Bucket grid with Trigger button, Schedule table |
+| `/opt/ops-portal/static/services.html` | 14.7 KB | Service control grid (Start/Stop/Restart per service), API health grid, Service audit log |
+| `/opt/ops-portal/static/css/ops.css` | 27.0 KB | Full theme with auth-overlay, confirm-dialog, toast, action-card, svc-card styles |
+| `/opt/ops-portal/static/js/app.js` | 11.6 KB | Auth module, API client, confirmation dialog, toast, utilities |
+
+## CSS classes for auth UI (in ops.css)
+
+- `.login-overlay` — full-screen fixed overlay, `display: none` by default, `.show` sets `display: flex`
+- `.login-card` — centered dialog, max-width 400px, dark card with logo + form
+- `.login-error` — red alert box, hidden by default, `.show` reveals it
+- `.confirm-overlay` — modal backdrop with blur, `.show` = `display: flex`
+- `.confirm-dialog` — centered card with title + message + action buttons
+- `.toast-container` — fixed top-right notification area
+- `.toast`, `.toast-success`, `.toast-error`, `.toast-info` — slide-in animations
+
+## Key quirks
+
+- **Login overlay is always in the HTML** — it's hidden by CSS `display: none`. The JS just toggles `.show` class. No page redirect on login; the overlay hides and the page stays.
+- **Auto-refresh stops when not authenticated** — every `setInterval` callback checks `Ops.isAuthenticated()` before fetching. Stale data stays visible until login.
+- **Caddy reload** after config change: `caddy reload --config /etc/caddy/Caddyfile` or `systemctl reload caddy`. Full restart (`systemctl stop/start`) only needed when adding new Caddy hosts, not modifying existing ones.
@@ -0,0 +1,71 @@
+# Grafana Dashboard API Patterns
+
+## Creating a dashboard via REST API
+
+After the Prometheus data source is created, provision dashboards via POST to `/api/dashboards/db`:
+
+```bash
+curl -s -X POST http://127.0.0.1:3002/api/dashboards/db \
+  -H "Content-Type: application/json" \
+  -u admin:admin \
+  -d @dashboard.json
+```
+
+Response: `{"id":..., "uid":"...", "slug":"...", "status":"success", "url":"/d/uid/title"}`
+
+## Common node_exporter panel queries
+
+### CPU Usage (stat or timeseries)
+```
+avg by(instance) (1 - rate(node_cpu_seconds_total{mode='idle'}[5m]))
+```
+
+### RAM Usage (gauge or timeseries)
+```
+1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
+```
+Bytes variant: `node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes`
+
+### Disk Usage (gauge)
+```
+1 - (node_filesystem_avail_bytes{mountpoint='/'} / node_filesystem_size_bytes{mountpoint='/'})
+```
+
+### Uptime (stat)
+```
+time() - node_boot_time_seconds
+```
+
+## Panel layout reference (gridPos)
+
+The dashboard grid is 24 columns wide. Common layouts:
+
+| Panel | gridPos | Width |
+|-------|---------|-------|
+| Single stat | `{h:8, w:6, x:0, y:0}` | 1/4 width |
+| Timeseries full | `{h:10, w:12, x:0, y:8}` | 1/2 width |
+| Two stat row | `{h:8, w:12, x:0, y:0}` + `{h:8, w:12, x:12, y:0}` | 1/2 each |
+
+## Adding a Prometheus data source
+
+```bash
+curl -s -X POST http://127.0.0.1:3002/api/datasources \
+  -H "Content-Type: application/json" \
+  -u admin:admin \
+  -d '{
+    "name":"Prometheus",
+    "type":"prometheus",
+    "url":"http://127.0.0.1:9090",
+    "access":"proxy",
+    "isDefault":true
+  }'
+```
+
+Response: `{"datasource":{"id":1,"uid":"...","name":"Prometheus",...},"message":"Datasource added"}`
+
+## Pitfalls
+
+- **Auth**: Use `-u admin:admin` (or whatever password was set with `GF_SECURITY_ADMIN_PASSWORD`). Default is `admin/admin`.
+- **Default data source**: Set `"isDefault":true` so panels don't need an explicit data source reference.
+- **Overwriting**: When re-importing, send `"overwrite": true` in the payload body (at the same level as `"dashboard"`).
+- **Time range**: Set `"time": {"from": "now-6h", "to": "now"}` and `"refresh": "30s"` at the `dashboard` level.
@@ -0,0 +1,72 @@
+# iOS PWA Push Notification Testing Guide
+
+Debugging web push notifications for the shark game on iOS.
+
+## Setup
+
+1. Load the test page at https://shark.iamgmb.com/push-test.html
+2. Share -> "Add to Home Screen"
+3. Open from home screen (not from Safari browser tab)
+
+## Expected subscription flow
+
+```
+Browser supports push -> SW registered -> Permission granted -> Push subscribed -> Subscription sent to server
+```
+
+## If subscribe succeeds but banner is invisible
+
+The push opt-in `<div id="pushBanner" style="display:none">` is hardcoded hidden in HTML. The `showNotificationBanner()` function must call `banner.style.display = 'block'` — without this explicit line, the banner state variable changes but the UI never reveals the prompt.
+
+**Check:** Does `showNotificationBanner()` have `banner.style.display = 'block';` at the top of the function body?
+
+## iOS PWA push limitation
+
+PushManager is NOT available in Safari's browser tab on iOS. The `FAIL: No PushManager` error is expected behavior on iPhone Safari. The user MUST:**
+
+1. Tap Share -> "Add to Home Screen"
+2. Open from home screen (PWA mode, fullscreen, no URL bar)
+3. Then push subscription works via Apple Push Service (APNs)
+
+## Common subscription failures
+The `user_id` column has a FK constraint to `users.id`. Test subscriptions use `user_id: NULL` — ensure the column allows NULL and the subscribe-noauth endpoint passes NULL.
+
+Fix:
+```sql
+-- Rebuild table with nullable user_id
+CREATE TABLE push_subscriptions_new (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    user_id INTEGER,
+    endpoint TEXT NOT NULL UNIQUE,
+    p256dh TEXT NOT NULL,
+    auth TEXT NOT NULL,
+    created_at TEXT DEFAULT (datetime('now'))
+);
+INSERT INTO push_subscriptions_new SELECT * FROM push_subscriptions;
+DROP TABLE push_subscriptions;
+ALTER TABLE push_subscriptions_new RENAME TO push_subscriptions;
+```
+
+### Server rejected subscription
+Check that the subscribe-noauth endpoint exists and accepts the POST body:
+```json
+{"endpoint": "https://...", "p256dh": "...", "auth": "..."}
+```
+
+## Test send after subcription
+
+```bash
+curl -s -X POST https://shark.iamgmb.com/api/push/test \
+  -H "Content-Type: application/json" \
+  -d '{}'
+```
+
+Expected: `{"sent": 1}`
+Failure: `{"sent": 0}` — check journald for `Push test failed: ...`
+
+## Debugging database
+
+```bash
+sqlite3 /root/shark-game/backend/game.db "SELECT count(*) FROM push_subscriptions;"
+sqlite3 /root/shark-game/backend/game.db "SELECT id, endpoint FROM push_subscriptions;"
+```
@@ -0,0 +1,57 @@
+# Ops Portal Jul 12 Cleanup Audit
+
+42 bugs found across 11 pages by Claude Sonnet 4-6 subagent.
+
+## Root Causes (5 issues causing 80% of bugs)
+
+### 1. `/js/utils.js` 404
+`utils.js` existed at `/var/www/ops/js/utils.js` but NOT at `/opt/ops-portal/static/js/` (the backend's JS directory). Every page referenced `/js/utils.js` which returned 404 via Caddy's `/js/*` file_server handler. Fix: `cp /var/www/ops/js/utils.js /opt/ops-portal/static/js/utils.js`
+
+### 2. `window.initNav` undefined
+`initNav()` is exported as `window.Ops.initNav` (via `app.js`), not `window.initNav`. Every page calling `window.initNav()` calls undefined. Fix: Pages must call `Ops.initNav()`, or export `window.initNav = initNav` in app.js.
+
+### 3. `logoutBtn` DOM lookup before nav fetch
+Pages services.html, servers.html, backups.html call `document.getElementById('logoutBtn')` synchronously before the `fetch('/nav.html')` callback completes. The nav (and `logoutBtn`) doesn't exist yet → TypeError. Fix: Move all `getElementById('logoutBtn')` calls inside the fetch callback.
+
+### 4. Pages missing required script files
+Some pages loaded only `app.js` (missing `utils.js`), others loaded only `utils.js` (missing `app.js`). Fix: EVERY page must load BOTH:
+```html
+<script src="/js/utils.js?v=2"></script>
+<script src="/js/app.js?v=2"></script>
+```
+
+### 5. No auth guard on 6 pages
+network.html, cron.html, config.html, logs.html, cost.html, fleettracker360.html have no login overlay or auth check. Fix: Add login overlay + `Ops.isAuthenticated()` guard.
+
+## Critical Bugs by Page
+
+| Page | Critical Bugs | File Path |
+|------|------|-----------|
+| index.html | 1 | /opt/ops-portal/static/index.html |
+| services.html | 3 | /opt/ops-portal/static/services.html |
+| servers.html | 2 | /opt/ops-portal/static/servers.html |
+| backups.html | 2 | /opt/ops-portal/static/backups.html |
+| network.html | 2 | /opt/ops-portal/static/network.html |
+| cron.html | 2 | /opt/ops-portal/static/cron.html |
+| config.html | 4 | /opt/ops-portal/static/config.html |
+| logs.html | 2 | /opt/ops-portal/static/logs.html |
+| audit.html | 2 | /opt/ops-portal/static/audit.html |
+| cost.html | 3 | /opt/ops-portal/static/cost.html |
+| fleettracker360.html | 0 | /opt/ops-portal/static/fleettracker360.html |
+
+## Quick Fixes Applied Jul 12
+- Copied utils.js to backend static dir → 404 fixed
+- Added cache busting `?v=2` to all JS/CSS
+- Fixed Grafana link from 127.0.0.1 to Tailscale IP
+- Added z-index: 1000 to nav-links
+- Added inline onclick fallback to nav-toggle
+- Hid "Invalid credentials" error text by default
+- Both JS files added to all 10 pages
+
+## Remaining Work
+- Export `window.initNav` properly
+- Fix `logoutBtn` timing on services/servers/backups
+- Add auth guards to 6 unprotected pages
+- Fix `window.fetchStatus` on cost.html
+- Fix `window.escapeHtml` on config.html
+- Fix audit.html `sessionStorage` → `localStorage`
@@ -0,0 +1,86 @@
+# Komodo Deployment (July 2026)
+
+Komodo Core v2.2.0 deployed on Core (netcup RS 2000) for multi-host Docker management.
+
+## Architecture
+
+- **Core** — Management server, installed as `/root/docker/komodo/` with docker-compose (MongoDB + Komodo Core)
+- **Periphery** — Agent installed on each managed host via `setup-periphery.py` script
+- **Port:** 9120
+- **Domain:** https://komodo.itpropartner.com
+- **Default login:** admin / changeme
+
+## Deployment Details
+
+### docker-compose.yml (v2.2.0 env vars)
+```yaml
+services:
+  mongo:
+    image: mongo
+    restart: unless-stopped
+    volumes:
+      - ./data/mongo-data:/data/db
+    networks:
+      - komodo-net
+    deploy:
+      resources:
+        limits:
+          memory: 1G
+
+  core:
+    image: ghcr.io/moghtech/komodo-core:2
+    restart: unless-stopped
+    ports:
+      - 9120:9120
+    depends_on:
+      - mongo
+    volumes:
+      - ./backups:/backups
+      - ./keys:/config/keys
+    environment:
+      - KOMODO_DATABASE_ADDRESS=mongo:27017
+      - KOMODO_LOCAL_AUTH=true
+      - KOMODO_INIT_ADMIN_USERNAME=admin
+      - KOMODO_INIT_ADMIN_PASSWORD=changeme
+      - KOMODO_FIRST_SERVER_NAME=core
+      - KOMODO_PERIPHERY_PUBLIC_KEY=file:/config/keys/periphery.pub
+      - KOMODO_HOST=https://komodo.itpropartner.com
+      - KOMODO_JWT_SECRET=<generated>
+      - KOMODO_WEBHOOK_SECRET=<generated>
+      - PUID=0
+      - PGID=0
+    networks:
+      - komodo-net
+    deploy:
+      resources:
+        limits:
+          memory: 512M
+
+networks:
+  komodo-net:
+    driver: bridge
+```
+
+### Env Var Changes from v1.x to v2.x
+- `MONGO_CONNECTION_STRING` → `KOMODO_DATABASE_ADDRESS`
+- `KOMODO_SERVER_PORT` → removed (defaults to 9120)
+- `KOMODO_SERVER_HOST` → removed
+- Must add `KOMODO_LOCAL_AUTH=true` for built-in auth
+- Must add `KOMODO_INIT_ADMIN_USERNAME` and `KOMODO_INIT_ADMIN_PASSWORD` for first-run admin user
+- Must add `KOMODO_FIRST_SERVER_NAME=core` to auto-name the management server
+
+### Caddy Config
+```
+komodo.itpropartner.com {
+    reverse_proxy 127.0.0.1:9120
+}
+```
+
+### Adding a Periphery Agent
+```bash
+curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py \
+  | python3 - \
+  --core-address="http://127.0.0.1:9120" \
+  --connect-as="<hostname>" \
+  --onboarding-key="<key-from-core-ui>"
+```
@@ -0,0 +1,53 @@
+# Ops Portal MIME Type Fix (Jul 12, 2026)
+
+## Problem
+All ops portal pages broke on mobile -- nav didn't initialize, login overlay stuck, service buttons unresponsive. Desktop with cached assets worked fine.
+
+## Root Cause
+FastAPI's `FileResponse()` returned Content-Type `application/json` for `.js` and `.css` files instead of `text/javascript` and `text/css`. Browsers strictly enforce MIME types for scripts/stylesheets -- the wrong type causes silent refusal to execute/apply.
+
+Additionally, JS/CSS files were created with `chmod 600` (root-only), so Caddy's static file handler returned 403 Forbidden, falling back to the backend which returned the wrong MIME type.
+
+## Fix -- Two Layers
+
+### Layer 1: Caddy static file handler (primary)
+Add `handle_path` blocks BEFORE `reverse_proxy`:
+```caddy
+ops.itpropartner.com {
+    handle_path /js/* {
+        root * /opt/ops-portal/static/js
+        file_server
+    }
+    handle_path /css/* {
+        root * /opt/ops-portal/static/css
+        file_server
+    }
+    reverse_proxy 127.0.0.1:8090
+}
+```
+
+### Layer 2: FastAPI fallback
+```python
+return FileResponse(str(fpath), media_type="application/javascript",
+    headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
+return FileResponse(str(fpath), media_type="text/css",
+    headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
+```
+
+### Prerequisite: File permissions
+```bash
+chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css
+```
+
+## Symptoms
+- Browser console: "Refused to execute script because its MIME type ('application/json') is not executable"
+- Works on desktop with cached assets, fails on mobile/cold load
+- All pages load HTML but nothing interactive works
+
+## Verification
+```bash
+curl -sI https://ops.itpropartner.com/js/app.js | grep -i content-type
+# Expected: text/javascript; charset=utf-8
+curl -sI https://ops.itpropartner.com/css/ops.css | grep -i content-type  
+# Expected: text/css; charset=utf-8
+```
@@ -0,0 +1,47 @@
+# Multi-Server Health Check API Pattern
+
+Added Jul 10, 2026 to ops portal. Replaced the single-server scraped-API health with ping-based live health for all 7 ITPP servers.
+
+## Backend (FastAPI endpoint)
+
+```python
+SERVERS = [
+    {"name": "Core", "ip": "152.53.192.33", "role": "Hermes, Docker, OSINT, Caddy", "tier": "Primary"},
+    {"name": "app1", "ip": "152.53.36.131", "role": "AI Stack", "tier": "Standard"},
+    {"name": "app2", "ip": "152.53.39.202", "role": "UNMS, UniFi, Traccar", "tier": "Standard"},
+    {"name": "app3", "ip": "152.53.241.111", "role": "WordPress", "tier": "Standard"},
+    {"name": "app1-bu", "ip": "5.161.114.8", "role": "Warm Standby", "tier": "Standby"},
+    {"name": "AI Box (legacy)", "ip": "178.156.167.181", "role": "LiteLLM (migrating)", "tier": "Legacy"},
+    {"name": "Docker Box (legacy)", "ip": "178.156.168.35", "role": "Uptime Kuma", "tier": "Legacy"},
+]
+
+@app.get("/api/servers/health")
+async def servers_health(user: str = Depends(get_current_user)):
+    results = []
+    for s in SERVERS:
+        try:
+            start = time.time()
+            result = subprocess.run(["ping", "-c", "1", "-W", "2", s["ip"]],
+                                    capture_output=True, text=True, timeout=5)
+            latency = round((time.time() - start) * 1000)
+            alive = result.returncode == 0
+        except:
+            alive = False
+            latency = None
+        results.append({...})
+    return {"servers": results, "count": len(results), "live": sum(...)}
+```
+
+## Frontend (vanilla JS)
+
+Page: `/opt/ops-portal/static/servers.html`
+Uses `Ops.apiFetch('/api/servers/health')` to get all 7 servers.
+Renders as card grid with border-left color (green=LIVE, red=OFFLINE).
+Shows: server name, status badge, role, IP, latency, tier.
+
+## Pitfalls
+
+- Ping requires root or CAP_NET_RAW. The ops-portal service runs as root, so ping works.
+- Some servers block ICMP (firewalls). If a server shows OFFLINE but is actually up, check UFW rules.
+- Legacy servers in migration status should still be monitored — they're running production services.
+- The API endpoint requires JWT auth (like all /api/* routes).
@@ -0,0 +1,83 @@
+# Nav Consistency Fix — July 12, 2026
+
+## Problem
+Multiple ops portal pages had erratic behavior on mobile — hamburger menu didn't work, nav links were missing or stale, some pages showed no data.
+
+## Root Causes Found (3 issues)
+
+### 1. Mixed nav strategies — inline vs fetched
+Some pages had the nav hardcoded inline in the HTML, while others fetched `/nav.html` dynamically. The inline navs were stale — missing Audit, Costs, and Grafana links. The fetched navs were always up to date.
+
+**Fix:** Every page should use:
+```html
+<div id="nav"></div>
+```
+And fetch nav at the bottom of the page script:
+```javascript
+fetch('/nav.html').then(r => r.text()).then(html => {
+    document.getElementById('nav').innerHTML = html;
+    if (typeof window.initNav === 'function') window.initNav();
+});
+```
+
+Pages affected: backups.html, servers.html, services.html (all had inline navs that were replaced)
+
+### 2. `Ops.initNav()` vs `window.initNav()` — TypeError kills nav
+The `initNav` function is defined in utils.js as `window.initNav`, NOT as `Ops.initNav`. Pages calling `Ops.initNav()` throw a ReferenceError that kills ALL JavaScript execution on the page — including the data loading and rendering code.
+
+**Fix:** Replace all `Ops.initNav()` with `window.initNav()`:
+```javascript
+// BROKEN:
+Ops.initNav();
+// FIXED:
+if (typeof window.initNav === 'function') window.initNav();
+```
+
+Pages affected: index.html, backups.html, servers.html, services.html
+
+### 3. CSS/JS MIME types — wrong content-type kills scripts
+The backend (FastAPI FileResponse) and Caddy reverse proxy served JS and CSS files with `Content-Type: application/json`. Browsers refuse to execute scripts or apply stylesheets with the wrong MIME type.
+
+**Root causes:**
+- FastAPI's `FileResponse()` without explicit `media_type` parameter sometimes resolves `.js` files as `application/json`
+- Caddy `handle_path /js/*` block was nested inside another `handle_path` block by accident (sed nesting bug)
+- File permissions were `600` (root-only) — Caddy returned 403, browser silently failed
+
+**Fix — Caddy approach (most reliable):**
+```caddy
+ops.itpropartner.com {
+    handle_path /data/* {
+        root * /var/www/ops/data/
+        file_server
+    }
+    handle_path /js/* {           # ← TOP LEVEL
+        root * /opt/ops-portal/static/js
+        file_server
+    }
+    handle_path /css/* {          # ← TOP LEVEL
+        root * /opt/ops-portal/static/css
+        file_server
+    }
+    reverse_proxy 127.0.0.1:8090
+}
+```
+
+**Plus fix file permissions:** `chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css`
+
+**Verification:**
+```bash
+curl -sI https://ops.itpropartner.com/js/app.js | grep -i content-type
+# Expected: text/javascript; charset=utf-8
+
+for page in / /services.html /servers.html /backups.html /audit.html /cost.html; do
+    echo "$page → $(curl -sS -o /dev/null -w '%{http_code}' https://ops.itpropartner.com$page)"
+done
+# All should return 200
+```
+
+## Prevention Checklist
+- Every new page: use `<div id="nav"></div>` + fetch pattern, never inline nav
+- Every new page: call `window.initNav()` (not `Ops.initNav()`) inside the fetch callback
+- After any Caddy config change: verify `handle_path` blocks are top-level, not nested
+- After adding any new static file: `chmod 644`
+- After any static file change: test with `curl -sI` to verify MIME type
@@ -0,0 +1,52 @@
+# Nav Consistency Pattern (Jul 12, 2026)
+
+## Principle
+
+All portal pages must load nav from `/nav.html` — the single source of truth for nav links, hamburger toggle, and nav CSS. The only exception is index.html (dashboard), which keeps an inline nav that matches nav.html link-for-link.
+
+## Current state (post-Jul 12 fix)
+
+| Page | Nav source | Status |
+|------|-----------|--------|
+| index.html | Inline (matches nav.html) | OK — all 11 links + hamburger |
+| services.html | Fetched | OK |
+| servers.html | Fetched | OK |
+| backups.html | Fetched | OK |
+| audit.html, cost.html, cron.html, network.html, config.html, logs.html | Fetched | OK |
+
+## Conversion procedure (inline → fetched)
+
+1. **Replace the `<nav class="ops-nav">...</nav>` block** with `<div id="nav"></div>`.
+
+2. **Move `initNav()` into the fetch callback.** The old pattern of calling `window.initNav()` before the nav HTML is injected is a race condition — it tries to wire up toggle listeners on elements that don't exist yet:
+   ```javascript
+   // WRONG (initNav runs before nav HTML exists):
+   window.initNav();
+   //  ... later ...
+   fetch('/nav.html')...
+
+   // RIGHT (initNav runs AFTER nav HTML is injected):
+   fetch('/nav.html').then(function(r) { return r.text(); }).then(function(h) {
+     document.getElementById('nav').innerHTML = h;
+     if (typeof window.initNav === 'function') window.initNav();
+   }).catch(function(e) { console.warn('Nav failed:', e); });
+   ```
+
+3. **Keep `window.initNav()`** — never use `Ops.initNav()` or bare `initNav()` inside a strict-IIFE. The nav init function is exported on `window` in `app.js`/`utils.js`.
+
+## Index.html: inline nav maintenance
+
+The dashboard keeps inline nav. When `/nav.html` gains or loses links, update index.html's inline nav to match. Key elements:
+- `class="nav-logo-text"` on the brand span
+- Hamburger toggle: `<button class="nav-toggle" id="navToggle" ...>`
+- All 11 links with `data-path` attributes
+- `class="nav-link"` and `target="_blank"` on Grafana
+
+## Verification checklist
+
+- [ ] `<div id="nav">` placeholder present (fetched pages only)
+- [ ] `initNav()` called AFTER `.innerHTML = h` in fetch callback
+- [ ] No orphan `window.initNav()` outside fetch block
+- [ ] index.html has all current nav.html links + hamburger toggle
+- [ ] HTTP 200 on all pages
+- [ ] `/nav.html` serves directly (not just through FastAPI proxy)
@@ -0,0 +1,50 @@
+# netcup SCP REST API Authentication
+
+The netcup Server Control Panel (SCP) REST API uses Keycloak (OIDC) password grant for authentication, NOT the CCP API key from Master Data.
+
+## Auth Flow
+
+```
+TOKEN=$(curl -s "https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token" \
+  -H "Content-Type: application/x-www-form-urlencoded" \
+  -d "grant_type=password&client_id=scp&username=${CUSTOMER_NUMBER}&password=${CCP_PASSWORD}&scope=openid" \
+  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
+```
+
+### Parameters
+
+| Field | Value | Source |
+|-------|-------|--------|
+| grant_type | password | OIDC resource owner password flow |
+| client_id | scp | Static SCP client ID |
+| username | Customer number | e.g. 389212 from CCP Master Data page |
+| password | CCP login password | Same as web login password |
+| scope | openid | Returns access_token + id_token + refresh_token |
+
+### Token properties
+
+- Access token expires in 300 seconds (5 min)
+- Refresh token expires in 1800 seconds (30 min)
+- Always obtain a fresh token each collection cycle
+
+## API Endpoints
+
+Base: https://www.servercontrolpanel.de/scp-core/api/v1/
+
+| Endpoint | Response |
+|----------|----------|
+| GET /servers | Array of servers with id, name, template, disabled, hostname |
+| GET /servers/{id} | Full detail: IPv4, IPv6, architecture, disks, RAM |
+
+## Known Working
+
+- RS 2000 G12 — ID 890903, 152.53.192.33
+- RS 4000 G12 — IDs 894147 (152.53.36.131), 894148 (152.53.39.202), 894149 (152.53.241.111)
+- Template name format: "RS {1000|2000|4000} G12"
+
+## Pitfalls
+
+- /scp-core/api/ NOT /scp-ui/api/ — /scp-ui/ returns HTML even with valid auth
+- www subdomain IS required — servercontrolpanel.de does NOT respond
+- CCP API Key is NOT for SCP — they use different auth systems
+- **Username is bare customer number, NOT `customer#<number>`.** Adding `customer#` prefix returns `invalid_grant` / `Invalid user credentials`. The Keycloak realm expects the bare numeric ID. This was discovered Jul 10, 2026 after multiple failed auth attempts.
@@ -0,0 +1,45 @@
+Generated Jul 9, 2026 by the Network Services Team. Full document at /root/.hermes/references/ops-portal-recommendations.md.
+
+## Architecture
+
+**Current:** Static HTML served by Caddy, reads JSON data file. Read-only.
+**Target:** FastAPI backend with auth, action endpoints, audit logging, RBAC.
+
+**Strategy:** Add action buttons to existing static pages immediately (via helper scripts), while planning the full rebuild.
+
+## Pages (13 total)
+
+8 existing + 5 new:
+- Dashboard, Servers, Backups, Network, Services, Cron Jobs, Logs, Config
+- **New:** Customer Management, Billing/Finance, Users/RBAC, Audit Log, Settings
+
+## Actions per page
+
+Each page should offer actions, not just views:
+- Restart a service
+- Trigger/restore backups
+- Add DNS records
+- Reboot routers
+- Create customers
+- Provision services
+- Create tickets
+
+## RBAC
+
+- Owner/Admin: everything
+- Tech/Support: tickets, server status, service restarts
+- Sales/Onboarding: customer activation, provisioning
+- Viewer: read-only
+
+## Integrations Roadmap
+
+- **Connected:** SyncroMSP, Cloudflare, Hetzner, netcup, Wasabi, admin-ai
+- **Pending:** RingLogix (phone), Stripe (billing), Xero (accounting), UniFi (API key saved), Hudu, RunCloud
+
+## Implementation Phases
+
+1. Action buttons on existing pages, VPN monitor widget
+2. Backend development (FastAPI + SQLite for audit)
+3. Customer management module
+4. RBAC + multi-user support
+5. Polish (Grafana dashboards, notifications, mobile)
@@ -0,0 +1,39 @@
+# Ops Portal Vision & Architecture
+
+Confirmed July 9, 2026. ops.itpropartner.com is where ITPP runs its business.
+
+## Core Vision
+
+"One page to do what I need to do" (Germaine). The ops portal is the single pane of glass for the entire IT Pro Partner business — monitoring, provisioning, customer lifecycle, billing, and infrastructure management.
+
+RBAC ensures different users see different things on the same platform:
+- **Admin (Germaine):** everything
+- **Tech/Support:** tickets, server status, network health — no billing, no provisioning
+- **Sales/Onboarding:** activation flow, order forms, provisioning status
+- **Viewer:** read-only, limited scope
+- **Tony:** DRE only (Debt Recovery Experts)
+
+## Architecture Evolution
+
+Phase 1 (current): FastAPI backend + static frontend at ops.itpropartner.com
+Phase 2 (next): Actionable pages with service restart, backup trigger, DNS management
+Phase 3 (future): Customer provisioning workflows, billing sync, automated activation
+
+## Confirmed Integrations
+
+Already connected: SyncroMSP, Cloudflare, Hetzner, netcup, Wasabi, admin-ai (LiteLLM)
+In discussion: RingLogix, Stripe, UniFi, Hudu
+Dropped: OpenRouter (already proxied through LiteLLM), OpenAI (LiteLLM already), Porkbun/Namecheap (Cloudflare sells domains at cost)
+
+## Monitoring Stack
+
+- Uptime-Kuma (existing on Hetzner Docker host) — embedded status page, has REST API
+- Prometheus (Core, deployed) — metrics collection at 127.0.0.1:9090
+- Grafana (Core, new) — dashboards at 127.0.0.1:3002 (port 3000 was taken by DocuSeal)
+- node_exporter on all servers — OS metrics (installed on Core so far)
+- VPS Threshold Alerts (deployed) — 80/90/95% RAM/disk/CPU alerts every 15 min via Telegram
+
+## Hardware Standard (Jul 2026)
+
+All new netcup servers: **RS 4000 G12** (12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe, ~$44/mo monthly billing in Manassas, VA).
+Exceptions: Core (existing RS 2000, light ops layer), app1-bu (Hetzner CPX11, warm standby).
@@ -0,0 +1,165 @@
+# Ops Portal Full Code Audit — July 12, 2026
+
+Auditor: Hermes | Pages: 11 | Total bugs: 42
+
+## The Four Root Causes
+
+Everything else flows from these four:
+
+### RC-1: `window.initNav` is undefined (affects ALL pages)
+`app.js` exports `initNav` only under `window.Ops.initNav`, not `window.initNav`.
+Every page's guard `typeof window.initNav === 'function'` evaluates FALSE.
+Nav hamburger, active-link, and logout wiring are dead on every page.
+
+**Fix (one line in app.js, inside IIFE before `window.Ops = {...}`):**
+```javascript
+window.initNav = initNav;
+```
+
+### RC-2: `utils.js` does not exist
+All pages reference `/js/utils.js?v=2` but the file is `/js/app.js`.
+The 404 is silent because `app.js` loads next and provides `window.Ops`.
+But if the server returns a hard error, all pages break.
+
+**Fix:** Rename `app.js` → `utils.js` OR update all `<script src="/js/utils.js">` to `app.js`.
+
+### RC-3: `logoutBtn` DOM lookup before nav fetch completes
+`services.html`, `servers.html`, `backups.html` all call
+`document.getElementById('logoutBtn').classList.add('show')` synchronously.
+`logoutBtn` lives in `nav.html` which is fetched asynchronously.
+Result: TypeError on every authenticated page load on those 3 pages.
+
+**Fix:** Move the `getElementById('logoutBtn')` call inside the fetch callback:
+```javascript
+fetch('/nav.html').then(r => r.text()).then(h => {
+  document.getElementById('nav').innerHTML = h;
+  document.getElementById('logoutBtn').classList.add('show'); // ← here
+  if (typeof window.initNav === 'function') window.initNav();
+});
+```
+
+### RC-4: 6 of 11 pages have NO auth guard
+network.html, cron.html, config.html, logs.html, cost.html, fleettracker360.html
+load and show data without any `Ops.isAuthenticated()` check.
+
+---
+
+## Full Bug Inventory by Page
+
+### index.html
+- B1 [HIGH] utils.js 404 (silent)
+- B2 [HIGH] window.initNav() call fails — window.initNav undefined
+- B3 [MED] navUser shown via inline style, inconsistent with .show pattern
+- B4 [MED] logoutBtn shown but never wired to logout (depends on initNav fix)
+
+### services.html
+- S1 [CRIT] utils.js 404
+- S2 [HIGH] login-overlay hardcoded class="show" — flash of login for auth'd users
+- S3 [HIGH] loginError "Invalid credentials" visible without display:none
+- S4 [CRIT] logoutBtn/navUser lookup before nav.html fetch completes → TypeError
+- S5 [MED]  window.initNav undefined (nav never initialized)
+- S6 [MED]  No guard for session expiry mid-session
+
+### servers.html
+- V1 [CRIT] utils.js 404
+- V2 [CRIT] logoutBtn.style.display = 'block' before nav loads → TypeError
+- V3 [HIGH] Login form inputs have no CSS classes, no form-group wrapper — unstyled
+- V4 [MED]  loginError always shows hardcoded "Invalid credentials" text
+- V5 [MED]  window.initNav undefined
+- V6 [LOW]  No <label> tags on login inputs (accessibility + autofill)
+- V7 [LOW]  login-logo uses emoji 🖥 instead of SVG (inconsistent)
+
+### backups.html
+- K1 [CRIT] utils.js 404
+- K2 [CRIT] logoutBtn.classList.add('show') before nav loads → TypeError
+- K3 [HIGH] login-overlay hardcoded class="show"
+- K4 [MED]  loginError no display:none
+- K5 [MED]  window.initNav undefined
+- K6 [HIGH] .summary-bar, .summary-item, .summary-label, .summary-val not in ops.css
+
+### network.html
+- N1 [CRIT] utils.js 404
+- N2 [HIGH] Scripts in <head> without defer (execute before DOM)
+- N3 [CRIT] NO auth guard — page shows all network data without login
+- N4 [MED]  window.initNav undefined
+- N5 [MED]  Grafana link uses Tailscale IP 100.71.155.7 — unreachable without Tailscale
+- N6 [LOW]  Inline <style> redefines .card, .badge, .status-dot (may conflict with ops.css)
+
+### cron.html
+- C1 [CRIT] utils.js 404
+- C2 [HIGH] Scripts in <head> without defer
+- C3 [CRIT] NO auth guard
+- C4 [HIGH] <main id="main-content"> placed AFTER the three <section> elements (inverted)
+- C5 [MED]  window.initNav undefined
+- C6 [LOW]  app.js loaded but Ops.* never used — all utilities re-implemented locally
+
+### config.html
+- G1 [CRIT] utils.js 404
+- G2 [CRIT] window.escapeHtml() called throughout — not exported by app.js → TypeError
+- G3 [HIGH] window.formatTime() used — not exported → falls back to Date.toLocaleString()
+- G4 [HIGH] fetchStatus() called — not defined anywhere → typeof guard prevents crash but live data never loads
+- G5 [CRIT] NO auth guard
+- G6 [MED]  window.initNav undefined
+- G7 [HIGH] Local initNav() defined inside IIFE shadows concept, calls window.initNav (self-referencing risk)
+- G8 [MED]  .content-area and .page-footer not in ops.css
+- G9 [MED]  Uses <header class="header"> but ops.css defines .page-header not .header
+
+### logs.html
+- L1 [CRIT] utils.js 404
+- L2 [HIGH] Scripts in <head> without defer
+- L3 [CRIT] NO auth guard
+- L4 [MED]  window.initNav undefined
+- L5 [MED]  <main> doesn't cleanly wrap content (navPlaceholder outside main, script inside </main>)
+
+### audit.html
+- A1 [CRIT] utils.js 404
+- A2 [CRIT] Uses sessionStorage for token — all other pages use localStorage — token not found on nav
+- A3 [MED]  app.js loaded but Ops.* never used — own escapeHtml/fmtTime re-implemented
+- A4 [MED]  logoutBtn in nav never wired (initNav not called)
+- A5 [MED]  window.initNav undefined
+- A6 [LOW]  Login form pre-filled with value="germaine" — leaks valid username
+
+### cost.html
+- T1 [CRIT] utils.js 404
+- T2 [CRIT] window.fetchStatus(...) called — not defined anywhere → TypeError on load → no data renders
+- T3 [CRIT] NO auth guard
+- T4 [MED]  app.js loaded but unused
+- T5 [MED]  window.initNav undefined
+- T6 [LOW]  Refresh button uses Unicode ↻ not SVG icon, .spinning animation doesn't apply
+
+### fleettracker360.html
+- F1 [HIGH] No app.js or utils.js loaded — no auth system
+- F2 [HIGH] NO auth guard
+- F3 [MED]  Missing viewport-fit=cover (all other pages have it)
+- F4 [LOW]  Hardcoded date "2026-07-12" in timestamp string
+- F5 [LOW]  COT hour calculation: `(utcHours + (-5) + 24).slice(-2)` fails at midnight (produces "24")
+- F6 [LOW]  Not linked from nav.html — orphaned page
+
+---
+
+## Cross-Cutting Issues
+
+| ID | Bug | Pages | Severity |
+|----|-----|-------|----------|
+| X1 | /js/utils.js 404 | All 10 auth pages | CRIT |
+| X2 | window.initNav undefined | All pages with fetch-nav | HIGH |
+| X3 | logoutBtn null before nav loads | services, servers, backups | CRIT |
+| X4 | No auth guard | network, cron, config, logs, cost, fleettracker | CRIT |
+| X5 | login-overlay hardcoded show class | services, servers, backups | HIGH |
+| X6 | Grafana link unreachable on mobile | nav.html + index.html | MED |
+| X7 | ops.css vs ops.css?v=2 version inconsistency | Mixed across pages | LOW |
+
+---
+
+## Priority Fix Order
+
+1. Add `window.initNav = initNav;` to app.js (fixes X2, B2, S5, V5, K5, N4, C5, G6, L4, A5, T5)
+2. Rename utils.js reference or file (fixes X1)
+3. Move logoutBtn lookups inside nav fetch callback (fixes X3)
+4. Replace `window.fetchStatus(...)` in cost.html with direct fetch (fixes T2)
+5. Add `window.escapeHtml = Ops.escapeHtml` to app.js or fix config.html (fixes G2)
+6. Change audit.html to use localStorage not sessionStorage (fixes A2)
+7. Add auth guard + login overlay to network, cron, config, logs, cost (fixes X4)
+8. Remove hardcoded `show` class from login-overlay in services/servers/backups HTML (fixes X5)
+9. Fix <main> placement in cron.html (fixes C4)
+10. Add .summary-bar CSS to ops.css (fixes K6)
@@ -0,0 +1,44 @@
+# Postfix SMTP Relay for RunCloud-Managed Servers
+
+When a RunCloud-managed server (like wphost02) has stuck email in Postfix queue because port 25 is blocked, relay through `mail.germainebrown.com:2525` with SASL auth and sender-domain rewriting.
+
+## Symptom
+`mailq` shows 60+ messages deferred with `(connect to ...:25: Connection timed out)`. The server tries to send directly to destination MX servers but port 25 is blocked.
+
+## Fix
+```bash
+# Configure relay
+postconf -e "relayhost = mail.germainebrown.com:2525"
+postconf -e "smtp_sasl_auth_enable = yes"
+postconf -e "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd"
+postconf -e "smtp_sasl_security_options = noanonymous"
+postconf -e "smtp_tls_security_level = encrypt"
+postconf -e "myorigin = /etc/mailname"
+
+# Auth credentials (use shonuff's SMTP account)
+echo "mail.germainebrown.com:2525 shonuff@germainebrown.com:Catches.bullets1985" > /etc/postfix/sasl_passwd
+chmod 600 /etc/postfix/sasl_passwd
+postmap /etc/postfix/sasl_passwd
+
+# Sender rewriting — MXroute requires real sender domain, not wphost02.local
+echo "apextrackexperience.com" > /etc/mailname
+echo "@wphost02 contact@apextrackexperience.com" > /etc/postfix/generic
+echo "root@wphost02 contact@apextrackexperience.com" >> /etc/postfix/generic
+postmap /etc/postfix/generic
+postconf -e "smtp_generic_maps = hash:/etc/postfix/generic"
+
+systemctl restart postfix
+postqueue -f   # flush the stuck queue
+```
+
+## Verification
+`mailq` says "Mail queue is empty". Logs show `relay=mail.germainebrown.com[65.109.144.50]:2525` with `status=sent`.
+
+## Note
+Old queued messages will get bounce notices (they were deferred too long). New emails flow through the relay cleanly. MXroute rejects `550 Envelope sender domain must be a fully qualified name` if the sender is `runcloud@wphost02` — the `smtp_generic_maps` rewrite fixes this.
+
+## Trigger Symptom Chain
+1. WP Mail SMTP on wphost02 configured for c1113726.sgvps.net:2525 (works for WP)
+2. PHP mail() falls back to Postfix, which does direct MX delivery on port 25
+3. Port 25 blocked → emails in deferred queue forever
+4. WPForms notifications and system emails pile up
@@ -0,0 +1,35 @@
+# Push Notification Testing — Shark Game
+
+## iOS Safari Limitation
+
+PushManager is NOT available in Safari on iPhone when browsing normally. It ONLY
+works when the page is launched from a Home Screen PWA icon (Add to Home Screen).
+
+Test flow for iOS:
+1. Visit shark.iamgmb.com/push-test.html
+2. Tap Share icon -> Add to Home Screen
+3. Open the test page from the Home Screen icon
+4. Tap Enable Push Notifications — all steps should pass
+
+## Standalone Test Page
+
+/push-test.html at /var/www/shark-game/push-test.html shows a full diagnostic
+log of each push setup step. No login required.
+
+## Subscription Endpoint
+
+POST /api/push/subscribe requires JWT auth.
+POST /api/push/subscribe-noauth stores under user_id=0 (for testing only).
+
+Both expect: {endpoint, p256dh, auth}
+
+## Banner Visibility Bug
+
+The pushBanner div starts with style="display:none". The JS function
+showNotificationBanner() must explicitly set banner.style.display = 'block'.
+Without this, the banner exists in the DOM but is never visible.
+
+## VAPID Keys
+
+Generated once, stored in /root/shark-game/backend/server.py as module-level
+variables. Public key exposed at GET /api/push/vapid-public-key.
@@ -0,0 +1,54 @@
+# Wasabi S3 Bucket Architecture
+
+Normalized Jul 8, 2026 from a single monolithic bucket into purpose-specific buckets.
+
+## Bucket Layout
+
+| Bucket | Purpose | Created | Initial versioning |
+|--------|---------|---------|-------------------|
+| `hermes-vps-backups` | Hermes live state, full backups, standby recovery | Pre-existing | ✅ Enabled Jul 8 |
+| `itpropartner-system-configs` | Caddyfile, systemd, SSH keys, scripts, .env files | **Manual (Wasabi Console)** | ✅ Enable after creation |
+| `itpropartner-docker-volumes` | Docuseal, Vaultwarden, Twenty, SearXNG, Ollama | **Manual (Wasabi Console)** | ✅ Enable after creation |
+| `mikrotik-ccr-backups` | Router configs, backup logs | Pre-existing | ✅ Enabled Jul 8 |
+| `itpropartner-backups` | Legacy orphaned data (no active writer) | Pre-existing | ✅ Enabled Jul 8 |
+
+**Note:** The Hermes-User IAM key lacks `s3:CreateBucket` permission. New buckets must be created in the Wasabi Console web UI. After creation, enable versioning via CLI:
+
+```bash
+source /opt/awscli-venv/bin/activate
+aws s3api put-bucket-versioning --bucket itpropartner-system-configs \
+  --versioning-configuration Status=Enabled \
+  --endpoint-url https://s3.us-east-1.wasabisys.com
+```
+
+## Data Formerly in `itpropartner-backups` (legacy, preserved)
+
+| Old path | Copied to |
+|----------|-----------|
+| `home-router/2026-07-07-config.rsc` | `mikrotik-ccr-backups/wisp-backups/configs/` |
+| `shonuff/*` (images) | `hermes-vps-backups/assets/` |
+
+No data deleted — old bucket preserved with versioning.
+
+## Sync Scripts
+
+| Script | Bucket | Schedule | What it syncs |
+|--------|--------|----------|--------------|
+| `hermes-live-sync.sh` | `hermes-vps-backups` | Every 15 min | Hermes config, skills, cron, sessions, profiles, .env, state.db |
+| `hermes-system-config-sync.sh` | `itpropartner-system-configs` | Every 15 min | Caddyfile, systemd services, ops scripts, SSH keys, .env files |
+| `hermes-docker-sync.sh` | `itpropartner-docker-volumes` | Every 15 min | Docuseal, Vaultwarden, Twenty, SearXNG, Ollama volume data |
+| `wisp-backup.py` | `mikrotik-ccr-backups` | Daily 6 AM | Router config exports |
+| `hermes-backup.sh` | `hermes-vps-backups/full/` | Daily 5 AM | Full tarball of everything |
+
+## Data collector tracking
+
+The ops-data-collector.py checks all 6 paths (5 buckets, with one split into sub-paths):
+
+```python
+check_s3_backup("s3://hermes-vps-backups/live/")        # → "hermes-vps-backups"
+check_s3_backup("s3://hermes-vps-backups/hermes-full-backup/")  # → "hermes-full-backups"
+check_s3_backup("s3://itpropartner-system-configs/")     # → "system-configs"
+check_s3_backup("s3://itpropartner-docker-volumes/")     # → "docker-volumes"
+check_s3_backup("s3://mikrotik-ccr-backups/")            # → "mikrotik-backups"
+check_s3_backup("s3://itpropartner-backups/")            # → "itpropartner-backups"
+```
@@ -0,0 +1,58 @@
+# Shark Game — Scoring Events & Push Notification Architecture
+
+## Point Values
+- Sighting: +2 pts
+- Bite: +5 pts
+- Fatality: +10 pts
+
+## Push Notification Lifecycle
+1. Scraper runs hourly (cron: `shark-scraper.sh`)
+2. Found events are scored in the backend
+3. `send_push_notification()` is called for each affected region
+4. Backend checks user's quiet hours and notification type preferences
+5. If notification is allowed, `webpush()` sends via VAPID keys through pywebpush
+6. Apple Push Service (APNs) delivers to iOS PWA, or browser push on desktop
+
+## Per-User Quiet Hours (default: 9 PM - 6 AM ET)
+```sql
+ALTER TABLE users ADD COLUMN quiet_hours_start TEXT DEFAULT '21:00';
+ALTER TABLE users ADD COLUMN quiet_hours_end TEXT DEFAULT '06:00';
+ALTER TABLE users ADD COLUMN notify_sightings INTEGER DEFAULT 1;
+ALTER TABLE users ADD COLUMN notify_bites INTEGER DEFAULT 1;
+ALTER TABLE users ADD COLUMN notify_fatalities INTEGER DEFAULT 1;
+ALTER TABLE users ADD COLUMN notify_draft INTEGER DEFAULT 1;
+ALTER TABLE users ADD COLUMN quiet_hours_enabled INTEGER DEFAULT 1;
+```
+
+## Draft Reminder Emails
+Cron job `shark-draft-reminder` runs every 15 minutes, checks leagues with upcoming drafts.
+
+Two reminders per player per league:
+- 24h before draft: "Your draft starts in 24 hours!"
+- 1h before draft: "Your draft starts in 1 hour!"
+
+**Tracking file:** `/root/shark-game/data/draft-reminders.json` — prevents duplicates.
+
+**Email:** Same pattern as boys-mail-monitor (shonuff@g, mail.germainebrown.com:2525, BCC g@germainebrown.com, random Sho'Nuff signature).
+
+## Desktop Layout
+The game is mobile-first but scales for desktop:
+- Max width: 1400px centered
+- Multi-column layout at 1024px+
+- Map + leaderboard side by side on widescreen
+- Mobile layout unchanged on small screens
+
+## Settings Page
+At `/settings.html` with:
+- Notification toggles (sightings, bites, fatalities, draft)
+- Quiet hours (start time, end time, enable toggle)
+- Profile info (display name, email)
+- League info
+
+## Help Page
+At `/help.html` with:
+- PWA setup guide (iOS steps, Android steps)
+- Push notification enablement guide
+- Quiet hours explanation
+- Scoring reference
+- Troubleshooting checklist
@@ -0,0 +1,27 @@
+# SiteGround SFTP Backup (July 2026)
+
+When SiteGround hosting is decommissioned (migrating to app3 netcup), all 21 sites were backed up to Wasabi S3 via SFTP.
+
+## Key Constraints
+
+- **No remote shell access.** SiteGround's SFTP on port 18765 blocks `exec_command()` — no remote `tar`, no `find | wc`, no `echo`. Only pure SFTP operations (listdir, read, write). This means per-file streaming over the SFTP channel, which is very slow for large WordPress sites with thousands of files.
+- **Directory structure:** Sites live at `/<domain>/<domain>/public_html` (nested directory with same name as site).
+- **Speed:** Each file requires a separate SFTP round-trip to open and read. The backup script uses `paramiko.SFTPClient` to recursively traverse directories and stream each file into a local `tarfile` in Python. For a 14K-file site, this takes 20-30 minutes.
+
+## SFTP Credentials
+
+Stored in the recovery manual and /root/.ssh/siteground.key (chmod 600). Key is DES-EDE3-CBC encrypted with passphrase `LoveMyBoys73!`.
+
+## Restore Process
+
+To restore a site from the S3 backup to a new host:
+1. Pull the backup: `aws s3 cp s3://hermes-vps-backups/siteground/<domain>/<domain>-<date>.tar.gz .`
+2. Extract: `tar xzf <domain>-<date>.tar.gz -C <target-dir>`
+3. Import MySQL database (needs a separate SQL dump — SFTP doesn't cover MySQL)
+4. Update wp-config.php with new DB credentials
+5. Point DNS to the new host IP
+6. Run `coolify` (or the site provisioning tool on app3) to wire up SSL
+
+## Note on MySQL
+
+Only site FILES were backed up via SFTP. The MySQL databases were NOT exported during the SFTP backup. To migrate a WordPress site fully, a MySQL dump is also needed from the SiteGround phpMyAdmin or via the existing RunCloud MySQL connection.
@@ -0,0 +1,26 @@
+# Static File Caching — Ops Portal
+
+**Pitfall (Jul 10, 2026):** The ops portal serves static HTML/JS/CSS through FastAPI's `FileResponse`. Without explicit `Cache-Control` headers, browsers aggressively cache these files. After editing `app.js`, `servers.html`, or `ops.css`, users see stale versions — even after hard refresh. The login page, server health, and API calls all appear broken because the browser is serving cached JavaScript.
+
+**Symptoms:**
+- User reports "pages won't refresh" despite confirmed deploys
+- `curl` returns HTTP 200 with correct content, but browser shows old version
+- JS changes (401 redirect fix, new endpoint calls) don't take effect
+- User has to manually Ctrl+Shift+R every page
+
+**Fix:** Add `Cache-Control: no-cache, no-store, must-revalidate` headers to all `FileResponse` calls in `server.py`:
+
+```python
+return FileResponse(str(fpath), headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
+```
+
+This applies to HTML pages, CSS, and JS. After adding headers, restart the service:
+```bash
+systemctl restart ops-portal
+```
+
+**Why this matters for ops:** The portal uses vanilla HTML/JS with no framework, no bundler, and no cache-busting hashes. Every file edit requires either a cache-control header or manual browser refresh. The header approach is reliable across all users.
+
+**Files affected:**
+- `/opt/ops-portal/server.py` — `serve_css()`, `serve_js()`, `serve_index()`, `serve_html_page()`
+- Any `@app.get("/{page}.html")` handler
@@ -0,0 +1,23 @@
+# SyncroMSP API Integration
+
+## Connection Details
+
+- **Base URL:** `https://itpropartner.syncromsp.com/api/v1/`
+- **Auth:** Bearer token `Ta9f9d1b462271a2f4-8d63a3f025eb89451edb16f2308c2e40`
+- **Admin user:** Germaine Brown (user_id 155629, email info@itpropartner.com)
+- **Admin:** Yes
+- **Subdomain:** itpropartner
+
+## Available Data
+
+| Endpoint | Response | Notes |
+|----------|----------|-------|
+| `/me` | user_token, user_email, user_name, user_id, admin | Auth verification |
+| `/customers` | `{customers: [...], total: N}` | 26 customers |
+| `/tickets` | `{total: N}` | Total across all statuses |
+
+## Data in ops portal
+
+- Added to `ops-data-collector.py` via `collect_syncromsp()` function
+- Rendered on `/services.html` — customer count, ticket count, status badge
+- Token stored in `/root/.hermes/.env` as SYNCROMSP_API_TOKEN
@@ -0,0 +1,35 @@
+# VPS Threshold Alert System — Jul 2026
+
+Automated monitoring system that checks RAM, disk, and CPU usage across all 10 servers every 15 minutes.
+
+## Script
+`/root/.hermes/scripts/vps-threshold-check.sh` (chmod 755)
+
+## Cron job
+`vps-threshold-check` — runs every 15 min, no_agent mode (stdout delivered as Telegram message)
+
+## Servers monitored (10 total)
+- 9 Hetzner servers (SSH via itpp-infra key, 5s timeout per host)
+- 1 local (core, netcup RS 2000)
+
+## Thresholds
+| Metric | Warning (80%) | Critical (90%) | Emergency (95%) |
+|--------|-------------|---------------|-----------------|
+| Disk | 80% | 90% | 95% |
+| RAM | 80% | 90% | 95% |
+| CPU load | 80% cores utilized | 90% | 95% |
+
+## Alert tracking
+`/root/.hermes/data/threshold-alerts.json` — keys format: `{hostname}_{metric}_{threshold}`
+- 24-hour cooldown per key
+- Auto-resets when metric drops below threshold
+- Re-alerts if metric crosses again after cooldown
+
+## Alert format (Telegram)
+```
+🚨 VPS Alert: ai.itpropartner.com
+  ⚠️ DISK at 92% (threshold: 90%)
+  ⚠️ RAM at 83% (threshold: 80%)
+```
+
+Multiple thresholds on one server are combined into a single message.
@@ -0,0 +1,68 @@
+#!/bin/bash
+# install-node-exporter.sh
+# Installs Prometheus Node Exporter as a systemd service on any Linux server.
+# Usage: bash install-node-exporter.sh [<host>]
+#   Without <host>: install locally
+#   With <host>: install via SSH (requires root SSH key at ~/.ssh/itpp-infra)
+
+set -euo pipefail
+
+NODE_EXPORTER_VERSION="1.8.2"
+ARCH="linux-amd64"
+
+install_local() {
+    if systemctl is-active --quiet node_exporter 2>/dev/null; then
+        echo "node_exporter already running — skipping install"
+        return 0
+    fi
+
+    echo "==> Downloading node_exporter ${NODE_EXPORTER_VERSION}..."
+    cd /tmp
+    wget -q "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.${ARCH}.tar.gz"
+    tar xzf "node_exporter-${NODE_EXPORTER_VERSION}.${ARCH}.tar.gz"
+
+    echo "==> Installing binary..."
+    cp "node_exporter-${NODE_EXPORTER_VERSION}.${ARCH}/node_exporter" /usr/local/bin/
+    chmod 755 /usr/local/bin/node_exporter
+
+    echo "==> Creating systemd service..."
+    cat > /etc/systemd/system/node_exporter.service << 'EOF'
+[Unit]
+Description=Prometheus Node Exporter
+After=network.target
+
+[Service]
+Type=simple
+User=nobody
+Group=nogroup
+ExecStart=/usr/local/bin/node_exporter \
+  --web.listen-address=:9100 \
+  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+    systemctl daemon-reload
+    systemctl enable node_exporter
+    systemctl start node_exporter
+    systemctl status node_exporter --no-pager -l | head -10
+
+    echo "==> Cleaning up..."
+    rm -rf "/tmp/node_exporter-${NODE_EXPORTER_VERSION}.${ARCH}"*
+
+    echo "==> node_exporter installed and running on port 9100"
+}
+
+if [ $# -eq 0 ]; then
+    install_local
+else
+    HOST="$1"
+    echo "==> Installing node_exporter on ${HOST}..."
+    # Copy install script and run remotely
+    scp -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra "$0" "root@${HOST}:/tmp/install-node-exporter.sh"
+    ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra "root@${HOST}" "bash /tmp/install-node-exporter.sh"
+    echo "==> node_exporter installed on ${HOST}"
+fi
@@ -0,0 +1,56 @@
+version: '3.8'
+
+services:
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    restart: unless-stopped
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+      - prometheus_data:/prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--storage.tsdb.path=/prometheus'
+      - '--storage.tsdb.retention.time=90d'
+    ports:
+      - "127.0.0.1:9090:9090"
+    deploy:
+      resources:
+        limits:
+          cpus: '1'
+          memory: 1G
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+    volumes:
+      - grafana_data:/var/lib/grafana
+      - ./grafana-provisioning:/etc/grafana/provisioning
+    environment:
+      - GF_SECURITY_ADMIN_USER=admin
+      - GF_SECURITY_ADMIN_PASSWORD=ITProPartnerMonitor!
+      - GF_INSTALL_PLUGINS=
+    ports:
+      - "127.0.0.1:3002:3000"
+    deploy:
+      resources:
+        limits:
+          cpus: '1'
+          memory: 512M
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+volumes:
+  prometheus_data:
+  grafana_data:
@@ -0,0 +1,16 @@
+# Grafana provisioning — Prometheus datasource
+# Mount this file to /etc/grafana/provisioning/datasources/prometheus.yml
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false
+    uid: prometheus
+    jsonData:
+      timeInterval: "60s"
+      queryTimeout: "30s"
+      httpMethod: POST
@@ -0,0 +1,46 @@
+# Prometheus configuration for IT Pro Partner infrastructure
+# Scrapes:
+#   1. Ops Portal backend (FastAPI /metrics)
+#   2. Node Exporter on Core server
+#   3. Node Exporter on each Hetzner server
+#   4. (Optional) Prometheus self-monitoring
+global:
+  scrape_interval: 60s      # Every 60s for all targets
+  evaluation_interval: 60s
+  scrape_timeout: 10s
+
+rule_files:
+  - "alerts.yml"
+
+scrape_configs:
+  # Self-monitoring
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # Ops Portal backend metrics
+  - job_name: 'ops-portal'
+    static_configs:
+      - targets: ['127.0.0.1:8090']
+    metrics_path: /metrics
+
+  # Core server (netcup) — node exporter on port 9100
+  - job_name: 'node-core'
+    static_configs:
+      - targets: ['127.0.0.1:9100']
+    metrics_path: /metrics
+
+  # Hetzner servers — node exporter on each
+  - job_name: 'node-hetzner'
+    static_configs:
+      - targets:
+        - '5.161.62.38:9100'     # wphost02
+        - '5.161.225.131:9100'   # unms
+        - '178.156.131.57:9100'  # unifi
+        - '178.156.130.130:9100' # hudu
+        - '178.156.149.32:9100'  # fleettracker360
+        - '178.156.168.35:9100'  # docker
+        - '178.156.167.181:9100' # ai
+        - '5.161.114.8:9100'     # app1-bu
+        - '87.99.144.163:9100'   # app1 (n8n)
+        - '87.99.159.142:9100'   # tony-vps