---
name: ops-portal-pipeline
version: 2.0.0
---

# Ops Portal — Architecture & Common Failure Modes

Created 2026-07-08 for ops.itpropartner.com. Updated 2026-07-12 with JS architecture fixes.

## Data Pipeline

Data Collector (every 5 min) → `/root/.hermes/scripts/ops-data-collector.py` → writes `/var/www/ops/data/ops-status.json` → dashboard pages fetch `/data/ops-status.json`

## JS Architecture

The portal uses TWO JavaScript files loaded on every page:

| File | Location | What it exports |
|------|----------|----------------|
| `utils.js` | `/opt/ops-portal/static/js/utils.js` | `window.initNav`, `window.fetchStatus`, `window.escapeHtml`, `window.statusBadge`, `window.fmtTime` |
| `app.js` | `/opt/ops-portal/static/js/app.js` | `window.Ops` (login, logout, isAuthenticated, getStatus, etc.), `window.initNav`, `window.escapeHtml`, `window.fmtTime`, `window.fetchStatus` |

**Both files MUST exist in the backend's static directory** (`/opt/ops-portal/static/js/`). A second copy lives at `/var/www/ops/js/` for Caddy's static file handler (controlled by `/etc/caddy/Caddyfile`). If either location goes out of sync, pages will 404 on one of the files.

**Critical failure (Jul 12):** `utils.js` existed at `/var/www/ops/js/utils.js` but NOT at `/opt/ops-portal/static/js/utils.js` — all pages referencing `/js/utils.js` got a 404. Fix: copy `utils.js` to the backend's static directory.

## Auth Flow

```
Page loads → app.js checks localStorage.getItem('ops_token')
  → token exists? → hide login overlay, load page data
  → no token? → show login overlay, wait for credentials
  → user submits → POST /api/auth/login → store token in localStorage → load page
```

**Token storage:** `localStorage` (NOT `sessionStorage`). `localStorage` persists across tabs and browser restarts.

**Credentials:** `germaine` / `itpp2026!` (stored in `/root/.hermes/.env` as `ADMIN_PASSWORD`).

## Nav Loading Pattern

Pages either have inline nav (index.html) or fetch nav dynamically:

**Fetched nav pattern (most pages):**
```html
<div id="nav"></div>
...
fetch('/nav.html').then(function(r) { return r.text(); }).then(function(h) {
    document.getElementById('nav').innerHTML = h;
    if (typeof Ops.initNav === 'function') Ops.initNav();
}).catch(function(e) { console.warn('Nav failed:', e); });
```

**The hamburger toggle** has an inline `onclick` handler that toggles `classList.toggle('open')` on `#navLinks`. The `Ops.initNav()` function used to ALSO add an `addEventListener` that did the same toggle — causing both handlers to fire on every click, cancelling each other out. **Fixed Jul 12:** removed the duplicate toggle from `initNav()`. initNav now only handles:
1. Active-link highlighting (`.nav-link.active`)
2. Closing the menu when tapping outside (`document click` listener)

## Common Failure Modes & Fixes

| Symptom | Root Cause | Fix |
|---------|------------|-----|
| Hamburger does nothing on mobile | Duplicate toggle handler (inline onclick + initNav addEventListener) | Remove addEventListener from initNav; inline onclick is sufficient |
| Hamburger does nothing (2) | `window.initNav` is undefined | Export `window.initNav = initNav` in app.js |
| Login overlay shows on every page load | `audit.html` uses `sessionStorage` not `localStorage` | Change to `localStorage` |
| Services/servers/backups crash on load | `getElementById('logoutBtn')` called before nav fetch completes | Wrap in interval or move inside fetch callback |
| cost.html blank on load | `window.fetchStatus` is undefined | Export `window.fetchStatus = fetchStatus` in app.js |
| config.html renders nothing | `window.escapeHtml` is undefined | Export `window.escapeHtml = escapeHtml` in app.js |
| JS/CSS files return 403 | File permissions are 600 | `chmod 644` on all static files |
| JS returns wrong content-type | Backend FileResponse without explicit media_type | Set `media_type` explicitly or serve through Caddy `file_server` |
| Pages load stale broken JS | Browser cached old version | Add `Cache-Control: no-cache` headers and `?v=N` on script tags |
| API shows 6/7 with one failing | `cloudflare_zones` returns a list, not "ok" string | Filter it out of `api_checks` in collector |
| Nav loaded but menu doesn't toggle | nav.html fetched but inline onclick not firing | Verify `onclick` attribute exists on toggle in nav.html |

## All Pages

/, /services.html, /servers.html, /backups.html, /network.html, /cron.html, /config.html, /logs.html, /audit.html, /cost.html, /fleettracker360.html, /nav.html