hermes-skills/skills/devops/infrastructure-automation/references/ops-portal-pipeline.md

name, version

name	version
ops-portal-pipeline	2.0.0

Ops Portal — Architecture & Common Failure Modes

Created 2026-07-08 for ops.itpropartner.com. Updated 2026-07-12 with JS architecture fixes.

Data Pipeline

Data Collector (every 5 min) → /root/.hermes/scripts/ops-data-collector.py → writes /var/www/ops/data/ops-status.json → dashboard pages fetch /data/ops-status.json

JS Architecture

The portal uses TWO JavaScript files loaded on every page:

File	Location	What it exports
utils.js	/opt/ops-portal/static/js/utils.js	window.initNav, window.fetchStatus, window.escapeHtml, window.statusBadge, window.fmtTime
app.js	/opt/ops-portal/static/js/app.js	window.Ops (login, logout, isAuthenticated, getStatus, etc.), window.initNav, window.escapeHtml, window.fmtTime, window.fetchStatus

Both files MUST exist in the backend's static directory (/opt/ops-portal/static/js/). A second copy lives at /var/www/ops/js/ for Caddy's static file handler (controlled by /etc/caddy/Caddyfile). If either location goes out of sync, pages will 404 on one of the files.

Critical failure (Jul 12): utils.js existed at /var/www/ops/js/utils.js but NOT at /opt/ops-portal/static/js/utils.js — all pages referencing /js/utils.js got a 404. Fix: copy utils.js to the backend's static directory.

Auth Flow

Page loads → app.js checks localStorage.getItem('ops_token')
  → token exists? → hide login overlay, load page data
  → no token? → show login overlay, wait for credentials
  → user submits → POST /api/auth/login → store token in localStorage → load page

Token storage: localStorage (NOT sessionStorage). localStorage persists across tabs and browser restarts.

Credentials: germaine / itpp2026! (stored in /root/.hermes/.env as ADMIN_PASSWORD).

Nav Loading Pattern

Pages either have inline nav (index.html) or fetch nav dynamically:

Fetched nav pattern (most pages):

<div id="nav"></div>
...
fetch('/nav.html').then(function(r) { return r.text(); }).then(function(h) {
    document.getElementById('nav').innerHTML = h;
    if (typeof Ops.initNav === 'function') Ops.initNav();
}).catch(function(e) { console.warn('Nav failed:', e); });

The hamburger toggle has an inline onclick handler that toggles classList.toggle('open') on #navLinks. The Ops.initNav() function used to ALSO add an addEventListener that did the same toggle — causing both handlers to fire on every click, cancelling each other out. Fixed Jul 12: removed the duplicate toggle from initNav(). initNav now only handles:

1. Active-link highlighting (.nav-link.active)
2. Closing the menu when tapping outside (document click listener)

Common Failure Modes & Fixes

Symptom	Root Cause	Fix
Hamburger does nothing on mobile	Duplicate toggle handler (inline onclick + initNav addEventListener)	Remove addEventListener from initNav; inline onclick is sufficient
Hamburger does nothing (2)	window.initNav is undefined	Export window.initNav = initNav in app.js
Login overlay shows on every page load	audit.html uses sessionStorage not localStorage	Change to localStorage
Services/servers/backups crash on load	getElementById('logoutBtn') called before nav fetch completes	Wrap in interval or move inside fetch callback
cost.html blank on load	window.fetchStatus is undefined	Export window.fetchStatus = fetchStatus in app.js
config.html renders nothing	window.escapeHtml is undefined	Export window.escapeHtml = escapeHtml in app.js
JS/CSS files return 403	File permissions are 600	chmod 644 on all static files
JS returns wrong content-type	Backend FileResponse without explicit media_type	Set media_type explicitly or serve through Caddy file_server
Pages load stale broken JS	Browser cached old version	Add Cache-Control: no-cache headers and ?v=N on script tags
API shows 6/7 with one failing	cloudflare_zones returns a list, not "ok" string	Filter it out of api_checks in collector
Nav loaded but menu doesn't toggle	nav.html fetched but inline onclick not firing	Verify onclick attribute exists on toggle in nav.html

All Pages

/, /services.html, /servers.html, /backups.html, /network.html, /cron.html, /config.html, /logs.html, /audit.html, /cost.html, /fleettracker360.html, /nav.html