hermes-recovery/references/ops-portal-recommendations.md

IT Pro Partner — Ops Portal Recommendations

Generated: 2026-07-09 Based on audit of current portal at /var/www/ops/ and ops-status.json collection pipeline

---

Table of Contents

1. Current State Summary
2. Recommended Pages (New & Improved)
3. Actions Per Page — What Users Should Be Able to Do
4. Integrations Roadmap
5. Data Already Flowing
6. RBAC Recommendations
7. Implementation Priorities

---

1. Current State Summary

Existing Pages (all read-only, no action buttons)

Page	URL	Status	What It Shows
Dashboard	/	✅ Live	Health bar, cron jobs, services, S3 backups, API health, server inventory, versions
Servers	/servers.html	✅ Live	netcup detail + Hetzner table (sortable), quick specs
Backups	/backups.html	✅ Live	S3 bucket status grid, backup schedules, Wasabi info, manual actions (text only)
Network	/network.html	✅ Live	DNS/Cloudflare zones, MikroTik (placeholder), Ubiquiti (placeholder)
Services	/services.html	✅ Live	Systemd service status grid, API health, SyncroMSP integration summary
Cron Jobs	/cron.html	✅ Live	Metrics overview, searchable job table with script viewer
Logs & Events	/logs.html	✅ Live	API timeline, service events, cron errors, failure summary with filters
Config	/config.html	✅ Live	Versions, config file viewer (Caddyfile, Hermes YAML), environment
Template	/template.html	✅ Minimal	Empty page with nav injection

Key Gaps Identified

• No action/control buttons anywhere — everything is read-only
• No customer management — no client list, activations, lifecycle
• No billing/finance — Stripe, Xero, invoices, payments
• No user/RBAC management — no login, no roles
• No audit log — no record of who did what
• No settings/integrations page — API keys, integrations, branding
• MikroTik routers — placeholder only
• Ubiquiti/UISP — placeholder only
• No backup restore buttons
• No service start/stop/restart
• No provisioning actions — no way to spin up a server or create a customer
• No DNS management actions — cannot add/remove records
• No support ticket creation

---

2. Recommended Pages

2.1 Dashboard (Improve Existing)

Current: Health bar, cron jobs, S3 backups, API health, server inventory, versions.

Improvements:

• Add role-aware widgets — show different cards depending on who's logged in
• Add quick-action toolbar — "Provision new server", "New customer", "Create ticket"
• Add alert/incident summary — current failures, threshold breaches
• Add recent activity feed — last 10 actions taken in the portal
• Add revenue/ billing snapshot for owner role
• Add weather/network status at a glance
• Add pinned services — most-watched services at top

2.2 Server Management (Improve Existing)

Current: netcup detail, Hetzner table with sorting, quick specs.

Improvements:

• Add provisioning actions per server:
  • Restart server (soft reboot via Hetzner API)
  • Power on/off
  • Create snapshot
  • Open SSH console (WebSSH / terminal-in-browser)
  • View resource graphs (CPU, RAM, disk over time)
• Add threshold alerts per server — show warning if disk/mem/CPU exceeds thresholds
• Add server tags/groups — tag servers by role (web, db, ai, monitoring)
• Add migration status — track Hetzner→netcup migration progress per server
• Show backup status per server (last Hetzner snapshot date)
• Show Hudu link for each server's documentation page
• Add network interface stats — bandwidth per interface

2.3 Customer Management (NEW)

Purpose: One place to see all ITPP customers and their lifecycle.

Data sources: SyncroMSP, Stripe, Hudu, DRE CRM

Display:

• Customer list/search table with: name, email, phone, plan/service tier, status (active/pending/suspended), billing status, last ticket date
• Customer detail page with:
  • Contact info
  • Active services/subscriptions
  • Ticket history (from SyncroMSP)
  • Billing history (from Stripe)
  • Device inventory (from SyncroMSP)
  • Notes / tags
  • Linked documentation (Hudu link)

Actions:

• Create new customer
• Suspend/reactivate customer
• Assign service tier
• Create ticket on behalf of customer
• View full billing history
• Link to SyncroMSP, Hudu, Stripe

2.4 Network (Improve Existing)

Current: Cloudflare zones + placeholders for MikroTik and Ubiquiti.

Improvements:

A) MikroTik Routers (unlock placeholder):

• Router list with: hostname, model, RouterOS version, uptime, CPU/load, memory, interfaces
• Interface table: name, status (up/down), traffic (tx/rx), errors
• Connected clients / DHCP leases
• Firewall rules overview (count, active)
• Backup status (link to MikroTik backup on Wasabi)

Actions:

• Reboot router
• Run configuration backup
• View interface traffic graphs
• Enable/disable interface
• View log tail

B) Ubiquiti / UISP (unlock placeholder):

• UISP device list: towers, access points, CPEs
• Client counts per tower
• Bandwidth utilization graphs
• Device health (online/offline, signal strength)
• UniFi controller status, access point health

Actions:

• Provision new CPE
• Block/stick client
• View real-time RF metrics
• Trigger site survey

C) DNS & Domains (add actions): Currently 13 Cloudflare zones already tracked: boxpilotlogistics.com, broadbandtx.com, debtrecoveryexperts.com, fleettracker360.com, forefrontisp.com, germainebrown.com, iamgmb.com, itpropartner.com, katiewattsdesign.com, localisp.com, texasinternetaccess.com, vigilanttac.com, voipsimplicity.com

Actions to add:

• Add/edit/delete DNS records (A, CNAME, MX, TXT, etc.)
• Enable/disable proxy (Cloudflare orange cloud)
• View DNS record count per zone
• Trigger Cloudflare cache purge
• Show SSL/TLS status per domain
• Domain expiry dates (if registrar API connected)

2.5 Backups (Improve Existing)

Current: S3 bucket status grid, schedules, Wasabi info, text-only manual actions.

Improvements:

• Add one-click restore button per bucket (list snapshots, pick restore point)
• Add trigger backup now button per bucket
• Show backup size per bucket (not just status)
• Add restore logs — history of recent restores
• Add backup retention policy display (how long each bucket keeps data)
• Add integrity check results — last verification timestamp and status
• Add restore to staging — restore to a temp directory for verification

Actions:

• Trigger backup → runs the relevant cron script immediately
• Restore from backup → pick a date/snapshot and restore
• Verify integrity → run checksum verification
• Download archive link to Wasabi console URL
• Set retention policy input

2.6 Services (Improve Existing)

Current: Systemd service grid, API health cards, SyncroMSP summary.

Improvements:

• Add start / stop / restart buttons per service
• Show service logs (journalctl tail)
• Show resource usage per service (memory, CPU, uptime)
• Add dependencies — what depends on this service being up
• Add enable/disable at boot toggle
• Add service health history — uptime %, last restart reason

Actions:

• Start service
• Stop service
• Restart service
• View live logs (journalctl -u)
• Enable/disable at boot
• Reload configuration (for services that support it)

2.7 Billing & Finance (NEW)

Purpose: Business owner's view of revenue, invoices, and accounting sync.

Data sources: Stripe, Xero

Display:

• Revenue overview — MTD, YTD, monthly breakdown (chart)
• Pending invoices — outstanding, overdue
• Payment history — recent transactions
• Customer billing summary — per customer: plan, last payment, balance
• Subscription health — active, past-due, canceled counts
• Accounting sync status — last Xero sync, any errors

Actions:

• View invoice detail
• Send invoice reminder
• Mark invoice as paid (manual override)
• Sync with Stripe
• Sync with Xero
• Export billing report (CSV)

2.8 DNS & Domains (NEW dedicated page — or merge into Network)

If kept as separate page:

• All Cloudflare zones with DNS record counts
• Domain registrar details (expiry, auto-renew)
• Quick-add DNS record form
• SSL/TLS settings per domain
• Page Rules / redirect rules

2.9 Users & RBAC (NEW)

Purpose: Who can access the portal and what they can see/do.

Display:

• User list table: name, email, role, last login, status (active/disabled)
• Role/permission matrix view
• Audit log of user actions

Actions:

• Create user
• Edit user role
• Disable/delete user
• View user activity log
• Reset password / send invite

2.10 Audit Log (NEW)

Purpose: Record every action taken in the portal.

Display:

• Searchable, filterable timeline of events
• Columns: timestamp, user, action, resource, details, IP
• Exportable to CSV

Data captures:

• Who restarted a service
• Who triggered a backup
• Who created a customer
• Who changed a DNS record
• Login/logout events
• Failed permission attempts

2.11 Settings (NEW)

Purpose: Configure the portal itself and connected integrations.

Sections:

• Integrations — enable/disable and configure API keys for:
  • SyncroMSP
  • Cloudflare
  • Hetzner
  • Stripe
  • Xero
  • Wasabi
  • UniFi
  • UISP
  • MikroTik
  • Hudu
  • RingLogix
  • RunCloud
• API Keys — generate/manage API keys for programmatic portal access
• Branding — logo, company name, colors, support phone/email
• Notifications — configure alert email/Telegram destinations
• Backup retention — set retention policies per bucket
• Data retention — how long to keep logs, audit trail, events

2.12 Support Tickets (NEW)

Purpose: ITPP internal support ticket management.

Data source: SyncroMSP

Display:

• Ticket queue: ID, customer, subject, status, priority, assignee, last update
• Ticket detail: full conversation, attachments, time tracking
• Ticket metrics: open/closed counts, average response time

Actions:

• Create ticket
• Assign ticket
• Update status
• Add note/internal note
• Close/resolve ticket
• Link to customer profile

2.13 DRE Claims Dashboard (NEW or link to internal DRE portal)

The DRE portal already exists at internal.debtrecoveryexperts.com. On the Ops portal, add a widget/card on the dashboard showing:

• Pending approvals count
• Recent claims
• Inbox message count (from dre-mail-poller)
• Link through to the full DRE portal

---

3. Actions Per Page

Page	Read-Only Information	Actions Available
Dashboard	Health summary, recent failures, quick stats	Quick action toolbar, drill into alerts
Servers	Server specs, status, IP, provider, type	Restart, power on/off, create snapshot, WebSSH, view resource graphs
Customers	Contact info, services, tickets, bills	Create, suspend, reactivate, create ticket, view billing
Network	Router status, interfaces, clients, DNS zones, device health	Reboot router, run backup, add DNS record, purge cache, enable/disable interface
Backups	Bucket status, last upload, age, schedule	Trigger backup, restore from snapshot, verify integrity, set retention
Services	Systemd status, API health, SyncroMSP stats	Start / stop / restart, view logs, enable/disable boot
Billing	Revenue, invoices, payments, subscriptions	Sync Stripe, sync Xero, send invoice reminder, export CSV
DNS	Zone list, record count, SSL status	Add/Edit/Delete record, toggle proxy, purge cache
Users/RBAC	User list, roles, last login	Create user, edit role, disable, view audit log
Audit Log	Full action timeline with user, action, resource	Search, filter, export
Settings	Integration status, versions, env	Add/rotate API keys, enable/disable integrations, set branding
Support Tickets	Queue, details, metrics	Create, assign, update status, add note, resolve
DRE	Pending approvals, claims, inbox count	Link to full DRE portal

---

4. Integrations Roadmap

Already Connected & Working

Integration	Status	Data Flow	Page(s) Using It
SyncroMSP	✅ Live	26 customers, tickets, devices via API	Dashboard, Services
Cloudflare	✅ Live	13 zones, DNS record status via API	Network, Dashboard
Hetzner	✅ Live	10 servers, status, type, IP via API	Servers, Dashboard
netcup	✅ Live	Server detail (template, IP, id) via API/config	Servers
Wasabi	✅ Live	6 backup buckets, last upload, age via AWS CLI	Backups, Dashboard
Admin-AI	✅ Live	API health check	Dashboard, Services

In Progress — API Keys Saved, Need Portal UI

Integration	Status	What to Build
UniFi	🔶 Saved	Add UniFi client count, access point health, client list to Network page
UISP (UNMS)	🔶 Saved	Add tower/CPE status, bandwidth to Network page

Planned — Need Connection & UI

Integration	Priority	What to Build
Stripe	🔴 High	Billing page: invoice list, revenue, subscription management
Xero	🔴 High	Billing page: accounting sync, P&L snapshot
MikroTik	🟡 Medium	Network page: RouterOS API for interface stats, DHCP, firewall
RingLogix	🟡 Medium	VoIP status, call metrics, DID inventory, phone management
Hudu	🟡 Medium	Link servers/customers to documentation, inline wiki viewer
RunCloud	🟢 Low	WordPress server management, site health, SSL

Future / Nice-to-Have

Integration	Use Case
Porkbun / Namecheap	Domain registration details, expiry dates, auto-renew
GitHub	Deploy status, latest commits, CI pipeline status
Grafana	Embed resource graphs, historical metrics
Uptime Kuma / Better Uptime	External monitoring status, incident history
OpenAI / AI credits	Usage tracking (Firecrawl already done, can add AI model usage)

---

5. Data Already Flowing

The ops-data-collector (Python, runs every 5 min) already collects:

{
  "timestamp": "ISO-8601",
  "collection_duration_ms": 10363,
  "overall": {
    "total_jobs": 21,
    "passing": 19,
    "failing": 2,
    "disk_used_pct": 6,
    "memory_used_pct": 41
  },
  "cron_jobs": [ /* 21 jobs with name, schedule, lastRun, status, script */ ],
  "services": { /* 8 systemd services: active/inactive */ },
  "disk_memory": { "disk_used_pct", "memory_used_pct" },
  "s3_backups": { /* 6 buckets: status, last_upload, age_hours */ },
  "api_checks": { /* admin-ai, caddy-config, port checks, cloudflare */ },
  "cloudflare_zones": [ /* 13 zone names + status */ ],
  "versions": { /* hermes, caddy, python, os */ },
  "routers": [ /* currently empty — placeholder */ ],
  "netcup_server": { /* id, template, ip */ },
  "hetzner_servers": [ /* 10 servers with name, status, type, ip, id */ ],
  "syncromsp": { /* customers: 26, tickets, devices, customer_list */ }
}

What the collector CANNOT currently collect:

• MikroTik RouterOS data (API key needed + collect every 1-5 min)
• Ubiquiti/UISP data (API key saved, but collector not extended yet)
• Stripe billing data (no API key in collector)
• Xero accounting data (no API key in collector)
• Per-server resource usage beyond disk/memory (needs SSH or agent)
• Historical metrics (needs time-series DB like Prometheus or simple log file)

---

6. RBAC Recommendations

6.1 Roles

Role	Description
Owner	Full access — see everything, do everything. Germaine.
Admin	Full access minus billing configuration and system settings. Tony.
Tech / Support	Servers, services, network, logs, tickets, customers (read). No billing, provisioning, settings.
Sales / Onboarding	Customer activation flow, order forms, provisioning status. No server internals, no billing.
Viewer / Auditor	Read-only across limited scope. No actions, no billing, no settings.
DRE Agent	DRE-specific pages only — claims, inbox, pending approvals. No infrastructure. Anita.

6.2 Page Access Matrix

Page / Feature	Owner	Admin	Tech/Support	Sales/Onboarding	Viewer	DRE Agent
Dashboard (full)	✅	✅	❌	❌	❌	❌
Dashboard (tech view)	✅	✅	✅	❌	✅	❌
Dashboard (sales view)	✅	✅	❌	✅	❌	❌
Dashboard (DRE view)	✅	❌	❌	❌	❌	✅
Servers — view	✅	✅	✅	❌	✅	❌
Servers — restart/power	✅	✅	✅	❌	❌	❌
Servers — snapshot/provision	✅	✅	❌	❌	❌	❌
Customers — view	✅	✅	✅	✅	✅	❌
Customers — create/edit/suspend	✅	✅	❌	✅	❌	❌
Customers — billing detail	✅	✅	❌	❌	❌	❌
Network — view	✅	✅	✅	❌	✅	❌
Network — reboot router/interface	✅	✅	✅	❌	❌	❌
Network — DNS edit	✅	✅	✅	❌	❌	❌
Backups — view	✅	✅	✅	❌	✅	❌
Backups — trigger/restore	✅	✅	❌	❌	❌	❌
Services — view	✅	✅	✅	❌	✅	❌
Services — start/stop/restart	✅	✅	✅	❌	❌	❌
Billing — view	✅	✅	❌	❌	❌	❌
Billing — modify/sync	✅	❌	❌	❌	❌	❌
Users/RBAC — manage	✅	❌	❌	❌	❌	❌
Users/RBAC — view	✅	✅	❌	❌	❌	❌
Audit Log — view	✅	✅	✅	❌	❌	❌
Settings — all	✅	❌	❌	❌	❌	❌
Settings — notification prefs	✅	✅	✅	❌	❌	❌
Tickets — view all	✅	✅	✅	✅	❌	❌
Tickets — create/assign	✅	✅	✅	✅	❌	❌
DRE Claims	✅	❌	❌	❌	❌	✅
Provisioning (new servers/customers)	✅	✅	❌	❌	❌	❌

6.3 Person-to-Role Mapping

Person	Role	Notes
Germaine (email@redacted)	Owner	Sees everything, can do everything. Needs billing, provisioning, and audit.
Tony (email@redacted)	Admin	Full access minus billing config and user management. Handles server operations, backups, customer management.
Anita	DRE Agent	DRE-focused view. Claims, pending approvals, inbox. No infrastructure.
Future Tech Hire	Tech/Support	Servers, services, network, tickets. No billing, provisioning, settings.
Future Sales Hire	Sales/Onboarding	Customer activation, provisioning status, order forms. No server internals.
Future Auditor	Viewer	Read-only, limited scope. Good for compliance or investor review.

6.4 RBAC Implementation Notes

• Start simple — use JWT or session tokens with role stored in the token
• No auth initially — phase 1: keep open but label sections as "Owner Only" / "Tech Only"
• Phase 2: add login with email/password or OAuth (Google/GitHub)
• Phase 3: full RBAC enforcement at API/backend level
• Consider using Hudu's SSO or SyncroMSP's user model as identity source to avoid yet another password
• Store role assignments in a simple JSON config file or small SQLite DB

---

7. Implementation Priorities

Phase 1: Add Actions to Existing Pages (Weeks 1-3)

1. Backups — add "Trigger Backup" and "Restore" buttons (calls existing scripts via SSH)
2. Services — add Start/Stop/Restart buttons (calls systemctl over SSH)
3. Servers — add Reboot + Snapshot buttons (calls Hetzner API)
4. Network — add DNS record management (calls Cloudflare API)

Phase 2: Unlock Placeholders (Weeks 3-5)

5. MikroTik — RouterOS API integration, live interface/uptime/leases data
6. Ubiquiti/UISP — Connect API, show tower status and client counts
7. Dashboard improvements — alert summary, recent activity, quick actions

Phase 3: New Pages — Foundation (Weeks 5-8)

8. Customer Management — list/search/create customers, linked to SyncroMSP
9. Audit Log — capture and display every action
10. Support Tickets — SyncroMSP ticket queue integration
11. DRE Dashboard widget — pending approvals, inbox count

Phase 4: New Pages — Business & Admin (Weeks 8-12)

12. Users & RBAC — login, role assignment, permission enforcement
13. Billing & Finance — Stripe revenue, invoice list, Xero sync
14. Settings — integration management, API keys, branding
15. Full RBAC enforcement — every API endpoint checks permissions

Phase 5: Polish & Optimization (Ongoing)

16. Add resource graphs (historical CPU/RAM/disk)
17. Add notification preferences per user (email/Telegram for specific alerts)
18. Add dark mode toggle (already dark theme, but add theme options)
19. Mobile-responsive improvements
20. Performance optimization (lazy load, paginate large tables, caching)

---

Architecture Notes for the Build Team

Current architecture:

• Static HTML pages served by Caddy (ops.itpropartner.com)
• All data from ops-status.json (generated every 5 min by Python collector)
• Vanilla JS fetches JSON and renders DOM
• Shared nav partial (nav.html) injected via fetch()
• Shared CSS (ops.css) and JS (utils.js)
• No backend API — purely static frontend reading JSON

Recommended evolution:

1. Keep static frontend for Phase 1 (quick action buttons can call API endpoints)
2. Add a lightweight backend (FastAPI or Node.js) for:
   • Authentication/sessions
   • Action endpoints (POST /api/restart-service, POST /api/trigger-backup, etc.)
   • Audit logging middleware
   • Role/permission checks
3. Options for backend:
   • FastAPI (Python) — matches existing ecosystem, easy to add SSH/cron exec
   • Express/PocketBase — simple, built-in auth
   • Remote game — use Caddy's reverse_proxy to a backend running on port 8084

Data pipeline options for actions:

• Direct API call from backend (best for Hetzner, Cloudflare, Stripe — they have HTTP APIs)
• SSH execute (for systemctl on Core and remote servers)
• Hermes cron trigger (for backup jobs — trigger the cron script directly)
• Webhook (for 3rd party services that support incoming webhooks)