hermes-recovery/references/ops-portal-recommendations.md

# IT Pro Partner — Ops Portal Recommendations

> Generated: 2026-07-09
> Based on audit of current portal at `/var/www/ops/` and ops-status.json collection pipeline

---

## Table of Contents

1. [Current State Summary](#1-current-state-summary)
2. [Recommended Pages (New & Improved)](#2-recommended-pages)
3. [Actions Per Page — What Users Should Be Able to Do](#3-actions-per-page)
4. [Integrations Roadmap](#4-integrations-roadmap)
5. [Data Already Flowing](#5-data-already-flowing)
6. [RBAC Recommendations](#6-rbac-recommendations)
7. [Implementation Priorities](#7-implementation-priorities)

---

## 1. Current State Summary

### Existing Pages (all read-only, no action buttons)

| Page | URL | Status | What It Shows |
|------|-----|--------|---------------|
| Dashboard | `/` | ✅ Live | Health bar, cron jobs, services, S3 backups, API health, server inventory, versions |
| Servers | `/servers.html` | ✅ Live | netcup detail + Hetzner table (sortable), quick specs |
| Backups | `/backups.html` | ✅ Live | S3 bucket status grid, backup schedules, Wasabi info, manual actions (text only) |
| Network | `/network.html` | ✅ Live | DNS/Cloudflare zones, MikroTik (placeholder), Ubiquiti (placeholder) |
| Services | `/services.html` | ✅ Live | Systemd service status grid, API health, SyncroMSP integration summary |
| Cron Jobs | `/cron.html` | ✅ Live | Metrics overview, searchable job table with script viewer |
| Logs & Events | `/logs.html` | ✅ Live | API timeline, service events, cron errors, failure summary with filters |
| Config | `/config.html` | ✅ Live | Versions, config file viewer (Caddyfile, Hermes YAML), environment |
| Template | `/template.html` | ✅ Minimal | Empty page with nav injection |

### Key Gaps Identified

- **No action/control buttons anywhere** — everything is read-only
- **No customer management** — no client list, activations, lifecycle
- **No billing/finance** — Stripe, Xero, invoices, payments
- **No user/RBAC management** — no login, no roles
- **No audit log** — no record of who did what
- **No settings/integrations page** — API keys, integrations, branding
- **MikroTik routers** — placeholder only
- **Ubiquiti/UISP** — placeholder only
- **No backup restore buttons**
- **No service start/stop/restart**
- **No provisioning actions** — no way to spin up a server or create a customer
- **No DNS management actions** — cannot add/remove records
- **No support ticket creation**

---

## 2. Recommended Pages

### 2.1 Dashboard (Improve Existing)

**Current:** Health bar, cron jobs, S3 backups, API health, server inventory, versions.

**Improvements:**
- Add **role-aware widgets** — show different cards depending on who's logged in
- Add **quick-action toolbar** — "Provision new server", "New customer", "Create ticket"
- Add **alert/incident summary** — current failures, threshold breaches
- Add **recent activity feed** — last 10 actions taken in the portal
- Add **revenue/ billing snapshot** for owner role
- Add **weather/network status** at a glance
- Add **pinned services** — most-watched services at top

### 2.2 Server Management (Improve Existing)

**Current:** netcup detail, Hetzner table with sorting, quick specs.

**Improvements:**
- Add **provisioning actions** per server:
  - Restart server (soft reboot via Hetzner API)
  - Power on/off
  - Create snapshot
  - Open SSH console (WebSSH / terminal-in-browser)
  - View resource graphs (CPU, RAM, disk over time)
- Add **threshold alerts per server** — show warning if disk/mem/CPU exceeds thresholds
- Add **server tags/groups** — tag servers by role (web, db, ai, monitoring)
- Add **migration status** — track Hetzner→netcup migration progress per server
- Show **backup status per server** (last Hetzner snapshot date)
- Show **Hudu link** for each server's documentation page
- Add **network interface stats** — bandwidth per interface

### 2.3 Customer Management (NEW)

**Purpose:** One place to see all ITPP customers and their lifecycle.

**Data sources:** SyncroMSP, Stripe, Hudu, DRE CRM

**Display:**
- Customer list/search table with: name, email, phone, plan/service tier, status (active/pending/suspended), billing status, last ticket date
- Customer detail page with:
  - Contact info
  - Active services/subscriptions
  - Ticket history (from SyncroMSP)
  - Billing history (from Stripe)
  - Device inventory (from SyncroMSP)
  - Notes / tags
  - Linked documentation (Hudu link)

**Actions:**
- Create new customer
- Suspend/reactivate customer
- Assign service tier
- Create ticket on behalf of customer
- View full billing history
- Link to SyncroMSP, Hudu, Stripe

### 2.4 Network (Improve Existing)

**Current:** Cloudflare zones + placeholders for MikroTik and Ubiquiti.

**Improvements:**

**A) MikroTik Routers (unlock placeholder):**
- Router list with: hostname, model, RouterOS version, uptime, CPU/load, memory, interfaces
- Interface table: name, status (up/down), traffic (tx/rx), errors
- Connected clients / DHCP leases
- Firewall rules overview (count, active)
- Backup status (link to MikroTik backup on Wasabi)

**Actions:**
- Reboot router
- Run configuration backup
- View interface traffic graphs
- Enable/disable interface
- View log tail

**B) Ubiquiti / UISP (unlock placeholder):**
- UISP device list: towers, access points, CPEs
- Client counts per tower
- Bandwidth utilization graphs
- Device health (online/offline, signal strength)
- UniFi controller status, access point health

**Actions:**
- Provision new CPE
- Block/stick client
- View real-time RF metrics
- Trigger site survey

**C) DNS & Domains (add actions):**
Currently 13 Cloudflare zones already tracked:
boxpilotlogistics.com, broadbandtx.com, debtrecoveryexperts.com, fleettracker360.com, forefrontisp.com, germainebrown.com, iamgmb.com, itpropartner.com, katiewattsdesign.com, localisp.com, texasinternetaccess.com, vigilanttac.com, voipsimplicity.com

**Actions to add:**
- Add/edit/delete DNS records (A, CNAME, MX, TXT, etc.)
- Enable/disable proxy (Cloudflare orange cloud)
- View DNS record count per zone
- Trigger Cloudflare cache purge
- Show SSL/TLS status per domain
- Domain expiry dates (if registrar API connected)

### 2.5 Backups (Improve Existing)

**Current:** S3 bucket status grid, schedules, Wasabi info, text-only manual actions.

**Improvements:**
- Add **one-click restore** button per bucket (list snapshots, pick restore point)
- Add **trigger backup now** button per bucket
- Show **backup size** per bucket (not just status)
- Add **restore logs** — history of recent restores
- Add **backup retention policy** display (how long each bucket keeps data)
- Add **integrity check results** — last verification timestamp and status
- Add **restore to staging** — restore to a temp directory for verification

**Actions:**
- **Trigger backup** → runs the relevant cron script immediately
- **Restore from backup** → pick a date/snapshot and restore
- **Verify integrity** → run checksum verification
- **Download archive** link to Wasabi console URL
- **Set retention policy** input

### 2.6 Services (Improve Existing)

**Current:** Systemd service grid, API health cards, SyncroMSP summary.

**Improvements:**
- Add **start / stop / restart** buttons per service
- Show **service logs** (journalctl tail)
- Show **resource usage** per service (memory, CPU, uptime)
- Add **dependencies** — what depends on this service being up
- Add **enable/disable at boot** toggle
- Add **service health history** — uptime %, last restart reason

**Actions:**
- Start service
- Stop service
- Restart service
- View live logs (journalctl -u)
- Enable/disable at boot
- Reload configuration (for services that support it)

### 2.7 Billing & Finance (NEW)

**Purpose:** Business owner's view of revenue, invoices, and accounting sync.

**Data sources:** Stripe, Xero

**Display:**
- **Revenue overview** — MTD, YTD, monthly breakdown (chart)
- **Pending invoices** — outstanding, overdue
- **Payment history** — recent transactions
- **Customer billing summary** — per customer: plan, last payment, balance
- **Subscription health** — active, past-due, canceled counts
- **Accounting sync status** — last Xero sync, any errors

**Actions:**
- View invoice detail
- Send invoice reminder
- Mark invoice as paid (manual override)
- Sync with Stripe
- Sync with Xero
- Export billing report (CSV)

### 2.8 DNS & Domains (NEW dedicated page — or merge into Network)

If kept as separate page:
- All Cloudflare zones with DNS record counts
- Domain registrar details (expiry, auto-renew)
- Quick-add DNS record form
- SSL/TLS settings per domain
- Page Rules / redirect rules

### 2.9 Users & RBAC (NEW)

**Purpose:** Who can access the portal and what they can see/do.

**Display:**
- User list table: name, email, role, last login, status (active/disabled)
- Role/permission matrix view
- Audit log of user actions

**Actions:**
- Create user
- Edit user role
- Disable/delete user
- View user activity log
- Reset password / send invite

### 2.10 Audit Log (NEW)

**Purpose:** Record every action taken in the portal.

**Display:**
- Searchable, filterable timeline of events
- Columns: timestamp, user, action, resource, details, IP
- Exportable to CSV

**Data captures:**
- Who restarted a service
- Who triggered a backup
- Who created a customer
- Who changed a DNS record
- Login/logout events
- Failed permission attempts

### 2.11 Settings (NEW)

**Purpose:** Configure the portal itself and connected integrations.

**Sections:**
- **Integrations** — enable/disable and configure API keys for:
  - SyncroMSP
  - Cloudflare
  - Hetzner
  - Stripe
  - Xero
  - Wasabi
  - UniFi
  - UISP
  - MikroTik
  - Hudu
  - RingLogix
  - RunCloud
- **API Keys** — generate/manage API keys for programmatic portal access
- **Branding** — logo, company name, colors, support phone/email
- **Notifications** — configure alert email/Telegram destinations
- **Backup retention** — set retention policies per bucket
- **Data retention** — how long to keep logs, audit trail, events

### 2.12 Support Tickets (NEW)

**Purpose:** ITPP internal support ticket management.

**Data source:** SyncroMSP

**Display:**
- Ticket queue: ID, customer, subject, status, priority, assignee, last update
- Ticket detail: full conversation, attachments, time tracking
- Ticket metrics: open/closed counts, average response time

**Actions:**
- Create ticket
- Assign ticket
- Update status
- Add note/internal note
- Close/resolve ticket
- Link to customer profile

### 2.13 DRE Claims Dashboard (NEW or link to internal DRE portal)

The DRE portal already exists at `internal.debtrecoveryexperts.com`. On the Ops portal, add a **widget/card** on the dashboard showing:
- Pending approvals count
- Recent claims
- Inbox message count (from dre-mail-poller)
- Link through to the full DRE portal

---

## 3. Actions Per Page

| Page | Read-Only Information | Actions Available |
|------|----------------------|-------------------|
| **Dashboard** | Health summary, recent failures, quick stats | Quick action toolbar, drill into alerts |
| **Servers** | Server specs, status, IP, provider, type | **Restart**, power on/off, create snapshot, WebSSH, view resource graphs |
| **Customers** | Contact info, services, tickets, bills | **Create**, suspend, reactivate, create ticket, view billing |
| **Network** | Router status, interfaces, clients, DNS zones, device health | **Reboot router**, run backup, add DNS record, purge cache, enable/disable interface |
| **Backups** | Bucket status, last upload, age, schedule | **Trigger backup**, **restore from snapshot**, verify integrity, set retention |
| **Services** | Systemd status, API health, SyncroMSP stats | **Start / stop / restart**, view logs, enable/disable boot |
| **Billing** | Revenue, invoices, payments, subscriptions | Sync Stripe, sync Xero, send invoice reminder, export CSV |
| **DNS** | Zone list, record count, SSL status | **Add/Edit/Delete record**, toggle proxy, purge cache |
| **Users/RBAC** | User list, roles, last login | **Create user**, edit role, disable, view audit log |
| **Audit Log** | Full action timeline with user, action, resource | Search, filter, export |
| **Settings** | Integration status, versions, env | **Add/rotate API keys**, enable/disable integrations, set branding |
| **Support Tickets** | Queue, details, metrics | **Create**, assign, update status, add note, resolve |
| **DRE** | Pending approvals, claims, inbox count | Link to full DRE portal |

---

## 4. Integrations Roadmap

### Already Connected & Working

| Integration | Status | Data Flow | Page(s) Using It |
|-------------|--------|-----------|------------------|
| **SyncroMSP** | ✅ Live | 26 customers, tickets, devices via API | Dashboard, Services |
| **Cloudflare** | ✅ Live | 13 zones, DNS record status via API | Network, Dashboard |
| **Hetzner** | ✅ Live | 10 servers, status, type, IP via API | Servers, Dashboard |
| **netcup** | ✅ Live | Server detail (template, IP, id) via API/config | Servers |
| **Wasabi** | ✅ Live | 6 backup buckets, last upload, age via AWS CLI | Backups, Dashboard |
| **Admin-AI** | ✅ Live | API health check | Dashboard, Services |

### In Progress — API Keys Saved, Need Portal UI

| Integration | Status | What to Build |
|-------------|--------|---------------|
| **UniFi** | 🔶 Saved | Add UniFi client count, access point health, client list to Network page |
| **UISP (UNMS)** | 🔶 Saved | Add tower/CPE status, bandwidth to Network page |

### Planned — Need Connection & UI

| Integration | Priority | What to Build |
|-------------|----------|---------------|
| **Stripe** | 🔴 High | Billing page: invoice list, revenue, subscription management |
| **Xero** | 🔴 High | Billing page: accounting sync, P&L snapshot |
| **MikroTik** | 🟡 Medium | Network page: RouterOS API for interface stats, DHCP, firewall |
| **RingLogix** | 🟡 Medium | VoIP status, call metrics, DID inventory, phone management |
| **Hudu** | 🟡 Medium | Link servers/customers to documentation, inline wiki viewer |
| **RunCloud** | 🟢 Low | WordPress server management, site health, SSL |

### Future / Nice-to-Have

| Integration | Use Case |
|-------------|----------|
| **Porkbun / Namecheap** | Domain registration details, expiry dates, auto-renew |
| **GitHub** | Deploy status, latest commits, CI pipeline status |
| **Grafana** | Embed resource graphs, historical metrics |
| **Uptime Kuma / Better Uptime** | External monitoring status, incident history |
| **OpenAI / AI credits** | Usage tracking (Firecrawl already done, can add AI model usage) |

---

## 5. Data Already Flowing

The ops-data-collector (Python, runs every 5 min) already collects:

```json
{
  "timestamp": "ISO-8601",
  "collection_duration_ms": 10363,
  "overall": {
    "total_jobs": 21,
    "passing": 19,
    "failing": 2,
    "disk_used_pct": 6,
    "memory_used_pct": 41
  },
  "cron_jobs": [ /* 21 jobs with name, schedule, lastRun, status, script */ ],
  "services": { /* 8 systemd services: active/inactive */ },
  "disk_memory": { "disk_used_pct", "memory_used_pct" },
  "s3_backups": { /* 6 buckets: status, last_upload, age_hours */ },
  "api_checks": { /* admin-ai, caddy-config, port checks, cloudflare */ },
  "cloudflare_zones": [ /* 13 zone names + status */ ],
  "versions": { /* hermes, caddy, python, os */ },
  "routers": [ /* currently empty — placeholder */ ],
  "netcup_server": { /* id, template, ip */ },
  "hetzner_servers": [ /* 10 servers with name, status, type, ip, id */ ],
  "syncromsp": { /* customers: 26, tickets, devices, customer_list */ }
}
```

**What the collector CANNOT currently collect:**
- MikroTik RouterOS data (API key needed + collect every 1-5 min)
- Ubiquiti/UISP data (API key saved, but collector not extended yet)
- Stripe billing data (no API key in collector)
- Xero accounting data (no API key in collector)
- Per-server resource usage beyond disk/memory (needs SSH or agent)
- Historical metrics (needs time-series DB like Prometheus or simple log file)

---

## 6. RBAC Recommendations

### 6.1 Roles

| Role | Description |
|------|-------------|
| **Owner** | Full access — see everything, do everything. Germaine. |
| **Admin** | Full access minus billing configuration and system settings. Tony. |
| **Tech / Support** | Servers, services, network, logs, tickets, customers (read). No billing, provisioning, settings. |
| **Sales / Onboarding** | Customer activation flow, order forms, provisioning status. No server internals, no billing. |
| **Viewer / Auditor** | Read-only across limited scope. No actions, no billing, no settings. |
| **DRE Agent** | DRE-specific pages only — claims, inbox, pending approvals. No infrastructure. Anita. |

### 6.2 Page Access Matrix

| Page / Feature | Owner | Admin | Tech/Support | Sales/Onboarding | Viewer | DRE Agent |
|----------------|-------|-------|-------------|-----------------|--------|-----------|
| Dashboard (full) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Dashboard (tech view) | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Dashboard (sales view) | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| Dashboard (DRE view) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Servers — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Servers — restart/power | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Servers — snapshot/provision | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Customers — view | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| Customers — create/edit/suspend | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| Customers — billing detail | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Network — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Network — reboot router/interface | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Network — DNS edit | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Backups — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Backups — trigger/restore | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Services — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Services — start/stop/restart | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Billing — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Billing — modify/sync | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Users/RBAC — manage | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Users/RBAC — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Audit Log — view | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Settings — all | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Settings — notification prefs | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Tickets — view all | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Tickets — create/assign | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DRE Claims | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Provisioning (new servers/customers) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |

### 6.3 Person-to-Role Mapping

| Person | Role | Notes |
|--------|------|-------|
| **Germaine** (email@redacted) | **Owner** | Sees everything, can do everything. Needs billing, provisioning, and audit. |
| **Tony** (email@redacted) | **Admin** | Full access minus billing config and user management. Handles server operations, backups, customer management. |
| **Anita** | **DRE Agent** | DRE-focused view. Claims, pending approvals, inbox. No infrastructure. |
| **Future Tech Hire** | **Tech/Support** | Servers, services, network, tickets. No billing, provisioning, settings. |
| **Future Sales Hire** | **Sales/Onboarding** | Customer activation, provisioning status, order forms. No server internals. |
| **Future Auditor** | **Viewer** | Read-only, limited scope. Good for compliance or investor review. |

### 6.4 RBAC Implementation Notes

- **Start simple** — use JWT or session tokens with role stored in the token
- **No auth initially** — phase 1: keep open but label sections as "Owner Only" / "Tech Only"
- **Phase 2**: add login with email/password or OAuth (Google/GitHub)
- **Phase 3**: full RBAC enforcement at API/backend level
- Consider using **Hudu's SSO** or **SyncroMSP's user model** as identity source to avoid yet another password
- Store role assignments in a simple JSON config file or small SQLite DB

---

## 7. Implementation Priorities

### Phase 1: Add Actions to Existing Pages (Weeks 1-3)

1. **Backups** — add "Trigger Backup" and "Restore" buttons (calls existing scripts via SSH)
2. **Services** — add Start/Stop/Restart buttons (calls systemctl over SSH)
3. **Servers** — add Reboot + Snapshot buttons (calls Hetzner API)
4. **Network** — add DNS record management (calls Cloudflare API)

### Phase 2: Unlock Placeholders (Weeks 3-5)

5. **MikroTik** — RouterOS API integration, live interface/uptime/leases data
6. **Ubiquiti/UISP** — Connect API, show tower status and client counts
7. **Dashboard improvements** — alert summary, recent activity, quick actions

### Phase 3: New Pages — Foundation (Weeks 5-8)

8. **Customer Management** — list/search/create customers, linked to SyncroMSP
9. **Audit Log** — capture and display every action
10. **Support Tickets** — SyncroMSP ticket queue integration
11. **DRE Dashboard widget** — pending approvals, inbox count

### Phase 4: New Pages — Business & Admin (Weeks 8-12)

12. **Users & RBAC** — login, role assignment, permission enforcement
13. **Billing & Finance** — Stripe revenue, invoice list, Xero sync
14. **Settings** — integration management, API keys, branding
15. **Full RBAC enforcement** — every API endpoint checks permissions

### Phase 5: Polish & Optimization (Ongoing)

16. Add resource graphs (historical CPU/RAM/disk)
17. Add notification preferences per user (email/Telegram for specific alerts)
18. Add dark mode toggle (already dark theme, but add theme options)
19. Mobile-responsive improvements
20. Performance optimization (lazy load, paginate large tables, caching)

---

## Architecture Notes for the Build Team

**Current architecture:**
- Static HTML pages served by Caddy (`ops.itpropartner.com`)
- All data from `ops-status.json` (generated every 5 min by Python collector)
- Vanilla JS fetches JSON and renders DOM
- Shared nav partial (`nav.html`) injected via `fetch()`
- Shared CSS (`ops.css`) and JS (`utils.js`)
- **No backend API** — purely static frontend reading JSON

**Recommended evolution:**
1. Keep static frontend for Phase 1 (quick action buttons can call API endpoints)
2. Add a lightweight backend (FastAPI or Node.js) for:
   - Authentication/sessions
   - Action endpoints (POST /api/restart-service, POST /api/trigger-backup, etc.)
   - Audit logging middleware
   - Role/permission checks
3. Options for backend:
   - **FastAPI** (Python) — matches existing ecosystem, easy to add SSH/cron exec
   - **Express/PocketBase** — simple, built-in auth
   - **Remote game** — use Caddy's `reverse_proxy` to a backend running on port 8084

**Data pipeline options for actions:**
- **Direct API call** from backend (best for Hetzner, Cloudflare, Stripe — they have HTTP APIs)
- **SSH execute** (for systemctl on Core and remote servers)
- **Hermes cron trigger** (for backup jobs — trigger the cron script directly)
- **Webhook** (for 3rd party services that support incoming webhooks)