Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
@@ -0,0 +1,549 @@
+# IT Pro Partner — Ops Portal Recommendations
+
+> Generated: 2026-07-09
+> Based on audit of current portal at `/var/www/ops/` and ops-status.json collection pipeline
+
+---
+
+## Table of Contents
+
+1. [Current State Summary](#1-current-state-summary)
+2. [Recommended Pages (New & Improved)](#2-recommended-pages)
+3. [Actions Per Page — What Users Should Be Able to Do](#3-actions-per-page)
+4. [Integrations Roadmap](#4-integrations-roadmap)
+5. [Data Already Flowing](#5-data-already-flowing)
+6. [RBAC Recommendations](#6-rbac-recommendations)
+7. [Implementation Priorities](#7-implementation-priorities)
+
+---
+
+## 1. Current State Summary
+
+### Existing Pages (all read-only, no action buttons)
+
+| Page | URL | Status | What It Shows |
+|------|-----|--------|---------------|
+| Dashboard | `/` | ✅ Live | Health bar, cron jobs, services, S3 backups, API health, server inventory, versions |
+| Servers | `/servers.html` | ✅ Live | netcup detail + Hetzner table (sortable), quick specs |
+| Backups | `/backups.html` | ✅ Live | S3 bucket status grid, backup schedules, Wasabi info, manual actions (text only) |
+| Network | `/network.html` | ✅ Live | DNS/Cloudflare zones, MikroTik (placeholder), Ubiquiti (placeholder) |
+| Services | `/services.html` | ✅ Live | Systemd service status grid, API health, SyncroMSP integration summary |
+| Cron Jobs | `/cron.html` | ✅ Live | Metrics overview, searchable job table with script viewer |
+| Logs & Events | `/logs.html` | ✅ Live | API timeline, service events, cron errors, failure summary with filters |
+| Config | `/config.html` | ✅ Live | Versions, config file viewer (Caddyfile, Hermes YAML), environment |
+| Template | `/template.html` | ✅ Minimal | Empty page with nav injection |
+
+### Key Gaps Identified
+
+- **No action/control buttons anywhere** — everything is read-only
+- **No customer management** — no client list, activations, lifecycle
+- **No billing/finance** — Stripe, Xero, invoices, payments
+- **No user/RBAC management** — no login, no roles
+- **No audit log** — no record of who did what
+- **No settings/integrations page** — API keys, integrations, branding
+- **MikroTik routers** — placeholder only
+- **Ubiquiti/UISP** — placeholder only
+- **No backup restore buttons**
+- **No service start/stop/restart**
+- **No provisioning actions** — no way to spin up a server or create a customer
+- **No DNS management actions** — cannot add/remove records
+- **No support ticket creation**
+
+---
+
+## 2. Recommended Pages
+
+### 2.1 Dashboard (Improve Existing)
+
+**Current:** Health bar, cron jobs, S3 backups, API health, server inventory, versions.
+
+**Improvements:**
+- Add **role-aware widgets** — show different cards depending on who's logged in
+- Add **quick-action toolbar** — "Provision new server", "New customer", "Create ticket"
+- Add **alert/incident summary** — current failures, threshold breaches
+- Add **recent activity feed** — last 10 actions taken in the portal
+- Add **revenue/ billing snapshot** for owner role
+- Add **weather/network status** at a glance
+- Add **pinned services** — most-watched services at top
+
+### 2.2 Server Management (Improve Existing)
+
+**Current:** netcup detail, Hetzner table with sorting, quick specs.
+
+**Improvements:**
+- Add **provisioning actions** per server:
+  - Restart server (soft reboot via Hetzner API)
+  - Power on/off
+  - Create snapshot
+  - Open SSH console (WebSSH / terminal-in-browser)
+  - View resource graphs (CPU, RAM, disk over time)
+- Add **threshold alerts per server** — show warning if disk/mem/CPU exceeds thresholds
+- Add **server tags/groups** — tag servers by role (web, db, ai, monitoring)
+- Add **migration status** — track Hetzner→netcup migration progress per server
+- Show **backup status per server** (last Hetzner snapshot date)
+- Show **Hudu link** for each server's documentation page
+- Add **network interface stats** — bandwidth per interface
+
+### 2.3 Customer Management (NEW)
+
+**Purpose:** One place to see all ITPP customers and their lifecycle.
+
+**Data sources:** SyncroMSP, Stripe, Hudu, DRE CRM
+
+**Display:**
+- Customer list/search table with: name, email, phone, plan/service tier, status (active/pending/suspended), billing status, last ticket date
+- Customer detail page with:
+  - Contact info
+  - Active services/subscriptions
+  - Ticket history (from SyncroMSP)
+  - Billing history (from Stripe)
+  - Device inventory (from SyncroMSP)
+  - Notes / tags
+  - Linked documentation (Hudu link)
+
+**Actions:**
+- Create new customer
+- Suspend/reactivate customer
+- Assign service tier
+- Create ticket on behalf of customer
+- View full billing history
+- Link to SyncroMSP, Hudu, Stripe
+
+### 2.4 Network (Improve Existing)
+
+**Current:** Cloudflare zones + placeholders for MikroTik and Ubiquiti.
+
+**Improvements:**
+
+**A) MikroTik Routers (unlock placeholder):**
+- Router list with: hostname, model, RouterOS version, uptime, CPU/load, memory, interfaces
+- Interface table: name, status (up/down), traffic (tx/rx), errors
+- Connected clients / DHCP leases
+- Firewall rules overview (count, active)
+- Backup status (link to MikroTik backup on Wasabi)
+
+**Actions:**
+- Reboot router
+- Run configuration backup
+- View interface traffic graphs
+- Enable/disable interface
+- View log tail
+
+**B) Ubiquiti / UISP (unlock placeholder):**
+- UISP device list: towers, access points, CPEs
+- Client counts per tower
+- Bandwidth utilization graphs
+- Device health (online/offline, signal strength)
+- UniFi controller status, access point health
+
+**Actions:**
+- Provision new CPE
+- Block/stick client
+- View real-time RF metrics
+- Trigger site survey
+
+**C) DNS & Domains (add actions):**
+Currently 13 Cloudflare zones already tracked:
+boxpilotlogistics.com, broadbandtx.com, debtrecoveryexperts.com, fleettracker360.com, forefrontisp.com, germainebrown.com, iamgmb.com, itpropartner.com, katiewattsdesign.com, localisp.com, texasinternetaccess.com, vigilanttac.com, voipsimplicity.com
+
+**Actions to add:**
+- Add/edit/delete DNS records (A, CNAME, MX, TXT, etc.)
+- Enable/disable proxy (Cloudflare orange cloud)
+- View DNS record count per zone
+- Trigger Cloudflare cache purge
+- Show SSL/TLS status per domain
+- Domain expiry dates (if registrar API connected)
+
+### 2.5 Backups (Improve Existing)
+
+**Current:** S3 bucket status grid, schedules, Wasabi info, text-only manual actions.
+
+**Improvements:**
+- Add **one-click restore** button per bucket (list snapshots, pick restore point)
+- Add **trigger backup now** button per bucket
+- Show **backup size** per bucket (not just status)
+- Add **restore logs** — history of recent restores
+- Add **backup retention policy** display (how long each bucket keeps data)
+- Add **integrity check results** — last verification timestamp and status
+- Add **restore to staging** — restore to a temp directory for verification
+
+**Actions:**
+- **Trigger backup** → runs the relevant cron script immediately
+- **Restore from backup** → pick a date/snapshot and restore
+- **Verify integrity** → run checksum verification
+- **Download archive** link to Wasabi console URL
+- **Set retention policy** input
+
+### 2.6 Services (Improve Existing)
+
+**Current:** Systemd service grid, API health cards, SyncroMSP summary.
+
+**Improvements:**
+- Add **start / stop / restart** buttons per service
+- Show **service logs** (journalctl tail)
+- Show **resource usage** per service (memory, CPU, uptime)
+- Add **dependencies** — what depends on this service being up
+- Add **enable/disable at boot** toggle
+- Add **service health history** — uptime %, last restart reason
+
+**Actions:**
+- Start service
+- Stop service
+- Restart service
+- View live logs (journalctl -u)
+- Enable/disable at boot
+- Reload configuration (for services that support it)
+
+### 2.7 Billing & Finance (NEW)
+
+**Purpose:** Business owner's view of revenue, invoices, and accounting sync.
+
+**Data sources:** Stripe, Xero
+
+**Display:**
+- **Revenue overview** — MTD, YTD, monthly breakdown (chart)
+- **Pending invoices** — outstanding, overdue
+- **Payment history** — recent transactions
+- **Customer billing summary** — per customer: plan, last payment, balance
+- **Subscription health** — active, past-due, canceled counts
+- **Accounting sync status** — last Xero sync, any errors
+
+**Actions:**
+- View invoice detail
+- Send invoice reminder
+- Mark invoice as paid (manual override)
+- Sync with Stripe
+- Sync with Xero
+- Export billing report (CSV)
+
+### 2.8 DNS & Domains (NEW dedicated page — or merge into Network)
+
+If kept as separate page:
+- All Cloudflare zones with DNS record counts
+- Domain registrar details (expiry, auto-renew)
+- Quick-add DNS record form
+- SSL/TLS settings per domain
+- Page Rules / redirect rules
+
+### 2.9 Users & RBAC (NEW)
+
+**Purpose:** Who can access the portal and what they can see/do.
+
+**Display:**
+- User list table: name, email, role, last login, status (active/disabled)
+- Role/permission matrix view
+- Audit log of user actions
+
+**Actions:**
+- Create user
+- Edit user role
+- Disable/delete user
+- View user activity log
+- Reset password / send invite
+
+### 2.10 Audit Log (NEW)
+
+**Purpose:** Record every action taken in the portal.
+
+**Display:**
+- Searchable, filterable timeline of events
+- Columns: timestamp, user, action, resource, details, IP
+- Exportable to CSV
+
+**Data captures:**
+- Who restarted a service
+- Who triggered a backup
+- Who created a customer
+- Who changed a DNS record
+- Login/logout events
+- Failed permission attempts
+
+### 2.11 Settings (NEW)
+
+**Purpose:** Configure the portal itself and connected integrations.
+
+**Sections:**
+- **Integrations** — enable/disable and configure API keys for:
+  - SyncroMSP
+  - Cloudflare
+  - Hetzner
+  - Stripe
+  - Xero
+  - Wasabi
+  - UniFi
+  - UISP
+  - MikroTik
+  - Hudu
+  - RingLogix
+  - RunCloud
+- **API Keys** — generate/manage API keys for programmatic portal access
+- **Branding** — logo, company name, colors, support phone/email
+- **Notifications** — configure alert email/Telegram destinations
+- **Backup retention** — set retention policies per bucket
+- **Data retention** — how long to keep logs, audit trail, events
+
+### 2.12 Support Tickets (NEW)
+
+**Purpose:** ITPP internal support ticket management.
+
+**Data source:** SyncroMSP
+
+**Display:**
+- Ticket queue: ID, customer, subject, status, priority, assignee, last update
+- Ticket detail: full conversation, attachments, time tracking
+- Ticket metrics: open/closed counts, average response time
+
+**Actions:**
+- Create ticket
+- Assign ticket
+- Update status
+- Add note/internal note
+- Close/resolve ticket
+- Link to customer profile
+
+### 2.13 DRE Claims Dashboard (NEW or link to internal DRE portal)
+
+The DRE portal already exists at `internal.debtrecoveryexperts.com`. On the Ops portal, add a **widget/card** on the dashboard showing:
+- Pending approvals count
+- Recent claims
+- Inbox message count (from dre-mail-poller)
+- Link through to the full DRE portal
+
+---
+
+## 3. Actions Per Page
+
+| Page | Read-Only Information | Actions Available |
+|------|----------------------|-------------------|
+| **Dashboard** | Health summary, recent failures, quick stats | Quick action toolbar, drill into alerts |
+| **Servers** | Server specs, status, IP, provider, type | **Restart**, power on/off, create snapshot, WebSSH, view resource graphs |
+| **Customers** | Contact info, services, tickets, bills | **Create**, suspend, reactivate, create ticket, view billing |
+| **Network** | Router status, interfaces, clients, DNS zones, device health | **Reboot router**, run backup, add DNS record, purge cache, enable/disable interface |
+| **Backups** | Bucket status, last upload, age, schedule | **Trigger backup**, **restore from snapshot**, verify integrity, set retention |
+| **Services** | Systemd status, API health, SyncroMSP stats | **Start / stop / restart**, view logs, enable/disable boot |
+| **Billing** | Revenue, invoices, payments, subscriptions | Sync Stripe, sync Xero, send invoice reminder, export CSV |
+| **DNS** | Zone list, record count, SSL status | **Add/Edit/Delete record**, toggle proxy, purge cache |
+| **Users/RBAC** | User list, roles, last login | **Create user**, edit role, disable, view audit log |
+| **Audit Log** | Full action timeline with user, action, resource | Search, filter, export |
+| **Settings** | Integration status, versions, env | **Add/rotate API keys**, enable/disable integrations, set branding |
+| **Support Tickets** | Queue, details, metrics | **Create**, assign, update status, add note, resolve |
+| **DRE** | Pending approvals, claims, inbox count | Link to full DRE portal |
+
+---
+
+## 4. Integrations Roadmap
+
+### Already Connected & Working
+
+| Integration | Status | Data Flow | Page(s) Using It |
+|-------------|--------|-----------|------------------|
+| **SyncroMSP** | ✅ Live | 26 customers, tickets, devices via API | Dashboard, Services |
+| **Cloudflare** | ✅ Live | 13 zones, DNS record status via API | Network, Dashboard |
+| **Hetzner** | ✅ Live | 10 servers, status, type, IP via API | Servers, Dashboard |
+| **netcup** | ✅ Live | Server detail (template, IP, id) via API/config | Servers |
+| **Wasabi** | ✅ Live | 6 backup buckets, last upload, age via AWS CLI | Backups, Dashboard |
+| **Admin-AI** | ✅ Live | API health check | Dashboard, Services |
+
+### In Progress — API Keys Saved, Need Portal UI
+
+| Integration | Status | What to Build |
+|-------------|--------|---------------|
+| **UniFi** | 🔶 Saved | Add UniFi client count, access point health, client list to Network page |
+| **UISP (UNMS)** | 🔶 Saved | Add tower/CPE status, bandwidth to Network page |
+
+### Planned — Need Connection & UI
+
+| Integration | Priority | What to Build |
+|-------------|----------|---------------|
+| **Stripe** | 🔴 High | Billing page: invoice list, revenue, subscription management |
+| **Xero** | 🔴 High | Billing page: accounting sync, P&L snapshot |
+| **MikroTik** | 🟡 Medium | Network page: RouterOS API for interface stats, DHCP, firewall |
+| **RingLogix** | 🟡 Medium | VoIP status, call metrics, DID inventory, phone management |
+| **Hudu** | 🟡 Medium | Link servers/customers to documentation, inline wiki viewer |
+| **RunCloud** | 🟢 Low | WordPress server management, site health, SSL |
+
+### Future / Nice-to-Have
+
+| Integration | Use Case |
+|-------------|----------|
+| **Porkbun / Namecheap** | Domain registration details, expiry dates, auto-renew |
+| **GitHub** | Deploy status, latest commits, CI pipeline status |
+| **Grafana** | Embed resource graphs, historical metrics |
+| **Uptime Kuma / Better Uptime** | External monitoring status, incident history |
+| **OpenAI / AI credits** | Usage tracking (Firecrawl already done, can add AI model usage) |
+
+---
+
+## 5. Data Already Flowing
+
+The ops-data-collector (Python, runs every 5 min) already collects:
+
+```json
+{
+  "timestamp": "ISO-8601",
+  "collection_duration_ms": 10363,
+  "overall": {
+    "total_jobs": 21,
+    "passing": 19,
+    "failing": 2,
+    "disk_used_pct": 6,
+    "memory_used_pct": 41
+  },
+  "cron_jobs": [ /* 21 jobs with name, schedule, lastRun, status, script */ ],
+  "services": { /* 8 systemd services: active/inactive */ },
+  "disk_memory": { "disk_used_pct", "memory_used_pct" },
+  "s3_backups": { /* 6 buckets: status, last_upload, age_hours */ },
+  "api_checks": { /* admin-ai, caddy-config, port checks, cloudflare */ },
+  "cloudflare_zones": [ /* 13 zone names + status */ ],
+  "versions": { /* hermes, caddy, python, os */ },
+  "routers": [ /* currently empty — placeholder */ ],
+  "netcup_server": { /* id, template, ip */ },
+  "hetzner_servers": [ /* 10 servers with name, status, type, ip, id */ ],
+  "syncromsp": { /* customers: 26, tickets, devices, customer_list */ }
+}
+```
+
+**What the collector CANNOT currently collect:**
+- MikroTik RouterOS data (API key needed + collect every 1-5 min)
+- Ubiquiti/UISP data (API key saved, but collector not extended yet)
+- Stripe billing data (no API key in collector)
+- Xero accounting data (no API key in collector)
+- Per-server resource usage beyond disk/memory (needs SSH or agent)
+- Historical metrics (needs time-series DB like Prometheus or simple log file)
+
+---
+
+## 6. RBAC Recommendations
+
+### 6.1 Roles
+
+| Role | Description |
+|------|-------------|
+| **Owner** | Full access — see everything, do everything. Germaine. |
+| **Admin** | Full access minus billing configuration and system settings. Tony. |
+| **Tech / Support** | Servers, services, network, logs, tickets, customers (read). No billing, provisioning, settings. |
+| **Sales / Onboarding** | Customer activation flow, order forms, provisioning status. No server internals, no billing. |
+| **Viewer / Auditor** | Read-only across limited scope. No actions, no billing, no settings. |
+| **DRE Agent** | DRE-specific pages only — claims, inbox, pending approvals. No infrastructure. Anita. |
+
+### 6.2 Page Access Matrix
+
+| Page / Feature | Owner | Admin | Tech/Support | Sales/Onboarding | Viewer | DRE Agent |
+|----------------|-------|-------|-------------|-----------------|--------|-----------|
+| Dashboard (full) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Dashboard (tech view) | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| Dashboard (sales view) | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| Dashboard (DRE view) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
+| Servers — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| Servers — restart/power | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Servers — snapshot/provision | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Customers — view | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| Customers — create/edit/suspend | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| Customers — billing detail | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Network — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| Network — reboot router/interface | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Network — DNS edit | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Backups — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| Backups — trigger/restore | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Services — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| Services — start/stop/restart | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Billing — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Billing — modify/sync | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Users/RBAC — manage | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Users/RBAC — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+| Audit Log — view | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Settings — all | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Settings — notification prefs | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
+| Tickets — view all | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
+| Tickets — create/assign | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
+| DRE Claims | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
+| Provisioning (new servers/customers) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
+
+### 6.3 Person-to-Role Mapping
+
+| Person | Role | Notes |
+|--------|------|-------|
+| **Germaine** (email@redacted) | **Owner** | Sees everything, can do everything. Needs billing, provisioning, and audit. |
+| **Tony** (email@redacted) | **Admin** | Full access minus billing config and user management. Handles server operations, backups, customer management. |
+| **Anita** | **DRE Agent** | DRE-focused view. Claims, pending approvals, inbox. No infrastructure. |
+| **Future Tech Hire** | **Tech/Support** | Servers, services, network, tickets. No billing, provisioning, settings. |
+| **Future Sales Hire** | **Sales/Onboarding** | Customer activation, provisioning status, order forms. No server internals. |
+| **Future Auditor** | **Viewer** | Read-only, limited scope. Good for compliance or investor review. |
+
+### 6.4 RBAC Implementation Notes
+
+- **Start simple** — use JWT or session tokens with role stored in the token
+- **No auth initially** — phase 1: keep open but label sections as "Owner Only" / "Tech Only"
+- **Phase 2**: add login with email/password or OAuth (Google/GitHub)
+- **Phase 3**: full RBAC enforcement at API/backend level
+- Consider using **Hudu's SSO** or **SyncroMSP's user model** as identity source to avoid yet another password
+- Store role assignments in a simple JSON config file or small SQLite DB
+
+---
+
+## 7. Implementation Priorities
+
+### Phase 1: Add Actions to Existing Pages (Weeks 1-3)
+
+1. **Backups** — add "Trigger Backup" and "Restore" buttons (calls existing scripts via SSH)
+2. **Services** — add Start/Stop/Restart buttons (calls systemctl over SSH)
+3. **Servers** — add Reboot + Snapshot buttons (calls Hetzner API)
+4. **Network** — add DNS record management (calls Cloudflare API)
+
+### Phase 2: Unlock Placeholders (Weeks 3-5)
+
+5. **MikroTik** — RouterOS API integration, live interface/uptime/leases data
+6. **Ubiquiti/UISP** — Connect API, show tower status and client counts
+7. **Dashboard improvements** — alert summary, recent activity, quick actions
+
+### Phase 3: New Pages — Foundation (Weeks 5-8)
+
+8. **Customer Management** — list/search/create customers, linked to SyncroMSP
+9. **Audit Log** — capture and display every action
+10. **Support Tickets** — SyncroMSP ticket queue integration
+11. **DRE Dashboard widget** — pending approvals, inbox count
+
+### Phase 4: New Pages — Business & Admin (Weeks 8-12)
+
+12. **Users & RBAC** — login, role assignment, permission enforcement
+13. **Billing & Finance** — Stripe revenue, invoice list, Xero sync
+14. **Settings** — integration management, API keys, branding
+15. **Full RBAC enforcement** — every API endpoint checks permissions
+
+### Phase 5: Polish & Optimization (Ongoing)
+
+16. Add resource graphs (historical CPU/RAM/disk)
+17. Add notification preferences per user (email/Telegram for specific alerts)
+18. Add dark mode toggle (already dark theme, but add theme options)
+19. Mobile-responsive improvements
+20. Performance optimization (lazy load, paginate large tables, caching)
+
+---
+
+## Architecture Notes for the Build Team
+
+**Current architecture:**
+- Static HTML pages served by Caddy (`ops.itpropartner.com`)
+- All data from `ops-status.json` (generated every 5 min by Python collector)
+- Vanilla JS fetches JSON and renders DOM
+- Shared nav partial (`nav.html`) injected via `fetch()`
+- Shared CSS (`ops.css`) and JS (`utils.js`)
+- **No backend API** — purely static frontend reading JSON
+
+**Recommended evolution:**
+1. Keep static frontend for Phase 1 (quick action buttons can call API endpoints)
+2. Add a lightweight backend (FastAPI or Node.js) for:
+   - Authentication/sessions
+   - Action endpoints (POST /api/restart-service, POST /api/trigger-backup, etc.)
+   - Audit logging middleware
+   - Role/permission checks
+3. Options for backend:
+   - **FastAPI** (Python) — matches existing ecosystem, easy to add SSH/cron exec
+   - **Express/PocketBase** — simple, built-in auth
+   - **Remote game** — use Caddy's `reverse_proxy` to a backend running on port 8084
+
+**Data pipeline options for actions:**
+- **Direct API call** from backend (best for Hetzner, Cloudflare, Stripe — they have HTTP APIs)
+- **SSH execute** (for systemctl on Core and remote servers)
+- **Hermes cron trigger** (for backup jobs — trigger the cron script directly)
+- **Webhook** (for 3rd party services that support incoming webhooks)