Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,119 @@
|
||||
# Web Standards & Accessibility — Ops Portal Checklist
|
||||
|
||||
Applied during the July 2026 audit to all 10 ops portal HTML files. Covers HTML5 validity, WCAG 2.1 AA, and ARIA compliance.
|
||||
|
||||
## Structural Requirements
|
||||
|
||||
### Document-Level
|
||||
- `<!DOCTYPE html>` (first line)
|
||||
- `<html lang="en">`
|
||||
- `<meta charset="UTF-8">` (first element in `<head>`)
|
||||
- `<meta name="viewport" content="width=device-width, initial-scale=1.0">`
|
||||
|
||||
### Semantic Elements
|
||||
| Element | When to Use |
|
||||
|---------|-------------|
|
||||
| `<main id="main-content" role="main">` | One per page, wraps primary content |
|
||||
| `<header>` | Page header (h1 + actions) |
|
||||
| `<footer>` | Page footer (last-updated timestamp) |
|
||||
| `<nav>` / `<nav role="navigation" aria-label="Main navigation">` | Navigation block |
|
||||
| `<section class="card" aria-labelledby="sectionId">` | Every card/section, paired with `<h2 id="sectionId">` |
|
||||
| `<h1>` → `<h2>` → `<h3>` | Strict hierarchy, no skipping |
|
||||
|
||||
### Section Card Pattern (before → after)
|
||||
|
||||
**Before (invalid):**
|
||||
```html
|
||||
<div class="card">
|
||||
<div class="card-header">Section Title</div>
|
||||
...
|
||||
</div>
|
||||
```
|
||||
|
||||
**After (valid):**
|
||||
```html
|
||||
<section class="card" aria-labelledby="myHeading">
|
||||
<h2 class="card-header" id="myHeading">Section Title</h2>
|
||||
...
|
||||
</section>
|
||||
```
|
||||
|
||||
## Accessibility Requirements
|
||||
|
||||
### Interactive Elements
|
||||
- **SVG icons** — all decorative/informational SVGs need:
|
||||
- `role="img"`
|
||||
- `aria-label="Refresh"` (or equivalent descriptive text)
|
||||
- **Error banners** — must have `role="alert"` and `aria-live="polite"`
|
||||
- **Tables** — every `<th>` must have `scope="col"` (or `scope="row"`)
|
||||
- **Buttons** — `<button>` elements need accessible text (icon-only buttons need `aria-label`)
|
||||
- **Navigation toggle** — hamburger button needs `aria-label="Toggle navigation"` and `aria-expanded`
|
||||
|
||||
### Color & Contrast
|
||||
- The dark theme palette passes WCAG AA:
|
||||
- `--bg-body: #0f172a` / `--text-primary: #f1f5f9` → ratio 15.1:1
|
||||
- `--accent: #f59e0b` (amber) on `--bg-card: #1e293b` → ratio 7.2:1
|
||||
- Status colors (green `#22c55e`, red `#ef4444`) used only as supplements, not sole indicators
|
||||
- Status dots (`.status-dot`) are always paired with text labels
|
||||
|
||||
### Focus & Keyboard
|
||||
- `.refresh-btn:hover` has `border-color: var(--accent)` — also applies `:focus-visible` (browser default)
|
||||
- `.filter-btn.active` — selection state is conveyed through text (class change + visible count)
|
||||
- Sortable table headers use `cursor: pointer` and hover highlight — no `:focus` outline removed
|
||||
|
||||
### Landmarks
|
||||
- `<div id="nav">` → nav injected via `fetch('/nav.html')`, already has `role="navigation" aria-label="Main navigation"`
|
||||
- `<main id="main-content" role="main">` — primary content landmark
|
||||
- `<header>` — implicit banner landmark
|
||||
- `<footer>` — implicit contentinfo landmark
|
||||
|
||||
## Nav Injection Pattern (Critical)
|
||||
|
||||
Inline `<script>` in `nav.html` **does not execute** when the file is loaded via `innerHTML`. Always:
|
||||
|
||||
1. **Place nav JS in `/js/utils.js`** as `window.initNav()`
|
||||
2. **Call `initNav()` after every nav injection:**
|
||||
|
||||
```html
|
||||
fetch('/nav.html')
|
||||
.then(r => r.text())
|
||||
.then(html => {
|
||||
document.getElementById('nav').innerHTML = html;
|
||||
if (typeof window.initNav === 'function') window.initNav();
|
||||
})
|
||||
```
|
||||
|
||||
`initNav()` handles:
|
||||
- Active link highlighting (via `data-path` attribute)
|
||||
- Hamburger toggle (`aria-expanded` management)
|
||||
- Click-outside-to-close
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Structural balance check
|
||||
for section balance:
|
||||
grep -Pc '<section\b' file.html
|
||||
grep -c '</section>' file.html
|
||||
|
||||
for main balance:
|
||||
grep -c '<main' file.html
|
||||
grep -c '</main>' file.html
|
||||
|
||||
for h2 balance:
|
||||
grep -Pc '<h2\b' file.html
|
||||
grep -c '</h2>' file.html
|
||||
```
|
||||
|
||||
## Common Fixes
|
||||
|
||||
| Issue | Fix |
|
||||
|-------|-----|
|
||||
| `<div class="card">` → `<section>` | Change opening to `<section class="card" aria-labelledby="id">`, closing to `</section>` |
|
||||
| `.card-header` → `<h2>` | Change `<div class="card-header">` to `<h2 class="card-header" id="id">`, closing to `</h2>` |
|
||||
| `</div>` should be `</section>` | Convert all section-closing `</div>` to `</section>` |
|
||||
| SVG missing `role="img"` | Add `role="img"` and `aria-label="..."` |
|
||||
| Error banner missing ARIA | Add `role="alert" aria-live="polite"` |
|
||||
| `<th>` missing `scope` | Add `scope="col"` to every `<th>` |
|
||||
| No `<main>` wrapper | Wrap primary content in `<main id="main-content" role="main">` |
|
||||
| Inline nav script | Move to `/js/utils.js` as `window.initNav()`, call after injection |
|
||||
@@ -0,0 +1,60 @@
|
||||
# Bulk node_exporter Install — July 2026 Rollout
|
||||
|
||||
## Server inventory
|
||||
|
||||
All 9 Hetzner servers received node_exporter v1.8.2 in a single session:
|
||||
|
||||
| Hostname | IP | Status |
|
||||
|----------|-----|--------|
|
||||
| app1-bu.itpropartner.com | 5.161.114.8 | Running |
|
||||
| ai.itpropartner.com | 178.156.167.181 | Running |
|
||||
| hudu.itpropartner.com | 178.156.130.130 | Running |
|
||||
| (unifi) | 178.156.131.57 | Running |
|
||||
| unms.forefrontwireless.com | 5.161.225.131 | Running |
|
||||
| wphost02 | 5.161.62.38 | Running |
|
||||
| app1.itpropartner.com | 87.99.144.163 | Running |
|
||||
| (docker) | 178.156.168.35 | Running |
|
||||
| (fleettrack360) | 178.156.149.32 | Running |
|
||||
|
||||
## Batch install pattern
|
||||
|
||||
Write two scripts:
|
||||
|
||||
1. **install-node-exporter.sh** — self-contained script that downloads, installs binary, creates systemd unit, enables/starts. The existing `scripts/install-node-exporter.sh` in this skill works.
|
||||
|
||||
2. **Wrapper script** — iterates over a list and runs SCP + SSH for each:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
SERVERS=(
|
||||
"hostname1:1.2.3.4"
|
||||
"hostname2:5.6.7.8"
|
||||
)
|
||||
KEY="/root/.ssh/itpp-infra"
|
||||
SCRIPT="/tmp/install-node-exporter.sh"
|
||||
|
||||
for entry in "${SERVERS[@]}"; do
|
||||
hostname="${entry%%:*}"
|
||||
ip="${entry##*:}"
|
||||
echo "=== Installing on $hostname ($ip) ==="
|
||||
scp -o StrictHostKeyChecking=no -i "$KEY" "$SCRIPT" "root@$ip:/tmp/"
|
||||
ssh -o StrictHostKeyChecking=no -i "$KEY" "root@$ip" "bash /tmp/install-node-exporter.sh"
|
||||
echo "✓ $hostname"
|
||||
done
|
||||
```
|
||||
|
||||
## Verifying all targets in Prometheus
|
||||
|
||||
After the config is updated and Prometheus reloaded:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
|
||||
"import json,sys;d=json.load(sys.stdin);[print(t['labels']['instance'],t['health']) for t in d['data']['activeTargets']]"
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Security scanners blocking raw IPs** in SSH/SCP commands. Using hostnames where available bypasses false-positive security warnings.
|
||||
- **All 9 installs can run sequentially** in ~90 seconds total — no parallelism needed.
|
||||
- **UFW/firewall** must allow incoming port 9100 from Core's IP, or Prometheus must scrape via hostname/SSH tunnel. The existing servers had no restrictive UFW rules blocking port 9100.
|
||||
- **node_exporter listens on all interfaces by default** (0.0.0.0:9100). If restricting to localhost, add `--web.listen-address=127.0.0.1:9100` to ExecStart and scrape via a local proxy or Tailscale.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Caddy Static File MIME Type Fix (Jul 12, 2026)
|
||||
|
||||
## Problem
|
||||
The ops portal at `ops.itpropartner.com` had JS and CSS files served with wrong MIME types (`application/json` instead of `text/javascript` / `text/css`). Browsers enforce correct MIME types for scripts and stylesheets — any JS file served as `application/json` is silently rejected. The entire `Ops` namespace never loaded, login overlays stayed visible, the mobile hamburger menu never worked, and pages appeared broken.
|
||||
|
||||
Root cause was twofold:
|
||||
|
||||
### 1. File permissions (the real fix)
|
||||
Caddy runs as the `caddy` system user. Files in `/opt/ops-portal/static/` were created by root with `chmod 600` — Caddy returned **403 Forbidden** for all static file requests. Even though a Caddy `handle_path` block was configured, the `file_server` handler couldn't read the files and returned 403.
|
||||
|
||||
**Fix:** `chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css`
|
||||
|
||||
### 2. Caddy handle_path nesting error (compounding factor)
|
||||
The `/js/*` and `/css/*` `handle_path` blocks were initially nested INSIDE the `/data/*` handler block due to a `sed` command error. They only triggered when the URL started with `/data/`, so `/js/app.js` never matched.
|
||||
|
||||
**Fix:** Extract to the top level of the Caddy site block, before `reverse_proxy`:
|
||||
```caddy
|
||||
ops.itpropartner.com {
|
||||
handle_path /data/* {
|
||||
root * /var/www/ops/data/
|
||||
file_server
|
||||
}
|
||||
handle_path /js/* {
|
||||
root * /opt/ops-portal/static/js
|
||||
file_server
|
||||
}
|
||||
handle_path /css/* {
|
||||
root * /opt/ops-portal/static/css
|
||||
file_server
|
||||
}
|
||||
reverse_proxy 127.0.0.1:8090
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Backend `media_type` workaround (not needed)
|
||||
The FastAPI backend has an `@app.get("/js/{file_path:path}")` route using `FileResponse`. Even with `media_type="application/javascript"` explicitly set, the response still returned `Content-Type: application/json` for unknown reasons (possibly a FastAPI 0.115.6 quirk, or the `media_type` parameter being ignored in certain response scenarios). The Caddy static file approach bypasses the backend entirely.
|
||||
|
||||
## Detection
|
||||
- Curl the JS URL: `curl -sI https://ops.itpropartner.com/js/app.js` — check `content-type` header
|
||||
- Expected: `text/javascript; charset=utf-8`
|
||||
- Broken: `application/json` (backend) or `text/html` (caddy error page) or empty (403/404)
|
||||
- Browser symptom: Pages load but no JS executes — login overlay stays visible, nav not working, "Ops is not defined" errors in console
|
||||
|
||||
## Prevention
|
||||
- Always `chmod 644` new static files added to `/opt/ops-portal/static/`
|
||||
- When adding a new Caddy `handle_path` block, verify it's at the TOP LEVEL of the site block, not indented inside another handler
|
||||
- After ANY Caddy config change: `caddy fmt --overwrite /etc/caddy/Caddyfile && caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
# Daily Feed Summary — Known Limitations
|
||||
|
||||
Repository file: `/root/.hermes/scripts/daily-feed-summary.py`
|
||||
Cron job: `daily-tech-digest` (11:00 daily via Hermes cron)
|
||||
|
||||
## Broken Sources
|
||||
|
||||
### Top Gear (`topgear.com/car-news`)
|
||||
- **Status:** [BROKEN] Scraping returns 0 articles
|
||||
- **Symptom:** The `TGLinkParser` finds no matching `<a href="/car-news/">` elements
|
||||
- **Likely cause:** Site structure changed — Top Gear uses a JS-heavy frontend (Next.js or similar) that renders article links dynamically. The current parser relies on finding `<a href="/car-news/...">` with a `title` attribute in the initial HTML, which no longer exists
|
||||
- **Impact:** Minor — Top Gear returns 0 articles silently. The digest continues without error
|
||||
- **Discovered:** Jul 6, 2026 (first documented failure)
|
||||
- **Fix required:** Switch to headless browser scraping (Playwright) or use an RSS/compat feed if available. LATER: Monitor if Top Gear adds RSS
|
||||
|
||||
## Python Dependencies
|
||||
|
||||
The script needs `feedparser` in the Hermes venv. See the SKILL.md "Hermes Cron Script Dependencies" section for install patterns.
|
||||
|
||||
## Verifying Full Output
|
||||
|
||||
```bash
|
||||
/root/hermes-assistant/venv/bin/python3 /root/.hermes/scripts/daily-feed-summary.py
|
||||
```
|
||||
|
||||
For email delivery:
|
||||
```bash
|
||||
/root/hermes-assistant/venv/bin/python3 /root/.hermes/scripts/daily-feed-summary.py --email
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
# Data Field Mapping — Collector Output vs Page JS Expectations
|
||||
|
||||
These are the most common post-W3C-audit failure points. Every time a page's JS is
|
||||
rewritten, re-verify these six paths against the ACTUAL JSON output.
|
||||
|
||||
## Canonical Mapping
|
||||
|
||||
| JS reads | Collector provides | Type |
|
||||
|----------|-------------------|------|
|
||||
| `data.cron_jobs` | `cron_jobs` | Array of {name, schedule, lastRun, status, script} |
|
||||
| `data.services` | `services` | Dict {service_name: status_string} |
|
||||
| `data.s3_backups` | `s3_backups` | Dict {bucket_name: {status, last_upload, age_hours}} |
|
||||
| `data.api_checks` | `api_checks` | Dict {check_name: status_string_or_emoji} |
|
||||
| `data.overall.disk_used_pct` | `overall.disk_used_pct` | Integer (0-100) |
|
||||
| `data.overall.memory_used_pct` | `overall.memory_used_pct` | Integer (0-100) |
|
||||
| `data.cloudflare_zones` | `cloudflare_zones` | Array of {name, status} |
|
||||
| `data.netcup_server` | `netcup_server` | Dict {id, template, ip} |
|
||||
| `data.hetzner_servers` | `hetzner_servers` | Array of {name, type, status, ip, id} |
|
||||
| `data.syncromsp` | `syncromsp` | Dict {status, customers, tickets, customer_list} |
|
||||
| `job.lastRun` | `lastRun` | String (ISO 8601 with TZ offset) — NOT `last_run` |
|
||||
|
||||
## Common wrong expectations (DO NOT use)
|
||||
- `scheduled_jobs` — use `cron_jobs`
|
||||
- `api_health` — use `api_checks` (dict, not array)
|
||||
- `disk_used_pct` at top level — it's inside `overall`
|
||||
- `memory_used_pct` at top level — it's inside `overall`
|
||||
- `job.last_run` (snake_case) — use `job.lastRun` (camelCase)
|
||||
- `services` as flat array — it's a dict, iterate `Object.keys(services)`
|
||||
- `s3_backups` as array — it's a dict {bucket_name: content}
|
||||
- `api_checks` as array — it's a dict {check_name: status_string}
|
||||
|
||||
## Root cause
|
||||
The W3C subagent writes new markup and JS from the task brief's schema description,
|
||||
not from the actual JSON file on disk. Always read the live file before writing JS.
|
||||
|
||||
## Verification command
|
||||
```bash
|
||||
python3 -c "import json; d=json.load(open('/var/www/ops/data/ops-status.json')); cj=d['cron_jobs'][0]; print('lastRun:', cj.get('lastRun','MISSING')); print('last_run:', cj.get('last_run','check')); print('services type:', type(d['services']).__name__)"
|
||||
```
|
||||
|
||||
## Post-audit checklist
|
||||
After ANY page markup change, before declaring complete:
|
||||
1. Run the verification command above
|
||||
2. Check all JS field references match the actual JSON output
|
||||
3. curl the page for HTTP 200
|
||||
4. curl the data endpoint for valid JSON
|
||||
5. Verify nav injection works (check for initNav in both the page and utils.js)
|
||||
6. Verify files have 644 perms (Caddy returns 403 on 600 files)
|
||||
@@ -0,0 +1,222 @@
|
||||
# FastAPI Ops Portal Backend Pattern
|
||||
|
||||
Reference architecture for the FastAPI backend built Jul 9, 2026 at `/opt/ops-portal/server.py`.
|
||||
|
||||
## Route ordering (CRITICAL)
|
||||
|
||||
FastAPI processes routes in definition order. The route list must be:
|
||||
|
||||
1. **API routes first** (`/api/auth/login`, `/api/health`, `/api/status`, ...)
|
||||
2. **Metrics** (`/metrics`)
|
||||
3. **Static assets** (`/css/{path}`, `/js/{path}`)
|
||||
4. **Page routes** (`/`, `/{page}.html`)
|
||||
|
||||
If a catch-all `/{page}.html` comes before `/api/*`, ALL API calls return "Page not found".
|
||||
|
||||
## Static file serving pattern
|
||||
|
||||
```python
|
||||
# Do NOT use app.mount("/", StaticFiles(...)) — this swallows API routes.
|
||||
# Instead, serve explicitly:
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
|
||||
@app.get("/css/{file_path:path}")
|
||||
async def serve_css(file_path: str):
|
||||
fpath = STATIC_DIR / "css" / file_path
|
||||
if fpath.exists() and fpath.is_file():
|
||||
return FileResponse(str(fpath))
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
|
||||
@app.get("/js/{file_path:path}")
|
||||
async def serve_js(file_path: str):
|
||||
fpath = STATIC_DIR / "js" / file_path
|
||||
if fpath.exists() and fpath.is_file():
|
||||
return FileResponse(str(fpath))
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
|
||||
@app.get("/")
|
||||
async def serve_index():
|
||||
index_path = STATIC_DIR / "index.html"
|
||||
if index_path.exists():
|
||||
return FileResponse(str(index_path))
|
||||
return JSONResponse({"status": "ok", "message": "Ops Portal API ready"})
|
||||
|
||||
@app.get("/{page}.html")
|
||||
async def serve_html_page(page: str):
|
||||
fpath = STATIC_DIR / f"{page}.html"
|
||||
if fpath.exists():
|
||||
return FileResponse(str(fpath))
|
||||
raise HTTPException(status_code=404, detail="Page not found")
|
||||
```
|
||||
|
||||
## JWT auth pattern
|
||||
|
||||
```python
|
||||
from jose import jwt, JWTError
|
||||
from fastapi import Depends, HTTPException, Request as FastAPIRequest
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
ADMIN_USERNAME = "germaine"
|
||||
ADMIN_PASSWORD = "..." # from .env
|
||||
JWT_SECRET = "..." # from .env
|
||||
JWT_ALGORITHM = "HS256"
|
||||
JWT_EXPIRE_HOURS = 24
|
||||
|
||||
def create_jwt(username: str) -> str:
|
||||
payload = {
|
||||
"sub": username,
|
||||
"iat": datetime.now(timezone.utc),
|
||||
"exp": datetime.now(timezone.utc) + timedelta(hours=JWT_EXPIRE_HOURS),
|
||||
"jti": secrets.token_hex(16),
|
||||
}
|
||||
return jwt.encode(payload, JWT_SECRET, algorithm=JWT_ALGORITHM)
|
||||
|
||||
async def get_current_user(request: FastAPIRequest) -> str:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
if not auth.startswith("Bearer "):
|
||||
raise HTTPException(status_code=401, detail="Missing or invalid Authorization header")
|
||||
token = auth[7:]
|
||||
payload = jwt.decode(token, JWT_SECRET, algorithms=[JWT_ALGORITHM])
|
||||
return payload["sub"]
|
||||
|
||||
# Usage:
|
||||
@app.get("/api/status")
|
||||
async def get_status(user: str = Depends(get_current_user)):
|
||||
...
|
||||
```
|
||||
|
||||
## Audit log pattern (SQLite)
|
||||
|
||||
```python
|
||||
import sqlite3
|
||||
|
||||
DB_PATH = "/opt/ops-portal/ops.db"
|
||||
|
||||
def get_db():
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
return conn
|
||||
|
||||
def init_db():
|
||||
conn = get_db()
|
||||
conn.executescript("""
|
||||
CREATE TABLE IF NOT EXISTS audit_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
user TEXT NOT NULL,
|
||||
action TEXT NOT NULL,
|
||||
target TEXT,
|
||||
result TEXT NOT NULL,
|
||||
detail TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS api_tokens (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
user TEXT NOT NULL,
|
||||
token_hash TEXT NOT NULL UNIQUE,
|
||||
created TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
last_used TEXT
|
||||
);
|
||||
""")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def log_audit(user: str, action: str, target, result: str, detail=None):
|
||||
conn = get_db()
|
||||
conn.execute(
|
||||
"INSERT INTO audit_log (user, action, target, result, detail) VALUES (?, ?, ?, ?, ?)",
|
||||
(user, action, target, result, detail)
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
```
|
||||
|
||||
## Service control pattern (systemd)
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
from fastapi import HTTPException
|
||||
|
||||
@app.post("/api/services/{name}/restart")
|
||||
async def restart_service(name: str, user: str = Depends(get_current_user)):
|
||||
svc = name if name.endswith(".service") else f"{name}.service"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["systemctl", "restart", svc],
|
||||
capture_output=True, text=True, timeout=30
|
||||
)
|
||||
success = result.returncode == 0
|
||||
log_audit(user, "restart_service", name, "ok" if success else "error", result.stderr[:500])
|
||||
if success:
|
||||
return {"status": "ok", "message": f"Service {name} restarted"}
|
||||
return JSONResponse(status_code=500, content={"status": "error", "message": result.stderr[:500]})
|
||||
except subprocess.TimeoutExpired:
|
||||
raise HTTPException(status_code=504, detail="Service restart timed out")
|
||||
```
|
||||
|
||||
## Prometheus counters
|
||||
|
||||
```python
|
||||
from prometheus_client import Counter, generate_latest, CONTENT_TYPE_LATEST
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
HTTP_REQUESTS = Counter("ops_portal_http_requests_total", "Total HTTP requests", ["method", "endpoint"])
|
||||
SERVICE_ACTIONS = Counter("ops_portal_service_actions_total", "Service actions", ["action", "service"])
|
||||
BACKUP_TRIGGERS = Counter("ops_portal_backup_triggers_total", "Backup triggers", ["bucket"])
|
||||
|
||||
@app.get("/metrics")
|
||||
async def metrics():
|
||||
return JSONResponse(
|
||||
content=generate_latest().decode(),
|
||||
media_type=CONTENT_TYPE_LATEST
|
||||
)
|
||||
```
|
||||
|
||||
## .env file layout
|
||||
|
||||
```
|
||||
# /opt/ops-portal/.env — chmod 600
|
||||
ADMIN_USERNAME=germaine
|
||||
ADMIN_PASSWORD=...
|
||||
JWT_SECRET=...
|
||||
CLOUDFLARE_API_TOKEN=...
|
||||
CLOUDFLARE_EMAIL=...
|
||||
HETZNER_API_TOKEN=...
|
||||
```
|
||||
|
||||
## Systemd service
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=ITPP Ops Portal Backend
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/ops-portal
|
||||
ExecStart=/opt/awscli-venv/bin/uvicorn server:app --host 127.0.0.1 --port 8090
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
EnvironmentFile=/opt/ops-portal/.env
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
## Caddy routing
|
||||
|
||||
```caddy
|
||||
ops.itpropartner.com {
|
||||
reverse_proxy 127.0.0.1:8090
|
||||
encode gzip
|
||||
log {
|
||||
output file /var/log/caddy/ops.log
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The entire Caddy block is just a reverse proxy — no static file serving, no API matchers. The FastAPI backend handles everything.
|
||||
@@ -0,0 +1,182 @@
|
||||
# Frontend Auth & Action Pattern — Ops Portal
|
||||
|
||||
Created: 2026-07-09
|
||||
Applies to: `/opt/ops-portal/static/` pages (index.html, servers.html, backups.html, services.html)
|
||||
|
||||
## Architecture
|
||||
|
||||
Each page is a self-contained HTML file served by FastAPI with:
|
||||
1. Inline login overlay (no separate login page)
|
||||
2. Inline nav (not fetched via nav.html partial — auth-aware logout button is tightly coupled)
|
||||
3. Single JS module: `/js/app.js` (exposed as `window.Ops`)
|
||||
|
||||
## JS Module (`/js/app.js`)
|
||||
|
||||
### Auth
|
||||
|
||||
```javascript
|
||||
// Login: stores JWT in localStorage['ops_token']
|
||||
await Ops.login('username', 'password');
|
||||
|
||||
// Check auth state
|
||||
if (Ops.isAuthenticated()) { ... }
|
||||
|
||||
// Logout — clears storage + page reload
|
||||
Ops.logout();
|
||||
```
|
||||
|
||||
### API Client
|
||||
|
||||
```javascript
|
||||
// All calls auto-attach Authorization header from stored token
|
||||
var data = await Ops.apiFetch('/api/status');
|
||||
var servers = await Ops.getServers();
|
||||
var audit = await Ops.getAuditLog(10);
|
||||
|
||||
// Service actions (all POST with JWT)
|
||||
await Ops.restartService('caddy');
|
||||
await Ops.stopService('hermes');
|
||||
await Ops.startService('ops-portal');
|
||||
|
||||
// Backup trigger
|
||||
await Ops.triggerBackup('hermes-vps-backups');
|
||||
|
||||
// Server reboot (by Hetzner server ID)
|
||||
await Ops.rebootServer(51140464);
|
||||
```
|
||||
|
||||
**401 handling:** `apiFetch()` automatically clears localStorage and calls `window.location.reload()` on 401 responses, showing the login overlay again.
|
||||
|
||||
### UI Components
|
||||
|
||||
```javascript
|
||||
// Toast notification (auto-dismiss 5s)
|
||||
Ops.toast('Server reboot initiated', 'success'); // 'success' | 'error' | 'info'
|
||||
|
||||
// Confirmation dialog (returns Promise<boolean>)
|
||||
var ok = await Ops.confirmDialog(
|
||||
'Reboot Server',
|
||||
'Are you sure you want to reboot wphost02?',
|
||||
'Reboot',
|
||||
'Cancel'
|
||||
);
|
||||
|
||||
// Button loading state
|
||||
Ops.spinning(document.getElementById('refreshBtn'), true);
|
||||
Ops.spinning(document.getElementById('refreshBtn'), false);
|
||||
|
||||
// Nav initialization (hamburger toggle + active link + logout)
|
||||
Ops.initNav();
|
||||
|
||||
// Formatting utilities
|
||||
Ops.fmtTime('2026-07-09T12:22:28-04:00'); // "Jul 9, 12:22 PM"
|
||||
Ops.fmtAge('...'); // "3h 12m ago"
|
||||
Ops.badgeHtml('ok'); // '<span class="badge badge-ok">✓ ok</span>'
|
||||
Ops.dotColor('running'); // 'dot-green'
|
||||
Ops.escapeHtml('<script>'); // '<script>'
|
||||
Ops.valOrDash(null); // '—'
|
||||
Ops.updateLastUpdated(); // Sets #lastUpdated textContent
|
||||
```
|
||||
|
||||
### Toast container (lazy-created)
|
||||
|
||||
If `#toastContainer` doesn't exist, `Ops.toast()` creates it as a `<div>` appended to `document.body`. Position: `fixed; top: 20px; right: 20px; z-index: 10001`. Styled via `ops.css` `.toast`, `.toast-success`, `.toast-error`, `.toast-info`.
|
||||
|
||||
## Page Template Pattern
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
|
||||
<meta name="theme-color" content="#0f1628">
|
||||
<title>Page — Operations Dashboard</title>
|
||||
<link rel="stylesheet" href="/css/ops.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<!-- Login Overlay -->
|
||||
<div class="login-overlay show" id="loginOverlay">
|
||||
<div class="login-card">
|
||||
<!-- logo, title, error div, form with #loginUser + #loginPass -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Navigation (inline, not fetched) -->
|
||||
<nav class="ops-nav" id="opsNav">
|
||||
<!-- brand, nav-links with SVG icons, logout button -->
|
||||
</nav>
|
||||
|
||||
<div class="page-wrapper">
|
||||
<header class="page-header">
|
||||
<h1>Page Title</h1>
|
||||
<div class="header-actions">
|
||||
<span class="last-updated" id="lastUpdated">—</span>
|
||||
<button class="refresh-btn" onclick="loadPageData()">Refresh</button>
|
||||
</div>
|
||||
</header>
|
||||
<!-- card sections with data -->
|
||||
</div>
|
||||
|
||||
<script src="/js/app.js"></script>
|
||||
<script>
|
||||
(function() {
|
||||
'use strict';
|
||||
|
||||
// Auth guard
|
||||
if (!Ops.isAuthenticated()) {
|
||||
document.getElementById('loginOverlay').classList.add('show');
|
||||
} else {
|
||||
document.getElementById('logoutBtn').classList.add('show');
|
||||
loadPageData();
|
||||
}
|
||||
|
||||
// Login handler
|
||||
window.handleLogin = async function(e) { ... await Ops.login(...) ... };
|
||||
|
||||
// Data loading
|
||||
window.loadPageData = async function() { ... };
|
||||
|
||||
// Action handlers with confirmation
|
||||
window.handleAction = async function(...) {
|
||||
if (await Ops.confirmDialog(...)) { ... await Ops.apiCall(...) ... Ops.toast(...) }
|
||||
};
|
||||
|
||||
Ops.initNav();
|
||||
// Auto-refresh
|
||||
setInterval(function() {
|
||||
if (Ops.isAuthenticated()) loadPageData();
|
||||
}, 60000);
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
## Files created Jul 9, 2026
|
||||
|
||||
| File | Size | Sections |
|
||||
|------|------|----------|
|
||||
| `/opt/ops-portal/static/index.html` | 19.3 KB | Health grid, Quick actions (restart/trigger), Servers summary, Cron jobs, Audit log |
|
||||
| `/opt/ops-portal/static/servers.html` | 15.0 KB | Quick specs (vCPUs/RAM), Sortable server table with click-to-expand details, Reboot button per server |
|
||||
| `/opt/ops-portal/static/backups.html` | 15.1 KB | Summary bar, Bucket grid with Trigger button, Schedule table |
|
||||
| `/opt/ops-portal/static/services.html` | 14.7 KB | Service control grid (Start/Stop/Restart per service), API health grid, Service audit log |
|
||||
| `/opt/ops-portal/static/css/ops.css` | 27.0 KB | Full theme with auth-overlay, confirm-dialog, toast, action-card, svc-card styles |
|
||||
| `/opt/ops-portal/static/js/app.js` | 11.6 KB | Auth module, API client, confirmation dialog, toast, utilities |
|
||||
|
||||
## CSS classes for auth UI (in ops.css)
|
||||
|
||||
- `.login-overlay` — full-screen fixed overlay, `display: none` by default, `.show` sets `display: flex`
|
||||
- `.login-card` — centered dialog, max-width 400px, dark card with logo + form
|
||||
- `.login-error` — red alert box, hidden by default, `.show` reveals it
|
||||
- `.confirm-overlay` — modal backdrop with blur, `.show` = `display: flex`
|
||||
- `.confirm-dialog` — centered card with title + message + action buttons
|
||||
- `.toast-container` — fixed top-right notification area
|
||||
- `.toast`, `.toast-success`, `.toast-error`, `.toast-info` — slide-in animations
|
||||
|
||||
## Key quirks
|
||||
|
||||
- **Login overlay is always in the HTML** — it's hidden by CSS `display: none`. The JS just toggles `.show` class. No page redirect on login; the overlay hides and the page stays.
|
||||
- **Auto-refresh stops when not authenticated** — every `setInterval` callback checks `Ops.isAuthenticated()` before fetching. Stale data stays visible until login.
|
||||
- **Caddy reload** after config change: `caddy reload --config /etc/caddy/Caddyfile` or `systemctl reload caddy`. Full restart (`systemctl stop/start`) only needed when adding new Caddy hosts, not modifying existing ones.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Grafana Dashboard API Patterns
|
||||
|
||||
## Creating a dashboard via REST API
|
||||
|
||||
After the Prometheus data source is created, provision dashboards via POST to `/api/dashboards/db`:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:3002/api/dashboards/db \
|
||||
-H "Content-Type: application/json" \
|
||||
-u admin:admin \
|
||||
-d @dashboard.json
|
||||
```
|
||||
|
||||
Response: `{"id":..., "uid":"...", "slug":"...", "status":"success", "url":"/d/uid/title"}`
|
||||
|
||||
## Common node_exporter panel queries
|
||||
|
||||
### CPU Usage (stat or timeseries)
|
||||
```
|
||||
avg by(instance) (1 - rate(node_cpu_seconds_total{mode='idle'}[5m]))
|
||||
```
|
||||
|
||||
### RAM Usage (gauge or timeseries)
|
||||
```
|
||||
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
|
||||
```
|
||||
Bytes variant: `node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes`
|
||||
|
||||
### Disk Usage (gauge)
|
||||
```
|
||||
1 - (node_filesystem_avail_bytes{mountpoint='/'} / node_filesystem_size_bytes{mountpoint='/'})
|
||||
```
|
||||
|
||||
### Uptime (stat)
|
||||
```
|
||||
time() - node_boot_time_seconds
|
||||
```
|
||||
|
||||
## Panel layout reference (gridPos)
|
||||
|
||||
The dashboard grid is 24 columns wide. Common layouts:
|
||||
|
||||
| Panel | gridPos | Width |
|
||||
|-------|---------|-------|
|
||||
| Single stat | `{h:8, w:6, x:0, y:0}` | 1/4 width |
|
||||
| Timeseries full | `{h:10, w:12, x:0, y:8}` | 1/2 width |
|
||||
| Two stat row | `{h:8, w:12, x:0, y:0}` + `{h:8, w:12, x:12, y:0}` | 1/2 each |
|
||||
|
||||
## Adding a Prometheus data source
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:3002/api/datasources \
|
||||
-H "Content-Type: application/json" \
|
||||
-u admin:admin \
|
||||
-d '{
|
||||
"name":"Prometheus",
|
||||
"type":"prometheus",
|
||||
"url":"http://127.0.0.1:9090",
|
||||
"access":"proxy",
|
||||
"isDefault":true
|
||||
}'
|
||||
```
|
||||
|
||||
Response: `{"datasource":{"id":1,"uid":"...","name":"Prometheus",...},"message":"Datasource added"}`
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Auth**: Use `-u admin:admin` (or whatever password was set with `GF_SECURITY_ADMIN_PASSWORD`). Default is `admin/admin`.
|
||||
- **Default data source**: Set `"isDefault":true` so panels don't need an explicit data source reference.
|
||||
- **Overwriting**: When re-importing, send `"overwrite": true` in the payload body (at the same level as `"dashboard"`).
|
||||
- **Time range**: Set `"time": {"from": "now-6h", "to": "now"}` and `"refresh": "30s"` at the `dashboard` level.
|
||||
@@ -0,0 +1,72 @@
|
||||
# iOS PWA Push Notification Testing Guide
|
||||
|
||||
Debugging web push notifications for the shark game on iOS.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Load the test page at https://shark.iamgmb.com/push-test.html
|
||||
2. Share -> "Add to Home Screen"
|
||||
3. Open from home screen (not from Safari browser tab)
|
||||
|
||||
## Expected subscription flow
|
||||
|
||||
```
|
||||
Browser supports push -> SW registered -> Permission granted -> Push subscribed -> Subscription sent to server
|
||||
```
|
||||
|
||||
## If subscribe succeeds but banner is invisible
|
||||
|
||||
The push opt-in `<div id="pushBanner" style="display:none">` is hardcoded hidden in HTML. The `showNotificationBanner()` function must call `banner.style.display = 'block'` — without this explicit line, the banner state variable changes but the UI never reveals the prompt.
|
||||
|
||||
**Check:** Does `showNotificationBanner()` have `banner.style.display = 'block';` at the top of the function body?
|
||||
|
||||
## iOS PWA push limitation
|
||||
|
||||
PushManager is NOT available in Safari's browser tab on iOS. The `FAIL: No PushManager` error is expected behavior on iPhone Safari. The user MUST:**
|
||||
|
||||
1. Tap Share -> "Add to Home Screen"
|
||||
2. Open from home screen (PWA mode, fullscreen, no URL bar)
|
||||
3. Then push subscription works via Apple Push Service (APNs)
|
||||
|
||||
## Common subscription failures
|
||||
The `user_id` column has a FK constraint to `users.id`. Test subscriptions use `user_id: NULL` — ensure the column allows NULL and the subscribe-noauth endpoint passes NULL.
|
||||
|
||||
Fix:
|
||||
```sql
|
||||
-- Rebuild table with nullable user_id
|
||||
CREATE TABLE push_subscriptions_new (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
user_id INTEGER,
|
||||
endpoint TEXT NOT NULL UNIQUE,
|
||||
p256dh TEXT NOT NULL,
|
||||
auth TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
INSERT INTO push_subscriptions_new SELECT * FROM push_subscriptions;
|
||||
DROP TABLE push_subscriptions;
|
||||
ALTER TABLE push_subscriptions_new RENAME TO push_subscriptions;
|
||||
```
|
||||
|
||||
### Server rejected subscription
|
||||
Check that the subscribe-noauth endpoint exists and accepts the POST body:
|
||||
```json
|
||||
{"endpoint": "https://...", "p256dh": "...", "auth": "..."}
|
||||
```
|
||||
|
||||
## Test send after subcription
|
||||
|
||||
```bash
|
||||
curl -s -X POST https://shark.iamgmb.com/api/push/test \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
Expected: `{"sent": 1}`
|
||||
Failure: `{"sent": 0}` — check journald for `Push test failed: ...`
|
||||
|
||||
## Debugging database
|
||||
|
||||
```bash
|
||||
sqlite3 /root/shark-game/backend/game.db "SELECT count(*) FROM push_subscriptions;"
|
||||
sqlite3 /root/shark-game/backend/game.db "SELECT id, endpoint FROM push_subscriptions;"
|
||||
```
|
||||
@@ -0,0 +1,57 @@
|
||||
# Ops Portal Jul 12 Cleanup Audit
|
||||
|
||||
42 bugs found across 11 pages by Claude Sonnet 4-6 subagent.
|
||||
|
||||
## Root Causes (5 issues causing 80% of bugs)
|
||||
|
||||
### 1. `/js/utils.js` 404
|
||||
`utils.js` existed at `/var/www/ops/js/utils.js` but NOT at `/opt/ops-portal/static/js/` (the backend's JS directory). Every page referenced `/js/utils.js` which returned 404 via Caddy's `/js/*` file_server handler. Fix: `cp /var/www/ops/js/utils.js /opt/ops-portal/static/js/utils.js`
|
||||
|
||||
### 2. `window.initNav` undefined
|
||||
`initNav()` is exported as `window.Ops.initNav` (via `app.js`), not `window.initNav`. Every page calling `window.initNav()` calls undefined. Fix: Pages must call `Ops.initNav()`, or export `window.initNav = initNav` in app.js.
|
||||
|
||||
### 3. `logoutBtn` DOM lookup before nav fetch
|
||||
Pages services.html, servers.html, backups.html call `document.getElementById('logoutBtn')` synchronously before the `fetch('/nav.html')` callback completes. The nav (and `logoutBtn`) doesn't exist yet → TypeError. Fix: Move all `getElementById('logoutBtn')` calls inside the fetch callback.
|
||||
|
||||
### 4. Pages missing required script files
|
||||
Some pages loaded only `app.js` (missing `utils.js`), others loaded only `utils.js` (missing `app.js`). Fix: EVERY page must load BOTH:
|
||||
```html
|
||||
<script src="/js/utils.js?v=2"></script>
|
||||
<script src="/js/app.js?v=2"></script>
|
||||
```
|
||||
|
||||
### 5. No auth guard on 6 pages
|
||||
network.html, cron.html, config.html, logs.html, cost.html, fleettracker360.html have no login overlay or auth check. Fix: Add login overlay + `Ops.isAuthenticated()` guard.
|
||||
|
||||
## Critical Bugs by Page
|
||||
|
||||
| Page | Critical Bugs | File Path |
|
||||
|------|------|-----------|
|
||||
| index.html | 1 | /opt/ops-portal/static/index.html |
|
||||
| services.html | 3 | /opt/ops-portal/static/services.html |
|
||||
| servers.html | 2 | /opt/ops-portal/static/servers.html |
|
||||
| backups.html | 2 | /opt/ops-portal/static/backups.html |
|
||||
| network.html | 2 | /opt/ops-portal/static/network.html |
|
||||
| cron.html | 2 | /opt/ops-portal/static/cron.html |
|
||||
| config.html | 4 | /opt/ops-portal/static/config.html |
|
||||
| logs.html | 2 | /opt/ops-portal/static/logs.html |
|
||||
| audit.html | 2 | /opt/ops-portal/static/audit.html |
|
||||
| cost.html | 3 | /opt/ops-portal/static/cost.html |
|
||||
| fleettracker360.html | 0 | /opt/ops-portal/static/fleettracker360.html |
|
||||
|
||||
## Quick Fixes Applied Jul 12
|
||||
- Copied utils.js to backend static dir → 404 fixed
|
||||
- Added cache busting `?v=2` to all JS/CSS
|
||||
- Fixed Grafana link from 127.0.0.1 to Tailscale IP
|
||||
- Added z-index: 1000 to nav-links
|
||||
- Added inline onclick fallback to nav-toggle
|
||||
- Hid "Invalid credentials" error text by default
|
||||
- Both JS files added to all 10 pages
|
||||
|
||||
## Remaining Work
|
||||
- Export `window.initNav` properly
|
||||
- Fix `logoutBtn` timing on services/servers/backups
|
||||
- Add auth guards to 6 unprotected pages
|
||||
- Fix `window.fetchStatus` on cost.html
|
||||
- Fix `window.escapeHtml` on config.html
|
||||
- Fix audit.html `sessionStorage` → `localStorage`
|
||||
@@ -0,0 +1,86 @@
|
||||
# Komodo Deployment (July 2026)
|
||||
|
||||
Komodo Core v2.2.0 deployed on Core (netcup RS 2000) for multi-host Docker management.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Core** — Management server, installed as `/root/docker/komodo/` with docker-compose (MongoDB + Komodo Core)
|
||||
- **Periphery** — Agent installed on each managed host via `setup-periphery.py` script
|
||||
- **Port:** 9120
|
||||
- **Domain:** https://komodo.itpropartner.com
|
||||
- **Default login:** admin / changeme
|
||||
|
||||
## Deployment Details
|
||||
|
||||
### docker-compose.yml (v2.2.0 env vars)
|
||||
```yaml
|
||||
services:
|
||||
mongo:
|
||||
image: mongo
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./data/mongo-data:/data/db
|
||||
networks:
|
||||
- komodo-net
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 1G
|
||||
|
||||
core:
|
||||
image: ghcr.io/moghtech/komodo-core:2
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- 9120:9120
|
||||
depends_on:
|
||||
- mongo
|
||||
volumes:
|
||||
- ./backups:/backups
|
||||
- ./keys:/config/keys
|
||||
environment:
|
||||
- KOMODO_DATABASE_ADDRESS=mongo:27017
|
||||
- KOMODO_LOCAL_AUTH=true
|
||||
- KOMODO_INIT_ADMIN_USERNAME=admin
|
||||
- KOMODO_INIT_ADMIN_PASSWORD=changeme
|
||||
- KOMODO_FIRST_SERVER_NAME=core
|
||||
- KOMODO_PERIPHERY_PUBLIC_KEY=file:/config/keys/periphery.pub
|
||||
- KOMODO_HOST=https://komodo.itpropartner.com
|
||||
- KOMODO_JWT_SECRET=<generated>
|
||||
- KOMODO_WEBHOOK_SECRET=<generated>
|
||||
- PUID=0
|
||||
- PGID=0
|
||||
networks:
|
||||
- komodo-net
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
|
||||
networks:
|
||||
komodo-net:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
### Env Var Changes from v1.x to v2.x
|
||||
- `MONGO_CONNECTION_STRING` → `KOMODO_DATABASE_ADDRESS`
|
||||
- `KOMODO_SERVER_PORT` → removed (defaults to 9120)
|
||||
- `KOMODO_SERVER_HOST` → removed
|
||||
- Must add `KOMODO_LOCAL_AUTH=true` for built-in auth
|
||||
- Must add `KOMODO_INIT_ADMIN_USERNAME` and `KOMODO_INIT_ADMIN_PASSWORD` for first-run admin user
|
||||
- Must add `KOMODO_FIRST_SERVER_NAME=core` to auto-name the management server
|
||||
|
||||
### Caddy Config
|
||||
```
|
||||
komodo.itpropartner.com {
|
||||
reverse_proxy 127.0.0.1:9120
|
||||
}
|
||||
```
|
||||
|
||||
### Adding a Periphery Agent
|
||||
```bash
|
||||
curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py \
|
||||
| python3 - \
|
||||
--core-address="http://127.0.0.1:9120" \
|
||||
--connect-as="<hostname>" \
|
||||
--onboarding-key="<key-from-core-ui>"
|
||||
```
|
||||
@@ -0,0 +1,53 @@
|
||||
# Ops Portal MIME Type Fix (Jul 12, 2026)
|
||||
|
||||
## Problem
|
||||
All ops portal pages broke on mobile -- nav didn't initialize, login overlay stuck, service buttons unresponsive. Desktop with cached assets worked fine.
|
||||
|
||||
## Root Cause
|
||||
FastAPI's `FileResponse()` returned Content-Type `application/json` for `.js` and `.css` files instead of `text/javascript` and `text/css`. Browsers strictly enforce MIME types for scripts/stylesheets -- the wrong type causes silent refusal to execute/apply.
|
||||
|
||||
Additionally, JS/CSS files were created with `chmod 600` (root-only), so Caddy's static file handler returned 403 Forbidden, falling back to the backend which returned the wrong MIME type.
|
||||
|
||||
## Fix -- Two Layers
|
||||
|
||||
### Layer 1: Caddy static file handler (primary)
|
||||
Add `handle_path` blocks BEFORE `reverse_proxy`:
|
||||
```caddy
|
||||
ops.itpropartner.com {
|
||||
handle_path /js/* {
|
||||
root * /opt/ops-portal/static/js
|
||||
file_server
|
||||
}
|
||||
handle_path /css/* {
|
||||
root * /opt/ops-portal/static/css
|
||||
file_server
|
||||
}
|
||||
reverse_proxy 127.0.0.1:8090
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 2: FastAPI fallback
|
||||
```python
|
||||
return FileResponse(str(fpath), media_type="application/javascript",
|
||||
headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
|
||||
return FileResponse(str(fpath), media_type="text/css",
|
||||
headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
|
||||
```
|
||||
|
||||
### Prerequisite: File permissions
|
||||
```bash
|
||||
chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css
|
||||
```
|
||||
|
||||
## Symptoms
|
||||
- Browser console: "Refused to execute script because its MIME type ('application/json') is not executable"
|
||||
- Works on desktop with cached assets, fails on mobile/cold load
|
||||
- All pages load HTML but nothing interactive works
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
curl -sI https://ops.itpropartner.com/js/app.js | grep -i content-type
|
||||
# Expected: text/javascript; charset=utf-8
|
||||
curl -sI https://ops.itpropartner.com/css/ops.css | grep -i content-type
|
||||
# Expected: text/css; charset=utf-8
|
||||
```
|
||||
@@ -0,0 +1,47 @@
|
||||
# Multi-Server Health Check API Pattern
|
||||
|
||||
Added Jul 10, 2026 to ops portal. Replaced the single-server scraped-API health with ping-based live health for all 7 ITPP servers.
|
||||
|
||||
## Backend (FastAPI endpoint)
|
||||
|
||||
```python
|
||||
SERVERS = [
|
||||
{"name": "Core", "ip": "152.53.192.33", "role": "Hermes, Docker, OSINT, Caddy", "tier": "Primary"},
|
||||
{"name": "app1", "ip": "152.53.36.131", "role": "AI Stack", "tier": "Standard"},
|
||||
{"name": "app2", "ip": "152.53.39.202", "role": "UNMS, UniFi, Traccar", "tier": "Standard"},
|
||||
{"name": "app3", "ip": "152.53.241.111", "role": "WordPress", "tier": "Standard"},
|
||||
{"name": "app1-bu", "ip": "5.161.114.8", "role": "Warm Standby", "tier": "Standby"},
|
||||
{"name": "AI Box (legacy)", "ip": "178.156.167.181", "role": "LiteLLM (migrating)", "tier": "Legacy"},
|
||||
{"name": "Docker Box (legacy)", "ip": "178.156.168.35", "role": "Uptime Kuma", "tier": "Legacy"},
|
||||
]
|
||||
|
||||
@app.get("/api/servers/health")
|
||||
async def servers_health(user: str = Depends(get_current_user)):
|
||||
results = []
|
||||
for s in SERVERS:
|
||||
try:
|
||||
start = time.time()
|
||||
result = subprocess.run(["ping", "-c", "1", "-W", "2", s["ip"]],
|
||||
capture_output=True, text=True, timeout=5)
|
||||
latency = round((time.time() - start) * 1000)
|
||||
alive = result.returncode == 0
|
||||
except:
|
||||
alive = False
|
||||
latency = None
|
||||
results.append({...})
|
||||
return {"servers": results, "count": len(results), "live": sum(...)}
|
||||
```
|
||||
|
||||
## Frontend (vanilla JS)
|
||||
|
||||
Page: `/opt/ops-portal/static/servers.html`
|
||||
Uses `Ops.apiFetch('/api/servers/health')` to get all 7 servers.
|
||||
Renders as card grid with border-left color (green=LIVE, red=OFFLINE).
|
||||
Shows: server name, status badge, role, IP, latency, tier.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Ping requires root or CAP_NET_RAW. The ops-portal service runs as root, so ping works.
|
||||
- Some servers block ICMP (firewalls). If a server shows OFFLINE but is actually up, check UFW rules.
|
||||
- Legacy servers in migration status should still be monitored — they're running production services.
|
||||
- The API endpoint requires JWT auth (like all /api/* routes).
|
||||
@@ -0,0 +1,83 @@
|
||||
# Nav Consistency Fix — July 12, 2026
|
||||
|
||||
## Problem
|
||||
Multiple ops portal pages had erratic behavior on mobile — hamburger menu didn't work, nav links were missing or stale, some pages showed no data.
|
||||
|
||||
## Root Causes Found (3 issues)
|
||||
|
||||
### 1. Mixed nav strategies — inline vs fetched
|
||||
Some pages had the nav hardcoded inline in the HTML, while others fetched `/nav.html` dynamically. The inline navs were stale — missing Audit, Costs, and Grafana links. The fetched navs were always up to date.
|
||||
|
||||
**Fix:** Every page should use:
|
||||
```html
|
||||
<div id="nav"></div>
|
||||
```
|
||||
And fetch nav at the bottom of the page script:
|
||||
```javascript
|
||||
fetch('/nav.html').then(r => r.text()).then(html => {
|
||||
document.getElementById('nav').innerHTML = html;
|
||||
if (typeof window.initNav === 'function') window.initNav();
|
||||
});
|
||||
```
|
||||
|
||||
Pages affected: backups.html, servers.html, services.html (all had inline navs that were replaced)
|
||||
|
||||
### 2. `Ops.initNav()` vs `window.initNav()` — TypeError kills nav
|
||||
The `initNav` function is defined in utils.js as `window.initNav`, NOT as `Ops.initNav`. Pages calling `Ops.initNav()` throw a ReferenceError that kills ALL JavaScript execution on the page — including the data loading and rendering code.
|
||||
|
||||
**Fix:** Replace all `Ops.initNav()` with `window.initNav()`:
|
||||
```javascript
|
||||
// BROKEN:
|
||||
Ops.initNav();
|
||||
// FIXED:
|
||||
if (typeof window.initNav === 'function') window.initNav();
|
||||
```
|
||||
|
||||
Pages affected: index.html, backups.html, servers.html, services.html
|
||||
|
||||
### 3. CSS/JS MIME types — wrong content-type kills scripts
|
||||
The backend (FastAPI FileResponse) and Caddy reverse proxy served JS and CSS files with `Content-Type: application/json`. Browsers refuse to execute scripts or apply stylesheets with the wrong MIME type.
|
||||
|
||||
**Root causes:**
|
||||
- FastAPI's `FileResponse()` without explicit `media_type` parameter sometimes resolves `.js` files as `application/json`
|
||||
- Caddy `handle_path /js/*` block was nested inside another `handle_path` block by accident (sed nesting bug)
|
||||
- File permissions were `600` (root-only) — Caddy returned 403, browser silently failed
|
||||
|
||||
**Fix — Caddy approach (most reliable):**
|
||||
```caddy
|
||||
ops.itpropartner.com {
|
||||
handle_path /data/* {
|
||||
root * /var/www/ops/data/
|
||||
file_server
|
||||
}
|
||||
handle_path /js/* { # ← TOP LEVEL
|
||||
root * /opt/ops-portal/static/js
|
||||
file_server
|
||||
}
|
||||
handle_path /css/* { # ← TOP LEVEL
|
||||
root * /opt/ops-portal/static/css
|
||||
file_server
|
||||
}
|
||||
reverse_proxy 127.0.0.1:8090
|
||||
}
|
||||
```
|
||||
|
||||
**Plus fix file permissions:** `chmod 644 /opt/ops-portal/static/js/*.js /opt/ops-portal/static/css/*.css`
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
curl -sI https://ops.itpropartner.com/js/app.js | grep -i content-type
|
||||
# Expected: text/javascript; charset=utf-8
|
||||
|
||||
for page in / /services.html /servers.html /backups.html /audit.html /cost.html; do
|
||||
echo "$page → $(curl -sS -o /dev/null -w '%{http_code}' https://ops.itpropartner.com$page)"
|
||||
done
|
||||
# All should return 200
|
||||
```
|
||||
|
||||
## Prevention Checklist
|
||||
- Every new page: use `<div id="nav"></div>` + fetch pattern, never inline nav
|
||||
- Every new page: call `window.initNav()` (not `Ops.initNav()`) inside the fetch callback
|
||||
- After any Caddy config change: verify `handle_path` blocks are top-level, not nested
|
||||
- After adding any new static file: `chmod 644`
|
||||
- After any static file change: test with `curl -sI` to verify MIME type
|
||||
@@ -0,0 +1,52 @@
|
||||
# Nav Consistency Pattern (Jul 12, 2026)
|
||||
|
||||
## Principle
|
||||
|
||||
All portal pages must load nav from `/nav.html` — the single source of truth for nav links, hamburger toggle, and nav CSS. The only exception is index.html (dashboard), which keeps an inline nav that matches nav.html link-for-link.
|
||||
|
||||
## Current state (post-Jul 12 fix)
|
||||
|
||||
| Page | Nav source | Status |
|
||||
|------|-----------|--------|
|
||||
| index.html | Inline (matches nav.html) | OK — all 11 links + hamburger |
|
||||
| services.html | Fetched | OK |
|
||||
| servers.html | Fetched | OK |
|
||||
| backups.html | Fetched | OK |
|
||||
| audit.html, cost.html, cron.html, network.html, config.html, logs.html | Fetched | OK |
|
||||
|
||||
## Conversion procedure (inline → fetched)
|
||||
|
||||
1. **Replace the `<nav class="ops-nav">...</nav>` block** with `<div id="nav"></div>`.
|
||||
|
||||
2. **Move `initNav()` into the fetch callback.** The old pattern of calling `window.initNav()` before the nav HTML is injected is a race condition — it tries to wire up toggle listeners on elements that don't exist yet:
|
||||
```javascript
|
||||
// WRONG (initNav runs before nav HTML exists):
|
||||
window.initNav();
|
||||
// ... later ...
|
||||
fetch('/nav.html')...
|
||||
|
||||
// RIGHT (initNav runs AFTER nav HTML is injected):
|
||||
fetch('/nav.html').then(function(r) { return r.text(); }).then(function(h) {
|
||||
document.getElementById('nav').innerHTML = h;
|
||||
if (typeof window.initNav === 'function') window.initNav();
|
||||
}).catch(function(e) { console.warn('Nav failed:', e); });
|
||||
```
|
||||
|
||||
3. **Keep `window.initNav()`** — never use `Ops.initNav()` or bare `initNav()` inside a strict-IIFE. The nav init function is exported on `window` in `app.js`/`utils.js`.
|
||||
|
||||
## Index.html: inline nav maintenance
|
||||
|
||||
The dashboard keeps inline nav. When `/nav.html` gains or loses links, update index.html's inline nav to match. Key elements:
|
||||
- `class="nav-logo-text"` on the brand span
|
||||
- Hamburger toggle: `<button class="nav-toggle" id="navToggle" ...>`
|
||||
- All 11 links with `data-path` attributes
|
||||
- `class="nav-link"` and `target="_blank"` on Grafana
|
||||
|
||||
## Verification checklist
|
||||
|
||||
- [ ] `<div id="nav">` placeholder present (fetched pages only)
|
||||
- [ ] `initNav()` called AFTER `.innerHTML = h` in fetch callback
|
||||
- [ ] No orphan `window.initNav()` outside fetch block
|
||||
- [ ] index.html has all current nav.html links + hamburger toggle
|
||||
- [ ] HTTP 200 on all pages
|
||||
- [ ] `/nav.html` serves directly (not just through FastAPI proxy)
|
||||
@@ -0,0 +1,50 @@
|
||||
# netcup SCP REST API Authentication
|
||||
|
||||
The netcup Server Control Panel (SCP) REST API uses Keycloak (OIDC) password grant for authentication, NOT the CCP API key from Master Data.
|
||||
|
||||
## Auth Flow
|
||||
|
||||
```
|
||||
TOKEN=$(curl -s "https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token" \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=password&client_id=scp&username=${CUSTOMER_NUMBER}&password=${CCP_PASSWORD}&scope=openid" \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Field | Value | Source |
|
||||
|-------|-------|--------|
|
||||
| grant_type | password | OIDC resource owner password flow |
|
||||
| client_id | scp | Static SCP client ID |
|
||||
| username | Customer number | e.g. 389212 from CCP Master Data page |
|
||||
| password | CCP login password | Same as web login password |
|
||||
| scope | openid | Returns access_token + id_token + refresh_token |
|
||||
|
||||
### Token properties
|
||||
|
||||
- Access token expires in 300 seconds (5 min)
|
||||
- Refresh token expires in 1800 seconds (30 min)
|
||||
- Always obtain a fresh token each collection cycle
|
||||
|
||||
## API Endpoints
|
||||
|
||||
Base: https://www.servercontrolpanel.de/scp-core/api/v1/
|
||||
|
||||
| Endpoint | Response |
|
||||
|----------|----------|
|
||||
| GET /servers | Array of servers with id, name, template, disabled, hostname |
|
||||
| GET /servers/{id} | Full detail: IPv4, IPv6, architecture, disks, RAM |
|
||||
|
||||
## Known Working
|
||||
|
||||
- RS 2000 G12 — ID 890903, 152.53.192.33
|
||||
- RS 4000 G12 — IDs 894147 (152.53.36.131), 894148 (152.53.39.202), 894149 (152.53.241.111)
|
||||
- Template name format: "RS {1000|2000|4000} G12"
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- /scp-core/api/ NOT /scp-ui/api/ — /scp-ui/ returns HTML even with valid auth
|
||||
- www subdomain IS required — servercontrolpanel.de does NOT respond
|
||||
- CCP API Key is NOT for SCP — they use different auth systems
|
||||
- **Username is bare customer number, NOT `customer#<number>`.** Adding `customer#` prefix returns `invalid_grant` / `Invalid user credentials`. The Keycloak realm expects the bare numeric ID. This was discovered Jul 10, 2026 after multiple failed auth attempts.
|
||||
@@ -0,0 +1,45 @@
|
||||
Generated Jul 9, 2026 by the Network Services Team. Full document at /root/.hermes/references/ops-portal-recommendations.md.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Current:** Static HTML served by Caddy, reads JSON data file. Read-only.
|
||||
**Target:** FastAPI backend with auth, action endpoints, audit logging, RBAC.
|
||||
|
||||
**Strategy:** Add action buttons to existing static pages immediately (via helper scripts), while planning the full rebuild.
|
||||
|
||||
## Pages (13 total)
|
||||
|
||||
8 existing + 5 new:
|
||||
- Dashboard, Servers, Backups, Network, Services, Cron Jobs, Logs, Config
|
||||
- **New:** Customer Management, Billing/Finance, Users/RBAC, Audit Log, Settings
|
||||
|
||||
## Actions per page
|
||||
|
||||
Each page should offer actions, not just views:
|
||||
- Restart a service
|
||||
- Trigger/restore backups
|
||||
- Add DNS records
|
||||
- Reboot routers
|
||||
- Create customers
|
||||
- Provision services
|
||||
- Create tickets
|
||||
|
||||
## RBAC
|
||||
|
||||
- Owner/Admin: everything
|
||||
- Tech/Support: tickets, server status, service restarts
|
||||
- Sales/Onboarding: customer activation, provisioning
|
||||
- Viewer: read-only
|
||||
|
||||
## Integrations Roadmap
|
||||
|
||||
- **Connected:** SyncroMSP, Cloudflare, Hetzner, netcup, Wasabi, admin-ai
|
||||
- **Pending:** RingLogix (phone), Stripe (billing), Xero (accounting), UniFi (API key saved), Hudu, RunCloud
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
1. Action buttons on existing pages, VPN monitor widget
|
||||
2. Backend development (FastAPI + SQLite for audit)
|
||||
3. Customer management module
|
||||
4. RBAC + multi-user support
|
||||
5. Polish (Grafana dashboards, notifications, mobile)
|
||||
@@ -0,0 +1,39 @@
|
||||
# Ops Portal Vision & Architecture
|
||||
|
||||
Confirmed July 9, 2026. ops.itpropartner.com is where ITPP runs its business.
|
||||
|
||||
## Core Vision
|
||||
|
||||
"One page to do what I need to do" (Germaine). The ops portal is the single pane of glass for the entire IT Pro Partner business — monitoring, provisioning, customer lifecycle, billing, and infrastructure management.
|
||||
|
||||
RBAC ensures different users see different things on the same platform:
|
||||
- **Admin (Germaine):** everything
|
||||
- **Tech/Support:** tickets, server status, network health — no billing, no provisioning
|
||||
- **Sales/Onboarding:** activation flow, order forms, provisioning status
|
||||
- **Viewer:** read-only, limited scope
|
||||
- **Tony:** DRE only (Debt Recovery Experts)
|
||||
|
||||
## Architecture Evolution
|
||||
|
||||
Phase 1 (current): FastAPI backend + static frontend at ops.itpropartner.com
|
||||
Phase 2 (next): Actionable pages with service restart, backup trigger, DNS management
|
||||
Phase 3 (future): Customer provisioning workflows, billing sync, automated activation
|
||||
|
||||
## Confirmed Integrations
|
||||
|
||||
Already connected: SyncroMSP, Cloudflare, Hetzner, netcup, Wasabi, admin-ai (LiteLLM)
|
||||
In discussion: RingLogix, Stripe, UniFi, Hudu
|
||||
Dropped: OpenRouter (already proxied through LiteLLM), OpenAI (LiteLLM already), Porkbun/Namecheap (Cloudflare sells domains at cost)
|
||||
|
||||
## Monitoring Stack
|
||||
|
||||
- Uptime-Kuma (existing on Hetzner Docker host) — embedded status page, has REST API
|
||||
- Prometheus (Core, deployed) — metrics collection at 127.0.0.1:9090
|
||||
- Grafana (Core, new) — dashboards at 127.0.0.1:3002 (port 3000 was taken by DocuSeal)
|
||||
- node_exporter on all servers — OS metrics (installed on Core so far)
|
||||
- VPS Threshold Alerts (deployed) — 80/90/95% RAM/disk/CPU alerts every 15 min via Telegram
|
||||
|
||||
## Hardware Standard (Jul 2026)
|
||||
|
||||
All new netcup servers: **RS 4000 G12** (12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe, ~$44/mo monthly billing in Manassas, VA).
|
||||
Exceptions: Core (existing RS 2000, light ops layer), app1-bu (Hetzner CPX11, warm standby).
|
||||
@@ -0,0 +1,165 @@
|
||||
# Ops Portal Full Code Audit — July 12, 2026
|
||||
|
||||
Auditor: Hermes | Pages: 11 | Total bugs: 42
|
||||
|
||||
## The Four Root Causes
|
||||
|
||||
Everything else flows from these four:
|
||||
|
||||
### RC-1: `window.initNav` is undefined (affects ALL pages)
|
||||
`app.js` exports `initNav` only under `window.Ops.initNav`, not `window.initNav`.
|
||||
Every page's guard `typeof window.initNav === 'function'` evaluates FALSE.
|
||||
Nav hamburger, active-link, and logout wiring are dead on every page.
|
||||
|
||||
**Fix (one line in app.js, inside IIFE before `window.Ops = {...}`):**
|
||||
```javascript
|
||||
window.initNav = initNav;
|
||||
```
|
||||
|
||||
### RC-2: `utils.js` does not exist
|
||||
All pages reference `/js/utils.js?v=2` but the file is `/js/app.js`.
|
||||
The 404 is silent because `app.js` loads next and provides `window.Ops`.
|
||||
But if the server returns a hard error, all pages break.
|
||||
|
||||
**Fix:** Rename `app.js` → `utils.js` OR update all `<script src="/js/utils.js">` to `app.js`.
|
||||
|
||||
### RC-3: `logoutBtn` DOM lookup before nav fetch completes
|
||||
`services.html`, `servers.html`, `backups.html` all call
|
||||
`document.getElementById('logoutBtn').classList.add('show')` synchronously.
|
||||
`logoutBtn` lives in `nav.html` which is fetched asynchronously.
|
||||
Result: TypeError on every authenticated page load on those 3 pages.
|
||||
|
||||
**Fix:** Move the `getElementById('logoutBtn')` call inside the fetch callback:
|
||||
```javascript
|
||||
fetch('/nav.html').then(r => r.text()).then(h => {
|
||||
document.getElementById('nav').innerHTML = h;
|
||||
document.getElementById('logoutBtn').classList.add('show'); // ← here
|
||||
if (typeof window.initNav === 'function') window.initNav();
|
||||
});
|
||||
```
|
||||
|
||||
### RC-4: 6 of 11 pages have NO auth guard
|
||||
network.html, cron.html, config.html, logs.html, cost.html, fleettracker360.html
|
||||
load and show data without any `Ops.isAuthenticated()` check.
|
||||
|
||||
---
|
||||
|
||||
## Full Bug Inventory by Page
|
||||
|
||||
### index.html
|
||||
- B1 [HIGH] utils.js 404 (silent)
|
||||
- B2 [HIGH] window.initNav() call fails — window.initNav undefined
|
||||
- B3 [MED] navUser shown via inline style, inconsistent with .show pattern
|
||||
- B4 [MED] logoutBtn shown but never wired to logout (depends on initNav fix)
|
||||
|
||||
### services.html
|
||||
- S1 [CRIT] utils.js 404
|
||||
- S2 [HIGH] login-overlay hardcoded class="show" — flash of login for auth'd users
|
||||
- S3 [HIGH] loginError "Invalid credentials" visible without display:none
|
||||
- S4 [CRIT] logoutBtn/navUser lookup before nav.html fetch completes → TypeError
|
||||
- S5 [MED] window.initNav undefined (nav never initialized)
|
||||
- S6 [MED] No guard for session expiry mid-session
|
||||
|
||||
### servers.html
|
||||
- V1 [CRIT] utils.js 404
|
||||
- V2 [CRIT] logoutBtn.style.display = 'block' before nav loads → TypeError
|
||||
- V3 [HIGH] Login form inputs have no CSS classes, no form-group wrapper — unstyled
|
||||
- V4 [MED] loginError always shows hardcoded "Invalid credentials" text
|
||||
- V5 [MED] window.initNav undefined
|
||||
- V6 [LOW] No <label> tags on login inputs (accessibility + autofill)
|
||||
- V7 [LOW] login-logo uses emoji 🖥 instead of SVG (inconsistent)
|
||||
|
||||
### backups.html
|
||||
- K1 [CRIT] utils.js 404
|
||||
- K2 [CRIT] logoutBtn.classList.add('show') before nav loads → TypeError
|
||||
- K3 [HIGH] login-overlay hardcoded class="show"
|
||||
- K4 [MED] loginError no display:none
|
||||
- K5 [MED] window.initNav undefined
|
||||
- K6 [HIGH] .summary-bar, .summary-item, .summary-label, .summary-val not in ops.css
|
||||
|
||||
### network.html
|
||||
- N1 [CRIT] utils.js 404
|
||||
- N2 [HIGH] Scripts in <head> without defer (execute before DOM)
|
||||
- N3 [CRIT] NO auth guard — page shows all network data without login
|
||||
- N4 [MED] window.initNav undefined
|
||||
- N5 [MED] Grafana link uses Tailscale IP 100.71.155.7 — unreachable without Tailscale
|
||||
- N6 [LOW] Inline <style> redefines .card, .badge, .status-dot (may conflict with ops.css)
|
||||
|
||||
### cron.html
|
||||
- C1 [CRIT] utils.js 404
|
||||
- C2 [HIGH] Scripts in <head> without defer
|
||||
- C3 [CRIT] NO auth guard
|
||||
- C4 [HIGH] <main id="main-content"> placed AFTER the three <section> elements (inverted)
|
||||
- C5 [MED] window.initNav undefined
|
||||
- C6 [LOW] app.js loaded but Ops.* never used — all utilities re-implemented locally
|
||||
|
||||
### config.html
|
||||
- G1 [CRIT] utils.js 404
|
||||
- G2 [CRIT] window.escapeHtml() called throughout — not exported by app.js → TypeError
|
||||
- G3 [HIGH] window.formatTime() used — not exported → falls back to Date.toLocaleString()
|
||||
- G4 [HIGH] fetchStatus() called — not defined anywhere → typeof guard prevents crash but live data never loads
|
||||
- G5 [CRIT] NO auth guard
|
||||
- G6 [MED] window.initNav undefined
|
||||
- G7 [HIGH] Local initNav() defined inside IIFE shadows concept, calls window.initNav (self-referencing risk)
|
||||
- G8 [MED] .content-area and .page-footer not in ops.css
|
||||
- G9 [MED] Uses <header class="header"> but ops.css defines .page-header not .header
|
||||
|
||||
### logs.html
|
||||
- L1 [CRIT] utils.js 404
|
||||
- L2 [HIGH] Scripts in <head> without defer
|
||||
- L3 [CRIT] NO auth guard
|
||||
- L4 [MED] window.initNav undefined
|
||||
- L5 [MED] <main> doesn't cleanly wrap content (navPlaceholder outside main, script inside </main>)
|
||||
|
||||
### audit.html
|
||||
- A1 [CRIT] utils.js 404
|
||||
- A2 [CRIT] Uses sessionStorage for token — all other pages use localStorage — token not found on nav
|
||||
- A3 [MED] app.js loaded but Ops.* never used — own escapeHtml/fmtTime re-implemented
|
||||
- A4 [MED] logoutBtn in nav never wired (initNav not called)
|
||||
- A5 [MED] window.initNav undefined
|
||||
- A6 [LOW] Login form pre-filled with value="germaine" — leaks valid username
|
||||
|
||||
### cost.html
|
||||
- T1 [CRIT] utils.js 404
|
||||
- T2 [CRIT] window.fetchStatus(...) called — not defined anywhere → TypeError on load → no data renders
|
||||
- T3 [CRIT] NO auth guard
|
||||
- T4 [MED] app.js loaded but unused
|
||||
- T5 [MED] window.initNav undefined
|
||||
- T6 [LOW] Refresh button uses Unicode ↻ not SVG icon, .spinning animation doesn't apply
|
||||
|
||||
### fleettracker360.html
|
||||
- F1 [HIGH] No app.js or utils.js loaded — no auth system
|
||||
- F2 [HIGH] NO auth guard
|
||||
- F3 [MED] Missing viewport-fit=cover (all other pages have it)
|
||||
- F4 [LOW] Hardcoded date "2026-07-12" in timestamp string
|
||||
- F5 [LOW] COT hour calculation: `(utcHours + (-5) + 24).slice(-2)` fails at midnight (produces "24")
|
||||
- F6 [LOW] Not linked from nav.html — orphaned page
|
||||
|
||||
---
|
||||
|
||||
## Cross-Cutting Issues
|
||||
|
||||
| ID | Bug | Pages | Severity |
|
||||
|----|-----|-------|----------|
|
||||
| X1 | /js/utils.js 404 | All 10 auth pages | CRIT |
|
||||
| X2 | window.initNav undefined | All pages with fetch-nav | HIGH |
|
||||
| X3 | logoutBtn null before nav loads | services, servers, backups | CRIT |
|
||||
| X4 | No auth guard | network, cron, config, logs, cost, fleettracker | CRIT |
|
||||
| X5 | login-overlay hardcoded show class | services, servers, backups | HIGH |
|
||||
| X6 | Grafana link unreachable on mobile | nav.html + index.html | MED |
|
||||
| X7 | ops.css vs ops.css?v=2 version inconsistency | Mixed across pages | LOW |
|
||||
|
||||
---
|
||||
|
||||
## Priority Fix Order
|
||||
|
||||
1. Add `window.initNav = initNav;` to app.js (fixes X2, B2, S5, V5, K5, N4, C5, G6, L4, A5, T5)
|
||||
2. Rename utils.js reference or file (fixes X1)
|
||||
3. Move logoutBtn lookups inside nav fetch callback (fixes X3)
|
||||
4. Replace `window.fetchStatus(...)` in cost.html with direct fetch (fixes T2)
|
||||
5. Add `window.escapeHtml = Ops.escapeHtml` to app.js or fix config.html (fixes G2)
|
||||
6. Change audit.html to use localStorage not sessionStorage (fixes A2)
|
||||
7. Add auth guard + login overlay to network, cron, config, logs, cost (fixes X4)
|
||||
8. Remove hardcoded `show` class from login-overlay in services/servers/backups HTML (fixes X5)
|
||||
9. Fix <main> placement in cron.html (fixes C4)
|
||||
10. Add .summary-bar CSS to ops.css (fixes K6)
|
||||
@@ -0,0 +1,44 @@
|
||||
# Postfix SMTP Relay for RunCloud-Managed Servers
|
||||
|
||||
When a RunCloud-managed server (like wphost02) has stuck email in Postfix queue because port 25 is blocked, relay through `mail.germainebrown.com:2525` with SASL auth and sender-domain rewriting.
|
||||
|
||||
## Symptom
|
||||
`mailq` shows 60+ messages deferred with `(connect to ...:25: Connection timed out)`. The server tries to send directly to destination MX servers but port 25 is blocked.
|
||||
|
||||
## Fix
|
||||
```bash
|
||||
# Configure relay
|
||||
postconf -e "relayhost = mail.germainebrown.com:2525"
|
||||
postconf -e "smtp_sasl_auth_enable = yes"
|
||||
postconf -e "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd"
|
||||
postconf -e "smtp_sasl_security_options = noanonymous"
|
||||
postconf -e "smtp_tls_security_level = encrypt"
|
||||
postconf -e "myorigin = /etc/mailname"
|
||||
|
||||
# Auth credentials (use shonuff's SMTP account)
|
||||
echo "mail.germainebrown.com:2525 shonuff@germainebrown.com:Catches.bullets1985" > /etc/postfix/sasl_passwd
|
||||
chmod 600 /etc/postfix/sasl_passwd
|
||||
postmap /etc/postfix/sasl_passwd
|
||||
|
||||
# Sender rewriting — MXroute requires real sender domain, not wphost02.local
|
||||
echo "apextrackexperience.com" > /etc/mailname
|
||||
echo "@wphost02 contact@apextrackexperience.com" > /etc/postfix/generic
|
||||
echo "root@wphost02 contact@apextrackexperience.com" >> /etc/postfix/generic
|
||||
postmap /etc/postfix/generic
|
||||
postconf -e "smtp_generic_maps = hash:/etc/postfix/generic"
|
||||
|
||||
systemctl restart postfix
|
||||
postqueue -f # flush the stuck queue
|
||||
```
|
||||
|
||||
## Verification
|
||||
`mailq` says "Mail queue is empty". Logs show `relay=mail.germainebrown.com[65.109.144.50]:2525` with `status=sent`.
|
||||
|
||||
## Note
|
||||
Old queued messages will get bounce notices (they were deferred too long). New emails flow through the relay cleanly. MXroute rejects `550 Envelope sender domain must be a fully qualified name` if the sender is `runcloud@wphost02` — the `smtp_generic_maps` rewrite fixes this.
|
||||
|
||||
## Trigger Symptom Chain
|
||||
1. WP Mail SMTP on wphost02 configured for c1113726.sgvps.net:2525 (works for WP)
|
||||
2. PHP mail() falls back to Postfix, which does direct MX delivery on port 25
|
||||
3. Port 25 blocked → emails in deferred queue forever
|
||||
4. WPForms notifications and system emails pile up
|
||||
@@ -0,0 +1,35 @@
|
||||
# Push Notification Testing — Shark Game
|
||||
|
||||
## iOS Safari Limitation
|
||||
|
||||
PushManager is NOT available in Safari on iPhone when browsing normally. It ONLY
|
||||
works when the page is launched from a Home Screen PWA icon (Add to Home Screen).
|
||||
|
||||
Test flow for iOS:
|
||||
1. Visit shark.iamgmb.com/push-test.html
|
||||
2. Tap Share icon -> Add to Home Screen
|
||||
3. Open the test page from the Home Screen icon
|
||||
4. Tap Enable Push Notifications — all steps should pass
|
||||
|
||||
## Standalone Test Page
|
||||
|
||||
/push-test.html at /var/www/shark-game/push-test.html shows a full diagnostic
|
||||
log of each push setup step. No login required.
|
||||
|
||||
## Subscription Endpoint
|
||||
|
||||
POST /api/push/subscribe requires JWT auth.
|
||||
POST /api/push/subscribe-noauth stores under user_id=0 (for testing only).
|
||||
|
||||
Both expect: {endpoint, p256dh, auth}
|
||||
|
||||
## Banner Visibility Bug
|
||||
|
||||
The pushBanner div starts with style="display:none". The JS function
|
||||
showNotificationBanner() must explicitly set banner.style.display = 'block'.
|
||||
Without this, the banner exists in the DOM but is never visible.
|
||||
|
||||
## VAPID Keys
|
||||
|
||||
Generated once, stored in /root/shark-game/backend/server.py as module-level
|
||||
variables. Public key exposed at GET /api/push/vapid-public-key.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Wasabi S3 Bucket Architecture
|
||||
|
||||
Normalized Jul 8, 2026 from a single monolithic bucket into purpose-specific buckets.
|
||||
|
||||
## Bucket Layout
|
||||
|
||||
| Bucket | Purpose | Created | Initial versioning |
|
||||
|--------|---------|---------|-------------------|
|
||||
| `hermes-vps-backups` | Hermes live state, full backups, standby recovery | Pre-existing | ✅ Enabled Jul 8 |
|
||||
| `itpropartner-system-configs` | Caddyfile, systemd, SSH keys, scripts, .env files | **Manual (Wasabi Console)** | ✅ Enable after creation |
|
||||
| `itpropartner-docker-volumes` | Docuseal, Vaultwarden, Twenty, SearXNG, Ollama | **Manual (Wasabi Console)** | ✅ Enable after creation |
|
||||
| `mikrotik-ccr-backups` | Router configs, backup logs | Pre-existing | ✅ Enabled Jul 8 |
|
||||
| `itpropartner-backups` | Legacy orphaned data (no active writer) | Pre-existing | ✅ Enabled Jul 8 |
|
||||
|
||||
**Note:** The Hermes-User IAM key lacks `s3:CreateBucket` permission. New buckets must be created in the Wasabi Console web UI. After creation, enable versioning via CLI:
|
||||
|
||||
```bash
|
||||
source /opt/awscli-venv/bin/activate
|
||||
aws s3api put-bucket-versioning --bucket itpropartner-system-configs \
|
||||
--versioning-configuration Status=Enabled \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
## Data Formerly in `itpropartner-backups` (legacy, preserved)
|
||||
|
||||
| Old path | Copied to |
|
||||
|----------|-----------|
|
||||
| `home-router/2026-07-07-config.rsc` | `mikrotik-ccr-backups/wisp-backups/configs/` |
|
||||
| `shonuff/*` (images) | `hermes-vps-backups/assets/` |
|
||||
|
||||
No data deleted — old bucket preserved with versioning.
|
||||
|
||||
## Sync Scripts
|
||||
|
||||
| Script | Bucket | Schedule | What it syncs |
|
||||
|--------|--------|----------|--------------|
|
||||
| `hermes-live-sync.sh` | `hermes-vps-backups` | Every 15 min | Hermes config, skills, cron, sessions, profiles, .env, state.db |
|
||||
| `hermes-system-config-sync.sh` | `itpropartner-system-configs` | Every 15 min | Caddyfile, systemd services, ops scripts, SSH keys, .env files |
|
||||
| `hermes-docker-sync.sh` | `itpropartner-docker-volumes` | Every 15 min | Docuseal, Vaultwarden, Twenty, SearXNG, Ollama volume data |
|
||||
| `wisp-backup.py` | `mikrotik-ccr-backups` | Daily 6 AM | Router config exports |
|
||||
| `hermes-backup.sh` | `hermes-vps-backups/full/` | Daily 5 AM | Full tarball of everything |
|
||||
|
||||
## Data collector tracking
|
||||
|
||||
The ops-data-collector.py checks all 6 paths (5 buckets, with one split into sub-paths):
|
||||
|
||||
```python
|
||||
check_s3_backup("s3://hermes-vps-backups/live/") # → "hermes-vps-backups"
|
||||
check_s3_backup("s3://hermes-vps-backups/hermes-full-backup/") # → "hermes-full-backups"
|
||||
check_s3_backup("s3://itpropartner-system-configs/") # → "system-configs"
|
||||
check_s3_backup("s3://itpropartner-docker-volumes/") # → "docker-volumes"
|
||||
check_s3_backup("s3://mikrotik-ccr-backups/") # → "mikrotik-backups"
|
||||
check_s3_backup("s3://itpropartner-backups/") # → "itpropartner-backups"
|
||||
```
|
||||
@@ -0,0 +1,58 @@
|
||||
# Shark Game — Scoring Events & Push Notification Architecture
|
||||
|
||||
## Point Values
|
||||
- Sighting: +2 pts
|
||||
- Bite: +5 pts
|
||||
- Fatality: +10 pts
|
||||
|
||||
## Push Notification Lifecycle
|
||||
1. Scraper runs hourly (cron: `shark-scraper.sh`)
|
||||
2. Found events are scored in the backend
|
||||
3. `send_push_notification()` is called for each affected region
|
||||
4. Backend checks user's quiet hours and notification type preferences
|
||||
5. If notification is allowed, `webpush()` sends via VAPID keys through pywebpush
|
||||
6. Apple Push Service (APNs) delivers to iOS PWA, or browser push on desktop
|
||||
|
||||
## Per-User Quiet Hours (default: 9 PM - 6 AM ET)
|
||||
```sql
|
||||
ALTER TABLE users ADD COLUMN quiet_hours_start TEXT DEFAULT '21:00';
|
||||
ALTER TABLE users ADD COLUMN quiet_hours_end TEXT DEFAULT '06:00';
|
||||
ALTER TABLE users ADD COLUMN notify_sightings INTEGER DEFAULT 1;
|
||||
ALTER TABLE users ADD COLUMN notify_bites INTEGER DEFAULT 1;
|
||||
ALTER TABLE users ADD COLUMN notify_fatalities INTEGER DEFAULT 1;
|
||||
ALTER TABLE users ADD COLUMN notify_draft INTEGER DEFAULT 1;
|
||||
ALTER TABLE users ADD COLUMN quiet_hours_enabled INTEGER DEFAULT 1;
|
||||
```
|
||||
|
||||
## Draft Reminder Emails
|
||||
Cron job `shark-draft-reminder` runs every 15 minutes, checks leagues with upcoming drafts.
|
||||
|
||||
Two reminders per player per league:
|
||||
- 24h before draft: "Your draft starts in 24 hours!"
|
||||
- 1h before draft: "Your draft starts in 1 hour!"
|
||||
|
||||
**Tracking file:** `/root/shark-game/data/draft-reminders.json` — prevents duplicates.
|
||||
|
||||
**Email:** Same pattern as boys-mail-monitor (shonuff@g, mail.germainebrown.com:2525, BCC g@germainebrown.com, random Sho'Nuff signature).
|
||||
|
||||
## Desktop Layout
|
||||
The game is mobile-first but scales for desktop:
|
||||
- Max width: 1400px centered
|
||||
- Multi-column layout at 1024px+
|
||||
- Map + leaderboard side by side on widescreen
|
||||
- Mobile layout unchanged on small screens
|
||||
|
||||
## Settings Page
|
||||
At `/settings.html` with:
|
||||
- Notification toggles (sightings, bites, fatalities, draft)
|
||||
- Quiet hours (start time, end time, enable toggle)
|
||||
- Profile info (display name, email)
|
||||
- League info
|
||||
|
||||
## Help Page
|
||||
At `/help.html` with:
|
||||
- PWA setup guide (iOS steps, Android steps)
|
||||
- Push notification enablement guide
|
||||
- Quiet hours explanation
|
||||
- Scoring reference
|
||||
- Troubleshooting checklist
|
||||
@@ -0,0 +1,27 @@
|
||||
# SiteGround SFTP Backup (July 2026)
|
||||
|
||||
When SiteGround hosting is decommissioned (migrating to app3 netcup), all 21 sites were backed up to Wasabi S3 via SFTP.
|
||||
|
||||
## Key Constraints
|
||||
|
||||
- **No remote shell access.** SiteGround's SFTP on port 18765 blocks `exec_command()` — no remote `tar`, no `find | wc`, no `echo`. Only pure SFTP operations (listdir, read, write). This means per-file streaming over the SFTP channel, which is very slow for large WordPress sites with thousands of files.
|
||||
- **Directory structure:** Sites live at `/<domain>/<domain>/public_html` (nested directory with same name as site).
|
||||
- **Speed:** Each file requires a separate SFTP round-trip to open and read. The backup script uses `paramiko.SFTPClient` to recursively traverse directories and stream each file into a local `tarfile` in Python. For a 14K-file site, this takes 20-30 minutes.
|
||||
|
||||
## SFTP Credentials
|
||||
|
||||
Stored in the recovery manual and /root/.ssh/siteground.key (chmod 600). Key is DES-EDE3-CBC encrypted with passphrase `LoveMyBoys73!`.
|
||||
|
||||
## Restore Process
|
||||
|
||||
To restore a site from the S3 backup to a new host:
|
||||
1. Pull the backup: `aws s3 cp s3://hermes-vps-backups/siteground/<domain>/<domain>-<date>.tar.gz .`
|
||||
2. Extract: `tar xzf <domain>-<date>.tar.gz -C <target-dir>`
|
||||
3. Import MySQL database (needs a separate SQL dump — SFTP doesn't cover MySQL)
|
||||
4. Update wp-config.php with new DB credentials
|
||||
5. Point DNS to the new host IP
|
||||
6. Run `coolify` (or the site provisioning tool on app3) to wire up SSL
|
||||
|
||||
## Note on MySQL
|
||||
|
||||
Only site FILES were backed up via SFTP. The MySQL databases were NOT exported during the SFTP backup. To migrate a WordPress site fully, a MySQL dump is also needed from the SiteGround phpMyAdmin or via the existing RunCloud MySQL connection.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Static File Caching — Ops Portal
|
||||
|
||||
**Pitfall (Jul 10, 2026):** The ops portal serves static HTML/JS/CSS through FastAPI's `FileResponse`. Without explicit `Cache-Control` headers, browsers aggressively cache these files. After editing `app.js`, `servers.html`, or `ops.css`, users see stale versions — even after hard refresh. The login page, server health, and API calls all appear broken because the browser is serving cached JavaScript.
|
||||
|
||||
**Symptoms:**
|
||||
- User reports "pages won't refresh" despite confirmed deploys
|
||||
- `curl` returns HTTP 200 with correct content, but browser shows old version
|
||||
- JS changes (401 redirect fix, new endpoint calls) don't take effect
|
||||
- User has to manually Ctrl+Shift+R every page
|
||||
|
||||
**Fix:** Add `Cache-Control: no-cache, no-store, must-revalidate` headers to all `FileResponse` calls in `server.py`:
|
||||
|
||||
```python
|
||||
return FileResponse(str(fpath), headers={"Cache-Control": "no-cache, no-store, must-revalidate"})
|
||||
```
|
||||
|
||||
This applies to HTML pages, CSS, and JS. After adding headers, restart the service:
|
||||
```bash
|
||||
systemctl restart ops-portal
|
||||
```
|
||||
|
||||
**Why this matters for ops:** The portal uses vanilla HTML/JS with no framework, no bundler, and no cache-busting hashes. Every file edit requires either a cache-control header or manual browser refresh. The header approach is reliable across all users.
|
||||
|
||||
**Files affected:**
|
||||
- `/opt/ops-portal/server.py` — `serve_css()`, `serve_js()`, `serve_index()`, `serve_html_page()`
|
||||
- Any `@app.get("/{page}.html")` handler
|
||||
@@ -0,0 +1,23 @@
|
||||
# SyncroMSP API Integration
|
||||
|
||||
## Connection Details
|
||||
|
||||
- **Base URL:** `https://itpropartner.syncromsp.com/api/v1/`
|
||||
- **Auth:** Bearer token `Ta9f9d1b462271a2f4-8d63a3f025eb89451edb16f2308c2e40`
|
||||
- **Admin user:** Germaine Brown (user_id 155629, email info@itpropartner.com)
|
||||
- **Admin:** Yes
|
||||
- **Subdomain:** itpropartner
|
||||
|
||||
## Available Data
|
||||
|
||||
| Endpoint | Response | Notes |
|
||||
|----------|----------|-------|
|
||||
| `/me` | user_token, user_email, user_name, user_id, admin | Auth verification |
|
||||
| `/customers` | `{customers: [...], total: N}` | 26 customers |
|
||||
| `/tickets` | `{total: N}` | Total across all statuses |
|
||||
|
||||
## Data in ops portal
|
||||
|
||||
- Added to `ops-data-collector.py` via `collect_syncromsp()` function
|
||||
- Rendered on `/services.html` — customer count, ticket count, status badge
|
||||
- Token stored in `/root/.hermes/.env` as SYNCROMSP_API_TOKEN
|
||||
@@ -0,0 +1,35 @@
|
||||
# VPS Threshold Alert System — Jul 2026
|
||||
|
||||
Automated monitoring system that checks RAM, disk, and CPU usage across all 10 servers every 15 minutes.
|
||||
|
||||
## Script
|
||||
`/root/.hermes/scripts/vps-threshold-check.sh` (chmod 755)
|
||||
|
||||
## Cron job
|
||||
`vps-threshold-check` — runs every 15 min, no_agent mode (stdout delivered as Telegram message)
|
||||
|
||||
## Servers monitored (10 total)
|
||||
- 9 Hetzner servers (SSH via itpp-infra key, 5s timeout per host)
|
||||
- 1 local (core, netcup RS 2000)
|
||||
|
||||
## Thresholds
|
||||
| Metric | Warning (80%) | Critical (90%) | Emergency (95%) |
|
||||
|--------|-------------|---------------|-----------------|
|
||||
| Disk | 80% | 90% | 95% |
|
||||
| RAM | 80% | 90% | 95% |
|
||||
| CPU load | 80% cores utilized | 90% | 95% |
|
||||
|
||||
## Alert tracking
|
||||
`/root/.hermes/data/threshold-alerts.json` — keys format: `{hostname}_{metric}_{threshold}`
|
||||
- 24-hour cooldown per key
|
||||
- Auto-resets when metric drops below threshold
|
||||
- Re-alerts if metric crosses again after cooldown
|
||||
|
||||
## Alert format (Telegram)
|
||||
```
|
||||
🚨 VPS Alert: ai.itpropartner.com
|
||||
⚠️ DISK at 92% (threshold: 90%)
|
||||
⚠️ RAM at 83% (threshold: 80%)
|
||||
```
|
||||
|
||||
Multiple thresholds on one server are combined into a single message.
|
||||
Reference in New Issue
Block a user