# Multi-Server Health Check API Pattern

Added Jul 10, 2026 to ops portal. Replaced the single-server scraped-API health with ping-based live health for all 7 ITPP servers.

## Backend (FastAPI endpoint)

```python
SERVERS = [
    {"name": "Core", "ip": "152.53.192.33", "role": "Hermes, Docker, OSINT, Caddy", "tier": "Primary"},
    {"name": "app1", "ip": "152.53.36.131", "role": "AI Stack", "tier": "Standard"},
    {"name": "app2", "ip": "152.53.39.202", "role": "UNMS, UniFi, Traccar", "tier": "Standard"},
    {"name": "app3", "ip": "152.53.241.111", "role": "WordPress", "tier": "Standard"},
    {"name": "app1-bu", "ip": "5.161.114.8", "role": "Warm Standby", "tier": "Standby"},
    {"name": "AI Box (legacy)", "ip": "178.156.167.181", "role": "LiteLLM (migrating)", "tier": "Legacy"},
    {"name": "Docker Box (legacy)", "ip": "178.156.168.35", "role": "Uptime Kuma", "tier": "Legacy"},
]

@app.get("/api/servers/health")
async def servers_health(user: str = Depends(get_current_user)):
    results = []
    for s in SERVERS:
        try:
            start = time.time()
            result = subprocess.run(["ping", "-c", "1", "-W", "2", s["ip"]],
                                    capture_output=True, text=True, timeout=5)
            latency = round((time.time() - start) * 1000)
            alive = result.returncode == 0
        except:
            alive = False
            latency = None
        results.append({...})
    return {"servers": results, "count": len(results), "live": sum(...)}
```

## Frontend (vanilla JS)

Page: `/opt/ops-portal/static/servers.html`
Uses `Ops.apiFetch('/api/servers/health')` to get all 7 servers.
Renders as card grid with border-left color (green=LIVE, red=OFFLINE).
Shows: server name, status badge, role, IP, latency, tier.

## Pitfalls

- Ping requires root or CAP_NET_RAW. The ops-portal service runs as root, so ping works.
- Some servers block ICMP (firewalls). If a server shows OFFLINE but is actually up, check UFW rules.
- Legacy servers in migration status should still be monitored — they're running production services.
- The API endpoint requires JWT auth (like all /api/* routes).