# Service Monitoring Patterns & Operations Notes (Jul 8 2026)

Collection of monitoring cron patterns used on Core for external services.
Each pattern is a no_agent cron that checks health and alerts on failure.

## Apex Mail Watchdog

**Purpose:** Monitor WP Mail SMTP email delivery on a remote WordPress site (wphost02).
**Every:** 5 min
**Script:** `/root/.hermes/scripts/apex-mail-watchdog.sh`
**Deliver:** origin (only messages on failure)

### Actions:
1. SSH to wphost02 via itpp-infra key
2. Send test email via SMTP (c1113726.sgvps.net:2525)
3. Check wp_wpmailsmtp_debug_events table for recent failures
4. Silent on success (no output = no message), only alerts on failure

### Common SMTP failure signs:
- WP Mail SMTP password PHP serialization length mismatch (`s:72` vs actual `s:13`)
- Sender address containing comma-separated emails (invalid `From:` header)
- SMTP server rejecting `MAIL FROM` for unowned domain

### Fix pattern:
1. Check `wp_wpmailsmtp_debug_events` for most recent error content
2. Test SMTP directly via Python: `smtplib.SMTP(host, 2525).starttls()`
3. If authentication fails: the PHP serialization of the password in `wp_options` is likely corrupted
4. SQL to fix: `UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\\\"pass\\\"', 's:13:\\\"pass\\\"') WHERE option_name = 'wp_mail_smtp'`

### Known failure mode (Jul 8 2026):
WP Mail SMTP password was stored with wrong PHP serialization length (`s:72` for 13-char password `apex.track!!`). WPForms `sender_address` had comma-separated emails (invalid `From:` header). Both fixed via SQL.

## Ops Data Collector

**Purpose:** Collect comprehensive infrastructure status data for the ops dashboard at ops.itpropartner.com
**Every:** 5 min
**Script:** `/root/.hermes/scripts/ops-data-collector.py`
**Deliver:** local (writes JSON file only)

### Data sources:
- Hermes cron jobs (from /root/.hermes/cron/jobs.json)
- Systemd service status via systemctl
- S3 backup timestamps via AWS CLI (wasabisys endpoint)
- API health checks (admin-ai, Caddy, ports, Cloudflare)
- Hetzner server list via API
- netcup server info via SCP API
- Local disk/memory usage
- Service versions

### JSON output:
/var/www/ops/data/ops-status.json -- consumed by the ops dashboard HTML page.

### Build pattern (Jul 8 2026):
- Collector: self-contained Python stdlib script, no external deps
- Dashboard: single HTML page, no CDN/frameworks, dark theme (#0f172a / #1e293b), mobile-first
- Site: Caddy reverse proxy at ops.itpropartner.com with HTTP-01 ACME challenge
- DNS for itpropartner.com is at SiteGround (not Cloudflare), so Caddy handles TLS directly
- Data/ directory gets CORS headers for cross-origin access

## Home Router Watchdog

**Purpose:** Verify home router is online.
**Every:** 5 min
**Script:** `/root/.hermes/scripts/home-router-watchdog.sh`
**Deliver:** origin

### Router Backup Pipeline
**Purpose:** Backup MikroTik router configs to S3.
**Every:** Daily at 6 AM
**Script:** run-wisp-backup.sh (referenced from /root/.hermes/cron/jobs.json)

### VPN dependencies for L2TP/IPsec router backup:
- xl2tpd package
- strongSwan-starter package
- IP in config must match the router's SSH bind address (often WireGuard tunnel IP, not LAN IP)

### Failure mode (Jul 8 2026):
Cron was using old home-router-backup.sh instead of proper run-wisp-backup.sh. Router SSH export produced stuck .in_progress files blocking SCP. Fix: switched cron, cleaned files, installed missing paramiko/xl2tpd/strongSwan.

## Warm Standby Sync

**Purpose:** Keep app1-bu's local state fresh for rapid failover.
**Every:** 10 min
**Script:** `/root/.hermes/scripts/hermes-standby-sync.sh` on app1-bu
**Triggered by:** System cron on app1-bu, not Hermes cron.

## Netcup SCP API Notes (Discovered Jul 8 2026)

The REST API lives at `https://www.servercontrolpanel.de/scp-core/api/v1/`. Auth is OIDC password grant via Keycloak at `https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token` using `client_id=scp` with username/password. The CCP API key from Master Data does NOT work with the SCP API.

Credentials stored in /root/.hermes/.env as NETCUP_CUSTOMER, NETCUP_PASSWORD, NETCUP_API_KEY.

## Infrastructure Consolidation Strategy (Jul 8 2026)

Hetzner new-server pricing is significantly inflated. API returns CPX11 at ~$17.49/mo vs historical ~$4. netcup root servers (RS 1000 at 4C/8G/256GB for ~$12.24/mo) offer better value. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.

## Domain/Portal Separation Rule (Established Jul 8 2026)

Service-related operational tools must be hosted on the provider's own domain, NOT a customer's/industry domain. Two-portal architecture:
- **ops.itpropartner.com** -- internal admin dashboard, infra monitoring, scripts, status pages (Cloudflare Access SSO protected)
- **portal.itpropartner.com** -- customer-facing portal
- **Never** put IT Pro Partner operational tools on customer/themed subdomains (e.g. debtrecoveryexperts.com)

This is a hard architectural rule. When deciding where a new dashboard, page, or tool lives, the default answer is ops.itpropartner.com unless the user explicitly says otherwise.

## Log Paths

| Watchdog | Log |
|----------|-----|
| Apex mail | /var/log/apex-mail-watchdog.log |
| Home router | /var/log/hermes-standby-watchdog.log |
| Standby sync | /var/log/hermes-standby-sync.log |
| Ops collector | /var/log/ops-collector.log |