hermes-skills/skills/devops/infrastructure-automation/references/service-monitoring-patterns.md

Service Monitoring Patterns & Operations Notes (Jul 8 2026)

Collection of monitoring cron patterns used on Core for external services. Each pattern is a no_agent cron that checks health and alerts on failure.

Apex Mail Watchdog

Purpose: Monitor WP Mail SMTP email delivery on a remote WordPress site (wphost02). Every: 5 min Script: /root/.hermes/scripts/apex-mail-watchdog.sh Deliver: origin (only messages on failure)

Actions:

1. SSH to wphost02 via itpp-infra key
2. Send test email via SMTP (c1113726.sgvps.net:2525)
3. Check wp_wpmailsmtp_debug_events table for recent failures
4. Silent on success (no output = no message), only alerts on failure

Common SMTP failure signs:

• WP Mail SMTP password PHP serialization length mismatch (s:72 vs actual s:13)
• Sender address containing comma-separated emails (invalid From: header)
• SMTP server rejecting MAIL FROM for unowned domain

Fix pattern:

1. Check wp_wpmailsmtp_debug_events for most recent error content
2. Test SMTP directly via Python: smtplib.SMTP(host, 2525).starttls()
3. If authentication fails: the PHP serialization of the password in wp_options is likely corrupted
4. SQL to fix: UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\\\"pass\\\"', 's:13:\\\"pass\\\"') WHERE option_name = 'wp_mail_smtp'

Known failure mode (Jul 8 2026):

WP Mail SMTP password was stored with wrong PHP serialization length (s:72 for 13-char password apex.track!!). WPForms sender_address had comma-separated emails (invalid From: header). Both fixed via SQL.

Ops Data Collector

Purpose: Collect comprehensive infrastructure status data for the ops dashboard at ops.itpropartner.com Every: 5 min Script: /root/.hermes/scripts/ops-data-collector.py Deliver: local (writes JSON file only)

Data sources:

• Hermes cron jobs (from /root/.hermes/cron/jobs.json)
• Systemd service status via systemctl
• S3 backup timestamps via AWS CLI (wasabisys endpoint)
• API health checks (admin-ai, Caddy, ports, Cloudflare)
• Hetzner server list via API
• netcup server info via SCP API
• Local disk/memory usage
• Service versions

JSON output:

/var/www/ops/data/ops-status.json -- consumed by the ops dashboard HTML page.

Build pattern (Jul 8 2026):

• Collector: self-contained Python stdlib script, no external deps
• Dashboard: single HTML page, no CDN/frameworks, dark theme (#0f172a / #1e293b), mobile-first
• Site: Caddy reverse proxy at ops.itpropartner.com with HTTP-01 ACME challenge
• DNS for itpropartner.com is at SiteGround (not Cloudflare), so Caddy handles TLS directly
• Data/ directory gets CORS headers for cross-origin access

Home Router Watchdog

Purpose: Verify home router is online. Every: 5 min Script: /root/.hermes/scripts/home-router-watchdog.sh Deliver: origin

Router Backup Pipeline

Purpose: Backup MikroTik router configs to S3. Every: Daily at 6 AM Script: run-wisp-backup.sh (referenced from /root/.hermes/cron/jobs.json)

VPN dependencies for L2TP/IPsec router backup:

• xl2tpd package
• strongSwan-starter package
• IP in config must match the router's SSH bind address (often WireGuard tunnel IP, not LAN IP)

Failure mode (Jul 8 2026):

Cron was using old home-router-backup.sh instead of proper run-wisp-backup.sh. Router SSH export produced stuck .in_progress files blocking SCP. Fix: switched cron, cleaned files, installed missing paramiko/xl2tpd/strongSwan.

Warm Standby Sync

Purpose: Keep app1-bu's local state fresh for rapid failover. Every: 10 min Script: /root/.hermes/scripts/hermes-standby-sync.sh on app1-bu Triggered by: System cron on app1-bu, not Hermes cron.

Netcup SCP API Notes (Discovered Jul 8 2026)

The REST API lives at https://www.servercontrolpanel.de/scp-core/api/v1/. Auth is OIDC password grant via Keycloak at https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token using client_id=scp with username/password. The CCP API key from Master Data does NOT work with the SCP API.

Credentials stored in /root/.hermes/.env as NETCUP_CUSTOMER, NETCUP_PASSWORD, NETCUP_API_KEY.

Infrastructure Consolidation Strategy (Jul 8 2026)

Hetzner new-server pricing is significantly inflated. API returns CPX11 at ~$17.49/mo vs historical ~$4. netcup root servers (RS 1000 at 4C/8G/256GB for ~$12.24/mo) offer better value. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.

Domain/Portal Separation Rule (Established Jul 8 2026)

Service-related operational tools must be hosted on the provider's own domain, NOT a customer's/industry domain. Two-portal architecture:

• ops.itpropartner.com -- internal admin dashboard, infra monitoring, scripts, status pages (Cloudflare Access SSO protected)
• portal.itpropartner.com -- customer-facing portal
• Never put IT Pro Partner operational tools on customer/themed subdomains (e.g. debtrecoveryexperts.com)

This is a hard architectural rule. When deciding where a new dashboard, page, or tool lives, the default answer is ops.itpropartner.com unless the user explicitly says otherwise.

Log Paths

Watchdog	Log
Apex mail	/var/log/apex-mail-watchdog.log
Home router	/var/log/hermes-standby-watchdog.log
Standby sync	/var/log/hermes-standby-sync.log
Ops collector	/var/log/ops-collector.log