114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
# Service Monitoring Patterns & Operations Notes (Jul 8 2026)
|
|
|
|
Collection of monitoring cron patterns used on Core for external services.
|
|
Each pattern is a no_agent cron that checks health and alerts on failure.
|
|
|
|
## Apex Mail Watchdog
|
|
|
|
**Purpose:** Monitor WP Mail SMTP email delivery on a remote WordPress site (wphost02).
|
|
**Every:** 5 min
|
|
**Script:** `/root/.hermes/scripts/apex-mail-watchdog.sh`
|
|
**Deliver:** origin (only messages on failure)
|
|
|
|
### Actions:
|
|
1. SSH to wphost02 via itpp-infra key
|
|
2. Send test email via SMTP (c1113726.sgvps.net:2525)
|
|
3. Check wp_wpmailsmtp_debug_events table for recent failures
|
|
4. Silent on success (no output = no message), only alerts on failure
|
|
|
|
### Common SMTP failure signs:
|
|
- WP Mail SMTP password PHP serialization length mismatch (`s:72` vs actual `s:13`)
|
|
- Sender address containing comma-separated emails (invalid `From:` header)
|
|
- SMTP server rejecting `MAIL FROM` for unowned domain
|
|
|
|
### Fix pattern:
|
|
1. Check `wp_wpmailsmtp_debug_events` for most recent error content
|
|
2. Test SMTP directly via Python: `smtplib.SMTP(host, 2525).starttls()`
|
|
3. If authentication fails: the PHP serialization of the password in `wp_options` is likely corrupted
|
|
4. SQL to fix: `UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\\\"pass\\\"', 's:13:\\\"pass\\\"') WHERE option_name = 'wp_mail_smtp'`
|
|
|
|
### Known failure mode (Jul 8 2026):
|
|
WP Mail SMTP password was stored with wrong PHP serialization length (`s:72` for 13-char password `apex.track!!`). WPForms `sender_address` had comma-separated emails (invalid `From:` header). Both fixed via SQL.
|
|
|
|
## Ops Data Collector
|
|
|
|
**Purpose:** Collect comprehensive infrastructure status data for the ops dashboard at ops.itpropartner.com
|
|
**Every:** 5 min
|
|
**Script:** `/root/.hermes/scripts/ops-data-collector.py`
|
|
**Deliver:** local (writes JSON file only)
|
|
|
|
### Data sources:
|
|
- Hermes cron jobs (from /root/.hermes/cron/jobs.json)
|
|
- Systemd service status via systemctl
|
|
- S3 backup timestamps via AWS CLI (wasabisys endpoint)
|
|
- API health checks (admin-ai, Caddy, ports, Cloudflare)
|
|
- Hetzner server list via API
|
|
- netcup server info via SCP API
|
|
- Local disk/memory usage
|
|
- Service versions
|
|
|
|
### JSON output:
|
|
/var/www/ops/data/ops-status.json -- consumed by the ops dashboard HTML page.
|
|
|
|
### Build pattern (Jul 8 2026):
|
|
- Collector: self-contained Python stdlib script, no external deps
|
|
- Dashboard: single HTML page, no CDN/frameworks, dark theme (#0f172a / #1e293b), mobile-first
|
|
- Site: Caddy reverse proxy at ops.itpropartner.com with HTTP-01 ACME challenge
|
|
- DNS for itpropartner.com is at SiteGround (not Cloudflare), so Caddy handles TLS directly
|
|
- Data/ directory gets CORS headers for cross-origin access
|
|
|
|
## Home Router Watchdog
|
|
|
|
**Purpose:** Verify home router is online.
|
|
**Every:** 5 min
|
|
**Script:** `/root/.hermes/scripts/home-router-watchdog.sh`
|
|
**Deliver:** origin
|
|
|
|
### Router Backup Pipeline
|
|
**Purpose:** Backup MikroTik router configs to S3.
|
|
**Every:** Daily at 6 AM
|
|
**Script:** run-wisp-backup.sh (referenced from /root/.hermes/cron/jobs.json)
|
|
|
|
### VPN dependencies for L2TP/IPsec router backup:
|
|
- xl2tpd package
|
|
- strongSwan-starter package
|
|
- IP in config must match the router's SSH bind address (often WireGuard tunnel IP, not LAN IP)
|
|
|
|
### Failure mode (Jul 8 2026):
|
|
Cron was using old home-router-backup.sh instead of proper run-wisp-backup.sh. Router SSH export produced stuck .in_progress files blocking SCP. Fix: switched cron, cleaned files, installed missing paramiko/xl2tpd/strongSwan.
|
|
|
|
## Warm Standby Sync
|
|
|
|
**Purpose:** Keep app1-bu's local state fresh for rapid failover.
|
|
**Every:** 10 min
|
|
**Script:** `/root/.hermes/scripts/hermes-standby-sync.sh` on app1-bu
|
|
**Triggered by:** System cron on app1-bu, not Hermes cron.
|
|
|
|
## Netcup SCP API Notes (Discovered Jul 8 2026)
|
|
|
|
The REST API lives at `https://www.servercontrolpanel.de/scp-core/api/v1/`. Auth is OIDC password grant via Keycloak at `https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token` using `client_id=scp` with username/password. The CCP API key from Master Data does NOT work with the SCP API.
|
|
|
|
Credentials stored in /root/.hermes/.env as NETCUP_CUSTOMER, NETCUP_PASSWORD, NETCUP_API_KEY.
|
|
|
|
## Infrastructure Consolidation Strategy (Jul 8 2026)
|
|
|
|
Hetzner new-server pricing is significantly inflated. API returns CPX11 at ~$17.49/mo vs historical ~$4. netcup root servers (RS 1000 at 4C/8G/256GB for ~$12.24/mo) offer better value. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
|
|
|
|
## Domain/Portal Separation Rule (Established Jul 8 2026)
|
|
|
|
Service-related operational tools must be hosted on the provider's own domain, NOT a customer's/industry domain. Two-portal architecture:
|
|
- **ops.itpropartner.com** -- internal admin dashboard, infra monitoring, scripts, status pages (Cloudflare Access SSO protected)
|
|
- **portal.itpropartner.com** -- customer-facing portal
|
|
- **Never** put IT Pro Partner operational tools on customer/themed subdomains (e.g. debtrecoveryexperts.com)
|
|
|
|
This is a hard architectural rule. When deciding where a new dashboard, page, or tool lives, the default answer is ops.itpropartner.com unless the user explicitly says otherwise.
|
|
|
|
## Log Paths
|
|
|
|
| Watchdog | Log |
|
|
|----------|-----|
|
|
| Apex mail | /var/log/apex-mail-watchdog.log |
|
|
| Home router | /var/log/hermes-standby-watchdog.log |
|
|
| Standby sync | /var/log/hermes-standby-sync.log |
|
|
| Ops collector | /var/log/ops-collector.log |
|