Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,113 @@
+# Service Monitoring Patterns & Operations Notes (Jul 8 2026)
+
+Collection of monitoring cron patterns used on Core for external services.
+Each pattern is a no_agent cron that checks health and alerts on failure.
+
+## Apex Mail Watchdog
+
+**Purpose:** Monitor WP Mail SMTP email delivery on a remote WordPress site (wphost02).
+**Every:** 5 min
+**Script:** `/root/.hermes/scripts/apex-mail-watchdog.sh`
+**Deliver:** origin (only messages on failure)
+
+### Actions:
+1. SSH to wphost02 via itpp-infra key
+2. Send test email via SMTP (c1113726.sgvps.net:2525)
+3. Check wp_wpmailsmtp_debug_events table for recent failures
+4. Silent on success (no output = no message), only alerts on failure
+
+### Common SMTP failure signs:
+- WP Mail SMTP password PHP serialization length mismatch (`s:72` vs actual `s:13`)
+- Sender address containing comma-separated emails (invalid `From:` header)
+- SMTP server rejecting `MAIL FROM` for unowned domain
+
+### Fix pattern:
+1. Check `wp_wpmailsmtp_debug_events` for most recent error content
+2. Test SMTP directly via Python: `smtplib.SMTP(host, 2525).starttls()`
+3. If authentication fails: the PHP serialization of the password in `wp_options` is likely corrupted
+4. SQL to fix: `UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\\\"pass\\\"', 's:13:\\\"pass\\\"') WHERE option_name = 'wp_mail_smtp'`
+
+### Known failure mode (Jul 8 2026):
+WP Mail SMTP password was stored with wrong PHP serialization length (`s:72` for 13-char password `apex.track!!`). WPForms `sender_address` had comma-separated emails (invalid `From:` header). Both fixed via SQL.
+
+## Ops Data Collector
+
+**Purpose:** Collect comprehensive infrastructure status data for the ops dashboard at ops.itpropartner.com
+**Every:** 5 min
+**Script:** `/root/.hermes/scripts/ops-data-collector.py`
+**Deliver:** local (writes JSON file only)
+
+### Data sources:
+- Hermes cron jobs (from /root/.hermes/cron/jobs.json)
+- Systemd service status via systemctl
+- S3 backup timestamps via AWS CLI (wasabisys endpoint)
+- API health checks (admin-ai, Caddy, ports, Cloudflare)
+- Hetzner server list via API
+- netcup server info via SCP API
+- Local disk/memory usage
+- Service versions
+
+### JSON output:
+/var/www/ops/data/ops-status.json -- consumed by the ops dashboard HTML page.
+
+### Build pattern (Jul 8 2026):
+- Collector: self-contained Python stdlib script, no external deps
+- Dashboard: single HTML page, no CDN/frameworks, dark theme (#0f172a / #1e293b), mobile-first
+- Site: Caddy reverse proxy at ops.itpropartner.com with HTTP-01 ACME challenge
+- DNS for itpropartner.com is at SiteGround (not Cloudflare), so Caddy handles TLS directly
+- Data/ directory gets CORS headers for cross-origin access
+
+## Home Router Watchdog
+
+**Purpose:** Verify home router is online.
+**Every:** 5 min
+**Script:** `/root/.hermes/scripts/home-router-watchdog.sh`
+**Deliver:** origin
+
+### Router Backup Pipeline
+**Purpose:** Backup MikroTik router configs to S3.
+**Every:** Daily at 6 AM
+**Script:** run-wisp-backup.sh (referenced from /root/.hermes/cron/jobs.json)
+
+### VPN dependencies for L2TP/IPsec router backup:
+- xl2tpd package
+- strongSwan-starter package
+- IP in config must match the router's SSH bind address (often WireGuard tunnel IP, not LAN IP)
+
+### Failure mode (Jul 8 2026):
+Cron was using old home-router-backup.sh instead of proper run-wisp-backup.sh. Router SSH export produced stuck .in_progress files blocking SCP. Fix: switched cron, cleaned files, installed missing paramiko/xl2tpd/strongSwan.
+
+## Warm Standby Sync
+
+**Purpose:** Keep app1-bu's local state fresh for rapid failover.
+**Every:** 10 min
+**Script:** `/root/.hermes/scripts/hermes-standby-sync.sh` on app1-bu
+**Triggered by:** System cron on app1-bu, not Hermes cron.
+
+## Netcup SCP API Notes (Discovered Jul 8 2026)
+
+The REST API lives at `https://www.servercontrolpanel.de/scp-core/api/v1/`. Auth is OIDC password grant via Keycloak at `https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token` using `client_id=scp` with username/password. The CCP API key from Master Data does NOT work with the SCP API.
+
+Credentials stored in /root/.hermes/.env as NETCUP_CUSTOMER, NETCUP_PASSWORD, NETCUP_API_KEY.
+
+## Infrastructure Consolidation Strategy (Jul 8 2026)
+
+Hetzner new-server pricing is significantly inflated. API returns CPX11 at ~$17.49/mo vs historical ~$4. netcup root servers (RS 1000 at 4C/8G/256GB for ~$12.24/mo) offer better value. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
+
+## Domain/Portal Separation Rule (Established Jul 8 2026)
+
+Service-related operational tools must be hosted on the provider's own domain, NOT a customer's/industry domain. Two-portal architecture:
+- **ops.itpropartner.com** -- internal admin dashboard, infra monitoring, scripts, status pages (Cloudflare Access SSO protected)
+- **portal.itpropartner.com** -- customer-facing portal
+- **Never** put IT Pro Partner operational tools on customer/themed subdomains (e.g. debtrecoveryexperts.com)
+
+This is a hard architectural rule. When deciding where a new dashboard, page, or tool lives, the default answer is ops.itpropartner.com unless the user explicitly says otherwise.
+
+## Log Paths
+
+| Watchdog | Log |
+|----------|-----|
+| Apex mail | /var/log/apex-mail-watchdog.log |
+| Home router | /var/log/hermes-standby-watchdog.log |
+| Standby sync | /var/log/hermes-standby-sync.log |
+| Ops collector | /var/log/ops-collector.log |