Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,110 @@
+# Apex WPForms Mail Debugging
+
+Quick reference for diagnosing and fixing Apex Track Experience WPForms email delivery failures.
+
+## Architecture
+
+| Component | Value |
+|-----------|-------|
+| Server | wphost02 (5.161.62.38, Hetzner CPX21) |
+| DB | MySQL via RunCloud |
+| SMTP | c1113726.sgvps.net:2525, STARTTLS |
+| SMTP user | contact@apextrackexperience.com |
+| SMTP pass | apex.track!! |
+| Plugin | WP Mail SMTP (Lite) |
+| Forms | NASA registration (270), Waiver (268) |
+| Last known SMTP creds | Username: `contact@apextrackexperience.com`, Password: `apex.track!!` |
+
+## Common Failure Modes
+
+### 1. PHP Serialized Password Mismatch (most common cause)
+
+WP Mail SMTP stores the SMTP password in the `wp_options` table as `wp_mail_smtp` in PHP serialized format. If the declared string length doesn't match the actual string, the plugin silently fails to authenticate.
+
+**Symptoms:** Form submissions appear successful in the UI, but no email is sent. The debug events table shows `event_type = 0` with no useful error text.
+
+**Check command:**
+```sql
+SELECT JSON_EXTRACT(option_value, '$.smtp.pass') FROM wp_options WHERE option_name = 'wp_mail_smtp';
+```
+
+The JSON_EXTRACT result shows the raw serialized PHP value. A broken entry looks like:
+```
+s:72:\"apex.track!!\"
+-- length 72 declared, but 'apex.track!!' is only 13 characters
+```
+
+**Fix command:**
+```sql
+UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\"apex.track!!\"', 's:13:\"apex.track!!\"') WHERE option_name = 'wp_mail_smtp';
+```
+
+### 2. WPForms sender_address
+
+Both forms had `contact@apextrackexperience.com, g@germainebrown.com` as the `sender_address`. This is invalid — `From:` headers require a single address. An SMTP server receiving two comma-separated addresses may reject the `MAIL FROM` command.
+
+**Check command (form 270 and 268):**
+```sql
+SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications') AS notifs FROM wp_posts WHERE ID IN (270, 268);
+```
+
+**Fix:** Set `sender_address` to `contact@apextrackexperience.com` only. The notification recipient `g@germainebrown.com` is configured separately as an email notification destination, not the sender.
+
+### 3. SMTP Network Reachability
+
+From wphost02:
+```bash
+# Test connection (not sending mail)
+timeout 5 bash -c 'echo | openssl s_client -connect c1113726.sgvps.net:2525 -starttls smtp'
+```
+
+The `CONNECTED` line should appear in the output.
+
+### 4. Debug Events Table
+
+WP Mail SMTP logs email delivery attempts to `wp_wpmailsmtp_debug_events`:
+```sql
+SELECT id, subject, status, date_sent, error_text FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 5;
+```
+
+- `event_type = 0` — email sent (or attempted)
+- `event_type = 1` — email failed
+- Check `created_at >= NOW() - INTERVAL 10 MINUTE` for recent attempts
+
+## Watchdog
+
+A cron job on Core monitors Apex mail every 5 minutes:
+- `/root/.hermes/scripts/apex-mail-watchdog.sh`
+- Tests SMTP LOGIN (does NOT send a test email — only verifies credentials)
+- Checks debug events for recent failures
+- Silent on success, alerts Germaine on failure
+- Do NOT use `tee -a` in the log function — it causes every log line to be delivered as a cron message
+
+## Watchdog Blindness (SSH-dependency failure mode)
+
+The watchdog runs FROM Core and SSHes to wphost02 (`root@5.161.62.38`, key `/root/.ssh/itpp-infra`, port 22) to test SMTP and read the debug table. Both steps require that SSH hop. **If SSH is refused/unreachable, the watchdog keeps "running" every 5 min but tests NOTHING** — the log fills with:
+
+```
+SMTP FAILED: ssh: connect to host 5.161.62.38 port 22: Connection refused
+```
+
+This is a monitoring blind spot, not a mail failure: the watchdog cannot distinguish "SMTP broken" from "I can't reach the box." When auditing Apex mail, ALWAYS confirm the SSH hop works before trusting a "FAIL" verdict:
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=5 root@5.161.62.38 'echo reachable'
+tail -5 /var/log/apex-mail-watchdog.log   # look for "connect to host ... Connection refused"
+```
+
+If SSH is refused: the box may be down, rebooting, mid-migration, or its SSH port/IP changed. **During the Hetzner->netcup migration this is expected churn** — wphost02 (Apex/WordPress) is slated to move to app3, which changes IP and possibly port. When a WordPress/mail host migrates, update `WPHOST` and any port in `apex-mail-watchdog.sh` and re-verify the hop. Do not report "Apex mail is down" from a `Connection refused` line alone.
+
+## Watching the Watchdog
+
+```bash
+# Check the watchdog cron
+cronjob action=list | grep apex
+
+# View the watchdog log
+cat /var/log/apex-mail-watchdog.log
+
+# The ops portal at ops.itpropartner.com/cron.html shows the job status live
+```