# Apex WPForms Mail Debugging

Quick reference for diagnosing and fixing Apex Track Experience WPForms email delivery failures.

## Architecture

| Component | Value |
|-----------|-------|
| Server | wphost02 (5.161.62.38, Hetzner CPX21) |
| DB | MySQL via RunCloud |
| SMTP | c1113726.sgvps.net:2525, STARTTLS |
| SMTP user | contact@apextrackexperience.com |
| SMTP pass | apex.track!! |
| Plugin | WP Mail SMTP (Lite) |
| Forms | NASA registration (270), Waiver (268) |
| Last known SMTP creds | Username: `contact@apextrackexperience.com`, Password: `apex.track!!` |

## Common Failure Modes

### 1. PHP Serialized Password Mismatch (most common cause)

WP Mail SMTP stores the SMTP password in the `wp_options` table as `wp_mail_smtp` in PHP serialized format. If the declared string length doesn't match the actual string, the plugin silently fails to authenticate.

**Symptoms:** Form submissions appear successful in the UI, but no email is sent. The debug events table shows `event_type = 0` with no useful error text.

**Check command:**
```sql
SELECT JSON_EXTRACT(option_value, '$.smtp.pass') FROM wp_options WHERE option_name = 'wp_mail_smtp';
```

The JSON_EXTRACT result shows the raw serialized PHP value. A broken entry looks like:
```
s:72:\"apex.track!!\"
-- length 72 declared, but 'apex.track!!' is only 13 characters
```

**Fix command:**
```sql
UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\"apex.track!!\"', 's:13:\"apex.track!!\"') WHERE option_name = 'wp_mail_smtp';
```

### 2. WPForms sender_address

Both forms had `contact@apextrackexperience.com, g@germainebrown.com` as the `sender_address`. This is invalid — `From:` headers require a single address. An SMTP server receiving two comma-separated addresses may reject the `MAIL FROM` command.

**Check command (form 270 and 268):**
```sql
SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications') AS notifs FROM wp_posts WHERE ID IN (270, 268);
```

**Fix:** Set `sender_address` to `contact@apextrackexperience.com` only. The notification recipient `g@germainebrown.com` is configured separately as an email notification destination, not the sender.

### 3. SMTP Network Reachability

From wphost02:
```bash
# Test connection (not sending mail)
timeout 5 bash -c 'echo | openssl s_client -connect c1113726.sgvps.net:2525 -starttls smtp'
```

The `CONNECTED` line should appear in the output.

### 4. Debug Events Table

WP Mail SMTP logs email delivery attempts to `wp_wpmailsmtp_debug_events`:
```sql
SELECT id, subject, status, date_sent, error_text FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 5;
```

- `event_type = 0` — email sent (or attempted)
- `event_type = 1` — email failed
- Check `created_at >= NOW() - INTERVAL 10 MINUTE` for recent attempts

## Watchdog

A cron job on Core monitors Apex mail every 5 minutes:
- `/root/.hermes/scripts/apex-mail-watchdog.sh`
- Tests SMTP LOGIN (does NOT send a test email — only verifies credentials)
- Checks debug events for recent failures
- Silent on success, alerts Germaine on failure
- Do NOT use `tee -a` in the log function — it causes every log line to be delivered as a cron message

## Watchdog Blindness (SSH-dependency failure mode)

The watchdog runs FROM Core and SSHes to wphost02 (`root@5.161.62.38`, key `/root/.ssh/itpp-infra`, port 22) to test SMTP and read the debug table. Both steps require that SSH hop. **If SSH is refused/unreachable, the watchdog keeps "running" every 5 min but tests NOTHING** — the log fills with:

```
SMTP FAILED: ssh: connect to host 5.161.62.38 port 22: Connection refused
```

This is a monitoring blind spot, not a mail failure: the watchdog cannot distinguish "SMTP broken" from "I can't reach the box." When auditing Apex mail, ALWAYS confirm the SSH hop works before trusting a "FAIL" verdict:

```bash
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=5 root@5.161.62.38 'echo reachable'
tail -5 /var/log/apex-mail-watchdog.log   # look for "connect to host ... Connection refused"
```

If SSH is refused: the box may be down, rebooting, mid-migration, or its SSH port/IP changed. **During the Hetzner->netcup migration this is expected churn** — wphost02 (Apex/WordPress) is slated to move to app3, which changes IP and possibly port. When a WordPress/mail host migrates, update `WPHOST` and any port in `apex-mail-watchdog.sh` and re-verify the hop. Do not report "Apex mail is down" from a `Connection refused` line alone.

## Watching the Watchdog

```bash
# Check the watchdog cron
cronjob action=list | grep apex

# View the watchdog log
cat /var/log/apex-mail-watchdog.log

# The ops portal at ops.itpropartner.com/cron.html shows the job status live
```