Apex WPForms Mail Debugging

Quick reference for diagnosing and fixing Apex Track Experience WPForms email delivery failures.

Architecture

Component	Value
Server	wphost02 (5.161.62.38, Hetzner CPX21)
DB	MySQL via RunCloud
SMTP	c1113726.sgvps.net:2525, STARTTLS
SMTP user	contact@apextrackexperience.com
SMTP pass	apex.track!!
Plugin	WP Mail SMTP (Lite)
Forms	NASA registration (270), Waiver (268)
Last known SMTP creds	Username: `contact@apextrackexperience.com`, Password: `apex.track!!`

Common Failure Modes

1. PHP Serialized Password Mismatch (most common cause)

WP Mail SMTP stores the SMTP password in the wp_options table as wp_mail_smtp in PHP serialized format. If the declared string length doesn't match the actual string, the plugin silently fails to authenticate.

Symptoms: Form submissions appear successful in the UI, but no email is sent. The debug events table shows event_type = 0 with no useful error text.

Check command:

SELECT JSON_EXTRACT(option_value, '$.smtp.pass') FROM wp_options WHERE option_name = 'wp_mail_smtp';

The JSON_EXTRACT result shows the raw serialized PHP value. A broken entry looks like:

s:72:\"apex.track!!\"
-- length 72 declared, but 'apex.track!!' is only 13 characters

Fix command:

UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\"apex.track!!\"', 's:13:\"apex.track!!\"') WHERE option_name = 'wp_mail_smtp';

2. WPForms sender_address

Both forms had contact@apextrackexperience.com, g@germainebrown.com as the sender_address. This is invalid — From: headers require a single address. An SMTP server receiving two comma-separated addresses may reject the MAIL FROM command.

Check command (form 270 and 268):

SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications') AS notifs FROM wp_posts WHERE ID IN (270, 268);

Fix: Set sender_address to contact@apextrackexperience.com only. The notification recipient g@germainebrown.com is configured separately as an email notification destination, not the sender.

3. SMTP Network Reachability

From wphost02:

# Test connection (not sending mail)
timeout 5 bash -c 'echo | openssl s_client -connect c1113726.sgvps.net:2525 -starttls smtp'

The CONNECTED line should appear in the output.

4. Debug Events Table

WP Mail SMTP logs email delivery attempts to wp_wpmailsmtp_debug_events:

SELECT id, subject, status, date_sent, error_text FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 5;

event_type = 0 — email sent (or attempted)
event_type = 1 — email failed
Check created_at >= NOW() - INTERVAL 10 MINUTE for recent attempts

Watchdog

A cron job on Core monitors Apex mail every 5 minutes:

/root/.hermes/scripts/apex-mail-watchdog.sh
Tests SMTP LOGIN (does NOT send a test email — only verifies credentials)
Checks debug events for recent failures
Silent on success, alerts Germaine on failure
Do NOT use tee -a in the log function — it causes every log line to be delivered as a cron message

Watchdog Blindness (SSH-dependency failure mode)

The watchdog runs FROM Core and SSHes to wphost02 (root@5.161.62.38, key /root/.ssh/itpp-infra, port 22) to test SMTP and read the debug table. Both steps require that SSH hop. If SSH is refused/unreachable, the watchdog keeps "running" every 5 min but tests NOTHING — the log fills with:

SMTP FAILED: ssh: connect to host 5.161.62.38 port 22: Connection refused

This is a monitoring blind spot, not a mail failure: the watchdog cannot distinguish "SMTP broken" from "I can't reach the box." When auditing Apex mail, ALWAYS confirm the SSH hop works before trusting a "FAIL" verdict:

ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=5 root@5.161.62.38 'echo reachable'
tail -5 /var/log/apex-mail-watchdog.log   # look for "connect to host ... Connection refused"

If SSH is refused: the box may be down, rebooting, mid-migration, or its SSH port/IP changed. During the Hetzner->netcup migration this is expected churn — wphost02 (Apex/WordPress) is slated to move to app3, which changes IP and possibly port. When a WordPress/mail host migrates, update WPHOST and any port in apex-mail-watchdog.sh and re-verify the hop. Do not report "Apex mail is down" from a Connection refused line alone.

Watching the Watchdog

# Check the watchdog cron
cronjob action=list | grep apex

# View the watchdog log
cat /var/log/apex-mail-watchdog.log

# The ops portal at ops.itpropartner.com/cron.html shows the job status live

4.6 KiB Raw Blame History