Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,57 @@
+# Apex WPForms Email Troubleshooting
+
+WPForms on apextrackexperience.com (wphost02, RunCloud CPX21) uses WP Mail SMTP plugin to send through c1113726.sgvps.net:2525.
+
+## Known Failure Modes
+
+### 1. PHP Serialization Corruption
+
+The password stored in WP Mail SMTP settings uses PHP serialized format. If the password contains special characters, the serialized length (s:N:) can become wrong:
+
+```
+s:72:"apex.track!!"   ← WRONG: password is 13 chars, not 72
+s:13:"apex.track!!"  ← CORRECT
+```
+
+**Symptom:** SMTP auth fails silently. WP Mail SMTP can't read the password. Registration emails never send. The site shows "form submitted" but no email arrives.
+
+**Fix:** UPDATE wp_options SET option_value = REPLACE(option_value, 's:WRONG_LEN:"', 's:CORRECT_LEN:"') WHERE option_name = 'wp_mail_smtp'
+
+### 2. Sender Address Contains Multiple Emails
+
+The sender_address field in form notifications must be a single email address. WPForms allows entering comma-separated emails but this creates an invalid From header:
+
+```
+"sender_address":"contact@apextrackexperience.com, g@germainebrown.com"  ← INVALID
+"sender_address":"contact@apextrackexperience.com"                       ← VALID
+```
+
+**Symptom:** SMTP server rejects with "Message rejected. domain.com is not currently owned by sender"
+
+**Fix:** UPDATE wp_posts SET post_content = REPLACE(post_content, 'sender_address":"...", g@...', 'sender_address":"contact@apextrackexperience.com') WHERE ID IN (form_ids)
+
+### 3. Password Format Changed by WP Mail SMTP Plugin Update
+
+Plugin updates can re-serialize the password with incorrect length. Check after every WP Mail SMTP or WPForms update.
+
+## Manual Verification
+
+```bash
+ssh -i /root/.ssh/itpp-infra root@5.161.62.38 '
+mysql -u apextrackexperience_1781549652 -p"K3E1ZZWvHDu0q8ZmoBCAhzKUZawEapdGBlbaPME1sOTKgGk9FCuYS" apextrackexperience_1781549652 -e "
+SELECT option_value FROM wp_options WHERE option_name = \"wp_mail_smtp\";
+" | grep -o "s:[0-9]*:\\\\\"apex" | head -1
+'
+```
+
+If the length (s:N) is wrong, apply fix #1.
+
+## Watchdog Cron
+
+The cron `apex-mail-watchdog` runs every 5 min on Core:
+- Sends test SMTP email via the Apex SMTP server
+- Checks wp_wpmailsmtp_debug_events for recent failures
+- Silent on success, alerts on failure
+
+Script: /root/.hermes/scripts/apex-mail-watchdog.sh
+Log: /var/log/apex-mail-watchdog.log
@@ -0,0 +1,46 @@
+# Cross-Session Project + Memory Audit Protocol
+
+Use when Germaine asks to review recent conversations, audit past work, reconcile inconsistent memory, or inventory current projects.
+
+## Trigger phrases
+- "take a look at my conversations over the week"
+- "audit past work"
+- "current projects need to be audited"
+- "memories are not consistent"
+- "what have we been working on"
+
+## Source order
+1. Verify the session database is readable: `/root/.hermes/state.db` with SQLite `PRAGMA quick_check` or `integrity_check`.
+2. Pull recent sessions grouped by date/source from `sessions` + `messages`.
+3. Read project/reference sources that already summarize work:
+   - `/root/.hermes/projects-log.md`
+   - `/root/projects/` project folders
+   - `/root/.hermes/references/*audit*`, `*inventory*`, `*plan*`, and issue logs
+4. Read current durable memory files and fact store when memory consistency is in scope:
+   - `/root/.hermes/memories/MEMORY.md`
+   - `/root/.hermes/memories/USER.md`
+   - fact store entries, if available
+5. Inspect active cron jobs when the week involved automation/model/config changes.
+6. Use `session_search` for targeted evidence windows, not as the only inventory source.
+
+## Audit output shape
+Keep Telegram output compact:
+- **Scope reviewed:** date range, source DB, integrity status
+- **Current projects found:** grouped list
+- **Memory issues:** stale facts, duplicates, secrets, procedural content in memory, junk facts
+- **Highest-risk audit targets:** Critical / High / Medium
+- **What was not changed:** explicitly state read-only if no modifications were made
+
+## Verification rules
+- Do not treat subagent summaries as proof. Check DB, files, cron state, or live systems yourself before reporting success.
+- If a subagent returns generic output or claims no access despite being given local paths, reject it and complete or re-delegate the work.
+- Prefer evidence handles: session IDs, file paths, job IDs, timestamps.
+- Never say a project is complete based only on `/root/.hermes/projects-log.md`; treat it as a lead for verification.
+
+## Common findings to check
+- Memory duplicates after consolidation cron
+- Plaintext secrets/API keys in memory/fact store
+- Stale server inventory after migrations/rebalances
+- Model/delegation config drift after model changes
+- Cron jobs pinned to stale or paid models
+- Subagent claims about deployments, S3 uploads, email sends, DNS, or service health
@@ -0,0 +1,47 @@
+# Live WPForms Verification Technique
+
+Demonstrated Jul 13, 2026 when verifying the Apex Track Experience Roebling Road registration form.
+
+## Problem
+Memory/plans said a vehicle selector field existed on the Apex registration form. Live verification proved it didn't. Need a reliable way to inspect WPForms on live WordPress sites.
+
+## Technique
+
+### 1. Enumerate all form fields via CSS classes
+```bash
+curl -sk --connect-timeout 10 'https://example.com/page/' \
+  | grep -oP 'wpforms-field-[a-zA-Z0-9_-]+' | sort -u
+```
+This extracts every field type (name, email, phone, address, payment-single, payment-total, layout, etc.) without needing to parse the DOM.
+
+### 2. Check for specific field/data presence
+```bash
+curl -sk --connect-timeout 10 'https://example.com/page/' \
+  | grep -i -c 'vehicle\|apex-vehicle\|car-select'
+```
+Returns count of matches — 0 means field/payload absent.
+
+### 3. Get form content (readable)
+```python
+web_extract(urls=["https://example.com/page/"], char_limit=10000)
+```
+Clean markdown output with form labels, field types, option lists, and payment amounts. Best for understanding what the user sees.
+
+### 4. Get raw form HTML for structure analysis
+```bash
+curl -sk --connect-timeout 10 'https://example.com/page/' \
+  | grep -i 'vehicle'
+```
+Shows surrounding HTML structure including field IDs, quantity selectors, order summary rows.
+
+## Application to Apex
+- Form #272 on /roebling-road/
+- Fields: name, email, phone, address, payment-single (×5 items), payment-total, paypal-commerce
+- No vehicle selector, no `apex-vehicle-field`, no make/model/year dropdown
+- "Additional Vehicle" is a quantity picker (0-5), not an identification field
+- Memory had claimed `apex-vehicle-field` CSS class existed — it doesn't
+
+## Pitfalls
+- Browser tools may fail (Chromium/DBus issues on headless servers) — always have curl fallback
+- `curl | python3` pipes are blocked by security scanning — use web_extract or curl+grep instead
+- WPForms loads field CSS even when the form isn't fully rendered — `wpforms-field-*` classes are reliable indicators of configured fields
@@ -0,0 +1,45 @@
+# Operational Claim Evidence
+
+Use this reference when reconstructing an incident or reporting completion of infrastructure work.
+
+## Evidence hierarchy
+
+1. Direct postcondition read from the affected system.
+2. Tool output with host identity, timestamp, and exit status.
+3. Corroborating logs or remote readback.
+4. User confirmation.
+5. Subagent summary or prior assistant statement -- lead only, never proof.
+
+## Claim-to-proof matrix
+
+- **Command ran**: tool output and exit status.
+- **Service restarted**: old/new PID or start timestamp, then health check.
+- **Host rebooted**: boot ID or boot time changed; uninterrupted chat does not prove or disprove reboot by itself.
+- **DNS migrated**: authoritative DNS answer and public hostname reaches intended host.
+- **TLS fixed**: normal certificate validation succeeds; `curl -k` cannot prove this.
+- **Gateway working**: process active, inbound update observed, model/provider produces a reply, and outbound delivery succeeds. Process presence alone is insufficient.
+- **Provider working**: authenticated minimal inference request succeeds. `/health`, DNS, TCP reachability, or model listing alone is insufficient.
+- **Backup valid**: exact object exists, size/checksum read back, archive integrity test passes, and required contents are present.
+- **Restore completed**: exact source artifact identified; restored files match; services and user-facing paths pass tests.
+- **Failover completed**: primary fenced, standby active, state freshness verified, and messaging works end to end.
+- **Cleanup freed space**: before/after filesystem or object usage values.
+
+## Incident timeline reconstruction
+
+When prior chat contains unsupported success claims:
+
+1. Treat assistant prose as allegations, not facts.
+2. Build the timeline from service journals, session tool records, remote state, object listings, and file mtimes.
+3. Separate actual events from claimed events.
+4. Retract contradicted claims explicitly.
+5. List unresolved state separately from completed work.
+
+## Reporting vocabulary
+
+- **Planned**: proposed only.
+- **Started**: execution evidence exists, postcondition pending.
+- **Verified**: postcondition independently confirmed.
+- **Unverified**: claim exists but evidence is incomplete.
+- **Failed/blocked**: action or verification failed; include the blocker.
+
+Never use "should be working" as a completion report.