Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,255 @@
+---
+name: imap-email-search-and-extract
+description: "Search, extract, and process emails from IMAP inbox — flight itineraries, attachment PDFs, confirmation codes, and structured data from HTML/plain-text emails."
+version: 1.2.0
+author: ShoNuff
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [email, imap, pdf, itinerary, flight, extraction]
+---
+
+# IMAP Email Search and Extract
+
+Search the user's IMAP inbox for specific emails and extract structured data.
+
+## Standard approach\n\n### Connection setup\n\nCredentials typically come from Himalaya password files (`/root/.config/himalaya/*.pass`). For WordPress-hosted sites (Apex Track Experience, ITPP customer sites), SMTP/IMAP credentials may instead be in the WordPress database — see `references/wordpress-smtp-credential-discovery.md`.\n\n```python
+import imaplib, email
+HOST = "mail.germainebrown.com"
+PORT = 993
+USER = "g@germainebrown.com"
+PW = open("/root/.config/himalaya/g-germainebrown.pass").read().strip()
+
+conn = imaplib.IMAP4_SSL(HOST, PORT)
+conn.login(USER, PW)
+conn.select("INBOX")
+```
+
+### Searching strategies
+
+**By sender and subject:**
+```python
+status, data = conn.search(None, 'FROM "Copa Airlines" SUBJECT "Cartagena"')
+```
+
+**By date range:**
+```python
+status, data = conn.search(None, '(SINCE "28-May-2026" BEFORE "5-Jul-2026")')
+```
+
+**Broad OR search across multiple keywords:**
+```python
+status, data = conn.search(None, 'OR OR SUBJECT "Cartagena" SUBJECT "flight" SUBJECT "itinerary"')
+```
+
+**Body text match:**
+```python
+status, data = conn.search(None, 'BODY "Cartagena"')
+```
+
+**Chain multiple filters:**
+```python
+# Sender + keyword across subject + body
+status, data = conn.search(None, 'FROM "Copa" OR SUBJECT "Cartagena" BODY "Cartagena"')
+```
+
+### Decoding email content
+
+**Get body (plain text or HTML):**
+```python
+def get_body(msg):
+    if msg.is_multipart():
+        for part in msg.walk():
+            ct = part.get_content_type()
+            if ct in ("text/plain", "text/html"):
+                payload = part.get_payload(decode=True)
+                if payload:
+                    return payload.decode('utf-8', errors='replace')
+    else:
+        payload = msg.get_payload(decode=True)
+        if payload:
+            return payload.decode('utf-8', errors='replace')
+    return ""
+```
+
+**Strip HTML tags for readability:**
+```python
+import re
+clean = re.sub(r'<[^>]+>', ' ', body)
+clean = re.sub(r'\s+', ' ', clean).strip()
+```
+
+### Extracting attachments (PDFs, images)
+
+```python
+for part in raw.walk():
+    fn = part.get_filename()
+    if fn and fn.endswith('.pdf'):
+        payload = part.get_payload(decode=True)
+        with open("/tmp/output.pdf", 'wb') as f:
+            f.write(payload)
+```
+
+### Parsing ticket/PDF data
+
+After extracting a PDF attachment, parse with `pdfminer`:
+
+```python
+from pdfminer.high_level import extract_text
+from io import BytesIO
+text = extract_text(BytesIO(pdf_bytes))
+
+# Extract structured info
+for line in text.split('\n'):
+    if 'Flight Number' in line or 'CM ' in line:
+        flights.append(line.strip())
+    if 'Departure' in line or 'Arrival' in line:
+        # next line has the date/time
+```
+
+**Install:** `pip3 install pdfminer.six`
+
+**See `references/pdf-attachment-extraction.md`** for Copa Airlines and other carrier-specific PDF extraction patterns, MIME multipart navigation, and pitfalls.
+
+### Flight itinerary extraction patterns
+
+Common fields to extract:
+
+| Field | Pattern |
+|---|---|
+| Flight number | `CM XXX` (Copa), `AA XXXX` (American), `UA XXXX` (United) |
+| Confirmation code | 6-char alphanumeric (e.g. `A4GSFD`) |
+| Departure time | `\d{1,2}:\d{2} (AM|PM)` |
+| Airport codes | 3-letter codes inbound from text: `(MCO|PTY|CTG|ATL|SAV|MIA|IAD)` |
+| Dates | `July? \d{1,2},? 2026` |
+
+### Travel itinerary assembly
+
+For full trip planning from extracted email data (flights + rental cars + hotels + activity research), see `references/travel-itinerary-building.md`. Covers:
+
+- Multi-sender flight extraction (Copa, Allegiant, Avis)
+- Account number hunt from HTML statements (Avis Wizard #)
+- Departure day timeline with drive time + traffic estimates
+- Markdown itinerary template
+- Pitfalls per carrier (Allegiant surveys, Copa MCO departure, AT&T SMS unreliability)
+- Real-time inbox monitoring during trip planning (Avis verification code → confirmation chain)
+- Travel plan adjustments by priority (same flight, PreCheck, no bags, gate arrival time)
+- Global Entry return procedure: customs clearance at MCO with GE kiosks (~10 min for 4 people, kiosk takes ~2 min/person, skip regular line, kids under 12 go through with parent)
+
+### Verification code reader
+
+When a service emails a security code (Avis login, banking), the agent can read the inbox faster than the user can find the email. See `references/avis-verification-code.md`.
+
+### Avis rental reservation extraction (email approval chain)
+
+A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.
+
+**Detection flow:**
+
+1. User says "check email for rental" — search `FROM "Avis" SINCE last-24h`
+2. Find the security code email (`Subject: "Your security code"`) — extract the code
+3. Moment after user logs in, Avis emails two confirmation emails (one per rental leg)
+4. Re-check inbox and extract reservation numbers, vehicle info, dates, costs
+
+**Avis confirmation key fields:**
+- Reservation # pattern: `\d+US\d+`
+- Vehicle: "Ford Explorer or Similar"
+- Pickup/Dropoff locations and times
+- Estimated total (includes taxes/fees)
+
+**Pitfalls:**
+- Avis sends security code FIRST — user needs code to log in before confirmations appear
+- Two separate confirmations arrive minutes apart (outbound leg + return leg)
+- Times in confirmation are local time zone
+- The estimated total includes taxes and fees
+
+### Global Entry return procedure
+
+A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.
+
+**Detection flow:**
+
+1. User says "check email for rental" — search `FROM "Avis" SINCE last-24h`
+2. Find the security code email (`Subject: "Your security code"`) — extract the 6-digit code
+3. Moment after user logs in, Avis emails **two confirmation emails** (one per rental)
+4. Re-check and find them — extract reservation numbers, vehicle info, dates, costs
+
+**Avis confirmation key fields:**
+
+```python
+# Extract from plain text:
+re.search(r'Reservation #(\d+US\d+)', body)
+re.search(r'Ford Explorer or Similar', body)
+re.search(r'Pick Up Location[^\n]*\n(.*?)\n', body, re.DOTALL)
+re.search(r'(\$[\d,]+\.\d{2})', body)
+```
+
+**Pitfalls:**
+- Avis sends the security code email FIRST — the user needs the code to log in before confirmations come through
+- Two separate confirmations arrive minutes apart (SAV→MCO one-way + MCO→SAV return)
+- The estimated total includes taxes and fees — strip `$` and commas for clean display
+- Times in confirmation are in local time zone (Eastern for SAV/MCO)
+
+### Global Entry return procedure
+
+When the user returns to the US with Global Entry, see `references/global-entry-return-procedure.md` for customs clearance timelines, kiosk procedure, and timeline placement in the itinerary.
+
+### Drive time estimation (Savannah → Orlando)
+
+For Savannah → MCO (~285 mi, Saturday July):
+
+| Condition | Time |
+|---|---|
+| No traffic | 4h 15m |
+| Light traffic | 4h 30m |
+| Moderate | 5h |
+| Heavy | 5h 30m |
+
+Jacksonville I-95 southbound backs up around lunchtime (11 AM - 1 PM) on Saturdays. Best departure: 8:30 AM. Worst: after 9 AM hits Jacksonville lunch rush.
+
+### Inbox cleaning — solicitor/trash triage
+
+For bulk classification of inbox messages as solicitation vs. keep, see `references/inbox-solicitation-triaging.md` and the runnable script at `scripts/inbox-solicitation-cleaner.py`.
+
+The script uses a multi-heuristic approach:
+
+1. **Allowlist** — internal senders, known legit domains (never flagged)
+2. **Subject-line keywords** — immediate signals ("insurance quote", "factoring", "load board")
+3. **Body content by sender domain class** — free-email senders vs. business-domain senders get different treatment
+4. **Combined signal accumulation** — threshold-based (≥2 signals = flagged)
+
+**Known false positives** include government registration notices (UCR.gov, login.gov), bank product emails with trucking keywords in boilerplate, and Slack onboarding templates. See the reference for the full false-positive table and mitigation strategies.
+
+**Move pattern:** COPY to target folder → STORE +FLAGS \\Deleted → EXPUNGE. Always use BODY.PEEK[] to avoid marking messages as read.
+
+### Headers-only fetch (efficient preview)
+
+```python
+status, msg_data = conn.fetch(num, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT DATE)])")
+hdr = msg_data[0][1].decode('utf-8', errors='replace')
+```
+
+### WPForms form submission delivery verification
+
+For WordPress sites using WPForms + WP Mail SMTP, see `references/wpforms-delivery-verification.md` for the complete pattern:
+1. Query `wp_wpforms_entries` in the WordPress DB for registrant names/emails
+2. Check the admin IMAP inbox for form notification emails (confirms pipeline triggered)
+3. Search for bounce messages via `FROM "mailer-daemon"` or `FROM "postmaster"`
+4. Cross-reference bounced recipient addresses with registrant emails
+5. Trace SPF/DKIM failures from bounce body back to DNS records
+6. Verify the SMTP relay IP is in the SPF record
+
+## Pitfalls
+
+- **Multiple flight confirmation emails have the same content** — Copa Airlines sends one PDF attachment per passenger. Both Germaine's and Anita's PDFs contain identical flight details; only the passenger name differs. To confirm both are on the same booking, compare the confirmation code (A4GSFD) across both emails.
+- **Emails are HTML by default** — plain text version may be empty or missing. Check both `text/plain` and `text/html` parts.
+- **PDFs are `multipart/mixed` subparts** — iterate `raw.walk()`, check `part.get_filename()` for `.pdf` extension.
+- **IMAP search dates use `DD-Mon-YYYY` format** — e.g. `28-May-2026`, not `2026-05-28` or `05/28/2026`.
+- **IMAP date search is based on internal date, not header Date** — Internaldate is when IMAP received the message, which is usually close to the Date header but not identical.
+- **Email HTML may use inline styles** — stripping all tags removes styling but keeps content. CSS class names (`style-Xia0GT4t_-text`) are noise — filter them out.
+- **Copies to self may have the same subject** — When the user forwards or BCCs themselves, you get duplicate-looking results. Check the Date/From fields to distinguish.
+- **Mailer-Daemon bounces are multipart** — The original message is included as an attachment. The bounce notification is text/plain, the original is text/rfc822-headers. Parse the root part for the bounce reason, the attachment for the original subject/body.
+- **`SPAM` folder not `INBOX.spam`** — IMAP folder names are case-sensitive. Check both `SPAM`, `INBOX.spam`, `Spam`, `Junk`.
+- **Multipart with many image attachments** — Copa Airlines ticket receipts include 6-7 inline images (logos, QR codes) alongside the PDF. The PDF is the last or second-to-last part in the MIME tree.
+- **HTML emails from Copa use content blocker redirects** — Links in the email go through `sendgrid.net` tracking URLs, not direct to Copa.com. Don't try to open them — extract data from the visible text in the email body.
+- **`get_content_charset()` may return `None`** — Fall back to `utf-8` or `iso-8859-1` for encoded payloads.