hermes-skills/skills/email/imap-email-search-and-extract/SKILL.md

---
name: imap-email-search-and-extract
description: "Search, extract, and process emails from IMAP inbox — flight itineraries, attachment PDFs, confirmation codes, and structured data from HTML/plain-text emails."
version: 1.2.0
author: ShoNuff
platforms: [linux]
metadata:
  hermes:
    tags: [email, imap, pdf, itinerary, flight, extraction]
---

# IMAP Email Search and Extract

Search the user's IMAP inbox for specific emails and extract structured data.

## Standard approach\n\n### Connection setup\n\nCredentials typically come from Himalaya password files (`/root/.config/himalaya/*.pass`). For WordPress-hosted sites (Apex Track Experience, ITPP customer sites), SMTP/IMAP credentials may instead be in the WordPress database — see `references/wordpress-smtp-credential-discovery.md`.\n\n```python
import imaplib, email
HOST = "mail.germainebrown.com"
PORT = 993
USER = "g@germainebrown.com"
PW = open("/root/.config/himalaya/g-germainebrown.pass").read().strip()

conn = imaplib.IMAP4_SSL(HOST, PORT)
conn.login(USER, PW)
conn.select("INBOX")
```

### Searching strategies

**By sender and subject:**
```python
status, data = conn.search(None, 'FROM "Copa Airlines" SUBJECT "Cartagena"')
```

**By date range:**
```python
status, data = conn.search(None, '(SINCE "28-May-2026" BEFORE "5-Jul-2026")')
```

**Broad OR search across multiple keywords:**
```python
status, data = conn.search(None, 'OR OR SUBJECT "Cartagena" SUBJECT "flight" SUBJECT "itinerary"')
```

**Body text match:**
```python
status, data = conn.search(None, 'BODY "Cartagena"')
```

**Chain multiple filters:**
```python
# Sender + keyword across subject + body
status, data = conn.search(None, 'FROM "Copa" OR SUBJECT "Cartagena" BODY "Cartagena"')
```

### Decoding email content

**Get body (plain text or HTML):**
```python
def get_body(msg):
    if msg.is_multipart():
        for part in msg.walk():
            ct = part.get_content_type()
            if ct in ("text/plain", "text/html"):
                payload = part.get_payload(decode=True)
                if payload:
                    return payload.decode('utf-8', errors='replace')
    else:
        payload = msg.get_payload(decode=True)
        if payload:
            return payload.decode('utf-8', errors='replace')
    return ""
```

**Strip HTML tags for readability:**
```python
import re
clean = re.sub(r'<[^>]+>', ' ', body)
clean = re.sub(r'\s+', ' ', clean).strip()
```

### Extracting attachments (PDFs, images)

```python
for part in raw.walk():
    fn = part.get_filename()
    if fn and fn.endswith('.pdf'):
        payload = part.get_payload(decode=True)
        with open("/tmp/output.pdf", 'wb') as f:
            f.write(payload)
```

### Parsing ticket/PDF data

After extracting a PDF attachment, parse with `pdfminer`:

```python
from pdfminer.high_level import extract_text
from io import BytesIO
text = extract_text(BytesIO(pdf_bytes))

# Extract structured info
for line in text.split('\n'):
    if 'Flight Number' in line or 'CM ' in line:
        flights.append(line.strip())
    if 'Departure' in line or 'Arrival' in line:
        # next line has the date/time
```

**Install:** `pip3 install pdfminer.six`

**See `references/pdf-attachment-extraction.md`** for Copa Airlines and other carrier-specific PDF extraction patterns, MIME multipart navigation, and pitfalls.

### Flight itinerary extraction patterns

Common fields to extract:

| Field | Pattern |
|---|---|
| Flight number | `CM XXX` (Copa), `AA XXXX` (American), `UA XXXX` (United) |
| Confirmation code | 6-char alphanumeric (e.g. `A4GSFD`) |
| Departure time | `\d{1,2}:\d{2} (AM|PM)` |
| Airport codes | 3-letter codes inbound from text: `(MCO|PTY|CTG|ATL|SAV|MIA|IAD)` |
| Dates | `July? \d{1,2},? 2026` |

### Travel itinerary assembly

For full trip planning from extracted email data (flights + rental cars + hotels + activity research), see `references/travel-itinerary-building.md`. Covers:

- Multi-sender flight extraction (Copa, Allegiant, Avis)
- Account number hunt from HTML statements (Avis Wizard #)
- Departure day timeline with drive time + traffic estimates
- Markdown itinerary template
- Pitfalls per carrier (Allegiant surveys, Copa MCO departure, AT&T SMS unreliability)
- Real-time inbox monitoring during trip planning (Avis verification code → confirmation chain)
- Travel plan adjustments by priority (same flight, PreCheck, no bags, gate arrival time)
- Global Entry return procedure: customs clearance at MCO with GE kiosks (~10 min for 4 people, kiosk takes ~2 min/person, skip regular line, kids under 12 go through with parent)

### Verification code reader

When a service emails a security code (Avis login, banking), the agent can read the inbox faster than the user can find the email. See `references/avis-verification-code.md`.

### Avis rental reservation extraction (email approval chain)

A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.

**Detection flow:**

1. User says "check email for rental" — search `FROM "Avis" SINCE last-24h`
2. Find the security code email (`Subject: "Your security code"`) — extract the code
3. Moment after user logs in, Avis emails two confirmation emails (one per rental leg)
4. Re-check inbox and extract reservation numbers, vehicle info, dates, costs

**Avis confirmation key fields:**
- Reservation # pattern: `\d+US\d+`
- Vehicle: "Ford Explorer or Similar"
- Pickup/Dropoff locations and times
- Estimated total (includes taxes/fees)

**Pitfalls:**
- Avis sends security code FIRST — user needs code to log in before confirmations appear
- Two separate confirmations arrive minutes apart (outbound leg + return leg)
- Times in confirmation are local time zone
- The estimated total includes taxes and fees

### Global Entry return procedure

A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.

**Detection flow:**

1. User says "check email for rental" — search `FROM "Avis" SINCE last-24h`
2. Find the security code email (`Subject: "Your security code"`) — extract the 6-digit code
3. Moment after user logs in, Avis emails **two confirmation emails** (one per rental)
4. Re-check and find them — extract reservation numbers, vehicle info, dates, costs

**Avis confirmation key fields:**

```python
# Extract from plain text:
re.search(r'Reservation #(\d+US\d+)', body)
re.search(r'Ford Explorer or Similar', body)
re.search(r'Pick Up Location[^\n]*\n(.*?)\n', body, re.DOTALL)
re.search(r'(\$[\d,]+\.\d{2})', body)
```

**Pitfalls:**
- Avis sends the security code email FIRST — the user needs the code to log in before confirmations come through
- Two separate confirmations arrive minutes apart (SAV→MCO one-way + MCO→SAV return)
- The estimated total includes taxes and fees — strip `$` and commas for clean display
- Times in confirmation are in local time zone (Eastern for SAV/MCO)

### Global Entry return procedure

When the user returns to the US with Global Entry, see `references/global-entry-return-procedure.md` for customs clearance timelines, kiosk procedure, and timeline placement in the itinerary.

### Drive time estimation (Savannah → Orlando)

For Savannah → MCO (~285 mi, Saturday July):

| Condition | Time |
|---|---|
| No traffic | 4h 15m |
| Light traffic | 4h 30m |
| Moderate | 5h |
| Heavy | 5h 30m |

Jacksonville I-95 southbound backs up around lunchtime (11 AM - 1 PM) on Saturdays. Best departure: 8:30 AM. Worst: after 9 AM hits Jacksonville lunch rush.

### Inbox cleaning — solicitor/trash triage

For bulk classification of inbox messages as solicitation vs. keep, see `references/inbox-solicitation-triaging.md` and the runnable script at `scripts/inbox-solicitation-cleaner.py`.

The script uses a multi-heuristic approach:

1. **Allowlist** — internal senders, known legit domains (never flagged)
2. **Subject-line keywords** — immediate signals ("insurance quote", "factoring", "load board")
3. **Body content by sender domain class** — free-email senders vs. business-domain senders get different treatment
4. **Combined signal accumulation** — threshold-based (≥2 signals = flagged)

**Known false positives** include government registration notices (UCR.gov, login.gov), bank product emails with trucking keywords in boilerplate, and Slack onboarding templates. See the reference for the full false-positive table and mitigation strategies.

**Move pattern:** COPY to target folder → STORE +FLAGS \\Deleted → EXPUNGE. Always use BODY.PEEK[] to avoid marking messages as read.

### Headers-only fetch (efficient preview)

```python
status, msg_data = conn.fetch(num, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT DATE)])")
hdr = msg_data[0][1].decode('utf-8', errors='replace')
```

### WPForms form submission delivery verification

For WordPress sites using WPForms + WP Mail SMTP, see `references/wpforms-delivery-verification.md` for the complete pattern:
1. Query `wp_wpforms_entries` in the WordPress DB for registrant names/emails
2. Check the admin IMAP inbox for form notification emails (confirms pipeline triggered)
3. Search for bounce messages via `FROM "mailer-daemon"` or `FROM "postmaster"`
4. Cross-reference bounced recipient addresses with registrant emails
5. Trace SPF/DKIM failures from bounce body back to DNS records
6. Verify the SMTP relay IP is in the SPF record

## Pitfalls

- **Multiple flight confirmation emails have the same content** — Copa Airlines sends one PDF attachment per passenger. Both Germaine's and Anita's PDFs contain identical flight details; only the passenger name differs. To confirm both are on the same booking, compare the confirmation code (A4GSFD) across both emails.
- **Emails are HTML by default** — plain text version may be empty or missing. Check both `text/plain` and `text/html` parts.
- **PDFs are `multipart/mixed` subparts** — iterate `raw.walk()`, check `part.get_filename()` for `.pdf` extension.
- **IMAP search dates use `DD-Mon-YYYY` format** — e.g. `28-May-2026`, not `2026-05-28` or `05/28/2026`.
- **IMAP date search is based on internal date, not header Date** — Internaldate is when IMAP received the message, which is usually close to the Date header but not identical.
- **Email HTML may use inline styles** — stripping all tags removes styling but keeps content. CSS class names (`style-Xia0GT4t_-text`) are noise — filter them out.
- **Copies to self may have the same subject** — When the user forwards or BCCs themselves, you get duplicate-looking results. Check the Date/From fields to distinguish.
- **Mailer-Daemon bounces are multipart** — The original message is included as an attachment. The bounce notification is text/plain, the original is text/rfc822-headers. Parse the root part for the bounce reason, the attachment for the original subject/body.
- **`SPAM` folder not `INBOX.spam`** — IMAP folder names are case-sensitive. Check both `SPAM`, `INBOX.spam`, `Spam`, `Junk`.
- **Multipart with many image attachments** — Copa Airlines ticket receipts include 6-7 inline images (logos, QR codes) alongside the PDF. The PDF is the last or second-to-last part in the MIME tree.
- **HTML emails from Copa use content blocker redirects** — Links in the email go through `sendgrid.net` tracking URLs, not direct to Copa.com. Don't try to open them — extract data from the visible text in the email body.
- **`get_content_charset()` may return `None`** — Fall back to `utf-8` or `iso-8859-1` for encoded payloads.