hermes-skills/skills/email/imap-email-search-and-extract/SKILL.md

name, description, version, author, platforms, metadata

name	description	version	author	platforms	metadata
imap-email-search-and-extract	Search, extract, and process emails from IMAP inbox — flight itineraries, attachment PDFs, confirmation codes, and structured data from HTML/plain-text emails.	1.2.0	ShoNuff	linux	hermes tags email imap pdf itinerary flight extraction

IMAP Email Search and Extract

Search the user's IMAP inbox for specific emails and extract structured data.

Standard approach\n\n### Connection setup\n\nCredentials typically come from Himalaya password files (/root/.config/himalaya/*.pass). For WordPress-hosted sites (Apex Track Experience, ITPP customer sites), SMTP/IMAP credentials may instead be in the WordPress database — see references/wordpress-smtp-credential-discovery.md.\n\n```python

import imaplib, email HOST = "mail.germainebrown.com" PORT = 993 USER = "g@germainebrown.com" PW = open("/root/.config/himalaya/g-germainebrown.pass").read().strip()

conn = imaplib.IMAP4_SSL(HOST, PORT) conn.login(USER, PW) conn.select("INBOX")

### Searching strategies

**By sender and subject:**
```python
status, data = conn.search(None, 'FROM "Copa Airlines" SUBJECT "Cartagena"')

By date range:

status, data = conn.search(None, '(SINCE "28-May-2026" BEFORE "5-Jul-2026")')

Broad OR search across multiple keywords:

status, data = conn.search(None, 'OR OR SUBJECT "Cartagena" SUBJECT "flight" SUBJECT "itinerary"')

Body text match:

status, data = conn.search(None, 'BODY "Cartagena"')

Chain multiple filters:

# Sender + keyword across subject + body
status, data = conn.search(None, 'FROM "Copa" OR SUBJECT "Cartagena" BODY "Cartagena"')

Decoding email content

Get body (plain text or HTML):

def get_body(msg):
    if msg.is_multipart():
        for part in msg.walk():
            ct = part.get_content_type()
            if ct in ("text/plain", "text/html"):
                payload = part.get_payload(decode=True)
                if payload:
                    return payload.decode('utf-8', errors='replace')
    else:
        payload = msg.get_payload(decode=True)
        if payload:
            return payload.decode('utf-8', errors='replace')
    return ""

Strip HTML tags for readability:

import re
clean = re.sub(r'<[^>]+>', ' ', body)
clean = re.sub(r'\s+', ' ', clean).strip()

Extracting attachments (PDFs, images)

for part in raw.walk():
    fn = part.get_filename()
    if fn and fn.endswith('.pdf'):
        payload = part.get_payload(decode=True)
        with open("/tmp/output.pdf", 'wb') as f:
            f.write(payload)

Parsing ticket/PDF data

After extracting a PDF attachment, parse with pdfminer:

from pdfminer.high_level import extract_text
from io import BytesIO
text = extract_text(BytesIO(pdf_bytes))

# Extract structured info
for line in text.split('\n'):
    if 'Flight Number' in line or 'CM ' in line:
        flights.append(line.strip())
    if 'Departure' in line or 'Arrival' in line:
        # next line has the date/time

Install: pip3 install pdfminer.six

See references/pdf-attachment-extraction.md for Copa Airlines and other carrier-specific PDF extraction patterns, MIME multipart navigation, and pitfalls.

Flight itinerary extraction patterns

Common fields to extract:

Field	Pattern
Flight number	CM XXX (Copa), AA XXXX (American), UA XXXX (United)
Confirmation code	6-char alphanumeric (e.g. A4GSFD)
Departure time	`\d{1,2}:\d{2} (AM
Airport codes	3-letter codes inbound from text: `(MCO
Dates	July? \d{1,2},? 2026

Travel itinerary assembly

For full trip planning from extracted email data (flights + rental cars + hotels + activity research), see references/travel-itinerary-building.md. Covers:

• Multi-sender flight extraction (Copa, Allegiant, Avis)
• Account number hunt from HTML statements (Avis Wizard #)
• Departure day timeline with drive time + traffic estimates
• Markdown itinerary template
• Pitfalls per carrier (Allegiant surveys, Copa MCO departure, AT&T SMS unreliability)
• Real-time inbox monitoring during trip planning (Avis verification code → confirmation chain)
• Travel plan adjustments by priority (same flight, PreCheck, no bags, gate arrival time)
• Global Entry return procedure: customs clearance at MCO with GE kiosks (~10 min for 4 people, kiosk takes ~2 min/person, skip regular line, kids under 12 go through with parent)

Verification code reader

When a service emails a security code (Avis login, banking), the agent can read the inbox faster than the user can find the email. See references/avis-verification-code.md.

Avis rental reservation extraction (email approval chain)

A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.

Detection flow:

1. User says "check email for rental" — search FROM "Avis" SINCE last-24h
2. Find the security code email (Subject: "Your security code") — extract the code
3. Moment after user logs in, Avis emails two confirmation emails (one per rental leg)
4. Re-check inbox and extract reservation numbers, vehicle info, dates, costs

Avis confirmation key fields:

• Reservation # pattern: \d+US\d+
• Vehicle: "Ford Explorer or Similar"
• Pickup/Dropoff locations and times
• Estimated total (includes taxes/fees)

Pitfalls:

• Avis sends security code FIRST — user needs code to log in before confirmations appear
• Two separate confirmations arrive minutes apart (outbound leg + return leg)
• Times in confirmation are local time zone
• The estimated total includes taxes and fees

Global Entry return procedure

A common pattern: user starts an Avis reservation, Avis sends a security code to their inbox, user asks the agent to check email. The agent finds the security code email, the user logs in, and shortly after Avis sends reservation confirmations.

Detection flow:

1. User says "check email for rental" — search FROM "Avis" SINCE last-24h
2. Find the security code email (Subject: "Your security code") — extract the 6-digit code
3. Moment after user logs in, Avis emails two confirmation emails (one per rental)
4. Re-check and find them — extract reservation numbers, vehicle info, dates, costs

Avis confirmation key fields:

# Extract from plain text:
re.search(r'Reservation #(\d+US\d+)', body)
re.search(r'Ford Explorer or Similar', body)
re.search(r'Pick Up Location[^\n]*\n(.*?)\n', body, re.DOTALL)
re.search(r'(\$[\d,]+\.\d{2})', body)

Pitfalls:

• Avis sends the security code email FIRST — the user needs the code to log in before confirmations come through
• Two separate confirmations arrive minutes apart (SAV→MCO one-way + MCO→SAV return)
• The estimated total includes taxes and fees — strip $ and commas for clean display
• Times in confirmation are in local time zone (Eastern for SAV/MCO)

Global Entry return procedure

When the user returns to the US with Global Entry, see references/global-entry-return-procedure.md for customs clearance timelines, kiosk procedure, and timeline placement in the itinerary.

Drive time estimation (Savannah → Orlando)

For Savannah → MCO (~285 mi, Saturday July):

Condition	Time
No traffic	4h 15m
Light traffic	4h 30m
Moderate	5h
Heavy	5h 30m

Jacksonville I-95 southbound backs up around lunchtime (11 AM - 1 PM) on Saturdays. Best departure: 8:30 AM. Worst: after 9 AM hits Jacksonville lunch rush.

Inbox cleaning — solicitor/trash triage

For bulk classification of inbox messages as solicitation vs. keep, see references/inbox-solicitation-triaging.md and the runnable script at scripts/inbox-solicitation-cleaner.py.

The script uses a multi-heuristic approach:

1. Allowlist — internal senders, known legit domains (never flagged)
2. Subject-line keywords — immediate signals ("insurance quote", "factoring", "load board")
3. Body content by sender domain class — free-email senders vs. business-domain senders get different treatment
4. Combined signal accumulation — threshold-based (≥2 signals = flagged)

Known false positives include government registration notices (UCR.gov, login.gov), bank product emails with trucking keywords in boilerplate, and Slack onboarding templates. See the reference for the full false-positive table and mitigation strategies.

Move pattern: COPY to target folder → STORE +FLAGS \Deleted → EXPUNGE. Always use BODY.PEEK[] to avoid marking messages as read.

Headers-only fetch (efficient preview)

status, msg_data = conn.fetch(num, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT DATE)])")
hdr = msg_data[0][1].decode('utf-8', errors='replace')

WPForms form submission delivery verification

For WordPress sites using WPForms + WP Mail SMTP, see references/wpforms-delivery-verification.md for the complete pattern:

1. Query wp_wpforms_entries in the WordPress DB for registrant names/emails
2. Check the admin IMAP inbox for form notification emails (confirms pipeline triggered)
3. Search for bounce messages via FROM "mailer-daemon" or FROM "postmaster"
4. Cross-reference bounced recipient addresses with registrant emails
5. Trace SPF/DKIM failures from bounce body back to DNS records
6. Verify the SMTP relay IP is in the SPF record

Pitfalls

• Multiple flight confirmation emails have the same content — Copa Airlines sends one PDF attachment per passenger. Both Germaine's and Anita's PDFs contain identical flight details; only the passenger name differs. To confirm both are on the same booking, compare the confirmation code (A4GSFD) across both emails.
• Emails are HTML by default — plain text version may be empty or missing. Check both text/plain and text/html parts.
• PDFs are multipart/mixed subparts — iterate raw.walk(), check part.get_filename() for .pdf extension.
• IMAP search dates use DD-Mon-YYYY format — e.g. 28-May-2026, not 2026-05-28 or 05/28/2026.
• IMAP date search is based on internal date, not header Date — Internaldate is when IMAP received the message, which is usually close to the Date header but not identical.
• Email HTML may use inline styles — stripping all tags removes styling but keeps content. CSS class names (style-Xia0GT4t_-text) are noise — filter them out.
• Copies to self may have the same subject — When the user forwards or BCCs themselves, you get duplicate-looking results. Check the Date/From fields to distinguish.
• Mailer-Daemon bounces are multipart — The original message is included as an attachment. The bounce notification is text/plain, the original is text/rfc822-headers. Parse the root part for the bounce reason, the attachment for the original subject/body.
• SPAM folder not INBOX.spam — IMAP folder names are case-sensitive. Check both SPAM, INBOX.spam, Spam, Junk.
• Multipart with many image attachments — Copa Airlines ticket receipts include 6-7 inline images (logos, QR codes) alongside the PDF. The PDF is the last or second-to-last part in the MIME tree.
• HTML emails from Copa use content blocker redirects — Links in the email go through sendgrid.net tracking URLs, not direct to Copa.com. Don't try to open them — extract data from the visible text in the email body.
• get_content_charset() may return None — Fall back to utf-8 or iso-8859-1 for encoded payloads.