Inbox Solicitation Triaging — Design & Pitfalls

Pattern for cleaning an inbox by classifying and moving solicitations.

Core Technique

Connect via IMAP (imaplib), select INBOX, fetch ALL messages
Iterate each message with BODY.PEEK[] (non-destructive read — doesn't mark as seen)
Run multi-heuristic solicitor detection on sender domain, subject line, and body text
Copy flagged to target folder, then +FLAGS \\Deleted + EXPUNGE from INBOX
Save detailed JSON report and print a grouped summary

Multi-Heuristic Detection Strategy

Four layered checks:

1. Allowlist Bypass (highest priority)

Internal senders (first-name patterns, company domain): skip entirely
Legit domains (boxpilotlogistics.com, etc.): skip entirely

2. Subject-Line Keywords (high signal)

Immediate giveaways: "insurance quote", "free quote", "factoring", "compliance", "load board", "mc number", "advertisement"

3. Body Content Analysis (by sender domain class)

Free/personal domains (gmail, yahoo, outlook, etc. — defined in FREE_DOMAINS set):

If sender has free domain AND references an external business domain in the body → strong solicitation signal
Also scan for solicitation keywords + phrases in body text

Business-domain senders (not free, not internal):

Count multiple trucking-solicitation signals (keywords + phrases)
Threshold: 2+ signals = solicitation

Transactional invoice guard: If subject/body contains ≥2 invoice keywords ("invoice", "receipt", "payment confirmation") and no other signals triggered, keep in inbox.

4. Combined Signal Accumulation

Body keywords: +1 each
Solicitation phrases: +2 each (trucking-specific phrases are stronger)
Subject keyword match: +1
Threshold: 2+ total = flagged

Known False-Positive Categories

Category	Why flagged	Typical senders	Action
Government registration notices	Body has "fmcsa", "usdot", "authority"	ucr.gov, login.gov, dot.gov	Move back — official notices
Bank product promos	Boilerplate contains "eld", "limited time"	americanexpress@blueprint.americanexpress.com	Review individually
Slack onboarding	Template has "eld", "limited time" in footer	no-reply@email.slackhq.com	Keep transactional, move promos
FMCSA process agent confirmations	Body has "boc-3", "process agent", "fmcsa"	fmcsaprocessagents.com	Keep if service was ordered

Pitfalls

HTML boilerplate noise: Some email templates (Amex, Slack) contain trucking keywords like "eld" in their footer HTML. The heuristic catches them as false positives. Mitigation: whitelist known transactional templates or strip common boilerplate before matching.
UCR.gov is legitimate but its subject/body triggers "compliance", "authority", "usdot". These are official registration notices, not solicitations.
BODY.PEEK[] vs BODY[]: Always use PEEK to avoid marking messages as read during review.
IMAP COPY + STORE \Deleted is the portable move pattern. Some servers support MOVE but not all.
conn.expunge() must be called after all deletions or the messages remain hidden but not freed.
Character encodings: Fall back through utf-8 → iso-8859-1 → windows-1252 when get_content_charset() returns None.
HTML-only emails: Need regex tag stripping (re.sub(r'<[^>]+>', ' ', body)) before keyword matching. CSS class names (style-Xia0GT4t_-text) are noise — filter them out.

Script

See scripts/inbox-solicitation-cleaner.py — a standalone runnable script with the full heuristic set. Adjust the Config block (host, credentials, folder name) and run:

python3 scripts/inbox-solicitation-cleaner.py

3.7 KiB Raw Permalink Blame History