# Inbox Solicitation Triaging — Design & Pitfalls

Pattern for cleaning an inbox by classifying and moving solicitations.

## Core Technique

1. **Connect via IMAP (imaplib)**, select INBOX, fetch ALL messages
2. **Iterate each message** with `BODY.PEEK[]` (non-destructive read — doesn't mark as seen)
3. **Run multi-heuristic solicitor detection** on sender domain, subject line, and body text
4. **Copy flagged to target folder**, then `+FLAGS \\Deleted` + `EXPUNGE` from INBOX
5. **Save detailed JSON report** and print a grouped summary

## Multi-Heuristic Detection Strategy

Four layered checks:

### 1. Allowlist Bypass (highest priority)
- **Internal senders** (first-name patterns, company domain): skip entirely
- **Legit domains** (`boxpilotlogistics.com`, etc.): skip entirely

### 2. Subject-Line Keywords (high signal)
Immediate giveaways: "insurance quote", "free quote", "factoring", "compliance", "load board", "mc number", "advertisement"

### 3. Body Content Analysis (by sender domain class)

**Free/personal domains** (gmail, yahoo, outlook, etc. — defined in FREE_DOMAINS set):
- If sender has free domain AND references an external business domain in the body → strong solicitation signal
- Also scan for solicitation keywords + phrases in body text

**Business-domain senders** (not free, not internal):
- Count multiple trucking-solicitation signals (keywords + phrases)
- Threshold: 2+ signals = solicitation

**Transactional invoice guard:** If subject/body contains ≥2 invoice keywords ("invoice", "receipt", "payment confirmation") and no other signals triggered, keep in inbox.

### 4. Combined Signal Accumulation
- Body keywords: +1 each
- Solicitation phrases: +2 each (trucking-specific phrases are stronger)
- Subject keyword match: +1
- Threshold: 2+ total = flagged

## Known False-Positive Categories

| Category | Why flagged | Typical senders | Action |
|---|---|---|---|
| Government registration notices | Body has "fmcsa", "usdot", "authority" | ucr.gov, login.gov, dot.gov | Move back — official notices |
| Bank product promos | Boilerplate contains "eld", "limited time" | americanexpress@blueprint.americanexpress.com | Review individually |
| Slack onboarding | Template has "eld", "limited time" in footer | no-reply@email.slackhq.com | Keep transactional, move promos |
| FMCSA process agent confirmations | Body has "boc-3", "process agent", "fmcsa" | fmcsaprocessagents.com | Keep if service was ordered |

## Pitfalls

- **HTML boilerplate noise**: Some email templates (Amex, Slack) contain trucking keywords like "eld" in their footer HTML. The heuristic catches them as false positives. Mitigation: whitelist known transactional templates or strip common boilerplate before matching.
- **UCR.gov is legitimate** but its subject/body triggers "compliance", "authority", "usdot". These are official registration notices, not solicitations.
- **BODY.PEEK[] vs BODY[]**: Always use PEEK to avoid marking messages as read during review.
- **IMAP COPY + STORE \\Deleted** is the portable move pattern. Some servers support `MOVE` but not all.
- **conn.expunge()** must be called after all deletions or the messages remain hidden but not freed.
- **Character encodings**: Fall back through `utf-8` → `iso-8859-1` → `windows-1252` when `get_content_charset()` returns None.
- **HTML-only emails**: Need regex tag stripping (`re.sub(r'<[^>]+>', ' ', body)`) before keyword matching. CSS class names (`style-Xia0GT4t_-text`) are noise — filter them out.

## Script

See `scripts/inbox-solicitation-cleaner.py` — a standalone runnable script with the full heuristic set. Adjust the Config block (host, credentials, folder name) and run:
```bash
python3 scripts/inbox-solicitation-cleaner.py
```