# AI-Powered News / Content Scraper Pipeline

General pattern for building a scraper that chains **web search (Firecrawl/SearXNG)** → **LLM classification** → **structured score output with dedup**.

## Architecture

```
Firecrawl search (6+ queries)
    │  └─ Returns {url, title, description}
    ▼
URL dedup (processed-urls.json)
    │  └─ Skip previously seen URLs
    ▼
Admin-AI LLM classification (deepseek-chat)
    │  └─ Returns JSON: {region, event_type, description, confidence}
    ▼
Region/category matcher (regex rules)
    │  └─ Maps classified region → game region ID
    ▼
pending-scores.json
    │  └─ Structured events with points, source_url, confidence
    ▼
Cron (daily 8 AM ET)
```

## Key Components

### 1. Search phase

Run multiple queries to maximize recall (Firecrawl returns 10 results per query max). Append today's date to get recency:

```python
SEARCH_QUERIES = [
    "shark attack today",
    "shark sighting",
    "shark bite",
    "shark incident",
]
today_str = date.today().isoformat()
for query in SEARCH_QUERIES:
    results = firecrawl_search(api_key, f"{query} {today_str}")
    time.sleep(0.5)  # rate-limit courtesy
```

**Dedup by URL** across queries — Firecrawl may return the same article for different queries. Use a `dict[str, dict]` keyed by URL.

### 2. Classification phase

Send title+description to the LLM. **Key design choices:**

- **System prompt** forces strict JSON output: `"Return valid JSON only — no markdown, no extra text"`
- **Schema** includes `event_type` with a sentinel value `"none"` for non-incidents
- **Temperature 0.1** for deterministic classification
- **Max tokens 300** — we only need the JSON response
- **Truncate article text** to ~3000 chars — Firecrawl descriptions are short
- **Parse JSON with regex fallback** — some models wrap in markdown code fences despite instruction:
  ```python
  json_match = re.search(r'\{.*\}', content, re.DOTALL)
  ```

### 3. Category/Region matching

Use regex with multiple fallback patterns per region. Try standalone names first, then broader patterns.

### 4. Structured output (pending-scores.json)

Each event gets `region_id`, `region_name`, `event_type`, `description`, `source_url`, `points`, `event_date`, `verified`, `confidence`, `classification_ts`.

### 5. Dedup (processed-urls.json)

Load → skip seen → append after classification → persist.

### 6. Cron wrapper

```bash
#!/bin/bash
cd /path/to/project || exit 1
source .venv/bin/activate
python3 scraper/scrape.py >> data/scraper.log 2>&1
```

TZ=America/New_York at crontab top.

## Points Scoring Convention

| Event Type | Points |
|---|---|
| sighting | 1 |
| bite | 5 |
| fatality | 10 |

## Pitfalls

- **Rate limit between LLM and search API calls** — 0.3-0.5s spacing
- **JSON parsing of LLM output** — always regex-fallback for code fences
- **Sentinel event_type "none"** — check both `!= "none"` and `is not None`
- **Filter pages that list historical incidents** — LLM tends to classify "attack map" pages as real incidents
- **Region regex ordering** — broad patterns first, standalone city names as fallbacks
- **Processed URLs accumulate** — archive URLs >90 days to prevent unbounded growth
- **Firecrawl API v1/v2 response structure differences** — check field names
- **State files need backup** — pending-scores.json and processed-urls.json are critical state for game/score systems

## Verification Pattern

Import the matching functions via `exec()` and test all region patterns in isolation — no API calls needed.