106 lines
3.5 KiB
Markdown
106 lines
3.5 KiB
Markdown
# AI-Powered News / Content Scraper Pipeline
|
|
|
|
General pattern for building a scraper that chains **web search (Firecrawl/SearXNG)** → **LLM classification** → **structured score output with dedup**.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Firecrawl search (6+ queries)
|
|
│ └─ Returns {url, title, description}
|
|
▼
|
|
URL dedup (processed-urls.json)
|
|
│ └─ Skip previously seen URLs
|
|
▼
|
|
Admin-AI LLM classification (deepseek-chat)
|
|
│ └─ Returns JSON: {region, event_type, description, confidence}
|
|
▼
|
|
Region/category matcher (regex rules)
|
|
│ └─ Maps classified region → game region ID
|
|
▼
|
|
pending-scores.json
|
|
│ └─ Structured events with points, source_url, confidence
|
|
▼
|
|
Cron (daily 8 AM ET)
|
|
```
|
|
|
|
## Key Components
|
|
|
|
### 1. Search phase
|
|
|
|
Run multiple queries to maximize recall (Firecrawl returns 10 results per query max). Append today's date to get recency:
|
|
|
|
```python
|
|
SEARCH_QUERIES = [
|
|
"shark attack today",
|
|
"shark sighting",
|
|
"shark bite",
|
|
"shark incident",
|
|
]
|
|
today_str = date.today().isoformat()
|
|
for query in SEARCH_QUERIES:
|
|
results = firecrawl_search(api_key, f"{query} {today_str}")
|
|
time.sleep(0.5) # rate-limit courtesy
|
|
```
|
|
|
|
**Dedup by URL** across queries — Firecrawl may return the same article for different queries. Use a `dict[str, dict]` keyed by URL.
|
|
|
|
### 2. Classification phase
|
|
|
|
Send title+description to the LLM. **Key design choices:**
|
|
|
|
- **System prompt** forces strict JSON output: `"Return valid JSON only — no markdown, no extra text"`
|
|
- **Schema** includes `event_type` with a sentinel value `"none"` for non-incidents
|
|
- **Temperature 0.1** for deterministic classification
|
|
- **Max tokens 300** — we only need the JSON response
|
|
- **Truncate article text** to ~3000 chars — Firecrawl descriptions are short
|
|
- **Parse JSON with regex fallback** — some models wrap in markdown code fences despite instruction:
|
|
```python
|
|
json_match = re.search(r'\{.*\}', content, re.DOTALL)
|
|
```
|
|
|
|
### 3. Category/Region matching
|
|
|
|
Use regex with multiple fallback patterns per region. Try standalone names first, then broader patterns.
|
|
|
|
### 4. Structured output (pending-scores.json)
|
|
|
|
Each event gets `region_id`, `region_name`, `event_type`, `description`, `source_url`, `points`, `event_date`, `verified`, `confidence`, `classification_ts`.
|
|
|
|
### 5. Dedup (processed-urls.json)
|
|
|
|
Load → skip seen → append after classification → persist.
|
|
|
|
### 6. Cron wrapper
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
cd /path/to/project || exit 1
|
|
source .venv/bin/activate
|
|
python3 scraper/scrape.py >> data/scraper.log 2>&1
|
|
```
|
|
|
|
TZ=America/New_York at crontab top.
|
|
|
|
## Points Scoring Convention
|
|
|
|
| Event Type | Points |
|
|
|---|---|
|
|
| sighting | 1 |
|
|
| bite | 5 |
|
|
| fatality | 10 |
|
|
|
|
## Pitfalls
|
|
|
|
- **Rate limit between LLM and search API calls** — 0.3-0.5s spacing
|
|
- **JSON parsing of LLM output** — always regex-fallback for code fences
|
|
- **Sentinel event_type "none"** — check both `!= "none"` and `is not None`
|
|
- **Filter pages that list historical incidents** — LLM tends to classify "attack map" pages as real incidents
|
|
- **Region regex ordering** — broad patterns first, standalone city names as fallbacks
|
|
- **Processed URLs accumulate** — archive URLs >90 days to prevent unbounded growth
|
|
- **Firecrawl API v1/v2 response structure differences** — check field names
|
|
- **State files need backup** — pending-scores.json and processed-urls.json are critical state for game/score systems
|
|
|
|
## Verification Pattern
|
|
|
|
Import the matching functions via `exec()` and test all region patterns in isolation — no API calls needed.
|