AI-Powered News / Content Scraper Pipeline

General pattern for building a scraper that chains web search (Firecrawl/SearXNG) → LLM classification → structured score output with dedup.

Architecture

Firecrawl search (6+ queries)
    │  └─ Returns {url, title, description}
    ▼
URL dedup (processed-urls.json)
    │  └─ Skip previously seen URLs
    ▼
Admin-AI LLM classification (deepseek-chat)
    │  └─ Returns JSON: {region, event_type, description, confidence}
    ▼
Region/category matcher (regex rules)
    │  └─ Maps classified region → game region ID
    ▼
pending-scores.json
    │  └─ Structured events with points, source_url, confidence
    ▼
Cron (daily 8 AM ET)

Key Components

1. Search phase

Run multiple queries to maximize recall (Firecrawl returns 10 results per query max). Append today's date to get recency:

SEARCH_QUERIES = [
    "shark attack today",
    "shark sighting",
    "shark bite",
    "shark incident",
]
today_str = date.today().isoformat()
for query in SEARCH_QUERIES:
    results = firecrawl_search(api_key, f"{query} {today_str}")
    time.sleep(0.5)  # rate-limit courtesy

Dedup by URL across queries — Firecrawl may return the same article for different queries. Use a dict[str, dict] keyed by URL.

2. Classification phase

Send title+description to the LLM. Key design choices:

System prompt forces strict JSON output: "Return valid JSON only — no markdown, no extra text"
Schema includes event_type with a sentinel value "none" for non-incidents
Temperature 0.1 for deterministic classification
Max tokens 300 — we only need the JSON response
Truncate article text to ~3000 chars — Firecrawl descriptions are short
Parse JSON with regex fallback — some models wrap in markdown code fences despite instruction:
```
json_match = re.search(r'\{.*\}', content, re.DOTALL)
```

3. Category/Region matching

Use regex with multiple fallback patterns per region. Try standalone names first, then broader patterns.

4. Structured output (pending-scores.json)

Each event gets region_id, region_name, event_type, description, source_url, points, event_date, verified, confidence, classification_ts.

5. Dedup (processed-urls.json)

Load → skip seen → append after classification → persist.

6. Cron wrapper

#!/bin/bash
cd /path/to/project || exit 1
source .venv/bin/activate
python3 scraper/scrape.py >> data/scraper.log 2>&1

TZ=America/New_York at crontab top.

Points Scoring Convention

Event Type	Points
sighting	1
bite	5
fatality	10

Pitfalls

Rate limit between LLM and search API calls — 0.3-0.5s spacing
JSON parsing of LLM output — always regex-fallback for code fences
Sentinel event_type "none" — check both != "none" and is not None
Filter pages that list historical incidents — LLM tends to classify "attack map" pages as real incidents
Region regex ordering — broad patterns first, standalone city names as fallbacks
Processed URLs accumulate — archive URLs >90 days to prevent unbounded growth
Firecrawl API v1/v2 response structure differences — check field names
State files need backup — pending-scores.json and processed-urls.json are critical state for game/score systems

Verification Pattern

Import the matching functions via exec() and test all region patterns in isolation — no API calls needed.

3.5 KiB Raw Permalink Blame History