Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,105 @@
+# AI-Powered News / Content Scraper Pipeline
+
+General pattern for building a scraper that chains **web search (Firecrawl/SearXNG)** → **LLM classification** → **structured score output with dedup**.
+
+## Architecture
+
+```
+Firecrawl search (6+ queries)
+    │  └─ Returns {url, title, description}
+    ▼
+URL dedup (processed-urls.json)
+    │  └─ Skip previously seen URLs
+    ▼
+Admin-AI LLM classification (deepseek-chat)
+    │  └─ Returns JSON: {region, event_type, description, confidence}
+    ▼
+Region/category matcher (regex rules)
+    │  └─ Maps classified region → game region ID
+    ▼
+pending-scores.json
+    │  └─ Structured events with points, source_url, confidence
+    ▼
+Cron (daily 8 AM ET)
+```
+
+## Key Components
+
+### 1. Search phase
+
+Run multiple queries to maximize recall (Firecrawl returns 10 results per query max). Append today's date to get recency:
+
+```python
+SEARCH_QUERIES = [
+    "shark attack today",
+    "shark sighting",
+    "shark bite",
+    "shark incident",
+]
+today_str = date.today().isoformat()
+for query in SEARCH_QUERIES:
+    results = firecrawl_search(api_key, f"{query} {today_str}")
+    time.sleep(0.5)  # rate-limit courtesy
+```
+
+**Dedup by URL** across queries — Firecrawl may return the same article for different queries. Use a `dict[str, dict]` keyed by URL.
+
+### 2. Classification phase
+
+Send title+description to the LLM. **Key design choices:**
+
+- **System prompt** forces strict JSON output: `"Return valid JSON only — no markdown, no extra text"`
+- **Schema** includes `event_type` with a sentinel value `"none"` for non-incidents
+- **Temperature 0.1** for deterministic classification
+- **Max tokens 300** — we only need the JSON response
+- **Truncate article text** to ~3000 chars — Firecrawl descriptions are short
+- **Parse JSON with regex fallback** — some models wrap in markdown code fences despite instruction:
+  ```python
+  json_match = re.search(r'\{.*\}', content, re.DOTALL)
+  ```
+
+### 3. Category/Region matching
+
+Use regex with multiple fallback patterns per region. Try standalone names first, then broader patterns.
+
+### 4. Structured output (pending-scores.json)
+
+Each event gets `region_id`, `region_name`, `event_type`, `description`, `source_url`, `points`, `event_date`, `verified`, `confidence`, `classification_ts`.
+
+### 5. Dedup (processed-urls.json)
+
+Load → skip seen → append after classification → persist.
+
+### 6. Cron wrapper
+
+```bash
+#!/bin/bash
+cd /path/to/project || exit 1
+source .venv/bin/activate
+python3 scraper/scrape.py >> data/scraper.log 2>&1
+```
+
+TZ=America/New_York at crontab top.
+
+## Points Scoring Convention
+
+| Event Type | Points |
+|---|---|
+| sighting | 1 |
+| bite | 5 |
+| fatality | 10 |
+
+## Pitfalls
+
+- **Rate limit between LLM and search API calls** — 0.3-0.5s spacing
+- **JSON parsing of LLM output** — always regex-fallback for code fences
+- **Sentinel event_type "none"** — check both `!= "none"` and `is not None`
+- **Filter pages that list historical incidents** — LLM tends to classify "attack map" pages as real incidents
+- **Region regex ordering** — broad patterns first, standalone city names as fallbacks
+- **Processed URLs accumulate** — archive URLs >90 days to prevent unbounded growth
+- **Firecrawl API v1/v2 response structure differences** — check field names
+- **State files need backup** — pending-scores.json and processed-urls.json are critical state for game/score systems
+
+## Verification Pattern
+
+Import the matching functions via `exec()` and test all region patterns in isolation — no API calls needed.