Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,152 @@
|
||||
# Extracting Content from Angular SPAs (Government / Statute Sites)
|
||||
|
||||
Many government websites — including the Texas Legislature Online — are built as **Angular Single-Page Applications (SPAs)**. The initial HTML page is a shell containing only CSS/JS bundles, navigation tree metadata, and skeleton markup. The actual content (statute text, bill text, search results) is loaded asynchronously via undocumented backend API calls.
|
||||
|
||||
## Identifying an Angular SPA Shell
|
||||
|
||||
Signs that you've hit an SPA shell, not the real content:
|
||||
|
||||
1. **Huge HTML file, tiny meaningful text**: The entire page is 200K–250K bytes but contains fewer than 100 meaningful text lines (e.g., just navigation tree labels).
|
||||
2. **`<app-root>` or `<app-home>` custom element**: Angular mounts the app on a custom element — the static HTML contains no content inside it.
|
||||
3. **`ng-state` JSON script tag**: Contains Angular server-side state, but close inspection shows it only holds navigation metadata (code lists, tree nodes), not the actual document text.
|
||||
4. **All real content is inside `<app-doc-viewer>`**: The document viewer component is empty in the static HTML — content loads via API after JavaScript executes.
|
||||
5. **Known backend API routes return 404**: The Angular app may call an API like `api/Statutes/Chapter?code=FI&chp=392` that doesn't exist at the static URL level (the Angular router intercepts it).
|
||||
|
||||
## Telltale Pattern: TX Legislature Online
|
||||
|
||||
The Texas statutes site (`https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm`) follows this pattern:
|
||||
|
||||
| Check | Result |
|
||||
|-------|--------|
|
||||
| Total HTML size | ~250 KB |
|
||||
| Meaningful text lines | ~84 (all navigation tree labels) |
|
||||
| `ng-state` content | Only `StatuteCode` lists (code names like "Finance Code") — no statute text |
|
||||
| `<app-doc-viewer>` | Empty `<!---->` in static HTML |
|
||||
| Backend API (`api/Statutes/Chapter`) | 404 |
|
||||
| Download links (DOCX, PDF) | Return 200 but serve HTML (bot wall/redirect) |
|
||||
|
||||
## Techniques to Try (in priority order)
|
||||
|
||||
### 1. Look for non-JavaScript versions or plain-text mirrors
|
||||
|
||||
Government sites often have alternate views or legacy versions:
|
||||
|
||||
```bash
|
||||
# Try the "print" or "show all" parameter
|
||||
curl -sL "https://statutes.capitol.texas.gov/GetStatute.aspx?Code=FI&Value=392&ShowAll=true" -o /tmp/page.html
|
||||
|
||||
# Try query parameters the old site might have used
|
||||
curl -sL "https://statutes.capitol.texas.gov/StatutesByCode.aspx?Code=FI&Value=392" -o /tmp/page.html
|
||||
|
||||
# Try different encoding paths
|
||||
curl -sL "https://statutes.capitol.texas.gov/Docs/FI/word/FI.392.docx" -o /tmp/doc.docx
|
||||
curl -sL "https://statutes.capitol.texas.gov/Download.aspx?Code=FI&Value=IV.392&type=pdf" -o /tmp/doc.pdf
|
||||
```
|
||||
|
||||
**⚠️ Pitfall**: Some "download" links return HTTP 200 but serve the Angular shell HTML (not the actual DOCX/PDF). Check the file's magic bytes: `head -c 100 /tmp/doc.docx` should start with `PK` (ZIP/DOCX), not `<!DOCTYPE html>`.
|
||||
|
||||
### 2. Try third-party legal databases
|
||||
|
||||
When the official site is locked behind an SPA, third-party aggregators may have the content in readable form:
|
||||
|
||||
| Source | Typical Format | Notes |
|
||||
|--------|---------------|-------|
|
||||
| **Justia** (`law.justia.com/codes/texas/...`) | HTML | May have Cloudflare challenge/bot protection |
|
||||
| **Cornell LII** (`www.law.cornell.edu/statutes/texas/...`) | HTML | URL structure may differ — try searching the site |
|
||||
| **FindLaw** | HTML | Generally permissive |
|
||||
| **Casetext** | HTML | May require registration |
|
||||
| **Hackers & Founders / GitHub** | Plain text / markdown | Community-maintained; may be stale |
|
||||
|
||||
```bash
|
||||
# Fetch a known URL pattern
|
||||
curl -sL "https://law.justia.com/codes/texas/finance-code/title-4/subtitle-b/chapter-392/" -o /tmp/justia.html
|
||||
|
||||
# If blocked by Cloudflare/challenge, check the first 500 bytes:
|
||||
head -c 500 /tmp/justia.html
|
||||
# "Just a moment..." = Cloudflare challenge — capture the page and report it
|
||||
```
|
||||
|
||||
### 3. Search for any `<script>` data payloads
|
||||
|
||||
Even in an Angular SPA, some content may be embedded in script tags or JSON:
|
||||
|
||||
```bash
|
||||
# Check ng-state (Angular Universal SSR)
|
||||
grep -oP 'id="ng-state"[^>]*>\K.*?(?=</script>)' /tmp/page.html | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for k, v in data.items():
|
||||
if isinstance(v, dict) and 'b' in v:
|
||||
for bk, bv in v['b'].items():
|
||||
# Print first 300 chars of each value
|
||||
print(f'{bk}: {str(bv)[:300]}')
|
||||
"
|
||||
|
||||
# Check for __NEXT_DATA__ (Next.js, not Angular but worth checking)
|
||||
grep -oP '__NEXT_DATA__[^>]*>\K.*?(?=</script>)' /tmp/page.html | head -c 2000
|
||||
|
||||
# Check for any JSON-like objects with content
|
||||
grep -oP '\{[^}]{100,}Chapter[^}]*\}' /tmp/page.html | head -5
|
||||
|
||||
# Check window.__INITIAL_STATE__
|
||||
grep -oP 'window\.__INITIAL_STATE__\s*=\s*\K.*?(?=;</script>)' /tmp/page.html | head -c 2000
|
||||
```
|
||||
|
||||
### 4. Probe the backend API
|
||||
|
||||
Angular apps often call a REST or JSON API. The backend may still work even if the Angular frontend doesn't render:
|
||||
|
||||
```bash
|
||||
# Common pattern for TX statutes
|
||||
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
|
||||
-H "Accept: application/json" -o /tmp/api.json
|
||||
|
||||
# Try with Referer header (anti-hotlinking)
|
||||
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
|
||||
-H "Accept: application/json" \
|
||||
-H "Referer: https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm" \
|
||||
-o /tmp/api.json
|
||||
```
|
||||
|
||||
If API returns 404, the route may not exist at that path — look for the actual API base URL in the JavaScript bundles (`chunk-*.js`):
|
||||
|
||||
```bash
|
||||
# Search for API paths in JS bundles
|
||||
grep -oP 'https?://[^"'"'"' ]*api[^"'"'"' ]*' /tmp/page.html | sort -u
|
||||
grep -oP '/api/[^"'"'"' ]+' /tmp/page.html | sort -u
|
||||
```
|
||||
|
||||
### 5. Render with a headless browser (last resort)
|
||||
|
||||
If none of the above works and the content is critical, you need a headless browser (Puppeteer, Playwright) to render the JavaScript and capture the DOM after Angular loads:
|
||||
|
||||
```bash
|
||||
# Not available via curl alone — this is a "requires browser" signal
|
||||
```
|
||||
|
||||
## When to Give Up and Report
|
||||
|
||||
Document the blocker honestly rather than fabricating content. Use this template:
|
||||
|
||||
> **Data gap**: The [source] is an Angular SPA that loads statute content via an undocumented backend API. The following approaches were attempted and failed:
|
||||
> - Direct HTML fetch: Only navigation shell, no content (~250KB of CSS/JS, ~84 lines of navigation text)
|
||||
> - `ng-state` JSON: Contained only metadata (code lists), not statute text
|
||||
> - Backend API probing: `/api/Statutes/Chapter?code=FI&chp=392` returned 404
|
||||
> - DOCX/PDF download links: Returned HTTP 200 but served the Angular HTML shell (bot wall)
|
||||
> - Third-party mirrors: [Justia/Cornell/etc.] had bot protection (Cloudflare challenge)
|
||||
>
|
||||
> **Impact**: Statute dates, specific section numbers, and exact wording of [key provisions] could not be independently verified against the live source.
|
||||
|
||||
## Summary Decision Tree
|
||||
|
||||
```
|
||||
Is the site an Angular SPA (ng-state, <app-root>, huge HTML + little text)?
|
||||
│
|
||||
├─ YES → Try 1: Legacy/alternate URL patterns (GetStatute.aspx?ShowAll=true)
|
||||
│ Try 2: Third-party legal aggregators (Justia, Cornell, FindLaw)
|
||||
│ Try 3: Script data payloads (ng-state, __NEXT_DATA__)
|
||||
│ Try 4: Backend API probing (look for /api/ paths in JS)
|
||||
│ Try 5: Report blocker — do NOT fabricate content
|
||||
│
|
||||
└─ NO → Use standard HTML extraction (web-research-fallback core technique)
|
||||
```
|
||||
Reference in New Issue
Block a user