153 lines
7.2 KiB
Markdown
153 lines
7.2 KiB
Markdown
# Extracting Content from Angular SPAs (Government / Statute Sites)
|
||
|
||
Many government websites — including the Texas Legislature Online — are built as **Angular Single-Page Applications (SPAs)**. The initial HTML page is a shell containing only CSS/JS bundles, navigation tree metadata, and skeleton markup. The actual content (statute text, bill text, search results) is loaded asynchronously via undocumented backend API calls.
|
||
|
||
## Identifying an Angular SPA Shell
|
||
|
||
Signs that you've hit an SPA shell, not the real content:
|
||
|
||
1. **Huge HTML file, tiny meaningful text**: The entire page is 200K–250K bytes but contains fewer than 100 meaningful text lines (e.g., just navigation tree labels).
|
||
2. **`<app-root>` or `<app-home>` custom element**: Angular mounts the app on a custom element — the static HTML contains no content inside it.
|
||
3. **`ng-state` JSON script tag**: Contains Angular server-side state, but close inspection shows it only holds navigation metadata (code lists, tree nodes), not the actual document text.
|
||
4. **All real content is inside `<app-doc-viewer>`**: The document viewer component is empty in the static HTML — content loads via API after JavaScript executes.
|
||
5. **Known backend API routes return 404**: The Angular app may call an API like `api/Statutes/Chapter?code=FI&chp=392` that doesn't exist at the static URL level (the Angular router intercepts it).
|
||
|
||
## Telltale Pattern: TX Legislature Online
|
||
|
||
The Texas statutes site (`https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm`) follows this pattern:
|
||
|
||
| Check | Result |
|
||
|-------|--------|
|
||
| Total HTML size | ~250 KB |
|
||
| Meaningful text lines | ~84 (all navigation tree labels) |
|
||
| `ng-state` content | Only `StatuteCode` lists (code names like "Finance Code") — no statute text |
|
||
| `<app-doc-viewer>` | Empty `<!---->` in static HTML |
|
||
| Backend API (`api/Statutes/Chapter`) | 404 |
|
||
| Download links (DOCX, PDF) | Return 200 but serve HTML (bot wall/redirect) |
|
||
|
||
## Techniques to Try (in priority order)
|
||
|
||
### 1. Look for non-JavaScript versions or plain-text mirrors
|
||
|
||
Government sites often have alternate views or legacy versions:
|
||
|
||
```bash
|
||
# Try the "print" or "show all" parameter
|
||
curl -sL "https://statutes.capitol.texas.gov/GetStatute.aspx?Code=FI&Value=392&ShowAll=true" -o /tmp/page.html
|
||
|
||
# Try query parameters the old site might have used
|
||
curl -sL "https://statutes.capitol.texas.gov/StatutesByCode.aspx?Code=FI&Value=392" -o /tmp/page.html
|
||
|
||
# Try different encoding paths
|
||
curl -sL "https://statutes.capitol.texas.gov/Docs/FI/word/FI.392.docx" -o /tmp/doc.docx
|
||
curl -sL "https://statutes.capitol.texas.gov/Download.aspx?Code=FI&Value=IV.392&type=pdf" -o /tmp/doc.pdf
|
||
```
|
||
|
||
**⚠️ Pitfall**: Some "download" links return HTTP 200 but serve the Angular shell HTML (not the actual DOCX/PDF). Check the file's magic bytes: `head -c 100 /tmp/doc.docx` should start with `PK` (ZIP/DOCX), not `<!DOCTYPE html>`.
|
||
|
||
### 2. Try third-party legal databases
|
||
|
||
When the official site is locked behind an SPA, third-party aggregators may have the content in readable form:
|
||
|
||
| Source | Typical Format | Notes |
|
||
|--------|---------------|-------|
|
||
| **Justia** (`law.justia.com/codes/texas/...`) | HTML | May have Cloudflare challenge/bot protection |
|
||
| **Cornell LII** (`www.law.cornell.edu/statutes/texas/...`) | HTML | URL structure may differ — try searching the site |
|
||
| **FindLaw** | HTML | Generally permissive |
|
||
| **Casetext** | HTML | May require registration |
|
||
| **Hackers & Founders / GitHub** | Plain text / markdown | Community-maintained; may be stale |
|
||
|
||
```bash
|
||
# Fetch a known URL pattern
|
||
curl -sL "https://law.justia.com/codes/texas/finance-code/title-4/subtitle-b/chapter-392/" -o /tmp/justia.html
|
||
|
||
# If blocked by Cloudflare/challenge, check the first 500 bytes:
|
||
head -c 500 /tmp/justia.html
|
||
# "Just a moment..." = Cloudflare challenge — capture the page and report it
|
||
```
|
||
|
||
### 3. Search for any `<script>` data payloads
|
||
|
||
Even in an Angular SPA, some content may be embedded in script tags or JSON:
|
||
|
||
```bash
|
||
# Check ng-state (Angular Universal SSR)
|
||
grep -oP 'id="ng-state"[^>]*>\K.*?(?=</script>)' /tmp/page.html | python3 -c "
|
||
import sys, json
|
||
data = json.load(sys.stdin)
|
||
for k, v in data.items():
|
||
if isinstance(v, dict) and 'b' in v:
|
||
for bk, bv in v['b'].items():
|
||
# Print first 300 chars of each value
|
||
print(f'{bk}: {str(bv)[:300]}')
|
||
"
|
||
|
||
# Check for __NEXT_DATA__ (Next.js, not Angular but worth checking)
|
||
grep -oP '__NEXT_DATA__[^>]*>\K.*?(?=</script>)' /tmp/page.html | head -c 2000
|
||
|
||
# Check for any JSON-like objects with content
|
||
grep -oP '\{[^}]{100,}Chapter[^}]*\}' /tmp/page.html | head -5
|
||
|
||
# Check window.__INITIAL_STATE__
|
||
grep -oP 'window\.__INITIAL_STATE__\s*=\s*\K.*?(?=;</script>)' /tmp/page.html | head -c 2000
|
||
```
|
||
|
||
### 4. Probe the backend API
|
||
|
||
Angular apps often call a REST or JSON API. The backend may still work even if the Angular frontend doesn't render:
|
||
|
||
```bash
|
||
# Common pattern for TX statutes
|
||
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
|
||
-H "Accept: application/json" -o /tmp/api.json
|
||
|
||
# Try with Referer header (anti-hotlinking)
|
||
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
|
||
-H "Accept: application/json" \
|
||
-H "Referer: https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm" \
|
||
-o /tmp/api.json
|
||
```
|
||
|
||
If API returns 404, the route may not exist at that path — look for the actual API base URL in the JavaScript bundles (`chunk-*.js`):
|
||
|
||
```bash
|
||
# Search for API paths in JS bundles
|
||
grep -oP 'https?://[^"'"'"' ]*api[^"'"'"' ]*' /tmp/page.html | sort -u
|
||
grep -oP '/api/[^"'"'"' ]+' /tmp/page.html | sort -u
|
||
```
|
||
|
||
### 5. Render with a headless browser (last resort)
|
||
|
||
If none of the above works and the content is critical, you need a headless browser (Puppeteer, Playwright) to render the JavaScript and capture the DOM after Angular loads:
|
||
|
||
```bash
|
||
# Not available via curl alone — this is a "requires browser" signal
|
||
```
|
||
|
||
## When to Give Up and Report
|
||
|
||
Document the blocker honestly rather than fabricating content. Use this template:
|
||
|
||
> **Data gap**: The [source] is an Angular SPA that loads statute content via an undocumented backend API. The following approaches were attempted and failed:
|
||
> - Direct HTML fetch: Only navigation shell, no content (~250KB of CSS/JS, ~84 lines of navigation text)
|
||
> - `ng-state` JSON: Contained only metadata (code lists), not statute text
|
||
> - Backend API probing: `/api/Statutes/Chapter?code=FI&chp=392` returned 404
|
||
> - DOCX/PDF download links: Returned HTTP 200 but served the Angular HTML shell (bot wall)
|
||
> - Third-party mirrors: [Justia/Cornell/etc.] had bot protection (Cloudflare challenge)
|
||
>
|
||
> **Impact**: Statute dates, specific section numbers, and exact wording of [key provisions] could not be independently verified against the live source.
|
||
|
||
## Summary Decision Tree
|
||
|
||
```
|
||
Is the site an Angular SPA (ng-state, <app-root>, huge HTML + little text)?
|
||
│
|
||
├─ YES → Try 1: Legacy/alternate URL patterns (GetStatute.aspx?ShowAll=true)
|
||
│ Try 2: Third-party legal aggregators (Justia, Cornell, FindLaw)
|
||
│ Try 3: Script data payloads (ng-state, __NEXT_DATA__)
|
||
│ Try 4: Backend API probing (look for /api/ paths in JS)
|
||
│ Try 5: Report blocker — do NOT fabricate content
|
||
│
|
||
└─ NO → Use standard HTML extraction (web-research-fallback core technique)
|
||
```
|