Extracting Content from Angular SPAs (Government / Statute Sites)

Many government websites — including the Texas Legislature Online — are built as Angular Single-Page Applications (SPAs). The initial HTML page is a shell containing only CSS/JS bundles, navigation tree metadata, and skeleton markup. The actual content (statute text, bill text, search results) is loaded asynchronously via undocumented backend API calls.

Identifying an Angular SPA Shell

Signs that you've hit an SPA shell, not the real content:

Huge HTML file, tiny meaningful text: The entire page is 200K–250K bytes but contains fewer than 100 meaningful text lines (e.g., just navigation tree labels).
<app-root> or <app-home> custom element: Angular mounts the app on a custom element — the static HTML contains no content inside it.
ng-state JSON script tag: Contains Angular server-side state, but close inspection shows it only holds navigation metadata (code lists, tree nodes), not the actual document text.
All real content is inside <app-doc-viewer>: The document viewer component is empty in the static HTML — content loads via API after JavaScript executes.
Known backend API routes return 404: The Angular app may call an API like api/Statutes/Chapter?code=FI&chp=392 that doesn't exist at the static URL level (the Angular router intercepts it).

Telltale Pattern: TX Legislature Online

The Texas statutes site (https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm) follows this pattern:

Check	Result
Total HTML size	~250 KB
Meaningful text lines	~84 (all navigation tree labels)
`ng-state` content	Only `StatuteCode` lists (code names like "Finance Code") — no statute text
`<app-doc-viewer>`	Empty `<!---->` in static HTML
Backend API (`api/Statutes/Chapter`)	404
Download links (DOCX, PDF)	Return 200 but serve HTML (bot wall/redirect)

Techniques to Try (in priority order)

1. Look for non-JavaScript versions or plain-text mirrors

Government sites often have alternate views or legacy versions:

# Try the "print" or "show all" parameter
curl -sL "https://statutes.capitol.texas.gov/GetStatute.aspx?Code=FI&Value=392&ShowAll=true" -o /tmp/page.html

# Try query parameters the old site might have used
curl -sL "https://statutes.capitol.texas.gov/StatutesByCode.aspx?Code=FI&Value=392" -o /tmp/page.html

# Try different encoding paths
curl -sL "https://statutes.capitol.texas.gov/Docs/FI/word/FI.392.docx" -o /tmp/doc.docx
curl -sL "https://statutes.capitol.texas.gov/Download.aspx?Code=FI&Value=IV.392&type=pdf" -o /tmp/doc.pdf

⚠️ Pitfall: Some "download" links return HTTP 200 but serve the Angular shell HTML (not the actual DOCX/PDF). Check the file's magic bytes: head -c 100 /tmp/doc.docx should start with PK (ZIP/DOCX), not <!DOCTYPE html>.

2. Try third-party legal databases

When the official site is locked behind an SPA, third-party aggregators may have the content in readable form:

Source	Typical Format	Notes
Justia (`law.justia.com/codes/texas/...`)	HTML	May have Cloudflare challenge/bot protection
Cornell LII (`www.law.cornell.edu/statutes/texas/...`)	HTML	URL structure may differ — try searching the site
FindLaw	HTML	Generally permissive
Casetext	HTML	May require registration
Hackers & Founders / GitHub	Plain text / markdown	Community-maintained; may be stale

# Fetch a known URL pattern
curl -sL "https://law.justia.com/codes/texas/finance-code/title-4/subtitle-b/chapter-392/" -o /tmp/justia.html

# If blocked by Cloudflare/challenge, check the first 500 bytes:
head -c 500 /tmp/justia.html
# "Just a moment..." = Cloudflare challenge — capture the page and report it

3. Search for any `<script>` data payloads

Even in an Angular SPA, some content may be embedded in script tags or JSON:

# Check ng-state (Angular Universal SSR)
grep -oP 'id="ng-state"[^>]*>\K.*?(?=</script>)' /tmp/page.html | python3 -c "
import sys, json
data = json.load(sys.stdin)
for k, v in data.items():
    if isinstance(v, dict) and 'b' in v:
        for bk, bv in v['b'].items():
            # Print first 300 chars of each value
            print(f'{bk}: {str(bv)[:300]}')
"

# Check for __NEXT_DATA__ (Next.js, not Angular but worth checking)
grep -oP '__NEXT_DATA__[^>]*>\K.*?(?=</script>)' /tmp/page.html | head -c 2000

# Check for any JSON-like objects with content
grep -oP '\{[^}]{100,}Chapter[^}]*\}' /tmp/page.html | head -5

# Check window.__INITIAL_STATE__
grep -oP 'window\.__INITIAL_STATE__\s*=\s*\K.*?(?=;</script>)' /tmp/page.html | head -c 2000

4. Probe the backend API

Angular apps often call a REST or JSON API. The backend may still work even if the Angular frontend doesn't render:

# Common pattern for TX statutes
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
  -H "Accept: application/json" -o /tmp/api.json

# Try with Referer header (anti-hotlinking)
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
  -H "Accept: application/json" \
  -H "Referer: https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm" \
  -o /tmp/api.json

If API returns 404, the route may not exist at that path — look for the actual API base URL in the JavaScript bundles (chunk-*.js):

# Search for API paths in JS bundles
grep -oP 'https?://[^"'"'"' ]*api[^"'"'"' ]*' /tmp/page.html | sort -u
grep -oP '/api/[^"'"'"' ]+' /tmp/page.html | sort -u

5. Render with a headless browser (last resort)

If none of the above works and the content is critical, you need a headless browser (Puppeteer, Playwright) to render the JavaScript and capture the DOM after Angular loads:

# Not available via curl alone — this is a "requires browser" signal

When to Give Up and Report

Document the blocker honestly rather than fabricating content. Use this template:

Data gap: The [source] is an Angular SPA that loads statute content via an undocumented backend API. The following approaches were attempted and failed:

Direct HTML fetch: Only navigation shell, no content (~250KB of CSS/JS, ~84 lines of navigation text)

ng-state JSON: Contained only metadata (code lists), not statute text

Backend API probing: /api/Statutes/Chapter?code=FI&chp=392 returned 404

DOCX/PDF download links: Returned HTTP 200 but served the Angular HTML shell (bot wall)

Third-party mirrors: [Justia/Cornell/etc.] had bot protection (Cloudflare challenge)

Impact: Statute dates, specific section numbers, and exact wording of [key provisions] could not be independently verified against the live source.

Summary Decision Tree

Is the site an Angular SPA (ng-state, <app-root>, huge HTML + little text)?
│
├─ YES → Try 1: Legacy/alternate URL patterns (GetStatute.aspx?ShowAll=true)
│         Try 2: Third-party legal aggregators (Justia, Cornell, FindLaw)
│         Try 3: Script data payloads (ng-state, __NEXT_DATA__)
│         Try 4: Backend API probing (look for /api/ paths in JS)
│         Try 5: Report blocker — do NOT fabricate content
│
└─ NO → Use standard HTML extraction (web-research-fallback core technique)

7.2 KiB Raw Permalink Blame History Unescape Escape