# Extracting Content from Angular SPAs (Government / Statute Sites)

Many government websites — including the Texas Legislature Online — are built as **Angular Single-Page Applications (SPAs)**. The initial HTML page is a shell containing only CSS/JS bundles, navigation tree metadata, and skeleton markup. The actual content (statute text, bill text, search results) is loaded asynchronously via undocumented backend API calls.

## Identifying an Angular SPA Shell

Signs that you've hit an SPA shell, not the real content:

1. **Huge HTML file, tiny meaningful text**: The entire page is 200K–250K bytes but contains fewer than 100 meaningful text lines (e.g., just navigation tree labels).
2. **`<app-root>` or `<app-home>` custom element**: Angular mounts the app on a custom element — the static HTML contains no content inside it.
3. **`ng-state` JSON script tag**: Contains Angular server-side state, but close inspection shows it only holds navigation metadata (code lists, tree nodes), not the actual document text.
4. **All real content is inside `<app-doc-viewer>`**: The document viewer component is empty in the static HTML — content loads via API after JavaScript executes.
5. **Known backend API routes return 404**: The Angular app may call an API like `api/Statutes/Chapter?code=FI&chp=392` that doesn't exist at the static URL level (the Angular router intercepts it).

## Telltale Pattern: TX Legislature Online

The Texas statutes site (`https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm`) follows this pattern:

| Check | Result |
|-------|--------|
| Total HTML size | ~250 KB |
| Meaningful text lines | ~84 (all navigation tree labels) |
| `ng-state` content | Only `StatuteCode` lists (code names like "Finance Code") — no statute text |
| `<app-doc-viewer>` | Empty `<!---->` in static HTML |
| Backend API (`api/Statutes/Chapter`) | 404 |
| Download links (DOCX, PDF) | Return 200 but serve HTML (bot wall/redirect) |

## Techniques to Try (in priority order)

### 1. Look for non-JavaScript versions or plain-text mirrors

Government sites often have alternate views or legacy versions:

```bash
# Try the "print" or "show all" parameter
curl -sL "https://statutes.capitol.texas.gov/GetStatute.aspx?Code=FI&Value=392&ShowAll=true" -o /tmp/page.html

# Try query parameters the old site might have used
curl -sL "https://statutes.capitol.texas.gov/StatutesByCode.aspx?Code=FI&Value=392" -o /tmp/page.html

# Try different encoding paths
curl -sL "https://statutes.capitol.texas.gov/Docs/FI/word/FI.392.docx" -o /tmp/doc.docx
curl -sL "https://statutes.capitol.texas.gov/Download.aspx?Code=FI&Value=IV.392&type=pdf" -o /tmp/doc.pdf
```

**⚠️ Pitfall**: Some "download" links return HTTP 200 but serve the Angular shell HTML (not the actual DOCX/PDF). Check the file's magic bytes: `head -c 100 /tmp/doc.docx` should start with `PK` (ZIP/DOCX), not `<!DOCTYPE html>`.

### 2. Try third-party legal databases

When the official site is locked behind an SPA, third-party aggregators may have the content in readable form:

| Source | Typical Format | Notes |
|--------|---------------|-------|
| **Justia** (`law.justia.com/codes/texas/...`) | HTML | May have Cloudflare challenge/bot protection |
| **Cornell LII** (`www.law.cornell.edu/statutes/texas/...`) | HTML | URL structure may differ — try searching the site |
| **FindLaw** | HTML | Generally permissive |
| **Casetext** | HTML | May require registration |
| **Hackers & Founders / GitHub** | Plain text / markdown | Community-maintained; may be stale |

```bash
# Fetch a known URL pattern
curl -sL "https://law.justia.com/codes/texas/finance-code/title-4/subtitle-b/chapter-392/" -o /tmp/justia.html

# If blocked by Cloudflare/challenge, check the first 500 bytes:
head -c 500 /tmp/justia.html
# "Just a moment..." = Cloudflare challenge — capture the page and report it
```

### 3. Search for any `<script>` data payloads

Even in an Angular SPA, some content may be embedded in script tags or JSON:

```bash
# Check ng-state (Angular Universal SSR)
grep -oP 'id="ng-state"[^>]*>\K.*?(?=</script>)' /tmp/page.html | python3 -c "
import sys, json
data = json.load(sys.stdin)
for k, v in data.items():
    if isinstance(v, dict) and 'b' in v:
        for bk, bv in v['b'].items():
            # Print first 300 chars of each value
            print(f'{bk}: {str(bv)[:300]}')
"

# Check for __NEXT_DATA__ (Next.js, not Angular but worth checking)
grep -oP '__NEXT_DATA__[^>]*>\K.*?(?=</script>)' /tmp/page.html | head -c 2000

# Check for any JSON-like objects with content
grep -oP '\{[^}]{100,}Chapter[^}]*\}' /tmp/page.html | head -5

# Check window.__INITIAL_STATE__
grep -oP 'window\.__INITIAL_STATE__\s*=\s*\K.*?(?=;</script>)' /tmp/page.html | head -c 2000
```

### 4. Probe the backend API

Angular apps often call a REST or JSON API. The backend may still work even if the Angular frontend doesn't render:

```bash
# Common pattern for TX statutes
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
  -H "Accept: application/json" -o /tmp/api.json

# Try with Referer header (anti-hotlinking)
curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \
  -H "Accept: application/json" \
  -H "Referer: https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm" \
  -o /tmp/api.json
```

If API returns 404, the route may not exist at that path — look for the actual API base URL in the JavaScript bundles (`chunk-*.js`):

```bash
# Search for API paths in JS bundles
grep -oP 'https?://[^"'"'"' ]*api[^"'"'"' ]*' /tmp/page.html | sort -u
grep -oP '/api/[^"'"'"' ]+' /tmp/page.html | sort -u
```

### 5. Render with a headless browser (last resort)

If none of the above works and the content is critical, you need a headless browser (Puppeteer, Playwright) to render the JavaScript and capture the DOM after Angular loads:

```bash
# Not available via curl alone — this is a "requires browser" signal
```

## When to Give Up and Report

Document the blocker honestly rather than fabricating content. Use this template:

> **Data gap**: The [source] is an Angular SPA that loads statute content via an undocumented backend API. The following approaches were attempted and failed:
> - Direct HTML fetch: Only navigation shell, no content (~250KB of CSS/JS, ~84 lines of navigation text)
> - `ng-state` JSON: Contained only metadata (code lists), not statute text
> - Backend API probing: `/api/Statutes/Chapter?code=FI&chp=392` returned 404
> - DOCX/PDF download links: Returned HTTP 200 but served the Angular HTML shell (bot wall)
> - Third-party mirrors: [Justia/Cornell/etc.] had bot protection (Cloudflare challenge)
> 
> **Impact**: Statute dates, specific section numbers, and exact wording of [key provisions] could not be independently verified against the live source.

## Summary Decision Tree

```
Is the site an Angular SPA (ng-state, <app-root>, huge HTML + little text)?
│
├─ YES → Try 1: Legacy/alternate URL patterns (GetStatute.aspx?ShowAll=true)
│         Try 2: Third-party legal aggregators (Justia, Cornell, FindLaw)
│         Try 3: Script data payloads (ng-state, __NEXT_DATA__)
│         Try 4: Backend API probing (look for /api/ paths in JS)
│         Try 5: Report blocker — do NOT fabricate content
│
└─ NO → Use standard HTML extraction (web-research-fallback core technique)
```
