# Extracting Content from Angular SPAs (Government / Statute Sites) Many government websites — including the Texas Legislature Online — are built as **Angular Single-Page Applications (SPAs)**. The initial HTML page is a shell containing only CSS/JS bundles, navigation tree metadata, and skeleton markup. The actual content (statute text, bill text, search results) is loaded asynchronously via undocumented backend API calls. ## Identifying an Angular SPA Shell Signs that you've hit an SPA shell, not the real content: 1. **Huge HTML file, tiny meaningful text**: The entire page is 200K–250K bytes but contains fewer than 100 meaningful text lines (e.g., just navigation tree labels). 2. **`` or `` custom element**: Angular mounts the app on a custom element — the static HTML contains no content inside it. 3. **`ng-state` JSON script tag**: Contains Angular server-side state, but close inspection shows it only holds navigation metadata (code lists, tree nodes), not the actual document text. 4. **All real content is inside ``**: The document viewer component is empty in the static HTML — content loads via API after JavaScript executes. 5. **Known backend API routes return 404**: The Angular app may call an API like `api/Statutes/Chapter?code=FI&chp=392` that doesn't exist at the static URL level (the Angular router intercepts it). ## Telltale Pattern: TX Legislature Online The Texas statutes site (`https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm`) follows this pattern: | Check | Result | |-------|--------| | Total HTML size | ~250 KB | | Meaningful text lines | ~84 (all navigation tree labels) | | `ng-state` content | Only `StatuteCode` lists (code names like "Finance Code") — no statute text | | `` | Empty `` in static HTML | | Backend API (`api/Statutes/Chapter`) | 404 | | Download links (DOCX, PDF) | Return 200 but serve HTML (bot wall/redirect) | ## Techniques to Try (in priority order) ### 1. Look for non-JavaScript versions or plain-text mirrors Government sites often have alternate views or legacy versions: ```bash # Try the "print" or "show all" parameter curl -sL "https://statutes.capitol.texas.gov/GetStatute.aspx?Code=FI&Value=392&ShowAll=true" -o /tmp/page.html # Try query parameters the old site might have used curl -sL "https://statutes.capitol.texas.gov/StatutesByCode.aspx?Code=FI&Value=392" -o /tmp/page.html # Try different encoding paths curl -sL "https://statutes.capitol.texas.gov/Docs/FI/word/FI.392.docx" -o /tmp/doc.docx curl -sL "https://statutes.capitol.texas.gov/Download.aspx?Code=FI&Value=IV.392&type=pdf" -o /tmp/doc.pdf ``` **⚠️ Pitfall**: Some "download" links return HTTP 200 but serve the Angular shell HTML (not the actual DOCX/PDF). Check the file's magic bytes: `head -c 100 /tmp/doc.docx` should start with `PK` (ZIP/DOCX), not ``. ### 2. Try third-party legal databases When the official site is locked behind an SPA, third-party aggregators may have the content in readable form: | Source | Typical Format | Notes | |--------|---------------|-------| | **Justia** (`law.justia.com/codes/texas/...`) | HTML | May have Cloudflare challenge/bot protection | | **Cornell LII** (`www.law.cornell.edu/statutes/texas/...`) | HTML | URL structure may differ — try searching the site | | **FindLaw** | HTML | Generally permissive | | **Casetext** | HTML | May require registration | | **Hackers & Founders / GitHub** | Plain text / markdown | Community-maintained; may be stale | ```bash # Fetch a known URL pattern curl -sL "https://law.justia.com/codes/texas/finance-code/title-4/subtitle-b/chapter-392/" -o /tmp/justia.html # If blocked by Cloudflare/challenge, check the first 500 bytes: head -c 500 /tmp/justia.html # "Just a moment..." = Cloudflare challenge — capture the page and report it ``` ### 3. Search for any `)' /tmp/page.html | python3 -c " import sys, json data = json.load(sys.stdin) for k, v in data.items(): if isinstance(v, dict) and 'b' in v: for bk, bv in v['b'].items(): # Print first 300 chars of each value print(f'{bk}: {str(bv)[:300]}') " # Check for __NEXT_DATA__ (Next.js, not Angular but worth checking) grep -oP '__NEXT_DATA__[^>]*>\K.*?(?=)' /tmp/page.html | head -c 2000 # Check for any JSON-like objects with content grep -oP '\{[^}]{100,}Chapter[^}]*\}' /tmp/page.html | head -5 # Check window.__INITIAL_STATE__ grep -oP 'window\.__INITIAL_STATE__\s*=\s*\K.*?(?=;)' /tmp/page.html | head -c 2000 ``` ### 4. Probe the backend API Angular apps often call a REST or JSON API. The backend may still work even if the Angular frontend doesn't render: ```bash # Common pattern for TX statutes curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \ -H "Accept: application/json" -o /tmp/api.json # Try with Referer header (anti-hotlinking) curl -sL "https://statutes.capitol.texas.gov/api/Statutes/Chapter?code=FI&chp=392" \ -H "Accept: application/json" \ -H "Referer: https://statutes.capitol.texas.gov/Docs/FI/htm/FI.392.htm" \ -o /tmp/api.json ``` If API returns 404, the route may not exist at that path — look for the actual API base URL in the JavaScript bundles (`chunk-*.js`): ```bash # Search for API paths in JS bundles grep -oP 'https?://[^"'"'"' ]*api[^"'"'"' ]*' /tmp/page.html | sort -u grep -oP '/api/[^"'"'"' ]+' /tmp/page.html | sort -u ``` ### 5. Render with a headless browser (last resort) If none of the above works and the content is critical, you need a headless browser (Puppeteer, Playwright) to render the JavaScript and capture the DOM after Angular loads: ```bash # Not available via curl alone — this is a "requires browser" signal ``` ## When to Give Up and Report Document the blocker honestly rather than fabricating content. Use this template: > **Data gap**: The [source] is an Angular SPA that loads statute content via an undocumented backend API. The following approaches were attempted and failed: > - Direct HTML fetch: Only navigation shell, no content (~250KB of CSS/JS, ~84 lines of navigation text) > - `ng-state` JSON: Contained only metadata (code lists), not statute text > - Backend API probing: `/api/Statutes/Chapter?code=FI&chp=392` returned 404 > - DOCX/PDF download links: Returned HTTP 200 but served the Angular HTML shell (bot wall) > - Third-party mirrors: [Justia/Cornell/etc.] had bot protection (Cloudflare challenge) > > **Impact**: Statute dates, specific section numbers, and exact wording of [key provisions] could not be independently verified against the live source. ## Summary Decision Tree ``` Is the site an Angular SPA (ng-state, , huge HTML + little text)? │ ├─ YES → Try 1: Legacy/alternate URL patterns (GetStatute.aspx?ShowAll=true) │ Try 2: Third-party legal aggregators (Justia, Cornell, FindLaw) │ Try 3: Script data payloads (ng-state, __NEXT_DATA__) │ Try 4: Backend API probing (look for /api/ paths in JS) │ Try 5: Report blocker — do NOT fabricate content │ └─ NO → Use standard HTML extraction (web-research-fallback core technique) ```