Update Forefront map implementation plan

.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md

@@ -0,0 +1,485 @@
+# Forefront Broadband Map Implementation Plan
+
+> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
+
+**Goal:** Build an internal Ops-hosted broadband availability map for ZIPs **75154 / Red Oak, Texas** and **75146 / Lancaster, Texas** showing address-level 300+ Mbps availability for AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier, and Forefront Wireless if Forefront data is provided.
+
+**Architecture:** Use FCC BDC fixed broadband availability as the reproducible baseline, normalize it into a local spatial database, and expose a small API + Leaflet/MapLibre web UI. Provider websites are validation sources, not the bulk-data foundation, because they are often CAPTCHA/ToS limited and not designed for bulk qualification.
+
+**Tech Stack:** Python/FastAPI, PostgreSQL + PostGIS or SQLite/GeoPackage for MVP, DuckDB for bulk CSV processing, Tippecanoe/PMTiles or GeoJSON for map layers, Leaflet or MapLibre GL JS, Caddy reverse proxy under Ops.
+
+---
+
+## Current Context / Assumptions
+
+- Target area: **ZIPs 75154 / Red Oak, Texas and 75146 / Lancaster, Texas**. Scope for 75146 is the **full ZIP**.
+- Minimum plan speed: **300 Mbps download or higher**.
+- Initial competitors: **AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier**.
+- Forefront Wireless should be added if we can get service-area GIS, tower/AP sector data, customer install data, or an address qualification/export file.
+- App is **internal first** under `ops.itpropartner.com`, not public customer-facing on day one.
+- Provider qualification access is limited to **public web pages** plus FCC/government data.
+- Results should include available details where sources provide them: speed, technology, price, install fee, contract terms, order URL, and phone number.
+- Services below 300 Mbps should be shown as **excluded under 300 Mbps** rather than treated as qualifying results.
+- Manual sampled validation is acceptable if provider websites block automation.
+- Initial validation address: `927 Pierce Road, Red Oak, TX 75154`, geocoded via OpenStreetMap/Nominatim to `32.5177279, -96.7677135`.
+- FCC BDC data is provider-reported and useful as a baseline, but it must be labeled as reported availability, not guaranteed installability.
+
+## Verified Source Findings
+
+### FCC Broadband Data Collection / National Broadband Map
+
+- The FCC Broadband Data page says the National Broadband Map provides information about services available to individual locations as reported by ISPs.
+- FCC public data APIs exist for downloadable BDC files, but require an FCC username and generated API token.
+- API docs identify:
+  - `GET /api/public/map/listAsOfDates`
+  - `GET /api/public/map/downloads/listAvailabilityData/{as_of_date}`
+  - `GET /api/public/map/downloads/downloadFile/{file_id}`
+- Availability download list supports filtering by category, subcategory, technology type, state/provider, etc.
+- Rate limit in the FCC API spec: **10 calls per minute**.
+
+### FCC fixed broadband availability files
+
+The FCC BDC download specification defines fixed broadband availability CSV fields including:
+
+- `frn`
+- `provider_id`
+- `brand_name`
+- `location_id`
+- `technology`
+- `max_advertised_download_speed`
+- `max_advertised_upload_speed`
+- `low_latency`
+- `business_residential_code`
+- `state_usps`
+- `block_geoid`
+- `h3_res8_id`
+
+Technology codes include:
+
+| Code | Technology |
+|---:|---|
+| 10 | Copper Wire |
+| 40 | Coaxial Cable / HFC |
+| 50 | Fiber to the Premises |
+| 60 | Geostationary Satellite |
+| 61 | Non-geostationary Satellite |
+| 70 | Unlicensed Terrestrial Fixed Wireless |
+| 71 | Licensed Terrestrial Fixed Wireless |
+| 72 | Licensed-by-Rule Terrestrial Fixed Wireless |
+| 0 | Other |
+
+### Fabric / address limitation
+
+- Public BDC availability records key on `location_id`.
+- The FCC help docs say to obtain address, coordinate, building type, and other location details beyond Location ID, access to the Broadband Serviceable Location Fabric is required.
+- Fabric access requires a license through FCC/CostQuest and can take up to ~2 weeks after entity info/request submission.
+- Without Fabric, we can still use `location_id`, block GEOID, H3 cell, ZIP/county/geography summaries, and geocoded user input — but exact address-to-BSL matching will be imperfect.
+
+### Census Geocoder
+
+- Census Geocoder provides public REST geocoding for U.S. addresses.
+- Single record endpoint form: `https://geocoding.geo.census.gov/geocoder/returntype/searchtype?parameters`
+- Batch mode supports up to **10,000 records** per batch file.
+- Useful for converting user-entered addresses to lat/lon and census geography, but it does **not** return FCC Fabric `location_id`.
+
+---
+
+## What Needs To Be Built
+
+### 1. Data ingestion pipeline
+
+**Purpose:** Pull, normalize, and refresh broadband availability data.
+
+**Components:**
+
+- `scripts/fcc_list_vintages.py`
+  - Calls FCC list-as-of-dates API.
+  - Selects latest availability vintage.
+
+- `scripts/fcc_download_availability.py`
+  - Downloads Texas fixed broadband availability data by technology/provider where available.
+  - Saves raw ZIP/CSV files under `data/raw/fcc/YYYY-MM-DD/`.
+  - Logs file IDs, provider IDs, source URL, download timestamp, hash.
+
+- `scripts/import_fcc_bdc.py`
+  - Imports CSVs into staging tables.
+  - Filters target providers.
+  - Filters `max_advertised_download_speed >= 300`.
+  - Normalizes technology codes.
+  - Produces clean availability table.
+
+- `scripts/import_fabric.py` **only if Fabric license/data is obtained**
+  - Imports county-level Fabric CSVs for Ellis County / Dallas County as needed.
+  - Joins BDC `location_id` to Fabric address/lat/lon.
+
+- `scripts/import_forefront.py`
+  - Imports Forefront service data if provided.
+  - Accepted input formats: CSV address list, customer/install export, tower coordinates + sector azimuth/beamwidth/radius, KMZ/KML, GeoJSON, shapefile, or coverage polygon.
+
+### 2. Spatial data store
+
+**MVP choice:** SQLite + GeoPackage if dataset stays small.
+
+**Better production choice:** PostgreSQL + PostGIS.
+
+Tables/views:
+
+| Table/View | Purpose |
+|---|---|
+| `raw_fcc_downloads` | downloaded file metadata and hashes |
+| `fcc_availability_raw` | raw BDC rows |
+| `provider_map` | provider aliases and FCC provider IDs |
+| `availability_clean` | normalized 300+ Mbps availability rows |
+| `fabric_locations` | BSL address/lat/lon/building fields if licensed Fabric is available |
+| `forefront_coverage` | Forefront-specific service areas/data |
+| `address_lookup_cache` | user address → lat/lon/geography/cache result |
+| `source_evidence` | URL, source type, confidence, retrieval timestamp |
+
+### 3. Backend API
+
+FastAPI endpoints:
+
+| Endpoint | Purpose |
+|---|---|
+| `GET /health` | service/database status |
+| `GET /providers` | list providers and metadata |
+| `GET /address?q=...` | geocode address and return availability |
+| `GET /availability?lat=...&lon=...` | coordinate-based lookup |
+| `GET /tiles/{z}/{x}/{y}` or static PMTiles | map layer data |
+| `GET /sources` | data source versions, hashes, timestamps |
+| `POST /admin/refresh` | manually trigger data refresh, internal only |
+
+Address lookup logic:
+
+1. Normalize user-entered address.
+2. Confirm address is inside/near ZIPs 75154 or 75146.
+3. Geocode via Census Geocoder first; fallback to Nominatim only if Census fails.
+4. If Fabric data exists: match nearest/normalized BSL and join by `location_id`.
+5. If no Fabric data: use nearest H3/block/provider footprint approximation and label confidence lower.
+6. Return 300+ Mbps providers as qualifying and show sub-300 Mbps services as **excluded under 300 Mbps** when source data exposes them.
+
+### 4. Frontend web app
+
+Internal Ops page:
+
+- Map centered on ZIPs 75154 and 75146.
+- Search box for address.
+- Provider filter toggles.
+- Result cards showing:
+  - provider
+  - max download/upload
+  - technology
+  - residential/business flag
+  - latency flag
+  - source
+  - confidence
+  - vintage date
+  - price, install fee, contract terms, order/contact URL, and phone number if verified/available
+- Source badge system:
+  - **High:** official provider address qualification or FCC BDC + licensed Fabric exact BSL match
+  - **Medium:** FCC BDC without Fabric exact address match, provider coverage map, GIS polygon
+  - **Low:** third-party directories, affiliate ISP comparison pages, inferred/SEO pages
+
+### 5. Admin / refresh workflow
+
+- Scheduled refresh when FCC releases new BDC data, likely twice per year.
+- Manual refresh command.
+- Data version page showing current vintage and import hashes.
+- Rollback to previous imported vintage.
+
+---
+
+## Data Acquisition Plan
+
+## A. FCC BDC data — primary baseline
+
+**Need:** FCC BDC API credentials.
+
+Steps:
+
+1. Create/use FCC CORES/BDC login.
+2. Generate NBM API token from `https://broadbandmap.fcc.gov/login` → Manage API Access.
+3. Store username/token in server `.env`.
+4. Call `listAsOfDates` to discover newest availability date.
+5. Call `listAvailabilityData/{as_of_date}` with filters:
+   - category: State or Provider
+   - technology_type: Fixed Broadband
+   - state: Texas / FIPS 48 where applicable
+6. Download Texas fixed broadband data files.
+7. Build provider ID map for AT&T, Spectrum, Rise, Brightspeed, Frontier.
+8. Import and filter:
+   - `state_usps = 'TX'`
+   - `max_advertised_download_speed >= 300`
+   - provider brand/provider ID in target list
+   - geography intersects ZIP 75154 / Red Oak region
+
+**Expected quality:** Medium to high, depending on whether we can join to Fabric.
+
+**Problem:** BDC availability data alone uses `location_id`; exact address display requires Fabric.
+
+## B. FCC/CostQuest Fabric — needed for exact address-level results
+
+**Need:** Fabric license/access for relevant counties/area.
+
+Steps:
+
+1. Determine license path:
+   - Forefront as broadband provider if eligible; or
+   - IT Pro Partner / client as other entity if challenge/research purpose fits.
+2. Request Fabric for required geography:
+   - At minimum: Ellis County and nearby ZIP 75154 area.
+   - ZIP 75154 may cross/neighbor multiple jurisdictions, so verify county boundary before final request.
+3. Import Fabric county CSVs.
+4. Join `fabric_locations.location_id = fcc_availability.location_id`.
+5. Build address-level searchable layer.
+
+**Expected quality:** High.
+
+**Known delay:** FCC help docs say delivery may take up to about two weeks after entity info/request submission.
+
+**If we do not get Fabric:** MVP can still work, but confidence drops. It becomes “reported availability near/within this area,” not exact install qualification.
+
+## C. Provider websites — validation and enrichment only
+
+Providers:
+
+- AT&T
+- Spectrum
+- Rise Broadband
+- Brightspeed
+- Frontier
+
+Use cases:
+
+- Spot-check known Red Oak addresses.
+- Collect official order/contact URLs.
+- Confirm whether FCC-reported service appears orderable.
+- Capture screenshots/evidence manually or semi-automated if allowed.
+
+Do **not** rely on scraping these sites for bulk data unless terms/API access allow it. Expect CAPTCHA, anti-bot controls, and inconsistent outputs.
+
+## D. Forefront Wireless internal data
+
+Need from Forefront, ideally one or more:
+
+| Data | Value |
+|---|---|
+| Tower/AP coordinates | Build RF/source layer |
+| Sector azimuth, beamwidth, downtilt, frequency, height | Estimate coverage polygon |
+| Service radius / install rules | Qualification model |
+| Existing customer/install addresses | Validate coverage and demand |
+| Failed install / no-LOS addresses | Exclusion/weakness layer |
+| CRM/export of leads | Sales planning overlay |
+| KML/KMZ/shapefile/GeoJSON coverage | Fastest map layer |
+
+Forefront data should be kept separate from FCC/provider public data so we do not mix internal truth with public reported coverage.
+
+## E. Boundaries/geocoding/base maps
+
+- ZIP 75154 and 75146 boundaries: Census TIGER/Line ZCTA or local GIS.
+- Address geocoding: Census Geocoder primary.
+- Base map: OpenStreetMap tiles or self-hosted tiles if public use grows.
+- Optional local GIS: Ellis County parcels/address points if publicly downloadable.
+
+---
+
+## Implementation Phases
+
+### Phase 0 — Decisions and access
+
+**Objective:** Remove ambiguity before building.
+
+Confirmed by Germaine:
+
+1. App audience: **internal**.
+2. Forefront should be included as a layer/reference provider.
+3. Include available service details where public/FCC/provider sources expose them.
+4. Lower-speed services should be shown as **excluded under 300 Mbps**.
+5. Manual sampled validation is acceptable if provider websites block automation.
+
+Still needed before build:
+
+1. FCC BDC username/token, or approval to create/request one.
+2. Decision on whether to pursue FCC/CostQuest Fabric access.
+3. Any Forefront coverage/install data when available.
+4. Confirm temporary URL if/when deployed; default assumption remains `ops.itpropartner.com/forefront-broadband-map/`.
+
+### Phase 1 — FCC data proof-of-concept
+
+**Objective:** Prove we can pull and parse target provider records.
+
+Tasks:
+
+1. Build FCC API client.
+2. Pull latest availability vintage list.
+3. List Texas fixed broadband downloads.
+4. Download target files.
+5. Import to DuckDB/SQLite.
+6. Filter to providers and speeds >=300 Mbps.
+7. Produce first table: provider, technology, count of qualifying locations, max speeds, vintage.
+
+Validation:
+
+- Verify source file hashes.
+- Verify row counts before/after filtering.
+- Verify provider IDs/brand aliases manually.
+
+### Phase 2 — Geography narrowing
+
+**Objective:** Limit data to ZIP 75154 / Red Oak.
+
+Two paths:
+
+- **With Fabric:** exact BSL address/coordinates → ZIP/spatial filter.
+- **Without Fabric:** use block GEOID/H3/ZIP boundary approximations → lower confidence.
+
+Validation:
+
+- Test against 5–10 known Red Oak addresses.
+- Confirm inside/outside ZIP behavior.
+
+### Phase 3 — API and UI MVP
+
+**Objective:** Build usable internal prototype.
+
+Tasks:
+
+1. FastAPI service with `/health`, `/address`, `/providers`, `/sources`.
+2. Static frontend with map and search.
+3. Provider filters and result cards.
+4. Source confidence badges.
+5. Deploy behind Caddy under Ops.
+6. Password protect if internal-only.
+
+Validation:
+
+- Search known addresses.
+- Compare returned providers against FCC map/provider websites.
+- Confirm every result shows source + confidence + vintage.
+
+### Phase 4 — Forefront layer
+
+**Objective:** Add Forefront’s own service intelligence.
+
+Tasks depend on data received:
+
+- If coverage polygons/KML exist: import directly.
+- If tower/sector data exists: generate approximate sector polygons.
+- If customer/install data exists: add point layer and anonymized heatmap.
+- If failed installs/no-LOS exists: add negative evidence layer.
+
+Validation:
+
+- Review with Forefront ops/sales.
+- Confirm no sensitive customer PII appears in public/internal UI unless approved.
+
+### Phase 5 — Data refresh and reporting
+
+**Objective:** Make it durable, not a one-off demo.
+
+Tasks:
+
+1. Add refresh script.
+2. Add import logs.
+3. Add rollback support.
+4. Add export/report:
+   - CSV of addresses/providers if address list supplied.
+   - Coverage gap report.
+   - Competitor overlap report.
+
+Validation:
+
+- Run refresh twice idempotently.
+- Verify no duplicate rows.
+- Verify previous vintage can be restored.
+
+---
+
+## Files Likely To Change / Be Created
+
+If this becomes a real build, create a dedicated project repo/folder, likely:
+
+```text
+/root/projects/forefront-broadband-map/
+  README.md
+  .env.example
+  docker-compose.yml
+  backend/
+    app/main.py
+    app/config.py
+    app/db.py
+    app/routes/address.py
+    app/routes/providers.py
+    app/routes/sources.py
+    app/services/geocode.py
+    app/services/availability.py
+    app/services/source_quality.py
+    tests/
+  frontend/
+    index.html
+    src/main.js
+    src/styles.css
+  scripts/
+    fcc_list_vintages.py
+    fcc_download_availability.py
+    import_fcc_bdc.py
+    import_fabric.py
+    import_forefront.py
+    build_tiles.py
+  data/
+    raw/
+    processed/
+  docs/
+    data-sources.md
+    source-confidence-policy.md
+    operations.md
+```
+
+Also keep canonical project note in:
+
+```text
+/root/projects/itpp-infrastructure/projects/forefront-broadband-map.md
+```
+
+---
+
+## Source Quality Ranking Policy
+
+| Rank | Source Type | Use |
+|---:|---|---|
+| 1 | Provider official address qualification API/page result | Highest confidence for individual address, if accessible and timestamped |
+| 2 | FCC BDC + licensed Fabric exact BSL join | Strong reproducible baseline; still provider-reported |
+| 3 | FCC BDC without Fabric exact address join | Good area/provider signal; not exact address certainty |
+| 4 | Official provider coverage map/page | Useful validation; often broad/incomplete |
+| 5 | Local/county GIS address/parcels | Good for geocoding/boundaries, not service availability |
+| 6 | Third-party ISP directories/comparison sites | Context only; do not use as truth |
+| 7 | Inferred RF/coverage model | Planning signal only until validated by installs/tests |
+
+---
+
+## Risks / Constraints
+
+1. **Fabric access is the hard gate for clean address-level accuracy.** Without it, exact address qualification is weaker.
+2. **Provider-reported FCC data can be wrong.** Must display source/vintage/confidence.
+3. **Provider websites are not reliable bulk data sources.** Use for validation, not scraping-first architecture.
+4. **ZIP boundaries are messy.** Need ZCTA/parcel/geocode handling and outside-scope warnings.
+5. **Forefront RF coverage may not equal installability.** Line-of-sight, foliage, CPE height, and sector load matter.
+6. **PII risk.** Forefront customer/install data must be anonymized or access-controlled.
+
+---
+
+## Open Questions
+
+1. Can Forefront obtain/provide FCC Fabric access, or should we request it separately?
+2. Which Forefront data format will come first: coverage polygons, tower/AP/sector/customer/install/no-LOS data, or address-level serviceability list?
+3. Should business-only and residential-only services be shown separately when the source distinguishes them, or merged into one availability card?
+4. What confidence threshold is acceptable for showing a provider as “available” versus “reported/possible”?
+5. Should the prototype remain unprotected as previously stated, or should Ops login be added once it is deployed?
+
+---
+
+## Recommended Next Move
+
+Do **Phase 1 first**: prove FCC BDC ingest and produce a filtered provider/speed table for ZIPs 75154 and 75146. In parallel, start Fabric access because it is the likely schedule bottleneck.
+
+If Fabric access is delayed, build a useful internal MVP with clear confidence labeling, then upgrade it to exact address-level matching once Fabric arrives.