From 6035fc44b896fd0a4f860e82542586acd22df6f8 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Jul 2026 22:59:45 -0400 Subject: [PATCH] Update Forefront map implementation plan --- ...-21_224054-forefront-broadband-map-plan.md | 485 ++++++++++++++++++ projects/forefront-broadband-map.md | 3 +- 2 files changed, 487 insertions(+), 1 deletion(-) create mode 100644 .hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md diff --git a/.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md b/.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md new file mode 100644 index 0000000..0bf1110 --- /dev/null +++ b/.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md @@ -0,0 +1,485 @@ +# Forefront Broadband Map Implementation Plan + +> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task. + +**Goal:** Build an internal Ops-hosted broadband availability map for ZIPs **75154 / Red Oak, Texas** and **75146 / Lancaster, Texas** showing address-level 300+ Mbps availability for AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier, and Forefront Wireless if Forefront data is provided. + +**Architecture:** Use FCC BDC fixed broadband availability as the reproducible baseline, normalize it into a local spatial database, and expose a small API + Leaflet/MapLibre web UI. Provider websites are validation sources, not the bulk-data foundation, because they are often CAPTCHA/ToS limited and not designed for bulk qualification. + +**Tech Stack:** Python/FastAPI, PostgreSQL + PostGIS or SQLite/GeoPackage for MVP, DuckDB for bulk CSV processing, Tippecanoe/PMTiles or GeoJSON for map layers, Leaflet or MapLibre GL JS, Caddy reverse proxy under Ops. + +--- + +## Current Context / Assumptions + +- Target area: **ZIPs 75154 / Red Oak, Texas and 75146 / Lancaster, Texas**. Scope for 75146 is the **full ZIP**. +- Minimum plan speed: **300 Mbps download or higher**. +- Initial competitors: **AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier**. +- Forefront Wireless should be added if we can get service-area GIS, tower/AP sector data, customer install data, or an address qualification/export file. +- App is **internal first** under `ops.itpropartner.com`, not public customer-facing on day one. +- Provider qualification access is limited to **public web pages** plus FCC/government data. +- Results should include available details where sources provide them: speed, technology, price, install fee, contract terms, order URL, and phone number. +- Services below 300 Mbps should be shown as **excluded under 300 Mbps** rather than treated as qualifying results. +- Manual sampled validation is acceptable if provider websites block automation. +- Initial validation address: `927 Pierce Road, Red Oak, TX 75154`, geocoded via OpenStreetMap/Nominatim to `32.5177279, -96.7677135`. +- FCC BDC data is provider-reported and useful as a baseline, but it must be labeled as reported availability, not guaranteed installability. + +## Verified Source Findings + +### FCC Broadband Data Collection / National Broadband Map + +- The FCC Broadband Data page says the National Broadband Map provides information about services available to individual locations as reported by ISPs. +- FCC public data APIs exist for downloadable BDC files, but require an FCC username and generated API token. +- API docs identify: + - `GET /api/public/map/listAsOfDates` + - `GET /api/public/map/downloads/listAvailabilityData/{as_of_date}` + - `GET /api/public/map/downloads/downloadFile/{file_id}` +- Availability download list supports filtering by category, subcategory, technology type, state/provider, etc. +- Rate limit in the FCC API spec: **10 calls per minute**. + +### FCC fixed broadband availability files + +The FCC BDC download specification defines fixed broadband availability CSV fields including: + +- `frn` +- `provider_id` +- `brand_name` +- `location_id` +- `technology` +- `max_advertised_download_speed` +- `max_advertised_upload_speed` +- `low_latency` +- `business_residential_code` +- `state_usps` +- `block_geoid` +- `h3_res8_id` + +Technology codes include: + +| Code | Technology | +|---:|---| +| 10 | Copper Wire | +| 40 | Coaxial Cable / HFC | +| 50 | Fiber to the Premises | +| 60 | Geostationary Satellite | +| 61 | Non-geostationary Satellite | +| 70 | Unlicensed Terrestrial Fixed Wireless | +| 71 | Licensed Terrestrial Fixed Wireless | +| 72 | Licensed-by-Rule Terrestrial Fixed Wireless | +| 0 | Other | + +### Fabric / address limitation + +- Public BDC availability records key on `location_id`. +- The FCC help docs say to obtain address, coordinate, building type, and other location details beyond Location ID, access to the Broadband Serviceable Location Fabric is required. +- Fabric access requires a license through FCC/CostQuest and can take up to ~2 weeks after entity info/request submission. +- Without Fabric, we can still use `location_id`, block GEOID, H3 cell, ZIP/county/geography summaries, and geocoded user input — but exact address-to-BSL matching will be imperfect. + +### Census Geocoder + +- Census Geocoder provides public REST geocoding for U.S. addresses. +- Single record endpoint form: `https://geocoding.geo.census.gov/geocoder/returntype/searchtype?parameters` +- Batch mode supports up to **10,000 records** per batch file. +- Useful for converting user-entered addresses to lat/lon and census geography, but it does **not** return FCC Fabric `location_id`. + +--- + +## What Needs To Be Built + +### 1. Data ingestion pipeline + +**Purpose:** Pull, normalize, and refresh broadband availability data. + +**Components:** + +- `scripts/fcc_list_vintages.py` + - Calls FCC list-as-of-dates API. + - Selects latest availability vintage. + +- `scripts/fcc_download_availability.py` + - Downloads Texas fixed broadband availability data by technology/provider where available. + - Saves raw ZIP/CSV files under `data/raw/fcc/YYYY-MM-DD/`. + - Logs file IDs, provider IDs, source URL, download timestamp, hash. + +- `scripts/import_fcc_bdc.py` + - Imports CSVs into staging tables. + - Filters target providers. + - Filters `max_advertised_download_speed >= 300`. + - Normalizes technology codes. + - Produces clean availability table. + +- `scripts/import_fabric.py` **only if Fabric license/data is obtained** + - Imports county-level Fabric CSVs for Ellis County / Dallas County as needed. + - Joins BDC `location_id` to Fabric address/lat/lon. + +- `scripts/import_forefront.py` + - Imports Forefront service data if provided. + - Accepted input formats: CSV address list, customer/install export, tower coordinates + sector azimuth/beamwidth/radius, KMZ/KML, GeoJSON, shapefile, or coverage polygon. + +### 2. Spatial data store + +**MVP choice:** SQLite + GeoPackage if dataset stays small. + +**Better production choice:** PostgreSQL + PostGIS. + +Tables/views: + +| Table/View | Purpose | +|---|---| +| `raw_fcc_downloads` | downloaded file metadata and hashes | +| `fcc_availability_raw` | raw BDC rows | +| `provider_map` | provider aliases and FCC provider IDs | +| `availability_clean` | normalized 300+ Mbps availability rows | +| `fabric_locations` | BSL address/lat/lon/building fields if licensed Fabric is available | +| `forefront_coverage` | Forefront-specific service areas/data | +| `address_lookup_cache` | user address → lat/lon/geography/cache result | +| `source_evidence` | URL, source type, confidence, retrieval timestamp | + +### 3. Backend API + +FastAPI endpoints: + +| Endpoint | Purpose | +|---|---| +| `GET /health` | service/database status | +| `GET /providers` | list providers and metadata | +| `GET /address?q=...` | geocode address and return availability | +| `GET /availability?lat=...&lon=...` | coordinate-based lookup | +| `GET /tiles/{z}/{x}/{y}` or static PMTiles | map layer data | +| `GET /sources` | data source versions, hashes, timestamps | +| `POST /admin/refresh` | manually trigger data refresh, internal only | + +Address lookup logic: + +1. Normalize user-entered address. +2. Confirm address is inside/near ZIPs 75154 or 75146. +3. Geocode via Census Geocoder first; fallback to Nominatim only if Census fails. +4. If Fabric data exists: match nearest/normalized BSL and join by `location_id`. +5. If no Fabric data: use nearest H3/block/provider footprint approximation and label confidence lower. +6. Return 300+ Mbps providers as qualifying and show sub-300 Mbps services as **excluded under 300 Mbps** when source data exposes them. + +### 4. Frontend web app + +Internal Ops page: + +- Map centered on ZIPs 75154 and 75146. +- Search box for address. +- Provider filter toggles. +- Result cards showing: + - provider + - max download/upload + - technology + - residential/business flag + - latency flag + - source + - confidence + - vintage date + - price, install fee, contract terms, order/contact URL, and phone number if verified/available +- Source badge system: + - **High:** official provider address qualification or FCC BDC + licensed Fabric exact BSL match + - **Medium:** FCC BDC without Fabric exact address match, provider coverage map, GIS polygon + - **Low:** third-party directories, affiliate ISP comparison pages, inferred/SEO pages + +### 5. Admin / refresh workflow + +- Scheduled refresh when FCC releases new BDC data, likely twice per year. +- Manual refresh command. +- Data version page showing current vintage and import hashes. +- Rollback to previous imported vintage. + +--- + +## Data Acquisition Plan + +## A. FCC BDC data — primary baseline + +**Need:** FCC BDC API credentials. + +Steps: + +1. Create/use FCC CORES/BDC login. +2. Generate NBM API token from `https://broadbandmap.fcc.gov/login` → Manage API Access. +3. Store username/token in server `.env`. +4. Call `listAsOfDates` to discover newest availability date. +5. Call `listAvailabilityData/{as_of_date}` with filters: + - category: State or Provider + - technology_type: Fixed Broadband + - state: Texas / FIPS 48 where applicable +6. Download Texas fixed broadband data files. +7. Build provider ID map for AT&T, Spectrum, Rise, Brightspeed, Frontier. +8. Import and filter: + - `state_usps = 'TX'` + - `max_advertised_download_speed >= 300` + - provider brand/provider ID in target list + - geography intersects ZIP 75154 / Red Oak region + +**Expected quality:** Medium to high, depending on whether we can join to Fabric. + +**Problem:** BDC availability data alone uses `location_id`; exact address display requires Fabric. + +## B. FCC/CostQuest Fabric — needed for exact address-level results + +**Need:** Fabric license/access for relevant counties/area. + +Steps: + +1. Determine license path: + - Forefront as broadband provider if eligible; or + - IT Pro Partner / client as other entity if challenge/research purpose fits. +2. Request Fabric for required geography: + - At minimum: Ellis County and nearby ZIP 75154 area. + - ZIP 75154 may cross/neighbor multiple jurisdictions, so verify county boundary before final request. +3. Import Fabric county CSVs. +4. Join `fabric_locations.location_id = fcc_availability.location_id`. +5. Build address-level searchable layer. + +**Expected quality:** High. + +**Known delay:** FCC help docs say delivery may take up to about two weeks after entity info/request submission. + +**If we do not get Fabric:** MVP can still work, but confidence drops. It becomes “reported availability near/within this area,” not exact install qualification. + +## C. Provider websites — validation and enrichment only + +Providers: + +- AT&T +- Spectrum +- Rise Broadband +- Brightspeed +- Frontier + +Use cases: + +- Spot-check known Red Oak addresses. +- Collect official order/contact URLs. +- Confirm whether FCC-reported service appears orderable. +- Capture screenshots/evidence manually or semi-automated if allowed. + +Do **not** rely on scraping these sites for bulk data unless terms/API access allow it. Expect CAPTCHA, anti-bot controls, and inconsistent outputs. + +## D. Forefront Wireless internal data + +Need from Forefront, ideally one or more: + +| Data | Value | +|---|---| +| Tower/AP coordinates | Build RF/source layer | +| Sector azimuth, beamwidth, downtilt, frequency, height | Estimate coverage polygon | +| Service radius / install rules | Qualification model | +| Existing customer/install addresses | Validate coverage and demand | +| Failed install / no-LOS addresses | Exclusion/weakness layer | +| CRM/export of leads | Sales planning overlay | +| KML/KMZ/shapefile/GeoJSON coverage | Fastest map layer | + +Forefront data should be kept separate from FCC/provider public data so we do not mix internal truth with public reported coverage. + +## E. Boundaries/geocoding/base maps + +- ZIP 75154 and 75146 boundaries: Census TIGER/Line ZCTA or local GIS. +- Address geocoding: Census Geocoder primary. +- Base map: OpenStreetMap tiles or self-hosted tiles if public use grows. +- Optional local GIS: Ellis County parcels/address points if publicly downloadable. + +--- + +## Implementation Phases + +### Phase 0 — Decisions and access + +**Objective:** Remove ambiguity before building. + +Confirmed by Germaine: + +1. App audience: **internal**. +2. Forefront should be included as a layer/reference provider. +3. Include available service details where public/FCC/provider sources expose them. +4. Lower-speed services should be shown as **excluded under 300 Mbps**. +5. Manual sampled validation is acceptable if provider websites block automation. + +Still needed before build: + +1. FCC BDC username/token, or approval to create/request one. +2. Decision on whether to pursue FCC/CostQuest Fabric access. +3. Any Forefront coverage/install data when available. +4. Confirm temporary URL if/when deployed; default assumption remains `ops.itpropartner.com/forefront-broadband-map/`. + +### Phase 1 — FCC data proof-of-concept + +**Objective:** Prove we can pull and parse target provider records. + +Tasks: + +1. Build FCC API client. +2. Pull latest availability vintage list. +3. List Texas fixed broadband downloads. +4. Download target files. +5. Import to DuckDB/SQLite. +6. Filter to providers and speeds >=300 Mbps. +7. Produce first table: provider, technology, count of qualifying locations, max speeds, vintage. + +Validation: + +- Verify source file hashes. +- Verify row counts before/after filtering. +- Verify provider IDs/brand aliases manually. + +### Phase 2 — Geography narrowing + +**Objective:** Limit data to ZIP 75154 / Red Oak. + +Two paths: + +- **With Fabric:** exact BSL address/coordinates → ZIP/spatial filter. +- **Without Fabric:** use block GEOID/H3/ZIP boundary approximations → lower confidence. + +Validation: + +- Test against 5–10 known Red Oak addresses. +- Confirm inside/outside ZIP behavior. + +### Phase 3 — API and UI MVP + +**Objective:** Build usable internal prototype. + +Tasks: + +1. FastAPI service with `/health`, `/address`, `/providers`, `/sources`. +2. Static frontend with map and search. +3. Provider filters and result cards. +4. Source confidence badges. +5. Deploy behind Caddy under Ops. +6. Password protect if internal-only. + +Validation: + +- Search known addresses. +- Compare returned providers against FCC map/provider websites. +- Confirm every result shows source + confidence + vintage. + +### Phase 4 — Forefront layer + +**Objective:** Add Forefront’s own service intelligence. + +Tasks depend on data received: + +- If coverage polygons/KML exist: import directly. +- If tower/sector data exists: generate approximate sector polygons. +- If customer/install data exists: add point layer and anonymized heatmap. +- If failed installs/no-LOS exists: add negative evidence layer. + +Validation: + +- Review with Forefront ops/sales. +- Confirm no sensitive customer PII appears in public/internal UI unless approved. + +### Phase 5 — Data refresh and reporting + +**Objective:** Make it durable, not a one-off demo. + +Tasks: + +1. Add refresh script. +2. Add import logs. +3. Add rollback support. +4. Add export/report: + - CSV of addresses/providers if address list supplied. + - Coverage gap report. + - Competitor overlap report. + +Validation: + +- Run refresh twice idempotently. +- Verify no duplicate rows. +- Verify previous vintage can be restored. + +--- + +## Files Likely To Change / Be Created + +If this becomes a real build, create a dedicated project repo/folder, likely: + +```text +/root/projects/forefront-broadband-map/ + README.md + .env.example + docker-compose.yml + backend/ + app/main.py + app/config.py + app/db.py + app/routes/address.py + app/routes/providers.py + app/routes/sources.py + app/services/geocode.py + app/services/availability.py + app/services/source_quality.py + tests/ + frontend/ + index.html + src/main.js + src/styles.css + scripts/ + fcc_list_vintages.py + fcc_download_availability.py + import_fcc_bdc.py + import_fabric.py + import_forefront.py + build_tiles.py + data/ + raw/ + processed/ + docs/ + data-sources.md + source-confidence-policy.md + operations.md +``` + +Also keep canonical project note in: + +```text +/root/projects/itpp-infrastructure/projects/forefront-broadband-map.md +``` + +--- + +## Source Quality Ranking Policy + +| Rank | Source Type | Use | +|---:|---|---| +| 1 | Provider official address qualification API/page result | Highest confidence for individual address, if accessible and timestamped | +| 2 | FCC BDC + licensed Fabric exact BSL join | Strong reproducible baseline; still provider-reported | +| 3 | FCC BDC without Fabric exact address join | Good area/provider signal; not exact address certainty | +| 4 | Official provider coverage map/page | Useful validation; often broad/incomplete | +| 5 | Local/county GIS address/parcels | Good for geocoding/boundaries, not service availability | +| 6 | Third-party ISP directories/comparison sites | Context only; do not use as truth | +| 7 | Inferred RF/coverage model | Planning signal only until validated by installs/tests | + +--- + +## Risks / Constraints + +1. **Fabric access is the hard gate for clean address-level accuracy.** Without it, exact address qualification is weaker. +2. **Provider-reported FCC data can be wrong.** Must display source/vintage/confidence. +3. **Provider websites are not reliable bulk data sources.** Use for validation, not scraping-first architecture. +4. **ZIP boundaries are messy.** Need ZCTA/parcel/geocode handling and outside-scope warnings. +5. **Forefront RF coverage may not equal installability.** Line-of-sight, foliage, CPE height, and sector load matter. +6. **PII risk.** Forefront customer/install data must be anonymized or access-controlled. + +--- + +## Open Questions + +1. Can Forefront obtain/provide FCC Fabric access, or should we request it separately? +2. Which Forefront data format will come first: coverage polygons, tower/AP/sector/customer/install/no-LOS data, or address-level serviceability list? +3. Should business-only and residential-only services be shown separately when the source distinguishes them, or merged into one availability card? +4. What confidence threshold is acceptable for showing a provider as “available” versus “reported/possible”? +5. Should the prototype remain unprotected as previously stated, or should Ops login be added once it is deployed? + +--- + +## Recommended Next Move + +Do **Phase 1 first**: prove FCC BDC ingest and produce a filtered provider/speed table for ZIPs 75154 and 75146. In parallel, start Fabric access because it is the likely schedule bottleneck. + +If Fabric access is delayed, build a useful internal MVP with clear confidence labeling, then upgrade it to exact address-level matching once Fabric arrives. diff --git a/projects/forefront-broadband-map.md b/projects/forefront-broadband-map.md index f33e0ae..ffec5de 100644 --- a/projects/forefront-broadband-map.md +++ b/projects/forefront-broadband-map.md @@ -28,6 +28,7 @@ Temporary hosting, if needed: **Ops** — likely `ops.itpropartner.com/forefront | Prototype access protection | Not needed now | | Manual sampled validation if provider sites block automation | Yes | | Forefront source data format | Still needs specific format; Germaine answered yes but not which type. Accept polygons, tower/AP/sector data, or address-level serviceability list. | +| Additional plan-response confirmation | Internal first; Forefront layer yes; include available result details; under-300 Mbps services should be marked excluded | ## Providers to include @@ -43,7 +44,7 @@ Temporary hosting, if needed: **Ops** — likely `ops.itpropartner.com/forefront 1. Open an interactive map of ZIPs **75154** and **75146**. Scope for **75146** is the full ZIP. 2. Search by street address. 3. See which providers offer **300+ Mbps** service at that address. -4. Show lower-speed services grayed out as excluded. +4. Show lower-speed services as **excluded under 300 Mbps** instead of hiding them. 5. View provider details when available: - download/upload speed - technology