itpp-infrastructure/.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md

Forefront Broadband Map Implementation Plan

For Hermes: Use subagent-driven-development skill to implement this plan task-by-task.

Goal: Build an internal Ops-hosted broadband availability map for ZIPs 75154 / Red Oak, Texas and 75146 / Lancaster, Texas showing address-level 300+ Mbps availability for AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier, and Forefront Wireless if Forefront data is provided.

Architecture: Use FCC BDC fixed broadband availability as the reproducible baseline, normalize it into a local spatial database, and expose a small API + Leaflet/MapLibre web UI. Provider websites are validation sources, not the bulk-data foundation, because they are often CAPTCHA/ToS limited and not designed for bulk qualification.

Tech Stack: Python/FastAPI, PostgreSQL + PostGIS or SQLite/GeoPackage for MVP, DuckDB for bulk CSV processing, Tippecanoe/PMTiles or GeoJSON for map layers, Leaflet or MapLibre GL JS, Caddy reverse proxy under Ops.

---

Current Context / Assumptions

• Target area: ZIPs 75154 / Red Oak, Texas and 75146 / Lancaster, Texas. Scope for 75146 is the full ZIP.
• Minimum plan speed: 300 Mbps download or higher.
• Initial competitors: AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier.
• Forefront Wireless should be added if we can get service-area GIS, tower/AP sector data, customer install data, or an address qualification/export file.
• App is internal first under ops.itpropartner.com, not public customer-facing on day one.
• Provider qualification access is limited to public web pages plus FCC/government data.
• Results should include available details where sources provide them: speed, technology, price, install fee, contract terms, order URL, and phone number.
• Services below 300 Mbps should be shown as excluded under 300 Mbps rather than treated as qualifying results.
• Manual sampled validation is acceptable if provider websites block automation.
• Initial validation address: 927 Pierce Road, Red Oak, TX 75154, geocoded via OpenStreetMap/Nominatim to 32.5177279, -96.7677135.
• FCC BDC data is provider-reported and useful as a baseline, but it must be labeled as reported availability, not guaranteed installability.

Verified Source Findings

FCC Broadband Data Collection / National Broadband Map

• The FCC Broadband Data page says the National Broadband Map provides information about services available to individual locations as reported by ISPs.
• FCC public data APIs exist for downloadable BDC files, but require an FCC username and generated API token.
• API docs identify:
  • GET /api/public/map/listAsOfDates
  • GET /api/public/map/downloads/listAvailabilityData/{as_of_date}
  • GET /api/public/map/downloads/downloadFile/{file_id}
• Availability download list supports filtering by category, subcategory, technology type, state/provider, etc.
• Rate limit in the FCC API spec: 10 calls per minute.

FCC fixed broadband availability files

The FCC BDC download specification defines fixed broadband availability CSV fields including:

• frn
• provider_id
• brand_name
• location_id
• technology
• max_advertised_download_speed
• max_advertised_upload_speed
• low_latency
• business_residential_code
• state_usps
• block_geoid
• h3_res8_id

Technology codes include:

Code	Technology
10	Copper Wire
40	Coaxial Cable / HFC
50	Fiber to the Premises
60	Geostationary Satellite
61	Non-geostationary Satellite
70	Unlicensed Terrestrial Fixed Wireless
71	Licensed Terrestrial Fixed Wireless
72	Licensed-by-Rule Terrestrial Fixed Wireless
0	Other

Fabric / address limitation

• Public BDC availability records key on location_id.
• The FCC help docs say to obtain address, coordinate, building type, and other location details beyond Location ID, access to the Broadband Serviceable Location Fabric is required.
• Fabric access requires a license through FCC/CostQuest and can take up to ~2 weeks after entity info/request submission.
• Without Fabric, we can still use location_id, block GEOID, H3 cell, ZIP/county/geography summaries, and geocoded user input — but exact address-to-BSL matching will be imperfect.

Census Geocoder

• Census Geocoder provides public REST geocoding for U.S. addresses.
• Single record endpoint form: https://geocoding.geo.census.gov/geocoder/returntype/searchtype?parameters
• Batch mode supports up to 10,000 records per batch file.
• Useful for converting user-entered addresses to lat/lon and census geography, but it does not return FCC Fabric location_id.

---

What Needs To Be Built

1. Data ingestion pipeline

Purpose: Pull, normalize, and refresh broadband availability data.

Components:

• scripts/fcc_list_vintages.py

  • Calls FCC list-as-of-dates API.
  • Selects latest availability vintage.

• scripts/fcc_download_availability.py

  • Downloads Texas fixed broadband availability data by technology/provider where available.
  • Saves raw ZIP/CSV files under data/raw/fcc/YYYY-MM-DD/.
  • Logs file IDs, provider IDs, source URL, download timestamp, hash.

• scripts/import_fcc_bdc.py

  • Imports CSVs into staging tables.
  • Filters target providers.
  • Filters max_advertised_download_speed >= 300.
  • Normalizes technology codes.
  • Produces clean availability table.

• scripts/import_fabric.py only if Fabric license/data is obtained

  • Imports county-level Fabric CSVs for Ellis County / Dallas County as needed.
  • Joins BDC location_id to Fabric address/lat/lon.

• scripts/import_forefront.py

  • Imports Forefront service data if provided.
  • Accepted input formats: CSV address list, customer/install export, tower coordinates + sector azimuth/beamwidth/radius, KMZ/KML, GeoJSON, shapefile, or coverage polygon.

2. Spatial data store

MVP choice: SQLite + GeoPackage if dataset stays small.

Better production choice: PostgreSQL + PostGIS.

Tables/views:

Table/View	Purpose
raw_fcc_downloads	downloaded file metadata and hashes
fcc_availability_raw	raw BDC rows
provider_map	provider aliases and FCC provider IDs
availability_clean	normalized 300+ Mbps availability rows
fabric_locations	BSL address/lat/lon/building fields if licensed Fabric is available
forefront_coverage	Forefront-specific service areas/data
address_lookup_cache	user address → lat/lon/geography/cache result
source_evidence	URL, source type, confidence, retrieval timestamp

3. Backend API

FastAPI endpoints:

Endpoint	Purpose
GET /health	service/database status
GET /providers	list providers and metadata
GET /address?q=...	geocode address and return availability
GET /availability?lat=...&lon=...	coordinate-based lookup
GET /tiles/{z}/{x}/{y} or static PMTiles	map layer data
GET /sources	data source versions, hashes, timestamps
POST /admin/refresh	manually trigger data refresh, internal only

Address lookup logic:

1. Normalize user-entered address.
2. Confirm address is inside/near ZIPs 75154 or 75146.
3. Geocode via Census Geocoder first; fallback to Nominatim only if Census fails.
4. If Fabric data exists: match nearest/normalized BSL and join by location_id.
5. If no Fabric data: use nearest H3/block/provider footprint approximation and label confidence lower.
6. Return 300+ Mbps providers as qualifying and show sub-300 Mbps services as excluded under 300 Mbps when source data exposes them.

4. Frontend web app

Internal Ops page:

• Map centered on ZIPs 75154 and 75146.
• Search box for address.
• Provider filter toggles.
• Result cards showing:
  • provider
  • max download/upload
  • technology
  • residential/business flag
  • latency flag
  • source
  • confidence
  • vintage date
  • price, install fee, contract terms, order/contact URL, and phone number if verified/available
• Source badge system:
  • High: official provider address qualification or FCC BDC + licensed Fabric exact BSL match
  • Medium: FCC BDC without Fabric exact address match, provider coverage map, GIS polygon
  • Low: third-party directories, affiliate ISP comparison pages, inferred/SEO pages

5. Admin / refresh workflow

• Scheduled refresh when FCC releases new BDC data, likely twice per year.
• Manual refresh command.
• Data version page showing current vintage and import hashes.
• Rollback to previous imported vintage.

---

Data Acquisition Plan

A. FCC BDC data — primary baseline

Need: FCC BDC API credentials.

Steps:

1. Create/use FCC CORES/BDC login.
2. Generate NBM API token from https://broadbandmap.fcc.gov/login → Manage API Access.
3. Store username/token in server .env.
4. Call listAsOfDates to discover newest availability date.
5. Call listAvailabilityData/{as_of_date} with filters:
   • category: State or Provider
   • technology_type: Fixed Broadband
   • state: Texas / FIPS 48 where applicable
6. Download Texas fixed broadband data files.
7. Build provider ID map for AT&T, Spectrum, Rise, Brightspeed, Frontier.
8. Import and filter:
   • state_usps = 'TX'
   • max_advertised_download_speed >= 300
   • provider brand/provider ID in target list
   • geography intersects ZIP 75154 / Red Oak region

Expected quality: Medium to high, depending on whether we can join to Fabric.

Problem: BDC availability data alone uses location_id; exact address display requires Fabric.

B. FCC/CostQuest Fabric — needed for exact address-level results

Need: Fabric license/access for relevant counties/area.

Steps:

1. Determine license path:
   • Forefront as broadband provider if eligible; or
   • IT Pro Partner / client as other entity if challenge/research purpose fits.
2. Request Fabric for required geography:
   • At minimum: Ellis County and nearby ZIP 75154 area.
   • ZIP 75154 may cross/neighbor multiple jurisdictions, so verify county boundary before final request.
3. Import Fabric county CSVs.
4. Join fabric_locations.location_id = fcc_availability.location_id.
5. Build address-level searchable layer.

Expected quality: High.

Known delay: FCC help docs say delivery may take up to about two weeks after entity info/request submission.

If we do not get Fabric: MVP can still work, but confidence drops. It becomes “reported availability near/within this area,” not exact install qualification.

C. Provider websites — validation and enrichment only

Providers:

• AT&T
• Spectrum
• Rise Broadband
• Brightspeed
• Frontier

Use cases:

• Spot-check known Red Oak addresses.
• Collect official order/contact URLs.
• Confirm whether FCC-reported service appears orderable.
• Capture screenshots/evidence manually or semi-automated if allowed.

Do not rely on scraping these sites for bulk data unless terms/API access allow it. Expect CAPTCHA, anti-bot controls, and inconsistent outputs.

D. Forefront Wireless internal data

Need from Forefront, ideally one or more:

Data	Value
Tower/AP coordinates	Build RF/source layer
Sector azimuth, beamwidth, downtilt, frequency, height	Estimate coverage polygon
Service radius / install rules	Qualification model
Existing customer/install addresses	Validate coverage and demand
Failed install / no-LOS addresses	Exclusion/weakness layer
CRM/export of leads	Sales planning overlay
KML/KMZ/shapefile/GeoJSON coverage	Fastest map layer

Forefront data should be kept separate from FCC/provider public data so we do not mix internal truth with public reported coverage.

E. Boundaries/geocoding/base maps

• ZIP 75154 and 75146 boundaries: Census TIGER/Line ZCTA or local GIS.
• Address geocoding: Census Geocoder primary.
• Base map: OpenStreetMap tiles or self-hosted tiles if public use grows.
• Optional local GIS: Ellis County parcels/address points if publicly downloadable.

---

Implementation Phases

Phase 0 — Decisions and access

Objective: Remove ambiguity before building.

Confirmed by Germaine:

1. App audience: internal.
2. Forefront should be included as a layer/reference provider.
3. Include available service details where public/FCC/provider sources expose them.
4. Lower-speed services should be shown as excluded under 300 Mbps.
5. Manual sampled validation is acceptable if provider websites block automation.

Still needed before build:

1. FCC BDC username/token, or approval to create/request one.
2. Decision on whether to pursue FCC/CostQuest Fabric access.
3. Any Forefront coverage/install data when available.
4. Confirm temporary URL if/when deployed; default assumption remains ops.itpropartner.com/forefront-broadband-map/.
5. No access protection is needed yet for the prototype.

Phase 1 — FCC data proof-of-concept

Objective: Prove we can pull and parse target provider records.

Tasks:

1. Build FCC API client.
2. Pull latest availability vintage list.
3. List Texas fixed broadband downloads.
4. Download target files.
5. Import to DuckDB/SQLite.
6. Filter to providers and speeds >=300 Mbps.
7. Produce first table: provider, technology, count of qualifying locations, max speeds, vintage.

Validation:

• Verify source file hashes.
• Verify row counts before/after filtering.
• Verify provider IDs/brand aliases manually.

Phase 2 — Geography narrowing

Objective: Limit data to ZIP 75154 / Red Oak.

Two paths:

• With Fabric: exact BSL address/coordinates → ZIP/spatial filter.
• Without Fabric: use block GEOID/H3/ZIP boundary approximations → lower confidence.

Validation:

• Test against 5–10 known Red Oak addresses.
• Confirm inside/outside ZIP behavior.

Phase 3 — API and UI MVP

Objective: Build usable internal prototype.

Tasks:

1. FastAPI service with /health, /address, /providers, /sources.
2. Static frontend with map and search.
3. Provider filters and result cards.
4. Source confidence badges.
5. Deploy behind Caddy under Ops.
6. Password protect if internal-only.

Validation:

• Search known addresses.
• Compare returned providers against FCC map/provider websites.
• Confirm every result shows source + confidence + vintage.

Phase 4 — Forefront layer

Objective: Add Forefront’s own service intelligence.

Tasks depend on data received:

• If coverage polygons/KML exist: import directly.
• If tower/sector data exists: generate approximate sector polygons.
• If customer/install data exists: add point layer and anonymized heatmap.
• If failed installs/no-LOS exists: add negative evidence layer.

Validation:

• Review with Forefront ops/sales.
• Confirm no sensitive customer PII appears in public/internal UI unless approved.

Phase 5 — Data refresh and reporting

Objective: Make it durable, not a one-off demo.

Tasks:

1. Add refresh script.
2. Add import logs.
3. Add rollback support.
4. Add export/report:
   • CSV of addresses/providers if address list supplied.
   • Coverage gap report.
   • Competitor overlap report.

Validation:

• Run refresh twice idempotently.
• Verify no duplicate rows.
• Verify previous vintage can be restored.

---

Files Likely To Change / Be Created

If this becomes a real build, create a dedicated project repo/folder, likely:

/root/projects/forefront-broadband-map/
  README.md
  .env.example
  docker-compose.yml
  backend/
    app/main.py
    app/config.py
    app/db.py
    app/routes/address.py
    app/routes/providers.py
    app/routes/sources.py
    app/services/geocode.py
    app/services/availability.py
    app/services/source_quality.py
    tests/
  frontend/
    index.html
    src/main.js
    src/styles.css
  scripts/
    fcc_list_vintages.py
    fcc_download_availability.py
    import_fcc_bdc.py
    import_fabric.py
    import_forefront.py
    build_tiles.py
  data/
    raw/
    processed/
  docs/
    data-sources.md
    source-confidence-policy.md
    operations.md

Also keep canonical project note in:

/root/projects/itpp-infrastructure/projects/forefront-broadband-map.md

---

Source Quality Ranking Policy

Rank	Source Type	Use
1	Provider official address qualification API/page result	Highest confidence for individual address, if accessible and timestamped
2	FCC BDC + licensed Fabric exact BSL join	Strong reproducible baseline; still provider-reported
3	FCC BDC without Fabric exact address join	Good area/provider signal; not exact address certainty
4	Official provider coverage map/page	Useful validation; often broad/incomplete
5	Local/county GIS address/parcels	Good for geocoding/boundaries, not service availability
6	Third-party ISP directories/comparison sites	Context only; do not use as truth
7	Inferred RF/coverage model	Planning signal only until validated by installs/tests

---

Risks / Constraints

1. Fabric access is the hard gate for clean address-level accuracy. Without it, exact address qualification is weaker.
2. Provider-reported FCC data can be wrong. Must display source/vintage/confidence.
3. Provider websites are not reliable bulk data sources. Use for validation, not scraping-first architecture.
4. ZIP boundaries are messy. Need ZCTA/parcel/geocode handling and outside-scope warnings.
5. Forefront RF coverage may not equal installability. Line-of-sight, foliage, CPE height, and sector load matter.
6. PII risk. Forefront customer/install data must be anonymized or access-controlled.

---

Open Questions

1. Can Forefront obtain/provide FCC Fabric access, or should we request it separately?
2. Which Forefront data format will come first: coverage polygons, tower/AP/sector/customer/install/no-LOS data, or address-level serviceability list?
3. Should business-only and residential-only services be shown separately when the source distinguishes them, or merged into one availability card?
4. What confidence threshold is acceptable for showing a provider as “available” versus “reported/possible”?
5. No access protection is needed yet for the prototype; revisit only if the app later exposes sensitive Forefront/customer data.

---

Recommended Next Move

Do Phase 1 first: prove FCC BDC ingest and produce a filtered provider/speed table for ZIPs 75154 and 75146. In parallel, start Fabric access because it is the likely schedule bottleneck.

If Fabric access is delayed, build a useful internal MVP with clear confidence labeling, then upgrade it to exact address-level matching once Fabric arrives.