itpp-infrastructure/.hermes/plans/2026-07-21_224054-forefront-broadband-map-plan.md

# Forefront Broadband Map Implementation Plan

> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.

**Goal:** Build an internal Ops-hosted broadband availability map for ZIPs **75154 / Red Oak, Texas** and **75146 / Lancaster, Texas** showing address-level 300+ Mbps availability for AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier, and Forefront Wireless if Forefront data is provided.

**Architecture:** Use FCC BDC fixed broadband availability as the reproducible baseline, normalize it into a local spatial database, and expose a small API + Leaflet/MapLibre web UI. Provider websites are validation sources, not the bulk-data foundation, because they are often CAPTCHA/ToS limited and not designed for bulk qualification.

**Tech Stack:** Python/FastAPI, PostgreSQL + PostGIS or SQLite/GeoPackage for MVP, DuckDB for bulk CSV processing, Tippecanoe/PMTiles or GeoJSON for map layers, Leaflet or MapLibre GL JS, Caddy reverse proxy under Ops.

---

## Current Context / Assumptions

- Target area: **ZIPs 75154 / Red Oak, Texas and 75146 / Lancaster, Texas**. Scope for 75146 is the **full ZIP**.
- Minimum plan speed: **300 Mbps download or higher**.
- Initial competitors: **AT&T, Spectrum, Rise Broadband, Brightspeed, Frontier**.
- Forefront Wireless should be added if we can get service-area GIS, tower/AP sector data, customer install data, or an address qualification/export file.
- App is **internal first** under `ops.itpropartner.com`, not public customer-facing on day one.
- Provider qualification access is limited to **public web pages** plus FCC/government data.
- Results should include available details where sources provide them: speed, technology, price, install fee, contract terms, order URL, and phone number.
- Services below 300 Mbps should be shown as **excluded under 300 Mbps** rather than treated as qualifying results.
- Manual sampled validation is acceptable if provider websites block automation.
- Initial validation address: `927 Pierce Road, Red Oak, TX 75154`, geocoded via OpenStreetMap/Nominatim to `32.5177279, -96.7677135`.
- FCC BDC data is provider-reported and useful as a baseline, but it must be labeled as reported availability, not guaranteed installability.

## Verified Source Findings

### FCC Broadband Data Collection / National Broadband Map

- The FCC Broadband Data page says the National Broadband Map provides information about services available to individual locations as reported by ISPs.
- FCC public data APIs exist for downloadable BDC files, but require an FCC username and generated API token.
- API docs identify:
  - `GET /api/public/map/listAsOfDates`
  - `GET /api/public/map/downloads/listAvailabilityData/{as_of_date}`
  - `GET /api/public/map/downloads/downloadFile/{file_id}`
- Availability download list supports filtering by category, subcategory, technology type, state/provider, etc.
- Rate limit in the FCC API spec: **10 calls per minute**.

### FCC fixed broadband availability files

The FCC BDC download specification defines fixed broadband availability CSV fields including:

- `frn`
- `provider_id`
- `brand_name`
- `location_id`
- `technology`
- `max_advertised_download_speed`
- `max_advertised_upload_speed`
- `low_latency`
- `business_residential_code`
- `state_usps`
- `block_geoid`
- `h3_res8_id`

Technology codes include:

| Code | Technology |
|---:|---|
| 10 | Copper Wire |
| 40 | Coaxial Cable / HFC |
| 50 | Fiber to the Premises |
| 60 | Geostationary Satellite |
| 61 | Non-geostationary Satellite |
| 70 | Unlicensed Terrestrial Fixed Wireless |
| 71 | Licensed Terrestrial Fixed Wireless |
| 72 | Licensed-by-Rule Terrestrial Fixed Wireless |
| 0 | Other |

### Fabric / address limitation

- Public BDC availability records key on `location_id`.
- The FCC help docs say to obtain address, coordinate, building type, and other location details beyond Location ID, access to the Broadband Serviceable Location Fabric is required.
- Fabric access requires a license through FCC/CostQuest and can take up to ~2 weeks after entity info/request submission.
- Without Fabric, we can still use `location_id`, block GEOID, H3 cell, ZIP/county/geography summaries, and geocoded user input — but exact address-to-BSL matching will be imperfect.

### Census Geocoder

- Census Geocoder provides public REST geocoding for U.S. addresses.
- Single record endpoint form: `https://geocoding.geo.census.gov/geocoder/returntype/searchtype?parameters`
- Batch mode supports up to **10,000 records** per batch file.
- Useful for converting user-entered addresses to lat/lon and census geography, but it does **not** return FCC Fabric `location_id`.

---

## What Needs To Be Built

### 1. Data ingestion pipeline

**Purpose:** Pull, normalize, and refresh broadband availability data.

**Components:**

- `scripts/fcc_list_vintages.py`
  - Calls FCC list-as-of-dates API.
  - Selects latest availability vintage.

- `scripts/fcc_download_availability.py`
  - Downloads Texas fixed broadband availability data by technology/provider where available.
  - Saves raw ZIP/CSV files under `data/raw/fcc/YYYY-MM-DD/`.
  - Logs file IDs, provider IDs, source URL, download timestamp, hash.

- `scripts/import_fcc_bdc.py`
  - Imports CSVs into staging tables.
  - Filters target providers.
  - Filters `max_advertised_download_speed >= 300`.
  - Normalizes technology codes.
  - Produces clean availability table.

- `scripts/import_fabric.py` **only if Fabric license/data is obtained**
  - Imports county-level Fabric CSVs for Ellis County / Dallas County as needed.
  - Joins BDC `location_id` to Fabric address/lat/lon.

- `scripts/import_forefront.py`
  - Imports Forefront service data if provided.
  - Accepted input formats: CSV address list, customer/install export, tower coordinates + sector azimuth/beamwidth/radius, KMZ/KML, GeoJSON, shapefile, or coverage polygon.

### 2. Spatial data store

**MVP choice:** SQLite + GeoPackage if dataset stays small.

**Better production choice:** PostgreSQL + PostGIS.

Tables/views:

| Table/View | Purpose |
|---|---|
| `raw_fcc_downloads` | downloaded file metadata and hashes |
| `fcc_availability_raw` | raw BDC rows |
| `provider_map` | provider aliases and FCC provider IDs |
| `availability_clean` | normalized 300+ Mbps availability rows |
| `fabric_locations` | BSL address/lat/lon/building fields if licensed Fabric is available |
| `forefront_coverage` | Forefront-specific service areas/data |
| `address_lookup_cache` | user address → lat/lon/geography/cache result |
| `source_evidence` | URL, source type, confidence, retrieval timestamp |

### 3. Backend API

FastAPI endpoints:

| Endpoint | Purpose |
|---|---|
| `GET /health` | service/database status |
| `GET /providers` | list providers and metadata |
| `GET /address?q=...` | geocode address and return availability |
| `GET /availability?lat=...&lon=...` | coordinate-based lookup |
| `GET /tiles/{z}/{x}/{y}` or static PMTiles | map layer data |
| `GET /sources` | data source versions, hashes, timestamps |
| `POST /admin/refresh` | manually trigger data refresh, internal only |

Address lookup logic:

1. Normalize user-entered address.
2. Confirm address is inside/near ZIPs 75154 or 75146.
3. Geocode via Census Geocoder first; fallback to Nominatim only if Census fails.
4. If Fabric data exists: match nearest/normalized BSL and join by `location_id`.
5. If no Fabric data: use nearest H3/block/provider footprint approximation and label confidence lower.
6. Return 300+ Mbps providers as qualifying and show sub-300 Mbps services as **excluded under 300 Mbps** when source data exposes them.

### 4. Frontend web app

Internal Ops page:

- Map centered on ZIPs 75154 and 75146.
- Search box for address.
- Provider filter toggles.
- Result cards showing:
  - provider
  - max download/upload
  - technology
  - residential/business flag
  - latency flag
  - source
  - confidence
  - vintage date
  - price, install fee, contract terms, order/contact URL, and phone number if verified/available
- Source badge system:
  - **High:** official provider address qualification or FCC BDC + licensed Fabric exact BSL match
  - **Medium:** FCC BDC without Fabric exact address match, provider coverage map, GIS polygon
  - **Low:** third-party directories, affiliate ISP comparison pages, inferred/SEO pages

### 5. Admin / refresh workflow

- Scheduled refresh when FCC releases new BDC data, likely twice per year.
- Manual refresh command.
- Data version page showing current vintage and import hashes.
- Rollback to previous imported vintage.

---

## Data Acquisition Plan

## A. FCC BDC data — primary baseline

**Need:** FCC BDC API credentials.

Steps:

1. Create/use FCC CORES/BDC login.
2. Generate NBM API token from `https://broadbandmap.fcc.gov/login` → Manage API Access.
3. Store username/token in server `.env`.
4. Call `listAsOfDates` to discover newest availability date.
5. Call `listAvailabilityData/{as_of_date}` with filters:
   - category: State or Provider
   - technology_type: Fixed Broadband
   - state: Texas / FIPS 48 where applicable
6. Download Texas fixed broadband data files.
7. Build provider ID map for AT&T, Spectrum, Rise, Brightspeed, Frontier.
8. Import and filter:
   - `state_usps = 'TX'`
   - `max_advertised_download_speed >= 300`
   - provider brand/provider ID in target list
   - geography intersects ZIP 75154 / Red Oak region

**Expected quality:** Medium to high, depending on whether we can join to Fabric.

**Problem:** BDC availability data alone uses `location_id`; exact address display requires Fabric.

## B. FCC/CostQuest Fabric — needed for exact address-level results

**Need:** Fabric license/access for relevant counties/area.

Steps:

1. Determine license path:
   - Forefront as broadband provider if eligible; or
   - IT Pro Partner / client as other entity if challenge/research purpose fits.
2. Request Fabric for required geography:
   - At minimum: Ellis County and nearby ZIP 75154 area.
   - ZIP 75154 may cross/neighbor multiple jurisdictions, so verify county boundary before final request.
3. Import Fabric county CSVs.
4. Join `fabric_locations.location_id = fcc_availability.location_id`.
5. Build address-level searchable layer.

**Expected quality:** High.

**Known delay:** FCC help docs say delivery may take up to about two weeks after entity info/request submission.

**If we do not get Fabric:** MVP can still work, but confidence drops. It becomes “reported availability near/within this area,” not exact install qualification.

## C. Provider websites — validation and enrichment only

Providers:

- AT&T
- Spectrum
- Rise Broadband
- Brightspeed
- Frontier

Use cases:

- Spot-check known Red Oak addresses.
- Collect official order/contact URLs.
- Confirm whether FCC-reported service appears orderable.
- Capture screenshots/evidence manually or semi-automated if allowed.

Do **not** rely on scraping these sites for bulk data unless terms/API access allow it. Expect CAPTCHA, anti-bot controls, and inconsistent outputs.

## D. Forefront Wireless internal data

Need from Forefront, ideally one or more:

| Data | Value |
|---|---|
| Tower/AP coordinates | Build RF/source layer |
| Sector azimuth, beamwidth, downtilt, frequency, height | Estimate coverage polygon |
| Service radius / install rules | Qualification model |
| Existing customer/install addresses | Validate coverage and demand |
| Failed install / no-LOS addresses | Exclusion/weakness layer |
| CRM/export of leads | Sales planning overlay |
| KML/KMZ/shapefile/GeoJSON coverage | Fastest map layer |

Forefront data should be kept separate from FCC/provider public data so we do not mix internal truth with public reported coverage.

## E. Boundaries/geocoding/base maps

- ZIP 75154 and 75146 boundaries: Census TIGER/Line ZCTA or local GIS.
- Address geocoding: Census Geocoder primary.
- Base map: OpenStreetMap tiles or self-hosted tiles if public use grows.
- Optional local GIS: Ellis County parcels/address points if publicly downloadable.

---

## Implementation Phases

### Phase 0 — Decisions and access

**Objective:** Remove ambiguity before building.

Confirmed by Germaine:

1. App audience: **internal**.
2. Forefront should be included as a layer/reference provider.
3. Include available service details where public/FCC/provider sources expose them.
4. Lower-speed services should be shown as **excluded under 300 Mbps**.
5. Manual sampled validation is acceptable if provider websites block automation.

Still needed before build:

1. FCC BDC username/token, or approval to create/request one.
2. Decision on whether to pursue FCC/CostQuest Fabric access.
3. Any Forefront coverage/install data when available.
4. Confirm temporary URL if/when deployed; default assumption remains `ops.itpropartner.com/forefront-broadband-map/`.

### Phase 1 — FCC data proof-of-concept

**Objective:** Prove we can pull and parse target provider records.

Tasks:

1. Build FCC API client.
2. Pull latest availability vintage list.
3. List Texas fixed broadband downloads.
4. Download target files.
5. Import to DuckDB/SQLite.
6. Filter to providers and speeds >=300 Mbps.
7. Produce first table: provider, technology, count of qualifying locations, max speeds, vintage.

Validation:

- Verify source file hashes.
- Verify row counts before/after filtering.
- Verify provider IDs/brand aliases manually.

### Phase 2 — Geography narrowing

**Objective:** Limit data to ZIP 75154 / Red Oak.

Two paths:

- **With Fabric:** exact BSL address/coordinates → ZIP/spatial filter.
- **Without Fabric:** use block GEOID/H3/ZIP boundary approximations → lower confidence.

Validation:

- Test against 5–10 known Red Oak addresses.
- Confirm inside/outside ZIP behavior.

### Phase 3 — API and UI MVP

**Objective:** Build usable internal prototype.

Tasks:

1. FastAPI service with `/health`, `/address`, `/providers`, `/sources`.
2. Static frontend with map and search.
3. Provider filters and result cards.
4. Source confidence badges.
5. Deploy behind Caddy under Ops.
6. Password protect if internal-only.

Validation:

- Search known addresses.
- Compare returned providers against FCC map/provider websites.
- Confirm every result shows source + confidence + vintage.

### Phase 4 — Forefront layer

**Objective:** Add Forefront’s own service intelligence.

Tasks depend on data received:

- If coverage polygons/KML exist: import directly.
- If tower/sector data exists: generate approximate sector polygons.
- If customer/install data exists: add point layer and anonymized heatmap.
- If failed installs/no-LOS exists: add negative evidence layer.

Validation:

- Review with Forefront ops/sales.
- Confirm no sensitive customer PII appears in public/internal UI unless approved.

### Phase 5 — Data refresh and reporting

**Objective:** Make it durable, not a one-off demo.

Tasks:

1. Add refresh script.
2. Add import logs.
3. Add rollback support.
4. Add export/report:
   - CSV of addresses/providers if address list supplied.
   - Coverage gap report.
   - Competitor overlap report.

Validation:

- Run refresh twice idempotently.
- Verify no duplicate rows.
- Verify previous vintage can be restored.

---

## Files Likely To Change / Be Created

If this becomes a real build, create a dedicated project repo/folder, likely:

```text
/root/projects/forefront-broadband-map/
  README.md
  .env.example
  docker-compose.yml
  backend/
    app/main.py
    app/config.py
    app/db.py
    app/routes/address.py
    app/routes/providers.py
    app/routes/sources.py
    app/services/geocode.py
    app/services/availability.py
    app/services/source_quality.py
    tests/
  frontend/
    index.html
    src/main.js
    src/styles.css
  scripts/
    fcc_list_vintages.py
    fcc_download_availability.py
    import_fcc_bdc.py
    import_fabric.py
    import_forefront.py
    build_tiles.py
  data/
    raw/
    processed/
  docs/
    data-sources.md
    source-confidence-policy.md
    operations.md
```

Also keep canonical project note in:

```text
/root/projects/itpp-infrastructure/projects/forefront-broadband-map.md
```

---

## Source Quality Ranking Policy

| Rank | Source Type | Use |
|---:|---|---|
| 1 | Provider official address qualification API/page result | Highest confidence for individual address, if accessible and timestamped |
| 2 | FCC BDC + licensed Fabric exact BSL join | Strong reproducible baseline; still provider-reported |
| 3 | FCC BDC without Fabric exact address join | Good area/provider signal; not exact address certainty |
| 4 | Official provider coverage map/page | Useful validation; often broad/incomplete |
| 5 | Local/county GIS address/parcels | Good for geocoding/boundaries, not service availability |
| 6 | Third-party ISP directories/comparison sites | Context only; do not use as truth |
| 7 | Inferred RF/coverage model | Planning signal only until validated by installs/tests |

---

## Risks / Constraints

1. **Fabric access is the hard gate for clean address-level accuracy.** Without it, exact address qualification is weaker.
2. **Provider-reported FCC data can be wrong.** Must display source/vintage/confidence.
3. **Provider websites are not reliable bulk data sources.** Use for validation, not scraping-first architecture.
4. **ZIP boundaries are messy.** Need ZCTA/parcel/geocode handling and outside-scope warnings.
5. **Forefront RF coverage may not equal installability.** Line-of-sight, foliage, CPE height, and sector load matter.
6. **PII risk.** Forefront customer/install data must be anonymized or access-controlled.

---

## Open Questions

1. Can Forefront obtain/provide FCC Fabric access, or should we request it separately?
2. Which Forefront data format will come first: coverage polygons, tower/AP/sector/customer/install/no-LOS data, or address-level serviceability list?
3. Should business-only and residential-only services be shown separately when the source distinguishes them, or merged into one availability card?
4. What confidence threshold is acceptable for showing a provider as “available” versus “reported/possible”?
5. Should the prototype remain unprotected as previously stated, or should Ops login be added once it is deployed?

---

## Recommended Next Move

Do **Phase 1 first**: prove FCC BDC ingest and produce a filtered provider/speed table for ZIPs 75154 and 75146. In parallel, start Fabric access because it is the likely schedule bottleneck.

If Fabric access is delayed, build a useful internal MVP with clear confidence labeling, then upgrade it to exact address-level matching once Fabric arrives.