Initial commit: MSD backend API, docs, mockups, README
This commit is contained in:
root
@@ -0,0 +1,88 @@
|
||||
# Anti-VRBO Playbook — Beating Vacation Rental Site Bot Protection
|
||||
|
||||
## Test Results (Aug 1, 2026)
|
||||
|
||||
| Site | Protection | Camofox Result | Notes |
|
||||
|------|-----------|---------------|-------|
|
||||
| **Airbnb** | Cloudflare + custom | **BEATEN** ✓ | Full page load, 5.7KB body, all content extracted. Camofox + DataImpulse proxy cleared Cloudflare with zero issues. |
|
||||
| **VRBO** | Imperva/Incapsula | **BLOCKED** ✗ | "Bot or Not?" challenge wall. Imperva blocks at the application layer even after Cloudflare passthrough. Challenge doesn't auto-resolve even after 45s wait. |
|
||||
| **Booking.com** | Akamai | **Untested** | — |
|
||||
| **Vacasa** | Unknown | **Untested** | — |
|
||||
| **Evolve** | Unknown | **Untested** | — |
|
||||
|
||||
## VRBO / Imperva — What We Tried
|
||||
|
||||
### Failed Approaches
|
||||
1. **Direct camofox navigation** — Landed on "Bot or Not?" challenge page with Imperva session ID
|
||||
2. **Extended wait (45s)** — Challenge regenerates but never auto-resolves
|
||||
3. **Snapshot inspection** — Imperva challenge page has almost zero interactive elements; just a logo link
|
||||
|
||||
### Why Imperva Is Harder Than Cloudflare
|
||||
- Cloudflare challenges happen at the edge (before the page loads) and Camoufox's C++-level fingerprint spoofing clears them
|
||||
- Imperva challenges happen at the application layer AFTER the page loads — they inject their own JS that runs fingerprinting on top of whatever the browser reports
|
||||
- Imperva's SDK (`window._Incapsula_Resource`) does behavioral analysis: mouse movements, scroll patterns, timing
|
||||
- The challenge ID changes on every request (different UUID each time), suggesting it's stateless at the edge but stateful in their backend
|
||||
|
||||
## Strategies to Beat VRBO
|
||||
|
||||
### Strategy 1: Headful Browser with Human Emulation (Most Promising)
|
||||
Instead of headless camofox, use a real Chrome/Chromium with CDP and emulate realistic human behavior:
|
||||
```
|
||||
1. Launch Chrome with puppeteer-real-browser or nodriver
|
||||
2. Navigate to VRBO
|
||||
3. Inject human-like mouse movements (bezier curves, not straight lines)
|
||||
4. Random scrolls and hover events
|
||||
5. Wait for Imperva challenge to auto-resolve (usually 5-15s in a real browser)
|
||||
6. Extract data once page loads normally
|
||||
```
|
||||
|
||||
### Strategy 2: VRBO API Reverse Engineering
|
||||
VRBO's frontend calls internal APIs. If we can find the endpoints:
|
||||
- Search: `vrbo.com/api/v1/search?...`
|
||||
- Property details: `vrbo.com/api/v1/properties/1313017`
|
||||
- Calendar: `vrbo.com/api/v1/properties/1313017/calendar`
|
||||
|
||||
Many of these may have weaker bot protection than the HTML pages. Imperva primarily protects the SSR HTML; API calls may use simpler auth.
|
||||
|
||||
### Strategy 3: Browserbase / Anchor (Cloudflare Web Bot Auth)
|
||||
If VRBO opts into Cloudflare's Bot Verifier program, Browserbase/Anchor can get a signed "verified bot" token. But this only works if VRBO explicitly allows it — unlikely for a booking site.
|
||||
|
||||
### Strategy 4: Alternative Data Sources
|
||||
Since Airbnb works perfectly with camofox, and most vacation rentals cross-list:
|
||||
- **Airbnb** (working): Property details, pricing, reviews, calendar, photos
|
||||
- **Google Travel / Hotel search**: Aggregates VRBO listings
|
||||
- **VRBO via Google cache**: `webcache.googleusercontent.com/search?q=cache:vrbo.com/1313017`
|
||||
|
||||
### Strategy 5: Proxy Rotation + Fingerprint Diversity
|
||||
Imperva may be flagging specific DataImpulse IP ranges. Try:
|
||||
- Different proxy providers (SOAX, Bright Data, IPRoyal)
|
||||
- Different geolocations
|
||||
- Different Camoufox fingerprint profiles
|
||||
- Session-level IP rotation
|
||||
|
||||
## Recommended Approach for Moore Sunny Daze
|
||||
|
||||
**Short-term (now):** Use Airbnb data since camofox clears it consistently. All the same content is there — Tim's listing is synced across platforms.
|
||||
|
||||
**Medium-term (this week):** Implement Strategy 1 (headful browser with human emulation) specifically for VRBO. This is the most reliable path.
|
||||
|
||||
**Long-term (product):** Build a unified scraper that tries Airbnb first, falls back to headful browser for VRBO, and caches results. This becomes the "Competitive Intelligence" feature of Beach Direct — show owners how their property compares to nearby listings on ALL platforms.
|
||||
|
||||
## Airbnb Data We Successfully Extracted
|
||||
|
||||
From the Airbnb listing (camofox, Aug 1 2026):
|
||||
- Full property description
|
||||
- 47 amenities list
|
||||
- 5.0 rating across all categories (cleanliness, accuracy, check-in, communication, location, value)
|
||||
- 4 reviews with text
|
||||
- Calendar availability for Aug-Sep 2026
|
||||
- Host: Ashley, 1 year hosting
|
||||
- Property photos from Airbnb CDN
|
||||
- Pricing: shown after date selection (requires interaction)
|
||||
|
||||
## Next Steps
|
||||
1. [x] Confirm camofox beats Airbnb
|
||||
2. [x] Document VRBO Imperva failure
|
||||
3. [ ] Test headful browser approach for VRBO
|
||||
4. [ ] Test Booking.com, Vacasa, Evolve
|
||||
5. [ ] Build unified scraper module for Beach Direct
|
||||
Reference in New Issue
Block a user