89 lines
4.6 KiB
Markdown
89 lines
4.6 KiB
Markdown
# Anti-VRBO Playbook — Beating Vacation Rental Site Bot Protection
|
|
|
|
## Test Results (Aug 1, 2026)
|
|
|
|
| Site | Protection | Camofox Result | Notes |
|
|
|------|-----------|---------------|-------|
|
|
| **Airbnb** | Cloudflare + custom | **BEATEN** ✓ | Full page load, 5.7KB body, all content extracted. Camofox + DataImpulse proxy cleared Cloudflare with zero issues. |
|
|
| **VRBO** | Imperva/Incapsula | **BLOCKED** ✗ | "Bot or Not?" challenge wall. Imperva blocks at the application layer even after Cloudflare passthrough. Challenge doesn't auto-resolve even after 45s wait. |
|
|
| **Booking.com** | Akamai | **Untested** | — |
|
|
| **Vacasa** | Unknown | **Untested** | — |
|
|
| **Evolve** | Unknown | **Untested** | — |
|
|
|
|
## VRBO / Imperva — What We Tried
|
|
|
|
### Failed Approaches
|
|
1. **Direct camofox navigation** — Landed on "Bot or Not?" challenge page with Imperva session ID
|
|
2. **Extended wait (45s)** — Challenge regenerates but never auto-resolves
|
|
3. **Snapshot inspection** — Imperva challenge page has almost zero interactive elements; just a logo link
|
|
|
|
### Why Imperva Is Harder Than Cloudflare
|
|
- Cloudflare challenges happen at the edge (before the page loads) and Camoufox's C++-level fingerprint spoofing clears them
|
|
- Imperva challenges happen at the application layer AFTER the page loads — they inject their own JS that runs fingerprinting on top of whatever the browser reports
|
|
- Imperva's SDK (`window._Incapsula_Resource`) does behavioral analysis: mouse movements, scroll patterns, timing
|
|
- The challenge ID changes on every request (different UUID each time), suggesting it's stateless at the edge but stateful in their backend
|
|
|
|
## Strategies to Beat VRBO
|
|
|
|
### Strategy 1: Headful Browser with Human Emulation (Most Promising)
|
|
Instead of headless camofox, use a real Chrome/Chromium with CDP and emulate realistic human behavior:
|
|
```
|
|
1. Launch Chrome with puppeteer-real-browser or nodriver
|
|
2. Navigate to VRBO
|
|
3. Inject human-like mouse movements (bezier curves, not straight lines)
|
|
4. Random scrolls and hover events
|
|
5. Wait for Imperva challenge to auto-resolve (usually 5-15s in a real browser)
|
|
6. Extract data once page loads normally
|
|
```
|
|
|
|
### Strategy 2: VRBO API Reverse Engineering
|
|
VRBO's frontend calls internal APIs. If we can find the endpoints:
|
|
- Search: `vrbo.com/api/v1/search?...`
|
|
- Property details: `vrbo.com/api/v1/properties/1313017`
|
|
- Calendar: `vrbo.com/api/v1/properties/1313017/calendar`
|
|
|
|
Many of these may have weaker bot protection than the HTML pages. Imperva primarily protects the SSR HTML; API calls may use simpler auth.
|
|
|
|
### Strategy 3: Browserbase / Anchor (Cloudflare Web Bot Auth)
|
|
If VRBO opts into Cloudflare's Bot Verifier program, Browserbase/Anchor can get a signed "verified bot" token. But this only works if VRBO explicitly allows it — unlikely for a booking site.
|
|
|
|
### Strategy 4: Alternative Data Sources
|
|
Since Airbnb works perfectly with camofox, and most vacation rentals cross-list:
|
|
- **Airbnb** (working): Property details, pricing, reviews, calendar, photos
|
|
- **Google Travel / Hotel search**: Aggregates VRBO listings
|
|
- **VRBO via Google cache**: `webcache.googleusercontent.com/search?q=cache:vrbo.com/1313017`
|
|
|
|
### Strategy 5: Proxy Rotation + Fingerprint Diversity
|
|
Imperva may be flagging specific DataImpulse IP ranges. Try:
|
|
- Different proxy providers (SOAX, Bright Data, IPRoyal)
|
|
- Different geolocations
|
|
- Different Camoufox fingerprint profiles
|
|
- Session-level IP rotation
|
|
|
|
## Recommended Approach for Moore Sunny Daze
|
|
|
|
**Short-term (now):** Use Airbnb data since camofox clears it consistently. All the same content is there — Tim's listing is synced across platforms.
|
|
|
|
**Medium-term (this week):** Implement Strategy 1 (headful browser with human emulation) specifically for VRBO. This is the most reliable path.
|
|
|
|
**Long-term (product):** Build a unified scraper that tries Airbnb first, falls back to headful browser for VRBO, and caches results. This becomes the "Competitive Intelligence" feature of Beach Direct — show owners how their property compares to nearby listings on ALL platforms.
|
|
|
|
## Airbnb Data We Successfully Extracted
|
|
|
|
From the Airbnb listing (camofox, Aug 1 2026):
|
|
- Full property description
|
|
- 47 amenities list
|
|
- 5.0 rating across all categories (cleanliness, accuracy, check-in, communication, location, value)
|
|
- 4 reviews with text
|
|
- Calendar availability for Aug-Sep 2026
|
|
- Host: Ashley, 1 year hosting
|
|
- Property photos from Airbnb CDN
|
|
- Pricing: shown after date selection (requires interaction)
|
|
|
|
## Next Steps
|
|
1. [x] Confirm camofox beats Airbnb
|
|
2. [x] Document VRBO Imperva failure
|
|
3. [ ] Test headful browser approach for VRBO
|
|
4. [ ] Test Booking.com, Vacasa, Evolve
|
|
5. [ ] Build unified scraper module for Beach Direct
|