Revenue Management Pricing & forecasting pipelines
4 min read Ingestion

Handling anti-bot challenges with Playwright

A competitor booking engine that renders its rate calendar with JavaScript and gates it behind a bot check will defeat a plain HTTP scraper every time. When a request-based fetch is not enough, a headless browser driven by Playwright can render the page, wait for the rates to load, and back off politely when a challenge appears. This guide covers that fallback path within Competitor Rate Scraping Pipelines, part of the Data Ingestion & OTA API Integration Workflows pillar. The goal is reliable extraction that respects target sites and commercial agreements — not evasion for its own sake.

Prerequisites

Step 1 — Launch a realistic browser context

Anti-bot systems fingerprint headless defaults. Launch a context with a real viewport, locale, and user agent so the page renders the way a genuine visitor’s would, and reuse the context across requests to the same domain to keep session cost down.

python 
from __future__ import annotations

import asyncio
import logging
from playwright.async_api import async_playwright, Browser, BrowserContext

logger = logging.getLogger("scrape.browser")


async def new_context(browser: Browser) -> BrowserContext:
    return await browser.new_context(
        viewport={"width": 1366, "height": 900},
        locale="en-US",
        user_agent=("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                    "AppleWebKit/537.36 (KHTML, like Gecko) "
                    "Chrome/120.0 Safari/537.36"),
    )

Step 2 — Render, wait for real content, and detect challenges

Navigate, then wait for the specific element that holds the rate rather than a fixed sleep — waiting on content is both faster and more robust. If a challenge page appears instead, detect it and signal a back-off rather than scraping garbage.

python 
class ChallengeDetected(Exception):
    pass


async def fetch_rate_html(context: BrowserContext, url: str,
                          rate_selector: str) -> str:
    page = await context.new_page()
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=20_000)
        title = (await page.title()).lower()
        if any(sig in title for sig in ("just a moment", "verify", "are you human")):
            raise ChallengeDetected(url)
        await page.wait_for_selector(rate_selector, timeout=15_000)
        return await page.inner_html(rate_selector)
    finally:
        await page.close()

Step 3 — Back off on challenges instead of hammering

A detected challenge is a signal to slow down, not to retry immediately. Apply exponential backoff and, after repeated challenges, route the target to a manual-review queue so a human can decide whether continued access is appropriate.

python 
async def fetch_with_backoff(context: BrowserContext, url: str, selector: str,
                             max_attempts: int = 3) -> str | None:
    for attempt in range(1, max_attempts + 1):
        try:
            return await fetch_rate_html(context, url, selector)
        except ChallengeDetected:
            wait = 2 ** attempt
            logger.warning("challenge on %s (attempt %d); backing off %ds",
                           url, attempt, wait)
            await asyncio.sleep(wait)
    logger.error("giving up on %s after %d challenges; routing to review",
                 url, max_attempts)
    return None

Backing off and escalating to review, rather than retrying aggressively, keeps the pipeline within acceptable-use norms and avoids the IP reputation damage that makes every future request harder.

Verification and testing

Playwright ships a test server pattern; here a local HTML fixture confirms the selector-wait and challenge detection without hitting a real site.

python 
import asyncio


async def _run(html_path: str, selector: str) -> str:
    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        context = await new_context(browser)
        result = await fetch_rate_html(context, f"file://{html_path}", selector)
        await browser.close()
        return result


def test_extracts_rate_block(tmp_path) -> None:
    fixture = tmp_path / "rates.html"
    fixture.write_text('<div id="rate">180</div>')
    html = asyncio.run(_run(str(fixture), "#rate"))
    assert "180" in html

Common pitfalls and edge cases

  • Fixed sleeps. sleep(5) is both slow and flaky; wait on the rate selector instead.
  • Leaking pages. Not closing pages exhausts browser memory over a long run; always close in a finally.
  • Ignoring challenges. Scraping a challenge page yields junk that poisons normalization; detect and back off.
  • One context per request. Recreating the browser context per URL is expensive; reuse per domain.
  • Terms of service. Headless rendering does not override a site’s terms; keep targets within agreed commercial boundaries and honor review escalations.