Browser Orchestrator

Captcha After Successful Fetches

Workers

Sessions

Agent brief Authentication Session lifecycle Fetch endpoint Result classification Pool guidance Examples

Agent brief

Browser Orchestrator API implementation brief

Base URL:
- Use the Scrappa browser-orchestrator base URL supplied by the environment or operator.
- Send every protected request with Authorization: Bearer ORCHESTRATOR_TOKEN.
- Always use an existing browser session for target traffic. Do not make a direct fetch fallback for protected or session-sensitive targets.

Core workflow:
1. Acquire an existing ready browser from the shared pool with POST /sessions/acquire.
2. If acquire returns 202 with status "starting", wait and call acquire again; do not fetch until a session is leased.
3. Fetch target URLs with POST /sessions/{id}/fetch.
4. Read result.ok, status, html/bodyText, hints.reasons, and hints.signals.
5. Treat hints.usable/classification as compatibility summaries only; Scrappa decides whether a mixed page is parseable.
6. Release leased sessions with POST /sessions/{id}/release when your caller is done.

Session acquire request:
POST /sessions/acquire
{
  "site": "trustpilot",
  "proxy_url": "http://user:pass@proxy.example:8080",
  "proxy_source": "mobile-proxies"
}
- The orchestrator verifies ready database sessions against the assigned worker before leasing.
- If a ready database session is missing on the worker, it is marked failed and skipped.
- If the shared pool already has a starting session for the requested proxy source, acquire returns that session with 202 instead of creating another.
- If the same proxy recently failed to start, acquire returns that failed session with 422 and last_error.
- New cold starts run a short proxy connectivity preflight before the worker starts a browser.
- If no verified ready session exists, acquire returns 202 with status "starting"; retry acquire before fetching.

Fetch request:
POST /sessions/{id}/fetch
{
  "url": "https://www.trustpilot.com/review/amazon.com",
  "timeout_ms": 30000,
  "headers": {
    "x-nextjs-data": "1"
  },
  "referrer": "https://www.trustpilot.com/",
  "bootstrap_url": "https://www.trustpilot.com/",
  "bootstrap_wait_ms": 500,
  "fetch_mode": "navigate"
}

Fetch modes:
- navigate: page.goto target URL and return the browser document.
- in_page: visit bootstrap_url first, then run browser-side fetch(target, { credentials: "include" }).
- navigate_argonaut: Realestate.com.au HTML profile that returns as soon as window.ArgonautExchange is present.
- Use in_page for middleware, JSON, or Next.js endpoints that require first-party cookies, referrer, or browser storage.
- Realestate.com.au URLs select navigate_argonaut automatically when fetch_mode is omitted.
- If fetch_mode is in_page and bootstrap_url is omitted, the worker bootstraps the target origin automatically.

Headers:
- Use headers for target fetch needs such as x-nextjs-data.
- Do not send Host, Connection, Content-Length, or Proxy-Authorization as custom fetch headers.
- In fetch_mode "in_page", custom headers are scoped to the browser-side target fetch and are not applied to the bootstrap navigation.

Fetch response signals:
- result.ok false means the worker caught a fetch/navigation failure. Inspect result.failure.
- result.html and result.bodyText are returned even when the page also has WAF/CAPTCHA markers.
- result.hints.usable and classification are backward-compatible summaries, not final parsing decisions.
- result.hints.signals reports objective page facts such as html_bytes, body_text_chars, has_waf_marker, has_captcha_marker, has_block_marker, has_challenge_marker, and has_mixed_content.
- hints.reasons contains machine-readable reasons such as no_usable_serp, google_sorry_url, unusual_traffic_text, aws_waf_marker, human_verification_text, http_405, captcha_marker, fetch_error, worker_fetch_error.

Structured fetch failures:
- Fetch failures return JSON with result.ok false; callers should not rely on HTTP 5xx for browser/runtime failures.
- failure.worker_id
- failure.session_id
- failure.target_url
- failure.upstream_status
- failure.final_url
- failure.page_title
- failure.body_preview
- failure.error

Shared browser pool guidance:
- The site field is compatibility metadata only; it does not select a warm pool or reserve browser history for one target.
- All ready sessions are eligible for all traffic, constrained only by proxy_source when a caller asks for one.
- Google Search and generic HTML pages should use fetch_mode "navigate" and decide parseability from result.ok, html/bodyText, hints.reasons, and hints.signals.
- JSON, middleware, and Next.js data endpoints should use fetch_mode "in_page" with a same-origin bootstrap_url. If bootstrap_url is omitted, the worker uses the target origin automatically.
- Passive WAF markers can appear on pages whose returned content is still parseable.
- WAF/challenge pages: mixed pages can contain useful text and active challenge markup. The orchestrator reports both; Scrappa owns the parse/use decision.

Do not treat a technically returned HTML document as automatically parseable. The orchestrator reports evidence; Scrappa decides how to use it.

Authentication

All Scrappa-facing API routes except health and the dashboard shell require the orchestrator token.

Authorization: Bearer ORCHESTRATOR_TOKEN

Session lifecycle

Acquire a ready session from the shared browser pool.
If acquire returns 202 starting, wait and retry acquire before fetching.
Fetch through that session.
Use response hints to decide whether the result is parseable.
Release the session when done.

POST /sessions/acquire
{
  "site": "booking",
  "proxy_url": "socks5://user:[email protected]:1080",
  "proxy_source": "mobile-proxies"
}

POST /sessions/{id}/release
{}

Fetch endpoint

Use navigate for document pages. Use in_page after a bootstrap visit for middleware, JSON, and protected Next.js data URLs.

POST /sessions/{id}/fetch
{
  "url": "https://example.com/protected/page",
  "timeout_ms": 30000,
  "headers": {
    "accept": "text/html,application/xhtml+xml"
  },
  "referrer": "https://example.com/",
  "bootstrap_url": "https://example.com/",
  "bootstrap_wait_ms": 500,
  "fetch_mode": "navigate"
}

Result signals

The worker returns content plus observed page-state markers. The scraper decides parseability.

if (!result.ok) {
  return {
    transportOk: false,
    failure: result.failure ?? null
  };
}

return {
  status: result.status,
  finalUrl: result.finalUrl,
  html: result.html,
  bodyText: result.bodyText,
  reasons: result.hints.reasons,
  signals: result.hints.signals,
  compatibilitySummary: {
    usable: result.hints.usable,
    classification: result.hints.classification
  }
};

Pool guidance

The site key is request metadata only. It does not select a dedicated browser pool.

Shared browser pool:
- every ready browser can serve every supported traffic type
- proxy_source can still constrain leasing to the requested proxy pool

Google Search and generic HTML:
- fetch_mode: "navigate"
- parseability is scraper-specific; classification is only a compatibility summary

JSON or middleware endpoint:
- fetch_mode: "in_page"
- bootstrap_url: a same-origin page that establishes cookies/storage
- custom headers apply only to the target browser-side fetch

AWS WAF / challenge-heavy sites:
- returned HTML can contain both real content and active challenge markers
- use hints.signals.has_mixed_content, reasons, status, html, and bodyText as evidence
- Scrappa decides whether partial/mixed content is useful for a specific parser

Examples

Google search:
{
  "site": "google_search",
  "fetch": {
    "url": "https://www.google.com/search?q=site%3Aexample.com",
    "timeout_ms": 30000,
    "fetch_mode": "navigate"
  }
}

Generic document page:
{
  "site": "example",
  "fetch": {
    "url": "https://example.com/articles/page",
    "timeout_ms": 30000,
    "fetch_mode": "navigate"
  }
}

Same-origin JSON endpoint:
{
  "site": "example",
  "fetch": {
    "url": "https://example.com/api/listings",
    "timeout_ms": 30000,
    "headers": {
      "accept": "application/json"
    },
    "bootstrap_url": "https://example.com/",
    "fetch_mode": "in_page"
  }
}

Trustpilot review page:
{
  "site": "trustpilot",
  "fetch": {
    "url": "https://www.trustpilot.com/review/amazon.com",
    "timeout_ms": 30000,
    "fetch_mode": "navigate"
  }
}

Trustpilot Next.js data request:
{
  "site": "trustpilot",
  "fetch": {
    "url": "https://www.trustpilot.com/_next/data/.../review/amazon.com.json",
    "timeout_ms": 30000,
    "headers": {
      "x-nextjs-data": "1"
    },
    "referrer": "https://www.trustpilot.com/review/amazon.com",
    "bootstrap_url": "https://www.trustpilot.com/review/amazon.com",
    "fetch_mode": "in_page"
  }
}

Booking search results:
{
  "site": "booking",
  "fetch": {
    "url": "https://www.booking.com/searchresults.html?ss=Paris",
    "timeout_ms": 45000,
    "bootstrap_url": "https://www.booking.com/",
    "fetch_mode": "navigate"
  }
}