scrape-diagnose

Guide: diagnose scraper 403 responses and empty HTML · Python Requests: diagnose 403 or empty HTML when the browser works

Scraper gets 403 although the page works in a browser? Returns 200 with empty HTML? scrape-diagnose determines whether request headers, JavaScript rendering, or a proxy is actually needed—and reports when none of them is.

It is an unofficial, open-source CLI for diagnosing public-page fetches. It catches ordinary HTTP failures and deceptive “successes” such as a 200 OK challenge page, an empty JavaScript shell, or a response missing the content you actually need.

It tests the URL directly first. On 403, it performs one free browser-header comparison before suggesting a provider. If either direct profile works, it recommends no paid proxy. An optional live mode can preflight and test the least expensive plausible ScraperAPI configuration under a cumulative preflight-credit ceiling.

The project is not affiliated with or endorsed by ScraperAPI.

Quick start

Node.js 20 or newer is required. A direct diagnosis needs no account or API key:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- https://example.com/ --expect-text "Example Domain"

The command starts with one direct, read-only GET; only a 403 adds one browser-header comparison. Each profile follows at most five public redirects. The classifier reads at most 1 MiB of response bytes and emits a redacted Markdown report. Target paths, query strings, fragments, response bodies, and marker values are never copied into the report.

Use --json for deterministic machine output:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- https://example.com/ --json

What it detects

• 401/402 authentication and paywall boundaries
• 403 blocks and 429 rate limits
• common bot-challenge and CAPTCHA shells
• false-200 pages with almost no visible content
• missing expected text
• empty, binary, server-error, timeout, and redirect-loop responses
• unsafe local, private, link-local, and metadata-service targets

The Markdown report includes every sanitized attempt, its HTTP result, configuration, estimated/observed credits, and stable finding codes. Live runs print sanitized phase progress to stderr while keeping stdout/JSON deterministic.

Fixture proof: 403 to minimum passing mode

This abbreviated transcript is generated from deterministic fixtures; it makes no live provider call and spends no credits:

$ scrape-diagnose "$TARGET_URL"
Decision: FAIL
Attempts:
  direct                  403  FAIL
  direct-browser-headers  403  FAIL
Finding: HTTP403

$ scrape-diagnose "$TARGET_URL" --live --max-total-credits 25
scrape-diagnose: cost preflight 1/3...
scrape-diagnose: paid probe 1/3 (estimated 1 credits)...   -> 403
scrape-diagnose: cost preflight 2/3...
scrape-diagnose: paid probe 2/3 (estimated 10 credits)...  -> 200 PASS
Minimum passing configuration: premium=true, render=false
Committed estimate: 11 / 25 credits

See Troubleshooting blocked and incomplete scrapes for 403 but browser works, 200 but empty HTML, and render vs premium decision paths.

A complete direct response returns exit code 0. A diagnosed failure returns 1; invalid or unsafe configuration returns 2.

Optional budgeted fallback

Live tests are opt-in. Put the key in SCRAPERAPI_KEY using your shell’s secure secret mechanism; this tool deliberately rejects --api-key and never auto-loads .env files.

Affiliate disclosure: If the diagnosis shows that you need a managed fallback and you do not already have an account, this ScraperAPI pricing link is an affiliate link. The maintainer may earn a commission if you sign up through it and later purchase service. The coarse sd-readme label identifies this README placement; it contains no user or session identifier. The link is never printed by the CLI or Action, opened automatically, or used to change a diagnostic result.

Check the account endpoint without exposing account fields:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- account --json

Ask ScraperAPI’s official preflight endpoint for the exact cost without scraping the target:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- cost https://example.com/ --render --json

Diagnose directly, then allow a bounded fallback ladder:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- https://example.com/ \
  --expect-text "Required marker" \
  --live \
  --max-total-credits 25 \
  --max-paid-probes 3

Every paid attempt gets an exact /account/urlcost preflight first. A request is not sent if its estimate would exceed the remaining cumulative budget. If the observed header differs from the estimate, the report warns and uses the observed value to reduce the remaining ladder budget. Provider-side redirect following is disabled, and the ladder stops on the first complete response. Rendering, premium proxies, and a country code can also be selected explicitly.

ScraperAPI documents request costs and the observed sa-credit-cost response header in its official credit guide. Provider pricing and plan limits can change; the tool uses the live preflight instead of hard-coding a price claim.

GitHub Action

The repository is also a dependency-free JavaScript action:

Use the Action only for development or testing of software associated with the repository that contains the workflow. It is not a generic uptime monitor, scheduled scraper, or proxy workload; those uses fall outside this Action’s supported scope and may conflict with GitHub-hosted-runner terms.

name: public-page-check
on:
  workflow_dispatch:

permissions: {}

jobs:
  diagnose:
    runs-on: ubuntu-latest
    steps:
      - uses: sanjayamaharjancodes/scrape-diagnose@abe305a38cee37a0287d5bb9d1097964759b8c72
        with:
          target-url: https://example.com/
          expect-text: Example Domain
          live: "false"

For live fallback, pass api-key: ${{ secrets.SCRAPERAPI_KEY }} and set live: "true". The Node 24 action masks the key before doing any work, does not read GITHUB_TOKEN, works with permissions: {}, writes only the configured report inside GITHUB_WORKSPACE, and makes no comments, commits, or uploads. If a required secret is unavailable in a fork or Dependabot context, it emits SKIPPED without a request by default.

See the complete Action guide for budget and failure-policy controls. Avoid pull_request_target for workflows that handle secrets.

Privacy and safety

• No telemetry, analytics, cookies, runtime dependencies, install hooks, browser downloads, or response-body storage.
• One target per run; at most two direct header profiles and three paid probes; the tool itself performs no hidden retries.
• No login, paywall, form submission, POST, extraction, crawling, interactive CAPTCHA solving, or local browser automation. Selecting provider rendering may cause ScraperAPI to use its remote browser infrastructure under its own service terms.
• Credentials come only from the process environment or Action secret input. Generated snippets reference environment variables rather than embedding values.
• JSON and Markdown include only the target scheme and hostname, coarse response metadata, findings, and integer credit counts.
• Tests use injected fetch fixtures and spend zero provider credits.

Only test pages you are authorized to access. A successful request does not grant permission to collect or reuse content. Read the privacy and safety design before using live mode in CI.

Development

npm install --ignore-scripts
npm test
npm run test:coverage
npm pack --dry-run

The public API is runDoctor(options) from src/index.mjs; it accepts injected fetchImpl and lookupImpl functions for deterministic tests. The report contract is documented in schemas/report.schema.json.

Scope and license

This is a focused diagnostic, not a scraping framework or proxy rotator. If it proves that direct fetching already works, that conclusion is a feature.

MIT licensed. See SECURITY.md for vulnerability reporting.