Skip to main content

Web Scraping API

Web Scraping API for Dynamic Websites

Render JavaScript pages in a browser and extract a defined set of fields. Give your pipeline structured results instead of another page to parse.

01 / OVERVIEW

What is Web Scraping API?

AdsCrawl Web Scraping API extracts specified fields from browser-rendered websites through POST /spa-extract. Define fields using DOM selectors or supported network response sources. When you need rendered markup or readable page content instead, use POST /html with the appropriate output mode.

Quick answers

Which scraping API should I use for JavaScript-heavy sites?

Use browser-rendered field extraction when the values you need appear after JavaScript runs. With AdsCrawl, POST /spa-extract can wait for a selector and return named DOM or supported network fields. For multi-step navigation, use Remote CDP. Judge the result by complete, valid fields on your target pages, not just an HTTP 200 response.

  1. Identify the required fields and a readiness selector; set waitFor.selector and define fields in the extraction request.
  2. Validate data, missingFields and field types before saving a record. A missing required field can return HTTP 422.
  3. Test a representative batch, recording retries and cost per valid record; put scheduling and concurrency control in your application.

SEE THE RESULT

From a JavaScript page to two usable fields.

Quotes to Scrape creates its quote elements with JavaScript. This recorded request reads the first quote and its author from the rendered DOM.

Captured on

returned fields
2
missing fields
0
HTTP
200
Source page captureVisit source
Quotes to Scrape JavaScript page captured by the AdsCrawl Screenshot API, showing rendered quotes and authors
Scroll to inspect the full captureOriginal PNG
JSON
{
  "data": {
    "author": "Albert Einstein",
    "quote": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”"
  },
  "missingFields": [],
  "mode": "extract",
  "page": {
    "url": "https://quotes.toscrape.com/js/",
    "title": "Quotes to Scrape"
  }
}

What to check

Compare data.quote and data.author with the first quote on the source page. An empty missingFields list means both requested fields were found in this run.

Try your own page →

A recorded public-page example. Results reflect the page at capture time and can change with the target website, loading conditions and request settings.

VALIDATE ONE COMPLETE TASK

Get verifiable JSON from a dynamic page

01

Wait for the target element

Call POST /spa-extract for https://quotes.toscrape.com/js/. Set waitFor.selector to .quote .text so field extraction waits for rendered content.

02

Define and validate fields

Use .quote .text for quote and .quote .author for author, both required with parse: string. Check a non-empty data.quote, data.author equal to Albert Einstein and an empty missingFields list.

03

Exercise the failure branch

Change a required field selector to a missing element in your test workflow and check that the result is rejected or handled as missing data. HTTP 200 alone does not establish that your data requirements were met.

The reference was captured on 2026-11-04 in one AdsCrawl run. It does not establish success rates, performance or cost across other sites. Download the request and result, then repeat the checks on your target page.

02 / USE CASES

Where it fits

01

Extract precise fields

Define CSS selectors for the names, headings or values your application needs. Give each field a stable name so downstream code does not depend on the complete page markup.

02

Read dynamic data

Use supported network fields when the page retrieves JSON while loading. Consult the API reference for matching and extraction rules.

03

Check completeness

Inspect missingFields alongside returned data. Detect changed selectors and incomplete loads before storing results in a monitoring or research pipeline.

03 / QUICKSTART

Start with one complete workflow

  1. Create an API key and store it in a server-side environment variable.
  2. Inspect your target page and identify a selector for each required field.
  3. Send POST /spa-extract with mode set to extract and the fields object.
  4. Check the returned values and missingFields before using the result.

Requires Bash, cURL and jq. Set the ADSCRAWL_API_KEY environment variable on your server first.

Full request and response reference →
cURL
curl --fail-with-body -sS \
  -X POST "https://api.adscrawl.net/spa-extract" \
  -H "x-api-key: $ADSCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "url": "https://www.adscrawl.net",
    "mode": "extract",
    "fields": {
      "title": {
        "source": "dom",
        "selector": "h1",
        "parse": "string"
      }
    }
  }' \
  --output extraction.json || exit 1

# Requires jq. Reject incomplete data before using the result.
jq -e '
  .mode == "extract" and
  (.missingFields | type == "array" and length == 0) and
  (.data.title | type == "string" and length > 0)
' extraction.json >/dev/null || {
  printf '%s\n' 'Extraction is incomplete. Check missingFields, selectors and wait settings.' >&2
  exit 1
}
# Read .data.title in your application; keep extracted content out of logs.

WORKFLOW GUIDE

Choose an output, then validate the fields

Use the endpoint that matches the result you need. An HTTP 200 response alone does not establish that all requested fields were found.

Illustrative successful response

An authored fixture following the extract response contract, not a recorded request result. Read the requested field from data.title.

{
  "mode": "extract",
  "page": {
    "url": "https://example.com/dashboard",
    "title": "Example Dashboard"
  },
  "data": {
    "title": "Example Dashboard"
  },
  "missingFields": []
}

Illustrative missing-field response

This optional-field fixture must fail the Quickstart's jq check. Marking the field required instead can return HTTP 422 with SPA_REQUIRED_FIELDS_MISSING.

{
  "mode": "extract",
  "page": {
    "url": "https://example.com/dashboard",
    "title": "Example Dashboard"
  },
  "data": {},
  "missingFields": [
    "title"
  ]
}

Check your result

  • The Quickstart rejects missing or empty title values and a non-empty missingFields list before the result is used.
  • For an unmatched selector, check the target markup and supported wait settings. Do not substitute page.title for the requested field.
  • Handle non-2xx responses separately; --fail-with-body exposes HTTP errors. Keep API keys, request payloads and extracted content out of public logs.

04 / LIMITS & BILLING

Know the boundaries

Sign up with 500 one-time credits. Hobby is $9/month, Starter $49/month and Pro $199/month. Launching a persistent Cloud Browser requires a paid plan.

Compare plans and allowances →
  • A target-site redesign can invalidate selectors. Validate the response rather than assuming every request returns every field.
  • Page readiness depends on the target site. Choose wait conditions within the endpoint's supported limits.
  • Use Remote CDP for multi-step navigation, clicks and forms. Field extraction is not a replacement for a browser script.
  • Requests use account credits and are subject to concurrency limits. Check your plan before scheduling a batch.

Keep API keys on the server. For HTTP 402 check your balance; for 429 reduce concurrency and retry with backoff.

05 / FAQ

Common questions

Should I use /html or /spa-extract?

Use /html for rendered HTML or readable article content. Use /spa-extract when you need a specified set of DOM or supported network fields.

What happens when a selector does not match?

Inspect missingFields and the returned data. Check the current selector and wait settings, then update your field definition if the target site's structure has changed.

Does a successful response mean my data is correct?

No. Validate required fields, types and any business rules before storing the result. Rendering and extraction do not establish that the source's claims are accurate.