Skip to main content

Authenticated Web Scraping

Authenticated Web Scraping with Reusable Browser Profiles

Return to an authorized account workflow with a saved browser profile. Inspect the login state, complete any required sign-in and stop the runtime when finished.

01 / OVERVIEW

What is Authenticated Web Scraping?

Authenticated web scraping retrieves content that requires a valid account session. AdsCrawl saved cloud browser profiles support persistent browser session workflows by retaining browser context between runs, while the target website remains responsible for whether a login is accepted.

Quick answers

How do I keep browser sessions alive across multiple cloud workers?

Keep session ownership outside an individual worker, and separate a running session from a saved profile. For AdsCrawl Cloud Browser, store the profile ID so later jobs can restart the same saved environment. This preserves supported profile state, not an indefinitely running browser or permanent login. Temporary Remote CDP sessions have a separate lifetime and do not automatically become saved profiles.

  1. In your job system, map each account to a profile ID and use a lease or lock so only one worker controls that profile at a time.
  2. Before continuing, query runtime status and verify the website's login state; API-key starts need an explicit valid proxy.
  3. After the job, stop the browser and confirm stopped before handing off ownership. Reauthenticate when the website requires it.

02 / USE CASES

Where it fits

01

Resume an account workflow

Reuse a saved profile for an account you are authorized to access. Confirm a signed-in page is visible before collecting content.

02

Handle interactive sign-in

Open the authenticated live viewer to inspect login state and complete required manual steps. A saved profile does not bypass MFA or session expiry.

03

Keep account contexts separate

Use separate profiles for workflows that require separate cookies and proxy settings. Avoid copying credentials or session data into public code examples.

03 / QUICKSTART

Start with one complete workflow

  1. Create or launch a saved browser profile with a valid explicit proxy; the example demonstrates the launch and stop lifecycle.
  2. Open runtime.connectUrl while signed in to AdsCrawl and complete the target site's authorized login flow.
  3. Verify the target account and content before continuing your collection workflow; launch alone does not extract data.
  4. Stop the runtime, confirm it is stopped, and keep the profile id for a later start instead of creating another profile.

Requires Bash, cURL and jq. Set the ADSCRAWL_API_KEY environment variable on your server first.

Full request and response reference →
cURL
# Replace the proxy address with your own proxy.
curl --fail-with-body -sS --max-time 200 \
  -X POST "https://api.adscrawl.net/cloud-browsers/launch" \
  -H "x-api-key: $ADSCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "proxy": { "server": "http://proxy.example.com:8080" },
    "tabs": ["https://www.adscrawl.net"]
  }'

# Open runtime.connectUrl from the response.
# When finished, replace <browser-id> with the returned id.
curl --fail-with-body -sS -X POST \
  "https://api.adscrawl.net/cloud-browsers/<browser-id>/stop" \
  -H "x-api-key: $ADSCRAWL_API_KEY"

# Requires jq. Replace <browser-id> with the same id used above.
# A successful stop request can still return status "stopping".
BROWSER_ID="<browser-id>"
stopped=false
for ((attempt=0; attempt<60; attempt++)); do
  detail=$(curl --fail-with-body -sS --max-time 30 \
    "https://api.adscrawl.net/cloud-browsers/$BROWSER_ID" \
    -H "x-api-key: $ADSCRAWL_API_KEY") || exit 1
  status=$(printf '%s' "$detail" | jq -er '.runtime.status') || exit 1
  if [ "$status" = "stopped" ]; then
    stopped=true
    break
  fi
  sleep 2
done
if [ "$stopped" != true ]; then
  printf '%s\n' 'Shutdown is not confirmed. Check this profile; do not launch a replacement.' >&2
  exit 1
fi

# Closing the viewer does not stop billing.
# If launch fails, inspect its returned id; do not repeat launch.

04 / LIMITS & BILLING

Know the boundaries

Sign up with 500 one-time credits. Hobby is $9/month, Starter $49/month and Pro $199/month. Launching a persistent Cloud Browser requires a paid plan.

Compare plans and allowances →
  • Saved browser context does not guarantee a permanent login. Sites can expire sessions or require another challenge.
  • The example manages a browser profile; you must implement the collection step using a supported workflow for your target.
  • Cloud Browser launch requires a paid plan and an explicit valid proxy for API-key requests.
  • A Cloud Browser profile is not automatically imported into a temporary Remote CDP session. Closing the viewer does not stop billing.

Keep API keys on the server. For HTTP 402 check your balance; for 429 reduce concurrency and retry with backoff.

05 / FAQ

Common questions

Does this bypass a login or MFA?

No. You need an authorized account and must satisfy the site's login requirements.

Is persistent context the same as a running browser?

No. A saved profile and a live runtime have different lifecycles and limits. Stop the runtime when finished.

What if the saved login has expired?

Open the profile, sign in again through the required flow and verify the account state before continuing.