AI Development
AdsCrawl for LlamaIndex
Prepare web evidence for a LlamaIndex RAG pipeline with AdsCrawl’s Python SDK. Fetch rendered Markdown, create a Document with source metadata, then split it into nodes before choosing your embedding model and index.
What you can do
Rendered page documents
Load Markdown into LlamaIndex Document objects with a stable URL-based document ID.
Source-preserving chunks
Use SentenceSplitter to create overlapping chunks whose metadata retains the page URL.
RAG ingestion input
Pass the resulting nodes to an index or ingestion pipeline configured for your application.
Before you start
- An AdsCrawl account with an API key and available credits.
- Python 3.10 or newer, preferably in a virtual environment.
SETUP METHOD 01
Prepare documents and nodes
Use the Python SDK for page retrieval and LlamaIndex Core for document conversion and splitting.
Install dependencies
Run this in your application directory.
pip install adscrawl llama-index-coreConfigure your API key
Set the variables in your server environment or local shell. Replace the placeholders with your own keys; keep them out of browser bundles and source control.
export ADSCRAWL_API_KEY="your_adscrawl_api_key"Save web_documents.py
The script creates one document, preserves source metadata and prints each text chunk. No embedding or LLM API is required for this step.
import os from adscrawl import AdsCrawl from llama_index.core import Document from llama_index.core.node_parser import SentenceSplitter client = AdsCrawl(api_key=os.environ["ADSCRAWL_API_KEY"]) url = "https://example.com" markdown = client.markdown({"url": url, "waitUntil": "domcontentloaded"}) if not markdown.strip(): raise ValueError("The page returned no text") document = Document(text=markdown, metadata={"source": url}, id_=url) splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64) nodes = splitter.get_nodes_from_documents([document]) for node in nodes: print(node.metadata["source"]) print(node.get_content())Run and connect the next step
Expect each node to carry https://example.com as source and contain the page text. For retrieval, configure an embedding model and vector store before indexing these nodes. A query engine also needs your chosen answer-generation model.
python web_documents.py
Usage & limits
- This example prepares RAG inputs; it does not create embeddings, persist an index or answer questions. Chunk size is measured in tokens. Model and storage usage are separate from AdsCrawl credits.
- Each page request uses AdsCrawl credits. These examples fetch one public URL; JavaScript-rendered pages may need a different waitUntil setting or additional browser interactions.
- Keep the source URL with the content. For longer pages, choose a chunking or context-budget strategy before passing text to a model.