ScrapeGraphAI

ScrapeGraphAI is the most popular open-source LLM-powered scraping library plus a hosted API of the same name. You give it a URL or a search query and a JSON schema describing the data you want; it spins up a graph of LLM-driven steps — fetch, render if needed, prune the DOM, extract, validate — and returns clean structured data. Because the extraction logic lives in prompts and pydantic schemas rather than CSS selectors, scrapers do not need to be rewritten when sites redesign, and a single script can scrape dozens of different domains with the same code. The hosted API adds managed proxies, headless browsers, rate limits, retries, and a few high-level endpoints (SmartScraper, SearchScraper, MarkdownScraper) that wrap common patterns. ScrapeGraphAI also publishes an official MCP server so Claude Desktop, Cursor, and other MCP-aware clients can extract structured web data without writing scraping code. The library is MIT licensed with 16k+ GitHub stars, and the hosted plans are usage-based per request with a free tier. Best fit for AI agents that need structured web data, RAG pipelines ingesting product catalogs or news, and ops teams replacing brittle Selenium scripts with self-healing LLM scrapers.