D4Vinci/Scrapling is an open-source project on GitHub, mainly written in Python. 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy It currently holds 86,006 stars and 8,810 forks with 6 open issues, and was last pushed on 2026-10-06 (repository created 2024-10-13).
Project Overview
Git Homed tracks it on the Today's Trending board.
GitHub Repository Details
Repository D4Vinci/Scrapling · default branch main · size 11421 KB · watchers 318 · source: GitHub REST API and repository README
Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume, automatic proxy rotation, and a crawl speed that adapts to how fast each website responds and backs off when it starts blocking you - all in a few lines of Python. One library, zero compromises.
Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar!
products = p.css('.product', auto_save=True) # Scrape data that survives website design changes!
products = p.css('.product', adaptive=True) # Later, if the website structure changes, pass adaptive=True to find them!
Or scale up to full crawls
from scrapling.spiders import Spider, Response
class MySpider(Spider):
name = "demo"
start_urls = ["https://example.com/"]
async def parse(self, response: Response):
for item in response.css('.product'):
yield {"title": item.css('h2::text').get()}
MySpider().start()
Platinum Sponsors
NodeMaven - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. Use code SCRAPLING35 for 35% discount.
Proxidize provides mobile and residential proxies for scraping, browser automation, SEO monitoring, AI agents, and data collection. Use code scrapling20 for 20% off.
Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for Akamai, DataDome, Kasada, and Incapsula. Simple API calls, no browser automation required.
Hey, we built BirdProxies because proxies shouldn't be complicated or overpriced. Fast residential and ISP proxies in 195+ locations, fair pricing, and real support. Try our FlappyBird game on the landing page for free data!
Evomi: residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass.
Scraper API for hassle-free results. MCP and N8N integrations are available.
TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets. Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off.
Close your laptop. Your scrapers keep running. PetroSky VPS - cloud servers built for nonstop automation. Windows and Linux machines with full control. From €6.99/mo.
Swiftproxy provides scalable residential proxies with 80M+ IPs across 195+ countries, delivering fast, reliable connections, automatic rotation, and strong anti-block performance. Free trial available.
Byteful supplies ethical residential, mobile, ISP, and datacenter proxies, plus scraping and AI automation tools. Use code SCRAPLING10: 10% off residential proxies.
ProxyLane: Reduce blocks and CAPTCHAs by 90%. 28M Premium Residential IPs for Heavy Web Scraping. 195 countries. City/ISP/ZIP targeting. Sticky/rotating sessions. Non-expiring traffic from $2/GB. Code SCRAPEANDTAKE: 25% off.
🔄 Multi-Session Support: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
💾 Pause & Resume: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
📡 Streaming Mode: Stream scraped items as they arrive via async for item in spider.stream() with real-time stats - ideal for UI, pipelines, and long-running crawls.
🛡️ Blocked Request Detection: Automatic detection and retry of blocked requests with customizable logic.
🚦 AutoThrottle: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what Retry-After asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
🤖 Robots.txt Compliance: Optional robots_txt_obey flag that respects Disallow, Crawl-delay, and Request-rate directives with per-domain caching.
🧪 Development Mode: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your parse() logic without re-hitting the target servers.
🧩 Ready-made Spider Templates: Skip the boilerplate with CrawlSpider for rule-based link following, SitemapSpider for sitemap/robots.txt-driven crawls, XMLFeedSpider/CSVFeedSpider for iterating XML/RSS and CSV feeds, and ShopifySpider to pull every product out of any Shopify store through its JSON API, one item per variant.
🔗 Link Extraction: A standalone LinkExtractor primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
📦 Built-in Export: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with result.items.to_json(), to_jsonl(), to_csv(), and to_xml().
Advanced Websites Fetching with Session Support
HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
Dynamic Loading: Fetch dynamic websites with full browser automation through the DynamicFetcher class supporting Playwright's Chromium and Google's Chrome.
Anti-bot Bypass: Advanced stealth capabilities with StealthyFetcher and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
Session Management: Persistent session support with FetcherSession, StealthySession, and DynamicSession classes for cookie and state management across requests.
Proxy Rotation: Built-in ProxyRotator with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
Domain & Ad Blocking: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
DNS Leak Prevention: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
Remote Browsers: Instead of launching a browser locally, connect to one that's already running through CDP with cdp_url, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with executable_path.
Background API Capture: Pass a URL pattern to capture_xhr, and all matching XHR/fetch responses the page makes while loading are collected for you as Response objects in response.captured_xhr - grab a site's API data without reverse-engineering the requests yourself.
Async Support: Complete async support across all fetchers and dedicated async session classes.
Adaptive Scraping
🔄 Smart Element Tracking: Relocate elements after website changes using intelligent similarity algorithms.
🎯 Smart Flexible Selection: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
🔍 Find Similar Elements: Automatically locate elements similar to found elements.
AI Features
🤖 MCP Server: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method), browser fetches, and stealth fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. (demo video)
🧠 Agent Skill: A ready-to-install Agent Skill that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
📚 RAG-ready Markdown: Turn any page into clean, sanitized, LLM-ready Markdown with one line (page.markdown()), or crawl a whole website into a Markdown corpus with the SiteToMarkdownSpider template, all without an LLM in the loop. (docs)
High-Performance & battle-tested Architecture
🚀 Lightning Fast: Optimized performance outperforming most Python scraping libraries.
🔋 Memory Efficient: Optimized data structures and lazy loading for a minimal memory footprint.
⚡ Fast JSON Serialization: 10x faster than the standard library.
🏗️ Battle tested: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.
Developer/Web Scraper Friendly Experience
🎯 Interactive Web Scraping Shell: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
🚀 Use it directly from the Terminal: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
🛠️ Rich Navigation API: Advanced DOM traversal with parent, sibling, and child navigation methods.
🧬 Enhanced Text Processing: Built-in regex, cleaning methods, and optimized string operations.
📝 Auto Selector Generation: Generate robust CSS/XPath selectors for any element.
🔌 Familiar API: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
🤝 Drop-in Scrapy Integration: Already invested in Scrapy? Decorate any callback with scrapling_response to parse the responses you already fetch with Scrapling's parser, no rewrite needed.
📘 Complete Type Coverage: Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with PyRight and MyPy with each change.
🔋 Ready Docker image: With each release, a Docker image containing all browsers is automatically built an
→
→
→
→
→
→
→
→