// the find
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Scrapling is a Python web scraping library that combines an HTML parser, multiple HTTP/browser fetchers (plain HTTP, Playwright stealth, full Chrome automation), and a Scrapy-style spider framework under one API. It targets developers who need to bypass bot detection without stitching together five separate libraries. The adaptive element tracking — where selectors auto-relocate after page structure changes — is the genuinely novel piece.
The tiered fetcher design is well thought out: Fetcher for plain HTTP with TLS fingerprint spoofing, StealthyFetcher for Cloudflare Turnstile bypass, DynamicFetcher for full Playwright — each with sync/async variants and session classes. The adaptive selector storage is the killer feature nobody else has: it saves element fingerprints so if a site redesigns and your CSS selector breaks, passing `adaptive=True` finds the element anyway using similarity matching. Parser performance is competitive with Parsel/lxml (benchmarks show ~1:1), while burying BeautifulSoup by 775x — relevant if you're parsing thousands of pages. The spider framework adds pause/resume via checkpoint files, AutoThrottle that backs off when it detects rate-limiting, and XHR capture that collects background API calls without reverse-engineering them.
The README is half sponsor gallery — nine platinum sponsor blocks before you reach the feature list. That's a signal about the project's incentive structure, and those proxy vendor relationships create obvious bias in which anti-bot strategies get prioritized. The adaptive element tracking requires a writable storage backend per-domain, which means stateless deployments (Lambda, containers) need extra wiring that isn't documented clearly. The parser's `auto_save=True` magic — silently writing element fingerprints to disk on first scrape — will surprise anyone who didn't read the adaptive scraping docs, and there's no obvious way to inspect or audit what got saved. Installation is genuinely annoying: base install excludes all fetchers, then you need a second `scrapling install` command to download browsers, which will break in CI unless you bake it into your Docker image.