// the find
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Crawlee is a Node.js/TypeScript web scraping framework from the team behind Apify. It unifies Cheerio, JSDOM, Playwright, and Puppeteer under one API with built-in request queuing, session management, proxy rotation, and autoscaling. If you're building a serious scraper that needs to survive bot detection, this is the most battle-tested option in the JS ecosystem.
The unified interface across HTTP and browser crawlers is genuinely well-designed — you can swap from Cheerio to Playwright by changing one class name and the rest of your request handler just works. The fingerprinting layer (TLS replication, browser-like headers, human-like fingerprints) is built in by default, not bolted on. Autoscaling based on CPU, memory, and event loop load means you don't have to tune concurrency manually for most use cases. The persistent request queue survives restarts, which matters when crawling millions of pages.
It's tightly coupled to the Apify ecosystem for cloud deployment — the local file storage is fine for prototyping but real production use will push you toward their platform. The monorepo has 15+ packages which makes the dependency graph unwieldy; a simple Cheerio scrape still pulls in the full `@crawlee/core` with autoscaling machinery you won't use. There's no built-in support for distributed crawling across multiple machines outside of Apify's own infrastructure. Documentation is thorough but the guides lean heavily on Apify-specific deployment scenarios, so self-hosted production patterns are underserved.