// the take
A lightweight Java crawler framework built on jsoup, letting you define page-to-POJO extraction with annotations instead of hand-rolling CSS selector parsing in every crawl. Aimed at Java developers who want a quick multi-threaded scraper for straightforward sites or structured data extraction, not at those building a full scraping platform.
Annotation-based mapping (@PageSelect/@PageFieldSelect) cuts a lot of the boilerplate of turning HTML into typed objects, and it supports extracting multiple PageVOs from a single page. The PageLoader abstraction is genuinely useful — Jsoup for fast static pages, Selenium+ChromeDriver for JS-rendered content, swappable without touching your extraction annotations. Dynamic proxy pool with pluggable routing (random/round-robin) and runtime-adjustable request params give it real flexibility for working around basic anti-scraping measures.
Distributed crawling isn't actually implemented — only LocalRunUrlPool ships in the box; getting a Redis-backed shared URL pool means writing that integration yourself. There's no built-in robots.txt respect or adaptive rate-limiting, just a manual 'active pause' and fixed retry count, so misconfigured crawls will get IPs banned easily. The codebase still carries a SeleniumPhantomjsPageLoader, and PhantomJS has been unmaintained for years — a sign some of the shipped strategies are stale rather than actively pruned. It's effectively a single-maintainer project (same author as xxl-job), so bus-factor and long-term support are real risks for anything beyond a side project.
We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →