// the take
Crawly is a Scrapy-style web crawling and scraping framework for Elixir, built on OTP so request scheduling, concurrency limits, and process supervision come from the BEAM instead of being hand-rolled. It's aimed at Elixir/Erlang shops that want to scrape data without reaching for Python, or that want scraping to live inside existing BEAM infrastructure.
The OTP foundation gives real benefits: per-domain concurrency limits, supervised worker processes, and crash isolation between spiders come for free instead of being application code you have to write. The middleware/pipeline architecture (DomainFilter, UniqueRequest, DuplicatesFilter, Validate, JSONEncoder, WriteToFile) is composable and will feel immediately familiar to anyone who's used Scrapy. Standalone mode lets ops run spiders as plain YAML files or mounted Elixir modules in a Docker container, so you don't need a mix project or Elixir knowledge just to operate it. It also ships a management UI and REST API for starting/stopping spiders and inspecting the request queue and extracted items, which most scraping libraries leave entirely to the user to build.
No push since July 2025 and the README's roadmap literally says "to be discussed" — that's over a year of silence on a project that was previously driven by one or two maintainers, so plan for you to be fixing your own bugs. Browser rendering isn't built in: it hands off to Splash, an external rendering service with its own maintenance questions, or a custom render server you run and operate yourself. The nicer Phoenix/LiveView management UI (CrawlyUI) is explicitly deprecated with nothing replacing it, leaving only the bare built-in admin for anything beyond start/stop/preview. The README asks users to report production usage rather than listing any, so there's little public evidence of this holding up at real scale outside the maintainer's own blog posts.
We read the README, the directory tree and a key source file in November 2026. Nobody pays to be listed. How we judge →