// the find
rmax/scrapy-redis
Redis-based components for Scrapy.
scrapy-redis replaces Scrapy's in-memory scheduler and duplicate filter with Redis-backed equivalents, letting multiple spider processes share a single crawl queue. It's the standard approach for scaling Scrapy horizontally when you need multi-process or multi-machine crawls without a full frontier system like Frontera.
The plug-and-play design is genuinely good — you swap scheduler and dupefilter via settings, no code changes required. The dupefilter uses a Redis set rather than an in-memory set, so it survives process crashes and restarts, which matters a lot in long-running crawls. Docker Compose example project is included and actually works, making it easy to spin up a local Redis + multiple spiders to see real distributed behavior. CI covers builds, checks, and tests as separate workflows, which keeps failure signal clear.
No built-in URL expiration — if you want to re-crawl something after N days, you have to manage that yourself or bolt on Frontera. The README's JSON-extension feature (FormRequest from Redis JSON payloads) reads like it was grafted in from a fork and documented as an afterthought; the install instructions even tell you to clone a different repo ('darkrho/scrapy-redis') to get it, which is confusing. Pickling request objects into Redis is the default serialization — it works but ties you to Python and makes inspecting queue contents painful. The project has been slow to move: issues sit open for months and the last real feature work was years ago.