// the find
TurboWay/spiderman
基于 scrapy-redis 的通用分布式爬虫框架
A scaffolding layer on top of scrapy-redis that handles the boilerplate of distributed web scraping: auto-table creation, code generation, and multi-sink pipelines (MySQL, Postgres, HBase, Kafka, Elasticsearch, HDFS, Doris, etc.). Aimed at teams running multi-site scraping operations who want a consistent project structure rather than reinventing pipelines per project.
The sink coverage is genuinely broad — relational DBs, Hive, HBase, Doris, Kafka, and flat files all work out of the box with a single pipeline enable flag. The built-in bloom filter deduplication is a practical choice for high-volume scraping where Redis memory for exact fingerprints becomes a problem. The metadata system (easy_meta.py) that auto-documents table schemas and field comments into a queryable store is a real operational win for teams managing many spiders. Switching between standalone and cluster mode is a single settings flag, which makes local development and production deployment use the same code.
Last meaningful commit was 2023, pinned to Python 3.6 and Scrapy 1.6 — both several major versions behind, and Scrapy 2.x had breaking middleware changes. The distributed mode requires SSH access to worker nodes with identical codebases and Python environments, which is a fragile deployment model; anyone on Kubernetes or Docker Swarm will find this unusable without significant rework. The deduplication default is 'no dedup, clear on restart', which is sensible for full crawls but will surprise anyone who expects incremental behavior. The README is entirely in Chinese with no English translation, which limits adoption outside that community.