// the find
my8100/scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:
ScrapydWeb is a Flask-based admin UI for managing Scrapyd clusters — the daemon that runs Scrapy spiders. It wraps the Scrapyd JSON API, adds a job dashboard with ECharts visualizations, integrates with LogParser for structured log analysis, and provides timer tasks and email alerting. It's for teams running distributed Scrapy scraping operations who want something better than curling the Scrapyd API directly.
The multi-node cluster management is the real value here — you can deploy, schedule, and stop jobs across dozens of Scrapyd nodes in a single click rather than hitting each one individually. The log analysis integration with LogParser is genuinely useful: it parses Scrapy's log format into structured stats (items scraped, errors, durations) and renders progress charts that make it easy to spot stuck or failing spiders. Auto-packaging is a practical time-saver — it zips your Scrapy project and deploys it without you needing to run `scrapyd-deploy` manually. The test suite is thorough for a project of this size, with separate test files per feature area and CircleCI integration.
The last meaningful release was years ago and the frontend bundles Element UI 2.4.6 (from 2018) vendored directly into the repo — you're shipping ancient Vue 2 components with no upgrade path in sight. Authentication is HTTP Basic Auth only, which means either you expose this on a private network or you're putting credentials in a browser prompt; there's no token-based auth, no RBAC, no per-user permissions. The Flask app runs synchronously and spawns background threads for polling, which means under any real load the GIL will hurt you — there's no async story here. The Heroku demo link is dead, which is a small but telling sign of maintenance posture.