// the find
SpiderClub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
A distributed proxy pool that scrapes free public proxies from the web, validates them per-site, and serves them via Redis. Built on Scrapy with a site-specific validation model — the same IP gets tested separately against each target site, which is actually the right call since proxy quality varies by destination. Aimed at teams running large-scale scrapers who need a self-hosted proxy layer.
The per-site validation approach is the standout design decision — instead of treating a proxy as globally good or bad, it maintains separate scored queues per target site, which matches how proxies actually behave in the wild. Prometheus + Grafana monitoring is wired in from the start, not bolted on later. The Squid integration as a language-agnostic proxy tier is a practical touch that lets non-Python scrapers use the pool. Docker Compose setup covers the full stack including Splash for JS rendering.
Abandoned since late 2022 and the README explicitly warns against using the master branch — you're told to pull from releases, which is a strange trust signal for a project you'd run in production. The entire system is tightly coupled to Redis; the README itself says 'if you need to replace Redis, measure that yourself,' which is not a great sign for operational flexibility. Free public proxy lists are increasingly useless — detection and blocking have gotten aggressive, and the yield from scraping public proxy sites has dropped sharply. No HTTP API for proxy retrieval; you need the Python client or Squid, which limits integration options.