// the find
Python3WebSpider/ProxyPool
An Efficient ProxyPool with Getter, Tester and Server
A self-hosted proxy pool that scrapes free proxy sites, tests them against a configurable URL, scores them in Redis by reliability, and exposes a Flask API to pull working proxies. Aimed at developers who want a learning project or who need a thin proxy rotation layer they control. The README is upfront that free proxy yield is terrible — often 1-2 usable proxies per hundred scraped.
Redis sorted set as the storage backend is a clean fit: score-based ranking, O(1) random pop, and TTL-via-eviction all fall out naturally. The crawler extension model is genuinely easy — drop a class inheriting BaseCrawler with a urls list and a parse generator and it auto-discovers. Docker Compose one-liner gets you Redis + all three processors supervised without touching your host. The GeoLite2 country filter is a nice touch for cases where you need region-specific exit IPs.
The public crawler list is heavily China-focused and most of those proxy sites rotate their structure or go offline frequently, so expect maintenance burden just to keep crawlers alive. Testing by default hits httpbin.org and Baidu — both rate-limit aggressively, so your test results degrade under any real load and the anonymous-IP check becomes unreliable when httpbin throttles you. Redis is a single point of failure with no replication config in the provided Compose file; if it restarts you lose all scored proxies and the pool goes cold. The Flask server runs with no auth by default and binds 0.0.0.0, which the README does warn about but the default is still a footgun for anyone who deploys this without reading carefully.