// the find
jhao104/proxy_pool
Python ProxyPool for web spider
A self-hosted proxy pool that scrapes free proxy lists, validates them continuously, and exposes them via a simple REST API backed by Redis. It's for Python web scrapers that need rotating proxies without paying for a service. Actively maintained, 23k stars, and the extension model is genuinely clean.
The fetcher plugin system is well-designed — drop a class inheriting BaseFetcher into fetcher/sources/ and it's picked up automatically without touching any config. The scheduler/server split makes operational sense: validation runs independently and doesn't block API requests. Docker Compose setup works out of the box, which is the right default for something with a Redis dependency. Test coverage exists and runs in CI, which is better than most scraping tools of this vintage.
Free proxies from these sources are genuinely bad — low availability, short-lived, and often blocked by any site worth scraping. The README's own availability ratings top out at three stars with most at one. If you actually need proxies for anything serious, this gets you a pool of marginally-working IPs you'll burn through fast. The /delete endpoint is a GET request that mutates state, which is just wrong. SSDB support exists but is clearly an afterthought — the client is there but integration tests for it suggest it may not be exercised much. No authentication on the API at all, so if you expose port 5010 anywhere beyond localhost, anyone can drain or manipulate your pool.