How many proxies do I need? A method in three counts

Why doesn't a rate limit decide how many proxies I need?

A rate limit decides how fast your job may go, not how many IPs it should use. When a site says "one request every 10 seconds" in its robots.txt, its terms or its API documentation, it means your crawler, whatever addresses it comes from.

Spreading requests over more IPs to exceed a limit the site has set is evading that limit. It breaks the site's rules, can harm a small server, and is against our acceptable use policy. The honest way to collect more data is a longer schedule, an official API or permission from the site owner.

So sizing starts from locations, spares and your own connections, and the site's pace becomes the speed limit of the job.

Count 1: how many locations do I need?

You need one IP per place whose view or latency matters to the job. A site that shows prices by country needs one IP in each country you track. A latency check of your own service from five regions needs five datacenter cities.

List the countries first, then the cities only where the city changes the result or the latency. RackProxies covers 22 countries and 34 datacenter cities with a page. Availability varies; the checkout shows what is online. The city guide helps you pick.

Count 2: how many spare and separate IPs?

Add one spare IP per location so a job keeps running when an IP has a problem. If an IP does not work or is unusable for a lawful use, write to support: we fix it or replace it with an IP in the same country, subject to stock; the spare covers the time in between.

Then add IPs for separation. Production and staging often need their own allowlisted addresses, and so do different clients you serve. A partner that allowlists your IPs usually wants a primary and a standby. These IPs are dedicated to you: Dedicated: during your rental, each IP is assigned to your account only and is not shared with other RackProxies customers.

Count 3: how many connections per IP?

Divide your peak number of open connections by the number you choose to run per IP, and round up. The per-IP figure is your decision: a lower one keeps each address's load modest and makes problems easier to trace.

Example: 2,000 sites, each fetched at a polite pace, with at most 200 connections open at any moment. If you choose at most 20 open connections per IP, you need 200 ÷ 20 = 10 IPs. We do not publish a connection limit per IP; fair use applies, and very heavy use that harms the network can be limited.

How long will a polite job take?

Multiply the pages you need from one site by the delay that site asks for. robots.txt is defined by RFC 9309; the Crawl-delay line is not part of that standard, but many sites still use it, and Python's urllib.robotparser reads it. The script below reads a site's rules and prints the time the job needs at that pace. Adding IPs does not change the answer.

"""Read a site's robots.txt and work out how long a polite job takes."""
import urllib.robotparser

SITE = 'https://example.com'
AGENT = 'price-monitor'   # the User-Agent your job sends
PAGES = 750               # pages you need from this site per day
DEFAULT_DELAY = 10        # your own pace in seconds when robots.txt sets none

robots = urllib.robotparser.RobotFileParser(SITE + '/robots.txt')
robots.read()

if not robots.can_fetch(AGENT, SITE + '/'):
    raise SystemExit('robots.txt disallows this path for ' + AGENT)

delay = robots.crawl_delay(AGENT) or DEFAULT_DELAY
rate = robots.request_rate(AGENT)
if rate:
    delay = max(delay, rate.seconds / rate.requests)

hours = PAGES * delay / 3600
print(f'One request every {delay:g} s for the whole job: {hours:.1f} h for {PAGES} pages')
print('More IPs do not shorten this: the pace applies to your crawler, not to each IP.')