Your Python scraper was working yesterday. Today the same script returns one 403 after another. I have seen this happen after just a few dozen requests. The website is not broken. It started treating your home IP address like a traffic spike.

This guide gives you a beginner-friendly way to do python web scraping with proxies. You will set up a python requests proxy, add proxy rotation python code so one IP does not take all the heat, and build a small scraping proxy pool that skips dead proxies. I use Python 3.8.10 and requests 2.32.4, so you can follow along on Windows, macOS, or Linux.

When Does a Scraper Need a Proxy?

Think of your IP address as the phone number your Python script calls from. If you call the same store 100 times in an hour from the same number, the store may stop picking up. That is what a 429 or 403 status code often means. A proxy gives you a different phone number for some of those calls.

You will likely need a proxy in three situations:

  1. Your IP gets blocked. The site starts returning 429 or 403 responses even though your code did not change.
  2. The content is served to visitors from certain regions. Your real IP is outside that region, so you never see the data.
  3. You plan to scrape many pages quickly. You want to spread requests across several IPs so one address does not trigger a rate limit.

If you have never used Python's requests library before, start with my Python web scraping for beginners guide. It covers the basic request loop you will build on here.

How Proxies Work in Plain English

A proxy is a middleman. Imagine you want to send a letter to a company but do not want your return address on the envelope. You mail the letter to a friend first. That friend takes your letter, puts it in a new envelope with their own return address, and mails it to the company. The company replies to your friend, and your friend forwards the reply to you.

A web proxy does the same thing for HTTP requests.

Your Python script sends a request to the proxy server instead of directly to the target website. The proxy forwards that request to the target site. The target site sees the proxy's IP address, not your home IP. The response comes back through the proxy and reaches your script.

That is the whole idea. You do not need to understand deep networking. You only need to know which proxy address to put in your code.

Free vs Paid Proxies: The Honest Version

Free proxy lists are easy to find. They are also easy to regret using.

A free proxy can disappear in the middle of your scrape. It can be slow because hundreds of other people use it. Some free proxy operators record the traffic that passes through their server, which is bad if you are sending any sensitive headers or session cookies. The worst case is not just slow scraping; it is unknowingly handing your request data to a stranger.

Paid proxies usually charge by bandwidth (gigabytes) or by pool size. You get support, a more stable IP pool, and a provider that has a reason to keep its service trustworthy.

Here is an honest comparison:

Free proxies Paid proxies
Uptime Random, often dead Usually stable
Speed Slow and crowded Consistent
Privacy Unknown; some log traffic Contract, less likely to snoop
Cost Zero dollars Per GB or monthly
Good for Testing on small sites Real projects

My realistic advice for beginners: do not buy a big proxy plan on day one. Practice with a free list on a small test site that allows scraping. Once you need reliability, move to a small paid plan. That saves you from debugging a script that is actually fine but using a dead free proxy.

Your First Request Through a Proxy

The requests library expects a proxies dictionary. It needs separate entries for http and https, even if both use the same proxy address. Here is the complete code:

import requests

# Replace with the target website you are allowed to scrape
url = "https://example.com/"

# A browser User-Agent is still important; see the anti-block guide
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

# Replace with a real proxy address and port
proxy_address = "proxy.example.com:8080"

# If your proxy requires a username and password, use the format below:
# "http://your_username:your_password@proxy.example.com:8080"
proxies = {
    "http": f"http://{proxy_address}",
    "https": f"http://{proxy_address}",
}

# Send a GET request through the proxy with a 10-second timeout
response = requests.get(url, headers=headers, proxies=proxies, timeout=10)

# Check if the request succeeded
if response.status_code == 200:
    print("Request worked through the proxy")
    print(response.text[:200])  # Print the first 200 characters of the page
else:
    print(f"Got status code: {response.status_code}")

Replace proxy.example.com:8080 with your real proxy address. Replace https://example.com/ with the page you actually want to scrape. The timeout=10 stops a slow proxy from hanging your script forever.

The commented username and password line shows the format for authenticated proxies. If your provider gave you credentials, put them before the proxy address with an @ symbol. Keep that string inside the http:// prefix, exactly as shown.

Build a Simple Proxy Pool for Proxy Rotation in Python

One proxy can also get blocked after too many requests. A simple fix is to keep a list of proxies and pick one randomly for each request. This is called proxy rotation. It spreads the traffic so no single IP looks too busy.

Here is a working script that scrapes five pages and rotates proxies using random.choice:

import random
import requests

# Replace with the real page URL you are allowed to scrape
url = "https://example.com/"

# A browser User-Agent helps; see the anti-block guide
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

# Replace these with real proxy addresses
proxy_list = [
    "proxy1.example.com:8080",
    "proxy2.example.com:8080",
    "proxy3.example.com:8080",
]

# Loop through several pages and pick a random proxy each time
for page in range(1, 6):
    # Choose a random proxy from the list
    proxy_address = random.choice(proxy_list)

    # Build the proxies dictionary for this request
    proxies = {
        "http": f"http://{proxy_address}",
        "https": f"http://{proxy_address}",
    }

    try:
        response = requests.get(url, headers=headers, proxies=proxies, timeout=10)
        print(f"Page {page}: {response.status_code}")
    except requests.RequestException as e:
        print(f"Page {page} failed with {proxy_address}: {e}")

This code does not stop on a bad proxy. It prints the error and moves to the next page. For a stronger setup, you can also read how to avoid getting blocked while scraping in Python. That guide covers headers, delays, and other anti-blocking tricks that work with proxies.

Fail Over to a Working Proxy

Random rotation is nice, but sometimes you need a stricter approach. If one proxy is dead and you want the script to skip it immediately and try another, use a failover loop.

Here is the pattern: try each proxy in order. If a request raises a requests.RequestException, print the error, move on to the next proxy. If one works, break out of the loop.

import requests

# Replace with the real target URL
url = "https://example.com/"

# A browser User-Agent helps; see the anti-block guide
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

# Replace with real proxy addresses; put several so one failure does not stop you
proxy_candidates = [
    "proxy1.example.com:8080",   # intentionally broken placeholder
    "proxy2.example.com:8080",
    "proxy3.example.com:8080",
]

response = None

# Try each proxy in order until one works
for proxy_address in proxy_candidates:
    proxies = {
        "http": f"http://{proxy_address}",
        "https": f"http://{proxy_address}",
    }

    try:
        print(f"Trying proxy: {proxy_address}")
        candidate_response = requests.get(url, headers=headers, proxies=proxies, timeout=10)
        # Raise an error for bad status codes if needed
        candidate_response.raise_for_status()
        response = candidate_response
        print(f"Success with {proxy_address}")
        break
    except requests.RequestException as e:
        print(f"Proxy {proxy_address} failed: {e}")
        # Move to the next proxy; optionally sleep a moment before retrying
        # import time; time.sleep(1)
        continue

if response is not None:
    print("Final status code:", response.status_code)
    # Do something with response.text here
else:
    print("All proxies failed. Check your proxy list and internet connection.")

The requests.RequestException catches common failures: timeouts, connection refusals, and HTTP errors like 407, 403, and 429. When a proxy returns a 407, that usually means the proxy needs a username and password. Go back to the previous code block and use the username:password@ format.

Keep It Legal and Polite

Proxies do not change the rules. If a site's robots.txt says no scraping, a proxy does not make it okay. If the Terms of Service forbid automated access, a proxy does not create a loophole.

Check https://example.com/robots.txt and read the site's ToS before you scrape. Keep your request rate low. Do not point a free proxy pool at a giant website and hammer it. A proxy hides your home IP from the target site, but it does not hide your ethical responsibility.

FAQ

Are free proxies safe?

Not always. Many free proxies are slow, unpredictable, or run by unknown operators. Some log your traffic. For practice on a small site, a free proxy can work. For anything with personal data, use a paid provider or no proxy at all. If you are asking "are free proxies safe for web scraping," the honest answer is: they are a risk, not a guarantee.

How many proxies do I need for scraping?

It depends on how many requests you send and how strict the target site is. For a small project, 5 to 10 proxies may be enough. For larger jobs, you may need dozens or a residential proxy pool. Start with a few and increase only if you see 429 or 403 responses.

Why does my proxy return a 407 error?

A 407 status code means Proxy Authentication Required. The proxy server is asking for a username and password. Check your proxy address string. It should look like http://username:password@proxy.example.com:8080. If your credentials contain special characters, URL-encode them first.

Do proxies work for HTTPS websites?

Yes. In requests, you set the https key in the proxies dictionary. The proxy forwards encrypted HTTPS traffic without reading the contents. That is why the same proxy address usually works for both http and https keys.

Can I run my own proxy server?

Yes. You can set up a small proxy on a VPS using software like Squid or a simple Python forwarding proxy. This gives you more control and a clean IP. The downside is that one server is still one IP. If that IP gets blocked, you are back to square one. Running your own proxy is a good learning project, but it is not a complete scraping proxy pool by itself.

Next Steps