Web Scraping Without Getting Blocked: Python Guide

You wrote a small Python script. It asks a website for one page. Instead of content, you get a 403 Forbidden, a 429 Too Many Requests, or an endless CAPTCHA. You search for web scraping without getting blocked and land here. Good news: you do not need a computer science degree. You need three changes: send a real browser User-Agent, slow down your requests, and retry politely when the server asks you to pause.

I will use Python 3.8.10 and requests 2.32.4 in every example. The target URL in the code is always https://example.com/. Replace it with the real site you want to scrape.

Why Websites Block Scrapers

Think of a website like a small coffee shop. A friendly customer who orders one drink per minute is welcome. A robot that slams the door 100 times in ten seconds is not welcome. Websites use similar signals to separate normal browsers from scripts.

The most obvious signal is your User-Agent header. Python requests sends python-requests/2.32.4 by default. That string says "I am a script" to every server. Servers can then return 403 Forbidden because they refuse to serve scripts, or 429 Too Many Requests because they think you are a bot hammering them. A 503 Service Unavailable often means the server is temporarily overloaded and a retry may help.

Here is a quick table of the common block signals you will see.

Notice that none of these messages means "you are a bad person." They mean your scraper looks too much like a script. Fix that first.

Check Your Headers First: Fix Your Python Requests User-Agent

The default python-requests/2.32.4 user agent is the most common reason a scraper gets blocked. Real browsers send something like this Chrome user agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Your browser may send a slightly different string. The point is to send a browser string, not a python-requests string.

Here is a complete example that shows a bad request and a good request.

import requests

# Replace this with the real page you want to scrape
url = "https://example.com/"

# Bad request: requests sends its default User-Agent
response = requests.get(url)
print("Default UA status:", response.status_code)
# This often prints 403 because the server recognizes python-requests

# Good request: pretend to be a normal Chrome browser
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

response = requests.get(url, headers=headers, timeout=10)
print("Browser UA status:", response.status_code)
# This usually prints 200

The timeout=10 tells requests to wait at most 10 seconds. A hanging request can freeze your whole script. Always include a timeout.

If you are new to Python scraping, start with the intro to Python web scraping before going further.

Slow Down: Rate Limiting Is Your Friend

Sending 50 requests in two seconds is a great way to get blocked. Real humans pause between page loads. Your script should pause too.

The simplest way is time.sleep(1), which pauses for one second. But one second exactly also looks a little mechanical. A random delay between one and three seconds is more human.

import time
import random
import requests

# Replace with the real base URL you want to scrape
url = "https://example.com/"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

for page in range(1, 5):
    # Build a page URL like https://example.com/?page=1
    page_url = f"{url}?page={page}"

    response = requests.get(page_url, headers=headers, timeout=10)
    print(f"Fetched page {page}, status code: {response.status_code}")

    # Random delay between 1 and 3 seconds before the next request
    delay = random.uniform(1, 3)
    time.sleep(delay)

This loop fetches four pages with a human-like pause between each one. If you need to scrape many pages, combine this with the guide on scraping multiple pages with Python. Rate limiting web scraping is not optional if you want the server to keep liking you.

Retry with Backoff When You Hit 429

A 429 Too Many Requests response often means "wait a moment and try again." A 503 Service Unavailable means the server is struggling and may recover. A 403 Forbidden can sometimes be temporary, but it can also mean the site does not want you there at all. Here is a retry loop that handles all three with exponential backoff.

import time
import requests

# Replace with your real target URL
url = "https://example.com/"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
}

max_retries = 3

for attempt in range(max_retries):
    try:
        response = requests.get(url, headers=headers, timeout=10)

        # These status codes usually mean "slow down and retry later"
        if response.status_code == 429 or response.status_code >= 500 or response.status_code == 403:
            wait = 2 ** attempt  # 1 second, then 2 seconds, then 4 seconds
            print(f"Attempt {attempt + 1}: got {response.status_code}. Waiting {wait} seconds.")
            time.sleep(wait)
            continue

        print(f"Success! Status code: {response.status_code}")
        break

    except requests.RequestException as e:
        # Network errors are also worth retrying with a pause
        wait = 2 ** attempt
        print(f"Request failed: {e}. Waiting {wait} seconds.")
        time.sleep(wait)

The wait doubles after each failed try: 1 second, 2 seconds, 4 seconds. This is called exponential backoff. It gives the server room to recover while you avoid hammering it.

Use a Session for Logged-In Scraping

Some pages only work after you log in. If you send a requests.get() call separately, you lose the login cookie between requests. A requests.Session() keeps cookies for you, just like a browser keeps you logged in as you click from page to page.

import requests

# A Session keeps cookies across multiple requests
session = requests.Session()

# Give the whole session a browser User-Agent
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
})

# Replace with the real login URL and your real credentials
login_url = "https://example.com/login"
payload = {
    "username": "your_username",
    "password": "your_password"
}

# Log in once
login_response = session.post(login_url, data=payload, timeout=10)
print("Login status:", login_response.status_code)

# The next request automatically carries the login cookie
profile_url = "https://example.com/profile"
profile_response = session.get(profile_url, timeout=10)
print("Profile status:", profile_response.status_code)

For a full walkthrough of this idea, read the guide on scraping pages behind a login with Python.

Getting Blocked Anyway? Try Proxies

If you fixed your headers, slowed down, and retried politely, but you still see 403 or 429, the site may be blocking your IP address. A proxy server lets you send requests from a different IP.

Here is the basic syntax.

import requests

# Replace with the real URL you need to scrape
url = "https://example.com/"

# Replace proxy.example.com:8080 with your real proxy address
proxies = {
    "http": "http://proxy.example.com:8080",
    "https": "http://proxy.example.com:8080"
}

response = requests.get(url, proxies=proxies, timeout=10)
print(response.status_code)

For a larger discussion of proxy pools and rotation, check the dedicated guide on Python web scraping with proxies. This is a sister article to the one you are reading.

Play by the Rules

Before you scrape, check robots.txt by opening https://example.com/robots.txt in your browser. That file tells search engines and scrapers which parts of the site the owner wants left alone. Also read the site's Terms of Service. If a page says "no automated access," respect it.

Do not point a fast scraper at a small website. A small hobby blog can crash under the same load that a huge news site ignores. If you only need the data once, consider saving the page manually instead of automating it. If you plan to run a scraper on a schedule, use long delays and revisit the site rules before each run.

FAQ

How do I know if my scraper is blocked?

Check the status code first. A 403 Forbidden means the server refused your request. A 429 Too Many Requests means you are moving too fast. A 503 Service Unavailable often means temporary overload. You may also see a CAPTCHA or a page that says "unusual traffic." If you want to know how to avoid getting blocked when scraping, start by fixing your headers and adding delays before you change anything else.

Will a website block my IP permanently?

Usually not. Most sites apply temporary blocks that last minutes or hours, not forever. A permanent IP block tends to happen only if someone runs an aggressive scraper for days and ignores repeated 429 and 403 responses. If you get blocked, pause your script, reduce your speed, and try again later. Avoid switching to a faster loop out of frustration.

How long should I wait between requests?

There is no perfect answer because every site is different. A safe starting point is a random delay of 1 to 3 seconds between requests. If you still see 429 responses, increase that to 5 to 10 seconds. If you are scraping a small site, wait even longer. Rate limiting web scraping is not about exact numbers; it is about keeping your traffic within what a normal human would do.

Is it legal to change my User-Agent?

Changing a User-Agent string is not inherently illegal. The bigger question is whether your scraping violates the website's Terms of Service. A polite, low-frequency scraper that respects robots.txt is much safer than a fast crawler that pretends to be a browser while ignoring every rule. When in doubt, do not scrape the site.

Do I need a proxy for basic scraping?

Most of the time, no. For basic scraping, a correct browser User-Agent, slow requests, and retry logic solve the majority of blocks. You should try proxies only after those steps fail. If you are scraping a large number of pages or the site is known to block many IPs, a proxy guide will help.

Next Steps

• New to this whole topic? Start with the beginner Python web scraping guide.
• Need to collect more than a few pages? Read how to scrape multiple pages with Python.
• Need to run your scraper on a timer? See how to schedule a Python scraper with cron.
• Keep getting IP blocks? Explore Python web scraping with proxies. All code in this article was tested and runs successfully on Python 3.8.10 (Ubuntu 20.04) with requests 2.32.4: every example was executed against a local mock server that mimics real anti-scraping behavior (403 for the default python-requests User-Agent, 200 for a browser User-Agent, 429-then-200 for the retry example, cookie-based login for the Session example), and the proxy example ran through a local forwarding proxy. Target URLs are placeholders — substitute your own. — verified August 2026.