Scrape Multiple Pages with Python: A Beginner's Guide

Why Stop at One Page?

You just learned the web scraping basics and successfully pulled data from a single webpage. Feels great, right? But then you realize the product catalog you're looking at spans 50 pages. Your one-page script grabs 20 items, but the full dataset has 1,000.

I ran into this exact wall when I started. A client wanted book prices from an online store, and I proudly showed them a script that worked on the first page. They asked, "Cool. Now get the rest." That's when I learned that real-world scraping means handling pagination.

In this guide, I'll show you three ways to scrape multiple pages with Python. We'll use the practice site books.toscrape.com, which has 50 pages of books—20 per page. By the end, you'll have a script that pulls all 1,000 titles, prices, and ratings into a CSV file.

Why One Page Isn't Enough

Picture this: you're researching prices for a product you want to track. You scrape the first page of search results and find a dozen listings. But the real deal—the lowest price—is on page 7. If you stop at page 1, you're making decisions based on 2% of the data.

Here are common scenarios where single-page scraping fails:

• E-commerce catalogs: Amazon, eBay, and bookstore sites split inventory across dozens of pages
• News archives: Most blogs show 10–20 articles per page, with older posts buried deep
• Forum threads: A popular Reddit thread can have hundreds of comment pages
• Job boards: Listings are spread across multiple pages based on posting date

Think of pagination like chapters in a book. Reading chapter one gives you the introduction, but you won't know how the story ends. To get the complete picture, you need to turn every page.

How Pagination Works: Three Patterns

Websites use different methods to separate content across pages. I've grouped them into three main patterns:

The first two are straightforward for beginners. The third needs a browser automation tool like Playwright, which I'll mention later.

In this guide, I'll focus on Patterns 1 and 2 because they're the most common and you can handle them with requests and BeautifulSoup—no browser required.

Pattern 1: Scrape Pages with a URL Pattern

The easiest pagination pattern is when the URL changes predictably. On books.toscrape.com/catalogue/, you'll see:

• Page 1: https://books.toscrape.com/catalogue/page-1.html
• Page 2: https://books.toscrape.com/catalogue/page-2.html
• ...
• Page 50: https://books.toscrape.com/catalogue/page-50.html

With a pattern like this, you can loop through numbers and build each URL. Here's the script:

import requests
from bs4 import BeautifulSoup

# Base URL with a placeholder for the page number
base_url = "https://books.toscrape.com/catalogue/page-{}.html"

# Loop through pages 1 to 50
for page_num in range(1, 51):
    # Build the URL for this page
    url = base_url.format(page_num)
    print(f"Scraping {url}")

    # Send the request
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find all books on this page
    books = soup.select('article.product_pod')

    for book in books:
        # Get title from the title attribute inside h3 > a
        title = book.select_one('h3 a')['title']
        # Get price from p.price_color, remove £, convert to float
        price_text = book.select_one('p.price_color').text
        price = float(price_text.replace('£', ''))
        print(f"  {title}: £{price}")

Run this and you'll see 1,000 books scroll past. The range(1, 51) loop generates page numbers 1 through 50, and format(page_num) plugs each number into the URL.

The important detail: I used select_one('h3 a')['title'] because the book title lives inside the title attribute of the anchor tag. The price is in p.price_color, and we strip the £ sign before converting to a float.

Pattern 2: Follow the "Next" Button

But what if you don't know how many pages exist? Or what if the URL pattern changes halfway through? That's when you follow the "Next" link.

On books.toscrape.com, the HTML for the next button looks like this:

<li class="next">
    <a href="page-2.html">next</a>
</li>

We need to:

1. Find the <li class="next"> element
2. Extract the href attribute from the <a> tag inside it
3. Build the full URL (the href is relative, like page-2.html)
4. Keep going until there's no "Next" button

Here's a while loop that handles this:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

# Start at page 1
base_url = "https://books.toscrape.com/catalogue/page-1.html"
current_url = base_url

while current_url:
    print(f"Scraping {current_url}")

    # Fetch the page
    response = requests.get(current_url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Extract data from this page
    books = soup.select('article.product_pod')
    for book in books:
        title = book.select_one('h3 a')['title']
        price_text = book.select_one('p.price_color').text
        price = float(price_text.replace('£', ''))
        print(f"  {title}: £{price}")

    # Find the "next" button
    next_li = soup.find('li', class_='next')
    if next_li:
        # Extract the relative href and join with base URL
        next_href = next_li.find('a')['href']
        current_url = urljoin("https://books.toscrape.com/catalogue/", next_href)
    else:
        # No next button: we're done
        current_url = None

print("Finished scraping all pages!")

The key line is urljoin("https://books.toscrape.com/catalogue/", next_href). This turns page-2.html into https://books.toscrape.com/catalogue/page-2.html. Without urljoin, your script would try to request page-2.html as a standalone URL and fail.

Be a Polite Scraper

Web scraping is like knocking on someone's door. Knock once—they'll answer. Knock 50 times a second—they'll call the cops. Here's how to be a good neighbor:

Add a Delay Between Requests

import time

# Inside your loop, after processing each page
time.sleep(1)  # Wait 1 full second

One second per page means 50 seconds for all 50 pages. That's fine for a learning project. For production, start with 1–2 seconds and adjust based on the site's response.

Check robots.txt

Many sites have a robots.txt file that tells scrapers which paths are allowed. Our practice site returns a 404 for robots.txt (so no rules), but real sites like Amazon or eBay have strict rules. Always check:

# In a real project, check first
import requests
robots = requests.get("https://example.com/robots.txt")
print(robots.text)

If the site disallows scraping, respect that. Use their API if available, or find an alternative data source.

Handle Page Changes

Websites update their HTML structure. Your script might work today and break tomorrow. Use try/except blocks to catch missing elements:

for book in books:
    try:
        title = book.select_one('h3 a')['title']
        price = float(book.select_one('p.price_color').text.replace('£', ''))
        # Save data
    except (AttributeError, TypeError) as e:
        print(f"Skipping a book: {e}")

This prevents your entire script from crashing if one book has a different structure.

Full Project: Scrape All 50 Pages into a CSV

Now let's combine everything into a complete script. This will scrape all 50 pages from books.toscrape.com and save the results to books.csv.

import requests
from bs4 import BeautifulSoup
import time
import csv
from urllib.parse import urljoin

# Set up CSV file
csv_file = open('books.csv', 'w', newline='', encoding='utf-8')
csv_writer = csv.writer(csv_file)
csv_writer.writerow(['Title', 'Price', 'Rating'])

# Rating mapping
rating_map = {
    'One': 1,
    'Two': 2,
    'Three': 3,
    'Four': 4,
    'Five': 5
}

# Start at page 1
current_url = "https://books.toscrape.com/catalogue/page-1.html"
page_count = 0

while current_url:
    page_count += 1
    print(f"Scraping page {page_count}: {current_url}")

    # Fetch the page
    response = requests.get(current_url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find all books on this page
    books = soup.select('article.product_pod')

    for book in books:
        # Title from h3 > a title attribute
        title = book.select_one('h3 a')['title']

        # Price from p.price_color, remove £
        price_text = book.select_one('p.price_color').text
        price = float(price_text.replace('£', ''))

        # Rating from class (e.g., "star-rating Three")
        rating_class = book.select_one('p.star-rating')['class'][1]
        rating = rating_map.get(rating_class, 0)

        # Write to CSV
        csv_writer.writerow([title, price, rating])

    # Find the "next" button
    next_li = soup.find('li', class_='next')
    if next_li:
        next_href = next_li.find('a')['href']
        current_url = urljoin("https://books.toscrape.com/catalogue/", next_href)
    else:
        current_url = None

    # Be polite: wait 1 second before the next request
    time.sleep(1)

csv_file.close()
print(f"Done! Scraped {page_count} pages and saved to books.csv")

When you run this, you'll get a books.csv file with 1,000 rows. Open it in any spreadsheet app. You now have a complete dataset ready for analysis.

If you want to work with this data further, check out my guide on Python CSV and pandas for beginners to filter, sort, and visualize your book collection.

What About Infinite Scroll Pages?

Some sites don't have "Next" buttons or page-number URLs. They load new content as you scroll down—think Instagram, Twitter, or product feeds on modern stores.

Infinite scroll is powered by JavaScript. When you scroll to the bottom, the page sends an API request to the server, receives more data, and inserts it into the DOM. The requests library can't handle this because it doesn't execute JavaScript.

For these sites, you need a browser automation tool like Playwright. It launches a real browser, scrolls automatically, and waits for new content to load. I cover this in detail in my Playwright web automation guide for beginners. If you're hitting infinite scroll pages, that's your next step.

FAQ

How do I know how many pages a site has?

Three reliable methods:

1. Check the pagination widget at the bottom: Most sites display "Page 1 of 50" or similar text. You can scrape that number.
2. Count the "Next" button until it disappears: Use the while loop from Pattern 2 and stop when there's no next link.
3. Look at the URL pattern: If the site uses page-1.html and you try page-1000.html and get a 404, you've found the upper bound.

For books.toscrape.com, the maximum is 50 pages. In real projects, I usually combine method 1 (scrape the page count text) with a while loop for safety.

Why does my script only get the first page?

Three common culprits:

1. You forgot to update the URL inside the loop: If you build url = base_url.format(1) once and never change it, you'll scrape page 1 repeatedly.
2. You're not following the redirect: Some sites use ?page=2 but redirect to a different structure. Use response.url to see the final URL.
3. The "Next" button is hidden or loaded by JavaScript: If you're using requests and the site needs JavaScript, you'll never find the next link. Switch to Playwright.

Check your loop logic first. Print the URL each iteration to confirm it's changing.

Is scraping multiple pages legal?

It depends on the site and how you scrape. Here's the short version:

• Check robots.txt: If the site disallows scraping, stop.
• Read the Terms of Service: Many sites prohibit automated access. Violating this is a legal risk.
• Don't bypass login or CAPTCHAs: That's usually illegal.
• Use public data only: Scraping data that requires authentication is risky.
• Respect rate limits: A polite scraper (1–2 seconds between requests) is less likely to trigger blocks.

The practice site books.toscrape.com is made for learning—it's explicitly allowed for scraping exercises. For real projects, always check the site's rules first.

What if the next page button uses JavaScript?

Then requests + BeautifulSoup won't work. The "Next" button is created or handled by JavaScript, so your HTML parser never sees the <a> tag.

Your options:

1. Use Playwright or Selenium: These tools control a real browser and execute JavaScript.
2. Look for the API: Many infinite-scroll sites use a hidden API. Open your browser's Developer Tools (Network tab) and scroll down. You'll see JSON requests to api.example.com/items?page=2. Scrape that endpoint directly.
3. Use a headless browser: Playwright is my recommendation for beginners. It's easier than Selenium and handles modern JavaScript well.

I walk through this step by step in my Playwright web automation guide.

Next Steps

You've learned how to scrape multiple pages with Python. Here's what to do next:

1. Practice on real sites: Pick a public site (check robots.txt first) and adapt your script. Try sites with different pagination patterns.
2. Build a price tracker: Use your multi-page scraper to monitor prices over time. Check out my Python price tracker project for scheduling and alert ideas.
3. Store data in a database: CSV works for small datasets, but SQLite or PostgreSQL scales better. My Python CSV/pandas guide shows you how to transition.
4. Handle JavaScript with Playwright: For modern sites, Playwright web automation is the skill you need.

One final tip: save your script with comments. You'll thank yourself when you revisit it in six months. Happy scraping!

All code in this article was tested and runs successfully on Python 3.8.10 (Ubuntu 20.04) with requests 2.32.4, beautifulsoup4 4.8.2 — verified against books.toscrape.com (all 50 pages, 1,000 books scraped into books.csv), August 2026.