Web Scraping with Python: A Beginner's Guide to Extracting Data from Any Website

Why Web Scraping?

You find a website with 50 pages of useful data. Manually copying each page would take hours. Web scraping automates this — a Python script pulls the data from all 50 pages in seconds.

Common use cases: - Price tracking — monitor product prices across stores - Job listings — collect openings from multiple job boards - Research — gather data for analysis or a personal project - Content aggregation — collect articles, quotes, or statistics

If you've followed our file renaming automation or Downloads organizer guides, you know Python excels at saving you time. Web scraping is the same idea — just pointed at the internet instead of your files.

What You'll Need

Only two libraries. requests fetches web pages; BeautifulSoup parses the HTML.

pip install requests beautifulsoup4

Your First Scraper: Extracting Page Titles

Let's start with the simplest possible scraper — pulling the title from any webpage:

import requests
from bs4 import BeautifulSoup

# Fetch the page
url = "https://books.toscrape.com"
response = requests.get(url)

# Parse the HTML
soup = BeautifulSoup(response.text, "html.parser")

# Extract the title
title = soup.title.text
print(f"Page title: {title}")

Output:

Page title: All products | Books to Scrape

Three lines of actual logic. Here's what each does: 1. requests.get(url) — downloads the page HTML, same as your browser 2. BeautifulSoup(response.text, "html.parser") — turns raw HTML into a searchable structure 3. soup.title.text — finds the <title> tag and extracts the text inside

Finding Elements: CSS Selectors and Methods

HTML pages are trees of elements. BeautifulSoup gives you multiple ways to find what you need:

# Find ONE element
soup.find("h1")               # First <h1> tag
soup.find("div", class_="book") # First <div class="book">
soup.select_one(".price")     # First element with class="price" (CSS selector)
soup.select_one("#main")      # First element with id="main"

# Find ALL matching elements
soup.find_all("a")            # All <a> tags (returns a list)
soup.find_all("li", class_="item")
soup.select(".product")       # All elements with class="product" (CSS selector)
soup.select("div.book > h3")  # All <h3> inside <div class="book"> (CSS selector)

CSS selectors are usually the easiest way. If you've ever styled a webpage, you already know the syntax: - .class-name — elements with a CSS class - #id-name — element with a specific ID - div > a — <a> tags directly inside a <div> - div a — <a> tags anywhere inside a <div>

A Real Example: Scraping Book Data

Let's scrape actual data from Books to Scrape, a practice website built for learning web scraping:

import requests
from bs4 import BeautifulSoup
import csv

url = "https://books.toscrape.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

# Each book is in an <article> with class "product_pod"
books = soup.find_all("article", class_="product_pod")

book_data = []

for book in books:
    # Title is inside <h3><a> — the <a> tag has a title attribute
    title = book.h3.a["title"]

    # Price is in <p class="price_color">
    price = book.find("p", class_="price_color").text

    # Availability is in <p class="instock availability">
    availability = book.find("p", class_="instock").text.strip()

    book_data.append([title, price, availability])

# Print results
for book in book_data:
    print(f"{book[0]} | {book[1]} | {book[2]}")

# Save to CSV
with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["Title", "Price", "Availability"])
    writer.writerows(book_data)

print(f"\nSaved {len(book_data)} books to books.csv")

Output (first few rows):

A Light in the Attic | £51.77 | In stock
Tipping the Velvet | £53.74 | In stock
Soumission | £50.10 | In stock
...
Saved 20 books to books.csv

Let's walk through the key extraction patterns:

Pro tip: right-click any element in your browser and choose Inspect — you'll see exactly which tags and classes contain the data you want.

Handling Multiple Pages (Pagination)

Most real scraping involves multiple pages. Here's the pattern:

import requests
from bs4 import BeautifulSoup

base_url = "https://books.toscrape.com/catalogue/page-{}.html"
all_books = []

for page_num in range(1, 6):  # Scrape pages 1-5
    url = base_url.format(page_num)
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")

    books = soup.find_all("article", class_="product_pod")

    for book in books:
        title = book.h3.a["title"]
        price = book.find("p", class_="price_color").text
        all_books.append({"title": title, "price": price})

    print(f"Page {page_num}: found {len(books)} books")

print(f"\nTotal: {len(all_books)} books scraped")

Important: always add a delay between page requests so you don't hammer the server:

import time

for page_num in range(1, 6):
    # ... scraping code ...
    time.sleep(1)  # Wait 1 second between pages

Handling Errors Gracefully

Real websites aren't always available. Your scraper should handle failures:

import requests
from requests.exceptions import RequestException, Timeout

def fetch_page(url, retries=3, timeout=10):
    """Fetch a page with retry logic."""
    for attempt in range(retries):
        try:
            response = requests.get(url, timeout=timeout)
            response.raise_for_status()  # Raise exception for 4xx/5xx
            return response
        except Timeout:
            print(f"Timeout on {url}, attempt {attempt + 1}/{retries}")
        except RequestException as e:
            print(f"Error fetching {url}: {e}")
        time.sleep(2)  # Wait before retry

    return None  # All retries failed

response = fetch_page("https://example.com/data")
if response is None:
    print("Failed to fetch the page after all retries")

Respect Robots.txt and Be Ethical

Before scraping any site, check https://example.com/robots.txt. This file tells crawlers which paths are off-limits:

User-agent: *
Disallow: /admin/
Disallow: /api/
Crawl-delay: 10

Rules of responsible scraping: 1. Check robots.txt — skip paths it disallows 2. Add delays — time.sleep(1) between requests minimum 3. Identify yourself — set a User-Agent header with your contact info 4. Don't re-scrape constantly — cache results, don't hit the same page every minute 5. Public data only — don't scrape behind logins, don't republish copyrighted content

Setting a custom User-Agent:

headers = {
    "User-Agent": "MyScraperBot/1.0 (your-email@example.com) — collecting public data for personal project"
}
response = requests.get(url, headers=headers)

Extracting Tables

Many websites put data in HTML tables. BeautifulSoup handles this easily:

# Find the table
table = soup.find("table")

# Extract all rows
rows = table.find_all("tr")

for row in rows:
    cells = row.find_all(["td", "th"])  # Both data and header cells
    cell_text = [cell.text.strip() for cell in cells]
    print(cell_text)

Scraping JavaScript-Heavy Sites

Some websites load data with JavaScript — requests.get() only gets the initial HTML. If you scrape and the data is missing, the site might need a different approach:

1. Check the Network tab in your browser's DevTools — the data may come from a hidden API endpoint you can call directly
2. Use Selenium or Playwright — these control a real browser and execute JavaScript (heavier but works everywhere)

For most beginner projects, requests + BeautifulSoup is enough. Websites that deliver content in their HTML (blogs, documentation, product listings, news articles) scrape perfectly with this approach.

Full Project: Price Tracker

Let's tie it together — a simple price tracker you can run daily:

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime

# This example scrapes Books to Scrape; adapt selectors for real sites
URL = "https://books.toscrape.com"
PRICE_LIMIT = 30.00  # Only track books under this price

response = requests.get(URL)
soup = BeautifulSoup(response.text, "html.parser")

books = soup.find_all("article", class_="product_pod")
bargains = []

for book in books:
    title = book.h3.a["title"]
    price_text = book.find("p", class_="price_color").text
    price = float(price_text.replace("£", ""))

    if price < PRICE_LIMIT:
        bargains.append([title, price, datetime.now().strftime("%Y-%m-%d")])

# Append to CSV (so you can run this daily)
with open("price_history.csv", "a", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    for b in bargains:
        writer.writerow(b)

print(f"Found {len(bargains)} bargains under £{PRICE_LIMIT}")

Run this daily with a cron job and you have a price history. Adapt the selectors to track real products you care about.

Next Steps

1. Automate your backups — combine scraping data with scheduled Python scripts
2. Organize your Downloads folder — more Python file operations practice
3. Try scraping a real site — Wikipedia tables, weather data, or product listings (check robots.txt first)
4. Learn Selenium — when you encounter JavaScript-heavy sites, Selenium controls a real browser

Web scraping opens up a world of data. Once you can pull anything from any webpage, you can build price trackers, job alert scripts, research datasets, and more — all while your computer does the tedious work.