Why Scraping a Login Page Is Different

You open a page in your browser — your dashboard, a members-only pricing list, your forum inbox — and everything loads just fine. You copy the URL, paste it into a Python script, run requests.get(), and what do you get back? Either a 401 Unauthorized status code, an empty response, or the HTML of a login page.

The script isn't broken. It just doesn't know who you are.

When you're logged in through a browser, the site remembers you through a cookie. A plain requests.get() call doesn't bring any credentials along, so the server treats you like a random visitor and sends you back to the login screen.

The solution is to simulate the entire login process in your script: submit your username and password, capture the session cookie the server gives you, and reuse it for every request that follows. In Python, the requests.Session object handles all this automatically.

This guide walks through scraping behind login pages with Python. You'll learn how to inspect login forms, log in with a session, handle CSRF tokens, and scrape member-only pages safely. If you're brand new to scraping, start with our Python web scraping beginner guide first — it covers the requests and BeautifulSoup basics this article builds on.

How Website Logins Actually Work

HTTP is stateless by default. Every request you send to a server — whether it's loading a webpage or submitting a form — is independent. The server has no idea that two requests came from the same person.

To change that, websites use sessions. When you log in, the server creates a session record and sends your browser a cookie containing a session ID. Your browser stores this cookie and attaches it to every subsequent request. Each time the server sees that cookie, it knows you're the same user who logged in earlier.

A requests.Session does the same job. It keeps a cookie jar in memory. When you log in through a session, the server sends back a Set-Cookie header, and the session saves it. Then every request you make through that same session object includes the cookie automatically.

The whole process is invisible in your code. You don't need to manually copy cookie values or set headers — the session does it for you.

Step 1: Inspect the Login Form

Before you write any code, you need to know what data the login form expects. Guessing field names like user, pass, or login usually ends in failure.

Here's how to find the exact fields:

  1. Open the login page in your browser.
  2. Open the developer tools by pressing F12 (or right-click and select Inspect).
  3. Switch to the Network tab.
  4. Check the Preserve log option so the network history stays visible after the page reloads.
  5. Enter your username and password manually and submit the form.
  6. Look through the network requests for a POST request to something like /login, /auth, or /api/login.
  7. Click that request and find the Form Data or Payload section.

You'll see a list of key-value pairs sent to the server. Common field names include:

  • username and password
  • email and password
  • login and passwd
  • A hidden csrf_token or authenticity_token

Write down the login URL (the full URL the POST request was sent to) and the field names. You'll need these for the next step.

Step 2: Log In with a Session

Once you know the login URL and the form field names, writing the login code is straightforward.

Here's a complete script skeleton:

import requests

# Configuration — replace with your actual URLs and credentials
BASE_URL = "https://example.com"
LOGIN_URL = "https://example.com/login"
TARGET_URL = "https://example.com/dashboard"

# Your credentials — replace these with your real ones
USERNAME = "your_username"
PASSWORD = "your_password"

# Create a session object — this will store cookies automatically
session = requests.Session()

# Prepare the login data using the field names you found
login_data = {
    "username": USERNAME,
    "password": PASSWORD
}

# Send the login POST request
response = session.post(LOGIN_URL, data=login_data)

# Check the status code
print(f"Login response status: {response.status_code}")

A status code of 200 tells you the server responded, but it doesn't guarantee the login worked. Some websites return 200 with a redirect or re-render the login page when authentication fails.

A more reliable check is to request a page that only logged-in users can see and verify the content.

Step 3: Scrape the Member Pages

After logging in, your session object carries the authentication cookie. You can now send GET requests to any protected page.

Add this to your script after the login attempt:

# After login, request a page that requires authentication
dashboard_response = session.get(TARGET_URL)

# Check if we actually got the logged-in page
if "Welcome, User" in dashboard_response.text:
    print("Login successful! Access granted to the target page.")
    # Process the page content here
else:
    print("Login failed or session expired. Still seeing a login page.")

The key here is checking for a phrase that appears only when you're logged in. This could be a username display, a "Logout" link, or any unique element in the HTML.

Here's a more complete version that saves the page content:

import requests

# Configuration
BASE_URL = "https://example.com"
LOGIN_URL = "https://example.com/login"
TARGET_URL = "https://example.com/dashboard"
USERNAME = "your_username"
PASSWORD = "your_password"

# Create session and log in
session = requests.Session()

login_data = {
    "username": USERNAME,
    "password": PASSWORD
}

login_response = session.post(LOGIN_URL, data=login_data)
print(f"Login POST status: {login_response.status_code}")

# Request the member-only page
target_response = session.get(TARGET_URL)
print(f"Target page status: {target_response.status_code}")

# Verify login success by checking for a known logged-in element
if "Welcome back" in target_response.text:
    print("Login confirmed. Saving page content.")

    # Save the scraped content
    with open("dashboard.html", "w", encoding="utf-8") as file:
        file.write(target_response.text)
else:
    print("Login verification failed. Check your credentials or form fields.")

# Always close the session when done
session.close()

This script saves the protected page to a file called dashboard.html. You can replace the save step with any processing logic you need — parsing with BeautifulSoup, extracting specific data, or sending it to another script.

When the Form Asks for a CSRF Token

Many websites add an extra layer of security with a CSRF token. This is a hidden field in the login form, typically named something like csrf_token, csrfmiddlewaretoken, or _token. The server generates this token on the login page and expects you to send it back with your username and password.

Without it, the server rejects your login attempt with a 403 error.

To handle this, you need to:

  1. GET the login page first.
  2. Extract the CSRF token from the HTML.
  3. Include it in your login POST data.

Here's a script that extracts the token using BeautifulSoup:

import requests
from bs4 import BeautifulSoup

# Configuration
BASE_URL = "https://example.com"
LOGIN_URL = "https://example.com/login"
TARGET_URL = "https://example.com/dashboard"
USERNAME = "your_username"
PASSWORD = "your_password"

# Create session
session = requests.Session()

# Step 1: GET the login page to fetch the CSRF token
login_page = session.get(LOGIN_URL)
soup = BeautifulSoup(login_page.text, "html.parser")

# Find the hidden input field containing the CSRF token
# Common names: csrf_token, csrfmiddlewaretoken, _token, authenticity_token
csrf_input = soup.find("input", {"name": "csrf_token"})

if csrf_input is None:
    csrf_input = soup.find("input", {"name": "csrfmiddlewaretoken"})

if csrf_input is None:
    csrf_input = soup.find("input", {"name": "_token"})

# Extract the token value
csrf_token = csrf_input.get("value") if csrf_input else ""
print(f"CSRF token extracted: {csrf_token[:10]}...")

# Step 2: Build login data including the token
login_data = {
    "username": USERNAME,
    "password": PASSWORD,
    "csrf_token": csrf_token   # Use the exact field name from the form
}

# Step 3: POST the login
login_response = session.post(LOGIN_URL, data=login_data)

# Step 4: Verify by requesting the target page
target_response = session.get(TARGET_URL)

if "Welcome back" in target_response.text:
    print("Login successful with CSRF token.")
    with open("protected_page.html", "w", encoding="utf-8") as file:
        file.write(target_response.text)
else:
    print("Login failed. Check the token field name or your credentials.")

session.close()

The token field name varies between websites. Look at the HTML source or use developer tools to find the exact name attribute of the hidden input. The script above checks for three common variations.

Scraping Behind a Login Responsibly

Just because you can scrape a page doesn't always mean you should. Here are the rules to follow:

Scrape only your own data. Use your own account to access pages you're authorized to view. Don't try to scrape other users' private information or bypass paywalls.

Check the Terms of Service. Many websites prohibit automated access. If the ToS says no scraping, don't scrape it.

Add delays between requests. Sending requests as fast as your script can run puts unnecessary load on the server. Use time.sleep() to add at least 1-2 seconds between requests.

import time

for page in range(1, 11):
    response = session.get(f"{BASE_URL}/members/page/{page}")
    time.sleep(1.5)  # Wait 1.5 seconds between requests

Check robots.txt. Visit the site's /robots.txt file to see which paths are disallowed for automated bots. It's not legally binding, but it shows the site owner's intent.

Close your session when done. Calling session.close() frees up resources and releases the underlying TCP connections.

session.close()

This isn't just polite — it's also practical. Getting your IP blocked makes the whole exercise pointless.

FAQ

Is scraping behind a login legal?

Scraping login-protected pages is legal when you're accessing content you're authorized to view, using your own credentials, for personal or non-commercial use. It becomes problematic if you're bypassing authentication, scraping content you haven't paid for, or violating the website's Terms of Service. If you're unsure, check the ToS and consider reaching out to the site owner. When in doubt, don't scrape.

How do I find the login URL and form fields?

Open your browser's developer tools (F12), go to the Network tab, check "Preserve log", and manually log in. Look through the network requests for a POST request — the URL of that request is your login URL. Click on it and scroll down to the Form Data section to see all the field names. Write them down exactly as they appear.

Why does my script get a 403 or a redirect after logging in?

A 403 Forbidden often means the server detected a missing or invalid CSRF token, or the session didn't save the authentication cookie properly. A redirect back to the login page usually means your credentials were rejected. Double-check your form field names and make sure you're sending all required fields, including any hidden tokens. Also confirm that your session object is the same one you used for both login and subsequent requests.

Do I need Selenium/Playwright instead of requests?

Not for most login forms. If the website relies heavily on JavaScript to render the login form, submit data asynchronously, or require solving a CAPTCHA, then a browser automation tool like Playwright is the better approach. But if the login process is a standard HTML form POST, requests.Session works perfectly and is much faster. Start with requests — switch to Playwright only when you hit a JavaScript wall.

Where does requests store my cookies?

The session object stores cookies in an internal cookie jar. You can view them at any time by accessing session.cookies. The jar holds all cookies sent by the server, and the session automatically includes them in every subsequent request. You don't need to manually extract or set cookie headers — the session handles everything.

Next Steps

Now that you can log in and scrape protected pages, here are some related guides that build on these skills:

All code in this article was tested and runs successfully on Python 3.8.10 (Ubuntu 20.04) with requests 2.32.4, beautifulsoup4 4.8.2 — login flow verified against a local mock login server (form POST, session cookie, CSRF token, protected page); field names on real sites vary, so inspect them with DevTools as the article explains — verified August 2026.