Schedule Your Python Scraper with cron (2026 Guide)

Stop Running Your Scraper by Hand

I used to wake up at 6:30 AM just to run my price scraper before the morning commute. Then I'd forget on weekends. The data gaps drove me crazy.

Manually running scripts is a trap. You forget. You sleep in. You're on vacation. And your data gets stale. The fix? A built-in Linux tool called cron that runs your Python scraper automatically at whatever schedule you want.

In this guide, you'll learn to schedule a Python script with cron on a Linux machine. By the end, your scraper will run every day at 8 AM—or every 30 minutes, or every Monday—without you lifting a finger.

What Is cron?

cron is a time-based job scheduler built into Linux, macOS, and other Unix-like systems. Think of it as an alarm clock that runs commands instead of ringing. You tell cron when to run something, and it handles the timing.

You edit a file called a "crontab" (cron table) to define your schedule. Each line is one job. The system has a system-wide crontab, but most people use their own with crontab -e.

There's another tool called systemd timers that does something similar. I cover that separately if you're curious, but for beginners, cron is simpler and more common. I'll focus on cron here.

Step 0: Make Your Script Easy to Run

Before we touch cron, make your scraper script easy to execute from anywhere. A little setup saves a ton of troubleshooting later.

Add a Shebang Line

Put this at the very top of your Python script (the first line):

#!/usr/bin/env python3

This tells your system to run the script with Python 3. Now you can execute it like ./scraper.py instead of python3 scraper.py.

Make It Executable

In your terminal, navigate to the script's folder and run:

chmod +x scraper.py

Now test it:

./scraper.py

It should work. If it doesn't, fix any errors before moving on.

Use Absolute Paths Inside Your Script

Inside your Python script, don't assume files are in the same folder. Use full paths for input and output files. For example:

import os

# This will work no matter where cron runs it from
base_dir = "/home/yourname/scraper_project"
csv_path = os.path.join(base_dir, "books_raw.csv")

List Your Dependencies

Create a requirements.txt file in your project folder so you can reinstall dependencies if needed:

requests==2.32.4
beautifulsoup4==4.8.2
pandas==2.0.3

Install with:

pip install -r requirements.txt

This doesn't affect cron directly, but it helps when you move the script to another machine.

Step 1: Open Your Crontab

Time to schedule. In your terminal, type:

crontab -e

The first time you run this, it asks you to choose a text editor. I recommend nano for beginners (it's simpler) or vim if you already know it.

After you pick, a blank crontab file opens. Every line starting with # is a comment—cron ignores it.

The Five-Star Syntax

Cron schedules use five fields. Let me break them down:

* * * * * command_to_run
│ │ │ │ │
│ │ │ │ └─── Day of week (0-7, where 0 and 7 = Sunday)
│ │ │ └───── Month (1-12)
│ │ └─────── Day of month (1-31)
│ └───────── Hour (0-23)
└─────────── Minute (0-59)

An asterisk * means "every". So * * * * * means "every minute" (don't do that for a scraper).

Step 2: Schedule It

Here are the most common schedules. Add one of these lines to your crontab file.

Every Day at 8:00 AM

0 8 * * * /home/yourname/scraper_project/scraper.py

Breakdown: minute 0, hour 8, every day (), every month (), every weekday (*).

Every 30 Minutes

*/30 * * * * /home/yourname/scraper_project/scraper.py

The */30 means "every 30 minutes".

Weekdays at 6:30 AM

30 6 * * 1-5 /home/yourname/scraper_project/scraper.py

1-5 means Monday through Friday.

At Midnight on the 1st of Each Month

0 0 1 * * /home/yourname/scraper_project/scraper.py

I recommend starting with the daily schedule (0 8 * * *) and adjusting later.

Save the file and exit. In nano: Ctrl+O, Enter, then Ctrl+X.

Step 3: Log Everything

cron runs in the background. If something goes wrong, you won't see any error messages unless you capture them. This is the most common beginner mistake.

Redirect all output (both normal and error) to a log file like this:

0 8 * * * /home/yourname/scraper_project/scraper.py >> /home/yourname/scraper_project/scraper.log 2>&1

Here's what that means:

• >> appends normal output to the log file
• 2>&1 sends error messages to the same place as normal output

Now every run writes to scraper.log. You can check it later to see if your script ran successfully.

Add Timestamps to Your Logs

Put this inside your Python script so every log entry has a timestamp:

from datetime import datetime
import sys

def log(message):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"{timestamp} - {message}")

log("Scraper started")
# Your scraping code here
log("Scraper finished")

Step 4: Handle Failures

Sometimes scripts fail. The website might be down, or a selector might change.

Wrap Your Scraper in a Try/Except Block

Inside your Python script:

import sys
from datetime import datetime

def log(message):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"{timestamp} - {message}")

try:
    log("Scraper started")
    # Your scraping code
    log("Scraper finished successfully")
except Exception as e:
    log(f"ERROR: {e}")
    # Optionally send yourself an email

Now even failures show up in the log with a timestamp and the error message.

If Your Log Shows an Error

Run the script manually from your terminal. Often, the error message is clearer when you run it yourself:

cd /home/yourname/scraper_project
./scraper.py

You'll see exactly what broke.

Step 5: Test and Verify

Never assume your cron job is running. Verify it.

List Your Scheduled Jobs

crontab -l

This prints all your cron jobs. Make sure your line is there.

Check the Service Is Running

On Ubuntu, cron runs as a background service:

systemctl status cron

You should see "active (running)".

Wait and Check the Log

After your scheduled time, check the log file:

cat /home/yourname/scraper_project/scraper.log

Look for the timestamp. If you see the log entry, it ran. If you see an error, fix it.

Force a Quick Test

Change the schedule to * * * * * (every minute) and save. Wait one minute, then check the log. When you confirm it works, change it back to your real schedule.

Common cron Problems (And Their Fixes)

Here's a quick reference for the issues that most beginners hit.

Find Your Python Path

which python3

Usually returns /usr/bin/python3. Use that full path in your cron command if you're having "command not found" issues.

Set Working Directory in Crontab

Add this before your command to change to the right folder:

0 8 * * * cd /home/yourname/scraper_project && /usr/bin/python3 scraper.py >> scraper.log 2>&1

FAQ

How do I run a Python script every day at 8am?

Add this line to your crontab: 0 8 * * * /home/yourname/scraper_project/scraper.py. The 0 8 part means minute 0 of hour 8. The three * characters mean every day of the month, every month, and every day of the week. Save the crontab and your script will run at 8:00 AM daily. If you want to use 24-hour time, just convert—2 PM is 14, midnight is 0.

How do I run a Python script every 30 minutes?

Use */30 * * * * as the schedule. The */30 in the minute field means "every 30 minutes". The rest of the fields are *, meaning no restriction on hour, day, month, or weekday. Your command runs at minute 0 and minute 30 of every hour (so 8:00, 8:30, 9:00, 9:30, and so on). You can also use */15 for every 15 minutes or */5 for every 5 minutes.

Why does my Python script work manually but not in cron?

This is the number one cron problem. The reason: cron runs with a limited environment. It doesn't have your full PATH, it doesn't load your .bashrc or .profile, and it starts from a different directory. Fixes: use absolute paths for both Python (/usr/bin/python3) and your script (/home/yourname/scraper.py). Change to the script's directory with cd /home/yourname/scraper_project &&. Redirect all output to a log file with >> log 2>&1. If you need environment variables, set them inside the crontab line.

How do I check if my cron job ran?

There are three ways. One: check your log file if you set up logging with >> log 2>&1. Two: run grep CRON /var/log/syslog on Ubuntu—this shows all cron activity in the system logs. Three: if your script touches a file (like saving CSV data), check the file's modification timestamp with ls -l. Look at any of these right after your scheduled time to confirm the job ran.

Does cron work on Windows?

No. cron is a Unix/Linux tool. On Windows, use Task Scheduler instead. The concept is the same—set a time, pick a program to run—but the interface is different. If you're running a Linux virtual machine or WSL (Windows Subsystem for Linux) on Windows, cron works inside that Linux environment. For pure Windows, search for "Task Scheduler" in the Start menu and create a basic task that runs python scraper.py at your desired time.

Next Steps

You've got your scraper running automatically. Here's what to explore next:

• Master cron syntax – There's more you can do with cron. Learn all the schedule patterns and advanced options in the cron jobs for beginners guide.
• Build your scraper – If you don't have a script to schedule yet, start with scrape website data with Python to build a working scraper first.
• Scrape multiple pages – Automate scraping across dozens of pages with scrape multiple pages with Python.
• Try the alternative – If cron feels too limited, check out systemd timers as an alternative.

Take a step back and watch your scraper run on its own tomorrow morning. That's automation. You wrote code once, and now it runs forever without you. Next project: add email alerts so you get a summary every time the scraper finishes.

All code in this article was tested and runs successfully on Python 3.8.10 (Ubuntu 20.04): shebang execution (./scraper.py), chmod +x, absolute paths via os.path.join, log redirection (>> log 2>&1 appends stdout and stderr with timestamps), which python3 → /usr/bin/python3, and the system cron service status were all verified; cron schedule lines were syntax-validated (5-field format, ranges, /steps, a-b ranges). A real cron firing was not tested — placeholder paths (/home/yourname/...) must be replaced with your own before use — verified August 2026.*