APIs vs Scraping: When to Call the Hidden API

Scraping vs API: What's the Difference?

Let's say you want data from a website. Maybe it's sports scores, product prices, or news headlines. You have two main ways to get it.

One way is web scraping. You download the HTML page, parse it, and pull out the pieces you need. It's like reading a restaurant menu from a photo someone took through the kitchen window. You can see everything, but you have to figure out what's what. It's messy, and if the menu changes, your photo is outdated.

The other way is using an API — Application Programming Interface. An API is like a waiter at that restaurant. You tell the waiter exactly what you want ("Give me today's specials"), and they bring you a clean, formatted answer. No guessing. No extra clutter. Just the data you asked for.

Here's a quick comparison:

The key difference? An API is built for machines to talk to machines. Scraping is a workaround when that official conversation isn't available.

Check for a Public API First

Before you even think about scraping, do this: search for [site name] + API docs. Seriously. Most major platforms have a public API, and they'd rather you use it than scrape their pages.

Take GitHub. It has a beautiful, well-documented API. And here's the best part — you can try it right now without any API key. Open your terminal and run:

# Fetch public data from a GitHub user without authentication
curl https://api.github.com/users/octocat

You'll get a JSON response with login, public_repos, followers, and more. It's clean, structured, and instant.

But there's a catch: without an API key, GitHub limits you to 60 requests per hour per IP address. That's fine for testing, but if you're building something serious, you'll want to register for a free key to get 5,000 requests per hour.

Here's a short list of sites with good public APIs:

• GitHub — user data, repos, issues (free, generous limits)
• OpenWeather — weather data (free tier available)
• Hacker News — top stories, comments (free, no key needed)
• Google Maps — locations, distances (requires key)

If you find an API, use it. It saves you time, bandwidth, and headaches.

How to Find a Hidden API

Sometimes there's no public documentation, but the website is clearly loading data dynamically. You click a button, the content updates without refreshing the page. That's JavaScript talking to a backend server.

That backend server is probably serving data over an API. It's just not public. It's a hidden API — an endpoint meant for the website's own frontend. But if the browser can call it, so can your Python script.

Here's how I find them, step by step:

1. Open the website in Chrome or Firefox.
2. Right-click anywhere and select Inspect.
3. Click the Network tab.
4. Refresh the page. You'll see a list of all network requests.
5. Filter by XHR or Fetch (these are the AJAX calls that load data).
6. Look for requests that return JSON. The response preview will show structured data like {"articles": [...]}.

Let's do a real example. Open ESPN's NBA news page in your browser. Open DevTools, go to Network, filter by XHR, and refresh. You'll see a request to:

https://site.api.espn.com/apis/site/v2/sports/basketball/nba/news

Click on that request. Go to the Preview tab. You'll see a clean JSON object with an articles array. Each article has headline, description, links, and published.

That's a hidden API. No API key. No documentation. But it's there, and we can call it.

A quick word of caution

Hidden APIs are not guaranteed. They can change without notice. They can disappear. They're built for the website, not for you. But in practice, they're often stable because the website itself depends on them.

Call a Hidden API with Python

Alright, let's write some code. We'll call that ESPN hidden API and print out the top headlines.

Make sure you have the requests library installed. If you don't, run:

# Install the requests library if you don't have it
pip install requests

Now create a Python script — let's call it espn_news.py:

import requests

# URL of the hidden API (ESPN NBA news)
url = "https://site.api.espn.com/apis/site/v2/sports/basketball/nba/news"

try:
    # Send a GET request with a 10-second timeout
    response = requests.get(url, timeout=10)

    # Raise an exception if the request failed (e.g., 404 or 500)
    response.raise_for_status()

    # Parse the JSON response into a Python dictionary
    data = response.json()

    # Navigate to the articles list
    articles = data.get("articles", [])

    # If there are no articles, let the user know
    if not articles:
        print("No articles found.")
    else:
        # Loop through the first 5 articles and print their headlines
        for article in articles[:5]:
            headline = article.get("headline", "No headline")
            published = article.get("published", "Unknown date")
            print(f"{headline} (Published: {published})")

except requests.exceptions.Timeout:
    print("The request timed out. Try again later.")
except requests.exceptions.HTTPError as e:
    print(f"HTTP error occurred: {e}")
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

Run it:

# Execute the script
python espn_news.py

You'll see something like:

2026 NBA buzz: Latest free agency and trade updates (Published: 2026-08-22T00:33:34Z)
Denver Nuggets offseason recap, early 2026-27 season preview (Published: 2026-08-22T04:55:25Z)
2026 NBA free agency: Grades for offseason signings, extensions (Published: 2026-08-21T22:41:07Z)
Nuggets signing 6-time All-Star DeMar DeRozan to 1-year deal (Published: 2026-08-21T22:54:03Z)
Klay Thompson to sign 2-year deal with Heat after Mavs buyout (Published: 2026-08-21T23:57:38Z)

That's a real run from August 2026 — your headlines will be different depending on the news of the day. That's it. You just called a hidden API with Python.

Breaking down the code

• requests.get(url, timeout=10) — sends the HTTP request. The timeout prevents your script from hanging forever.
• response.raise_for_status() — checks if the response was successful. If not, it raises an exception.
• response.json() — parses the JSON response into a Python dictionary.
• data.get("articles", []) — safely tries to get the "articles" key. If it doesn't exist, it returns an empty list.

This pattern works for almost any JSON API — hidden or public.

When Scraping Is Still the Right Choice

APIs are great, but they're not always available. Here are real scenarios where scraping beats APIs:

1. No API exists. Many smaller sites, local business directories, or government portals don't have an API. If the data is only visible as HTML, scraping is your only option.

2. The API requires payment. Some APIs are expensive. Twitter's enterprise tier costs thousands per month. If you just need a small dataset, scraping might be free (though you should check the ToS).

3. The data is in the HTML. Sometimes the API returns incomplete data. The full product description, reviews, or images might only be in the HTML. In that case, you'll scrape the page to get everything.

4. You need a snapshot of the page. Maybe you want to save exactly what the user sees — formatting, layout, images. An API gives you raw data. Scraping gives you a visual copy.

If you're new to scraping, I've written a web scraping with Python guide that walks you through BeautifulSoup and extracting data from HTML.

The Golden Rules

Whether you're using an API or scraping, follow these rules. They'll keep you out of trouble and make your scripts last longer.

1. Read the Terms of Service. Some websites explicitly forbid scraping in their ToS. Others don't care as long as you're respectful. Check before you build something that could get your IP banned.

2. Respect rate limits. If an API says "60 requests per hour", don't try to make 61. You'll get blocked. Use time.sleep() in your Python script to space out requests.

3. Don't hit the server too hard. This applies to both APIs and scraping. If you're making thousands of requests per minute, you're essentially DDoSing the site. Slow down. Add delays. Cache results when you can.

4. Cache results when possible. If the data doesn't change every second, save it to a file or database. Your script can check the cache before making a new request. This saves you from hitting rate limits and reduces server load.

5. Have a fallback. APIs break. Hidden APIs change. Scraping targets get redesigned. Always assume your data source might stop working tomorrow. Have a backup plan — even if it's just an error message that logs the issue.

If you're scheduling your data collection, check out how to schedule your scraper with cron to run it automatically without manual intervention.

API vs Scraping: Decision Flow

Here's a mental flowchart I use:

FAQ

Is it legal to use a hidden API?

It depends on the website's Terms of Service. A hidden API is not illegal just because it's undocumented — but if the ToS says "you may not access our services programmatically," then using it could be a violation. I've seen many developers use hidden APIs for personal projects without issue. Just don't build a commercial product on one. And if the website asks you to stop, stop.

Do I need an API key to use an API?

Not always. Public APIs like the GitHub user endpoint work without authentication (though with limited requests). Most commercial APIs require a key so they can track your usage and enforce limits. Hidden APIs usually don't need a key because they're designed for the website's own frontend — but that also means they're not officially supported.

Why is scraping slower than calling an API?

When you scrape, your script downloads the entire HTML page — images, styles, ads, scripts, everything. Then it parses all that clutter to find the tiny piece of data you need. An API returns only the data you requested, in a lightweight format like JSON. Less data = faster response.

What is a rate limit?

A rate limit is a cap on how many requests you can make in a certain time window. For example, GitHub's anonymous API allows 60 requests per hour. If you exceed that, the API returns an error until the limit resets. Rate limits protect the server from being overloaded by too many requests.

What if the hidden API stops working?

It happens. The website updates its frontend, changes the endpoint URL, or modifies the JSON structure. When that happens, your script breaks. That's the price of relying on an unofficial interface. Your best bet is to switch to scraping the HTML as a fallback, or look for a public API alternative. Always write your scripts with error handling so they fail gracefully instead of crashing.

Next Steps

Now that you've learned how to choose between APIs and scraping, here's what to explore next:

• Web scraping with Python — if you need to extract data from HTML when no API exists.
• Save scraped data to Excel — once you have the data, learn how to store it in spreadsheets.
• Schedule your scraper with cron — automate your data collection to run daily without manual work.
• Clean scraped data with pandas — tidy up messy datasets after you've collected them.

All code in this article was tested and runs successfully on Python 3.8.10 (Ubuntu 20.04) with requests 2.32.4 — verified against the live GitHub API (api.github.com/users/octocat) and the ESPN hidden API (site.api.espn.com/apis/site/v2/sports/basketball/nba/news), August 2026.