PythonMastery
intermediate 20 min read · lesson 2 of 14 in Python How-To

Web Scraping with requests + BeautifulSoup

1 · The lesson

read

Scraping is two problems, one after the other: fetch the page (HTTP) and find the data inside it (HTML parsing). Many tutorials explain one half well and skip over the other. A scraper that is polite and doesn't break easily needs both. It also needs to behave well enough that the site's admin has no reason to notice it.

The mental model: you're writing a small, courteous robot. It pretends to be a browser, asks for pages, waits its turn, extracts the bits you want, and saves them in a structure you can re-use. Twenty minutes from now you'll have one.

1. The Two-Library Stack

Most Python scrapers use the same two libraries, and have done for about ten years:

  • requests: fetches the page. It is synchronous, easy to use and has sensible defaults.
  • beautifulsoup4: parses the HTML. It copes well with messy markup and makes it easy to move around the page.
bash
pip install requests beautifulsoup4

The simplest end-to-end scrape:

python
import requests
from bs4 import BeautifulSoup

r = requests.get("https://example.com")
r.raise_for_status()                 # raise an error on any 4xx or 5xx status (Section 3)
soup = BeautifulSoup(r.text, "html.parser")
print(soup.title.text)               # "Example Domain"

Only three lines do the real work. The rest of this lesson makes that script reliable and polite, and able to cope with the ways real pages go wrong.

2. Always Send a Real User-Agent

By default, requests sends User-Agent: python-requests/2.31.0 (with your installed version number). Many sites reply with 403 Forbidden as soon as they see that header, because it is a clear sign of a bot. The fix is one line:

python
HEADERS = {
    "User-Agent": "Mozilla/5.0 (compatible; HedyBot/1.0; +https://example.com/bot)",
    "Accept-Language": "en-GB,en;q=0.9",
}

r = requests.get("https://example.com", headers=HEADERS, timeout=10)
+ setup added so this can run · defines requests
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

requests = _AutoMock('requests')

Two principles at play:

  • Identify yourself honestly. Putting a contact URL in the UA is good manners. If your scraper misbehaves, the admin can email you instead of blocking your IP address.
  • Look enough like a browser to not get blocked. A plain Mozilla/5.0 is the minimum for sites with simple bot filters.

Some sites need the full browser UA string. You can copy yours from navigator.userAgent in DevTools. Don't pretend to be Googlebot. That is faking a known crawler, and sites block it faster than an honest bot.

3. Always Check the Status Code

python
r = requests.get(url, headers=HEADERS, timeout=10)

# Option A: let it raise an error on any 4xx or 5xx status
r.raise_for_status()

# Option B: check the code yourself when you want to handle some codes
if r.status_code == 404:
    return None
if r.status_code >= 500:
    raise RuntimeError("server problem; try again later")
r.raise_for_status()

The classic bug: you forget the check, and BeautifulSoup parses an HTML error page without any complaint. Your selectors return nothing, your code thinks the site has no data, and you spend an hour debugging the selector instead of noticing the 503.

r.raise_for_status() raises requests.HTTPError on any non-2xx. Combine with a try/except if you want to log-and-continue rather than crash. See Exceptions for how to do that well.

4. Parsing with BeautifulSoup: Find, Find All, Select

BeautifulSoup gives you two ways to locate elements: the find / find_all API and CSS selectors via select. Both are useful, but selectors are usually more concise.

python
from bs4 import BeautifulSoup

html = """
<html><body>
  <div class="quote" data-id="1">
    <span class="text">The unexamined life is not worth living.</span>
    <span class="author">Socrates</span>
    <a class="tag" href="/tag/wisdom">wisdom</a>
    <a class="tag" href="/tag/life">life</a>
  </div>
</body></html>
"""

soup = BeautifulSoup(html, "html.parser")

# find and find_all search by tag name and attributes
first = soup.find("span", class_="text")
print(first.text)                                    # The unexamined life...
quotes = soup.find_all("div", class_="quote")        # list of matching tags

# CSS selectors are short, and you may know them from web work
print(soup.select_one("div.quote span.text").text)
for a in soup.select("div.quote a.tag"):
    print(a.text, "→", a.get("href"))

The handful of accessors you'll use over and over:

AccessorWhat it does
tag.textAll text inside the tag, joined.
tag.get("href")Read an attribute. Returns None if missing.
tag["href"]Same, but raises KeyError if missing. Prefer .get().
tag.parentWalk up one level.
tag.childrenIterate direct children (mixes tags and NavigableStrings).
tag.find_next_sibling("li")Next <li> sibling.

Use class_= (with the underscore) in find calls, because class is a Python keyword. With selectors, the dotted form (.quote) is the normal CSS.

5. Build Selectors That Don't Break Easily

The biggest reason scrapers break in production: someone changed a CSS class name and your selector silently returned None.

Prefer, in this order:

1. Semantic tags + structure: article > h2, main .entry-title.
2. data-* attributes: the site usually adds these for its own JS or tests, so they rarely change: [data-testid="price"].
3. IDs: usually stable, but sometimes generated (e.g. user-7821).
4. Class names: last choice. Avoid utility classes (text-sm flex-1) in particular, because they change often.

python
# Fragile: depends on class names that a build tool generated
price = soup.select_one(".sc-1a3b2c4d-5 .x9k").text

# Better: uses an attribute that someone probably added on purpose
price = soup.select_one("[data-testid='product-price']").text
+ setup added so this can run · defines soup
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

soup = _AutoMock('soup')

When the data isn't tagged semantically, your fallback is the structural anchor: find a stable parent, then navigate down. "The <dt> that says 'Price', then the next <dd>":

python
dt = soup.find("dt", string="Price")
price = dt.find_next_sibling("dd").text.strip()
+ setup added so this can run · defines soup
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

soup = _AutoMock('soup')

This pattern survives a lot of CSS refactors because it reads the page the way a human does.

6. Pagination: Don't Stop at Page One

Most sites use one of two kinds of pagination.

Numbered pages are easy:

python
def scrape_pages(base_url, max_pages=10):
    results = []
    for page in range(1, max_pages + 1):
        r = requests.get(f"{base_url}?page={page}", headers=HEADERS, timeout=10)
        if r.status_code == 404:
            break                                # past the last page
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "html.parser")
        items = soup.select("article")
        if not items:
            break                                # an empty page means we are done
        results.extend(extract(item) for item in items)
        time.sleep(1)                            # be polite to the server (Section 8)
    return results
+ setup added so this can run · defines BeautifulSoup, requests, HEADERS, time, extract
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def BeautifulSoup(*_a, **_kw):
    print('-> BeautifulSoup() called')
    return _AutoMock('BeautifulSoup()')
requests = _AutoMock('requests')
HEADERS = _AutoMock('HEADERS')
time = _AutoMock('time')
def extract(*_a, **_kw):
    print('-> extract() called')
    return _AutoMock('extract()')

Next-link pagination: follow the next link until there isn't one:

python
url = "https://example.com/blog"
while url:
    r = requests.get(url, headers=HEADERS, timeout=10)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    yield from extract_items(soup)
    next_link = soup.select_one("a[rel='next']")
    url = next_link["href"] if next_link else None
    time.sleep(1)

Always cap iterations with max_pages or a sanity counter. If the site's pagination ever has a bug where page N links to page N, a scraper without a cap loops forever.

7. Respect robots.txt

robots.txt is the site's machine-readable "please don't scrape these paths". You're not legally bound by it everywhere, but ignoring it is rude and often a Terms-of-Service violation. The stdlib parses it for you:

python
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

UA = "HedyBot/1.0"
if rp.can_fetch(UA, "https://example.com/private/"):
    fetch(...)
else:
    print("denied by robots.txt")
+ setup added so this can run · defines fetch
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def fetch(*_a, **_kw):
    print('-> fetch() called')
    return _AutoMock('fetch()')

Cache the parsed result for each domain. Fetching robots.txt again before every request is wasteful, and it adds load to the site. Check once per scraper run. Refresh hourly for long-running jobs.

8. Rate Limiting and Courtesy

A scraper sending 100 requests per second to a small site looks like a denial-of-service (DoS) attack, and it has the same effect. A few lines of time.sleep often decide whether you get the data or get banned.

python
import time

for url in urls:
    fetch(url)
    time.sleep(0.5)                              # at most 2 requests per second, a polite default
+ setup added so this can run · defines urls, fetch
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

urls = ["alpha", "beta", "gamma"]
def fetch(*_a, **_kw):
    print('-> fetch() called')
    return _AutoMock('fetch()')

Better: rate-limit per domain. If you scrape example.com and other.org in parallel, each gets its own budget:

python
from collections import defaultdict
import time

last_request = defaultdict(float)
MIN_GAP = 0.5                                    # seconds between requests to the same domain

def polite_get(url, headers=HEADERS):
    from urllib.parse import urlparse
    domain = urlparse(url).netloc
    wait = MIN_GAP - (time.monotonic() - last_request[domain])
    if wait > 0:
        time.sleep(wait)
    last_request[domain] = time.monotonic()
    return requests.get(url, headers=headers, timeout=10)
+ setup added so this can run · defines HEADERS, requests
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

HEADERS = _AutoMock('HEADERS')
requests = _AutoMock('requests')

For larger scrapers, look at the ratelimit package or implement a token bucket. For a one-off, time.sleep is fine.

9. Retries with Exponential Backoff

Networks fail, and servers go down for a few seconds. Your scraper shouldn't stop because of one temporary 503.

A version you write yourself, which is good enough for most jobs:

python
import time
import requests

def fetch_with_retries(url, max_attempts=4):
    for attempt in range(max_attempts):
        try:
            r = requests.get(url, headers=HEADERS, timeout=10)
            if r.status_code < 500 and r.status_code != 429:
                return r                         # 2xx, 3xx and other 4xx codes: don't retry
        except requests.RequestException:
            pass
        if attempt < max_attempts - 1:           # no point waiting after the last try
            time.sleep(2 ** attempt)             # 1s, 2s, 4s
    raise RuntimeError(f"gave up after {max_attempts} attempts: {url}")
+ setup added so this can run · defines HEADERS
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

HEADERS = _AutoMock('HEADERS')

The production-grade version uses urllib3.util.Retry mounted on a session:

python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=4,
    backoff_factor=1,                            # 0s, 2s, 4s, 8s between retries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
+ setup added so this can run · defines requests
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

requests = _AutoMock('requests')

Retry 5xx and 429 (rate-limited). Don't retry other 4xx errors: they mean something is wrong with your request, and sending it again won't fix it. Cap total attempts so a permanent outage doesn't loop forever.

10. Use a Session: Cookies, Pooling, Defaults

For more than two requests to the same host, use a requests.Session. It reuses the same TCP connection (much faster), keeps cookies between requests (needed for any page behind a login), and lets you set headers in one place.

python
with requests.Session() as s:
    s.headers.update(HEADERS)                    # applied to every request
    s.get("https://example.com/login")           # cookies stored automatically
    r = s.get("https://example.com/dashboard")
    r.raise_for_status()
    process(r.text)
+ setup added so this can run · defines HEADERS, process, requests
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

HEADERS = _AutoMock('HEADERS')
def process(*_a, **_kw):
    print('-> process() called')
    return _AutoMock('process()')
requests = _AutoMock('requests')

On big scrapes, reusing connections is reason enough. A new TCP handshake for every request adds tens of milliseconds and extra CPU work. Add the Retry adapter from Section 9 and you have a solid base to build on.

11. When the Page Is JavaScript-Rendered

requests fetches the initial HTML response. If the page loads its real content with JavaScript after that (as most React, Vue and Svelte single-page apps do), your soup will be almost empty: just a loading shell.

Two signs that you have this problem:

  • view-source: in your browser shows mostly empty <div id="root"> containers.
  • The data you see in DevTools doesn't appear in requests.get(url).text.

What you can do instead:

  • Find the underlying API. Open DevTools → Network → XHR. Often the page gets its data from a JSON endpoint that you can call directly. This is much cleaner than scraping the rendered DOM. (See APIs.)
  • Use a real browser via Playwright or Selenium. playwright is the modern choice: await page.goto(url), wait for the selector, then read the rendered HTML. It is slower and uses more memory, but it is reliable.

requests + BeautifulSoup works for pages whose content is in the HTML the server sends. Pages built by JavaScript need a headless browser, and scripts that ignore this break easily.

12. Saving the Output

Once you've extracted data, save it in a structured format. JSONL is a good default because you can write it one record at a time. CSV is for data that will go into a spreadsheet. Saving the raw HTML is for when you might need to parse the pages again later:

python
import json
from pathlib import Path

out = Path("quotes.jsonl")
with out.open("w", encoding="utf-8") as f:
    for record in records:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")
+ setup added so this can run · defines records
records = ["alpha", "beta", "gamma"]

For tabular output and the full set of pitfalls (encoding, newline modes, streaming), see CSV & JSON. To save the raw HTML, use path.write_bytes(r.content). You can then parse it again offline whenever you need to.

The law on scraping is often unclear, and it differs from country to country. A short, practical checklist:

  • Read the Terms of Service. Many explicitly forbid automated access. Violating them rarely lands you in court, but it can get your account or IP banned.
  • Respect robots.txt (Section 7).
  • Don't republish copyrighted content. Extracting facts is generally OK, but reposting whole articles is not.
  • Avoid personal data. Names, emails, addresses scraped from public pages are still personal data under GDPR and similar laws.
  • Rate-limit yourself so the site barely notices you (Section 8). Costing a small site real money in bandwidth is the fastest way to get blocked.
  • Cache aggressively. If you re-parse the same page in development, save the HTML once and read it from disk on subsequent runs. Each test run will be faster, and the site gets less traffic from you.

When in doubt: would you be embarrassed if the site's admin saw your traffic in their logs? If yes, slow it down or stop.

14. Common Mistakes

1. No User-Agent set
The default python-requests/2.x UA gets a 403 on many real sites. One line of headers prevents it.

2. No rate limiting
Sending requests as fast as possible gets your IP banned and puts load on the site. time.sleep(0.5) minimum.

3. Brittle selectors based on utility classes
.sc-1a3b2c4d-5 will change next deployment. Anchor on semantics, data-*, or stable IDs first.

4. Not checking r.status_code
Parsing an HTML error page silently produces "no results" bugs that take hours to diagnose. r.raise_for_status() or branch explicitly.

5. Hardcoding cookies/session state
Pasting a session cookie into your script works for ten minutes, then expires. Use requests.Session and log in properly, or use the site's API.

6. except: pass around the whole loop
You'll catch the KeyboardInterrupt you sent to stop the runaway scraper. Catch specific exceptions (requests.RequestException for network errors, AttributeError for missing selectors) and log them. See Exceptions.

7. No timeout
requests.get(url) with no timeout= can hang forever if the server stops responding mid-stream. Always set one.

8. Re-fetching during development
While you adjust your selector, every save fetches the page again, so you might hit the site 50 times. Save the HTML to disk once and test against that copy. It is faster and more polite.

🎯 Your Turn: Scrape quotes.toscrape.com

https://quotes.toscrape.com/ is a sandbox site built exactly for this. It has numbered pages at /page/1/, /page/2/, …, and structured quote markup. Write scrape_quotes(url, max_pages=5) that returns a list of dicts:

python
[
    {"quote": "The world as we have created it...",
     "author": "Albert Einstein",
     "tags": ["change", "deep-thoughts", "thinking", "world"]},
    ...
]

Constraints:

  • One-second sleep between page fetches.
  • Send a real User-Agent.
  • Stop early when a page has zero quotes (you've gone past the last one).
  • Set a timeout on every request.

Skeleton:

python
import time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; LearnerBot/1.0)"}

def scrape_quotes(base_url="https://quotes.toscrape.com", max_pages=5):
    results = []
    for page in range(1, max_pages + 1):
        url = f"{base_url}/page/{page}/"
        # TODO 1: GET with headers and timeout; check status
        # TODO 2: parse with BeautifulSoup
        # TODO 3: find each quote block; extract text, author, tags
        # TODO 4: break early if the page yields no quotes
        # TODO 5: sleep 1 second between pages
        ...
    return results
Hint 1: Inspect the markup Each quote is wrapped in <div class="quote">. Inside: <span class="text"> for the quote, <small class="author"> for the author, and <a class="tag"> elements for each tag. Use select with CSS selectors, because they are short and easy to read.
Hint 2: Stripping the curly quotes The site wraps quotes in U+201C / U+201D curly quotes. .text.strip("“”") removes them. Or leave them in and clean the data later.
Show full solution
python
import time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; LearnerBot/1.0)"}

def scrape_quotes(base_url="https://quotes.toscrape.com", max_pages=5):
    """Scrape quotes from quotes.toscrape.com, politely."""
    results = []
    for page in range(1, max_pages + 1):
        url = f"{base_url}/page/{page}/"
        r = requests.get(url, headers=HEADERS, timeout=10)
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "html.parser")
        blocks = soup.select("div.quote")
        if not blocks:
            break                                       # past the last page
        for block in blocks:
            results.append({
                "quote":  block.select_one("span.text").text.strip("“”"),
                "author": block.select_one("small.author").text,
                "tags":   [a.text for a in block.select("a.tag")],
            })
        time.sleep(1)                                   # be polite
    return results


# Demo
quotes = scrape_quotes(max_pages=2)
print(f"scraped {len(quotes)} quotes")
for q in quotes[:2]:
    print(f"- {q['author']}: {q['quote'][:60]}... [{', '.join(q['tags'])}]")

What you did:

  • Sent a real User-Agent in every request, instead of the default python-requests one.
  • Checked the response with raise_for_status() so a 5xx error fails loudly instead of producing empty results.
  • Set a timeout=10 so a stalled server doesn't hang your script forever.
  • Used CSS selectors with select and select_one.
  • Broke early when a page had zero quotes, which means there is no more data.
  • Slept one second between pages, so the load on the site stays small.

Every scraper for a static site has this same structure: loop, fetch, check, parse, extract, sleep. Once you know it, most of the work on a new site is finding the right selectors.

For a real production scraper you'd also: wrap retries with urllib3.util.Retry, store results incrementally (so a crash doesn't lose everything), and cache fetched HTML to disk during development. The lesson above shows how to do all three.

What You Learned

  • The stack: requests for fetching, BeautifulSoup for parsing. Three lines do the simple case.
  • Always set a User-Agent and a timeout. Always check r.status_code or call raise_for_status().
  • Selectors: .find(), .find_all(), .select(), .select_one(). Navigate with .text, .get("href"), .parent.
  • Stable selectors prefer semantic tags, data-* attributes, and structural anchors over utility class names.
  • Pagination: loop ?page=N until empty, or follow a[rel='next'] until missing. Cap iterations.
  • Politeness: respect robots.txt, sleep between requests, rate-limit per domain.
  • Retries: urllib3.util.Retry mounted on a Session, or hand-rolled exponential backoff. Retry 5xx and 429, never other 4xx errors.
  • Sessions reuse connections and keep cookies. Use one whenever you make several requests.
  • JS-rendered pages need Playwright/Selenium, or look for the underlying API (APIs).
  • Save structured output as JSON, CSV or JSONL. See CSV & JSON.
  • Scrape responsibly: check the ToS, copyright and personal data rules, and limit your request rate.

Next: REST APIs. When a site has a JSON endpoint, use the API instead of scraping.

Found this useful? Share it with a friend who’s learning Python.