Selenium Scraping in Python: A Practical Selenium 4 Tutorial

Selenium scraping means using a real, script-controlled browser to load a page, let its JavaScript run, and read the finished content. To do it in Python, install Selenium with pip, start Chrome with webdriver.Chrome(), find elements with find_element(By.CSS_SELECTOR, ...), wait for content with WebDriverWait, and loop through pages.

Use it when a page only shows its data after JavaScript runs or after you click something. If the data is already in the HTML, a plain HTTP request is faster and lighter, and you don't need a browser at all.

This tutorial builds one working scraper from start to finish. By the end you'll have a script that:

• Opens a JavaScript-rendered site in headless Chrome
• Waits for the content instead of guessing with sleep
• Follows the Next link across pages
• Saves the results to a CSV

All the code runs against Quotes to Scrape, a sandbox built for people learning scraping, so you can practise without touching anyone's production site.

When Selenium is the right scraping tool

Selenium is a browser automation tool, widely used for testing. For scraping, the trade-off is simple: you get a complete browser, and you pay for it in speed and memory.

Three situations justify that cost:

• The content is rendered by JavaScript. The first HTML response is an empty shell, and scripts fill it in.
• You have to interact. The data appears only after clicking a tab, a filter, or a Load More button.
• You're already a Selenium shop. Your team has test code and infrastructure, and reusing it beats learning a new stack.

Here's how to see the first case for yourself. The sandbox has two versions of the same page. Fetching each with requests and counting the quote cards shows the difference. This check needs two extra packages, so run pip install requests beautifulsoup4 first:

import requests
from bs4 import BeautifulSoup

for url in ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/js/"]:
    html = requests.get(url, timeout=10).text
    cards = BeautifulSoup(html, "html.parser").select("div.quote")
    print(url, len(cards))

The plain page returns 10 cards. The /js/ page returns 0, because the quotes don't exist in the HTML until a browser runs the page's script. That second page is the one that needs Selenium, so it's the one this tutorial scrapes.

Looking for the wider map of browser automation tools? Our guide to web automation tools compares frameworks, recorders, workflow tools, and AI agents. This post stays on one job: writing a scraper in Selenium.

Set up Selenium 4 in Python

You need Python and a browser. The current Selenium release requires Python 3.10 or newer.

python3 -m venv .venv
source .venv/bin/activate
pip install selenium

That's the whole install. You don't have to download ChromeDriver by hand, because Selenium Manager has shipped with Selenium since version 4.6 and handles it for you. Its docs also say it can discover, download, and cache Chrome, Firefox, or Edge if the browser isn't installed.

Old tutorials still rank with Selenium 3 code, and two patterns fail on a current install:

Find elements with By locators

Selenium 4 has two lookup methods. find_element returns the first match and raises NoSuchElementException if nothing matches. find_elements returns a list, which is empty if nothing matches.

Here's the smallest useful script:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://quotes.toscrape.com/")

for card in driver.find_elements(By.CSS_SELECTOR, "div.quote"):
    text = card.find_element(By.CSS_SELECTOR, "span.text").text
    author = card.find_element(By.CSS_SELECTOR, "small.author").text
    print(author, "-", text)

driver.quit()

Notice that the inner lookups run on card, not on driver. Searching inside an element limits the search to that element, which keeps each author paired with the right quote.

To find a selector, right-click the content in Chrome, choose Inspect, and look at the surrounding tags. CSS selectors cover most scraping. By.ID is the most stable when a useful ID exists, and By.XPATH helps when you need to match on visible text.

Pick selectors that describe meaning, such as div.quote. Long auto-generated class names often change on the next deploy, and your scraper breaks with them.

Wait for content instead of sleeping

Most beginner Selenium bugs are timing bugs. The script looks for an element a moment before JavaScript has created it, and it fails even though the page looks fine when you open it yourself.

Selenium's waiting strategies give you two main fixes:

• Implicit wait: a global setting that makes every element lookup retry for a set time.
• Explicit wait: a loop that polls for one specific condition, such as "this element is present."

Use explicit waits for scraping. They say what you're waiting for, and they return the moment the condition is met:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote")))

If the quotes don't appear within 10 seconds, this raises TimeoutException.

Pick one approach and stick with it. Selenium's docs warn against mixing the two, saying that an implicit wait of 10 seconds with an explicit wait of 15 seconds could time out after 20 seconds.

Handle pagination and save to CSV

Pagination adds a trap. After you click Next, the old page's elements are still on screen for a moment. If you read straight away, you can scrape page 1 twice.

The fix is to keep a reference to an element from the current page, click, and wait for that element to go stale. EC.staleness_of tells you the old element has left the page, and then you can wait for the new content.

Here's the complete scraper, with headless mode switched on. It stops after three pages:

import csv
import time

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1280,900")

driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 10)
rows = []

try:
    driver.get("https://quotes.toscrape.com/js/")

    for page in range(1, 4):
        wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote")))

        for card in driver.find_elements(By.CSS_SELECTOR, "div.quote"):
            rows.append({
                "text": card.find_element(By.CSS_SELECTOR, "span.text").text,
                "author": card.find_element(By.CSS_SELECTOR, "small.author").text,
                "page": page,
            })

        try:
            next_link = driver.find_element(By.CSS_SELECTOR, "li.next a")
        except NoSuchElementException:
            break  # no Next link means this was the last page

        old_card = driver.find_element(By.CSS_SELECTOR, "div.quote")
        next_link.click()
        wait.until(EC.staleness_of(old_card))
        time.sleep(2)  # a polite pause between pages
finally:
    driver.quit()

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["text", "author", "page"])
    writer.writeheader()
    writer.writerows(rows)

print(f"Saved {len(rows)} quotes")

Run it and you get 30 unique quotes in quotes.csv, 10 from each page.

Three details are worth copying into your own scripts:

• try and finally: driver.quit() runs even if the scrape crashes, so you don't leave stray browser processes behind.
• A page cap: range(1, 4) stops a bug from turning into an endless crawl. Raise it once the script works.
• time.sleep(2): this isn't there for timing, since the waits handle that. It's there to slow your request rate.

Run Selenium in headless mode

Headless mode runs the browser with no visible window, which is what you want on a server or in a scheduled job. The script above sets it with one argument:

options.add_argument("--headless=new")

You may also see plain --headless in other tutorials. It works too. Since Chrome 132, the old headless mode lives in a separate chrome-headless-shell binary, so --headless on regular Chrome no longer gets you the old behavior. --headless=new still runs fine on current Chrome.

You may see options.headless = True in older code too. Selenium deprecated that convenience method in 4.8, and the Python changelog records its removal in 4.13. On a current install that line raises no error and does nothing, so Chrome simply opens a visible window. Set the mode through an argument instead.

While you're writing a scraper, leave headless off. Watching the real browser shows you cookie banners, overlays, and layout changes that explain most failures. Switch it on once the script works.

Selenium vs requests and Beautiful Soup vs Playwright

Selenium is often the wrong choice, and it's worth checking before you commit. Work through these in order.

Step 1: look for an API. Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload the page. Many JavaScript sites load their data from a JSON endpoint. The sandbox's infinite-scroll page does exactly that: https://quotes.toscrape.com/api/quotes?page=1 returns clean JSON. Check the site's terms before you rely on an endpoint like this, and prefer a documented public API when one exists.

Step 2: try requests with Beautiful Soup. If the data is in the raw HTML, this is the best option. There's no browser to start, so it's much faster and easier to run at volume.

Step 3: use a browser only when the first two fail. Then you're choosing between Selenium and Playwright.

For a brand-new scraper, Playwright's auto-waiting means fewer timing bugs out of the box. Selenium still makes sense if your team already runs it, if you want to reuse automated testing tools and Selenium Grid for remote browsers, or if you work in Ruby, which Selenium's install docs cover and Playwright's language list doesn't.

Some people don't want to write a scraper at all. If your goal is turning pages into markdown for an AI model, a hosted or open-source crawler will get you there faster. Our list of Firecrawl alternatives covers those options. If you want an AI agent to drive the browser, see our Playwright MCP server guide.

Scrape responsibly: robots.txt, terms, and rate limits

Before you point Selenium at a real site, go through this list.

Read robots.txt. This file tells crawlers which paths a site owner wants them to stay out of. The Robots Exclusion Protocol states that these rules are not a form of access authorization, so it isn't a lock. Treat it as a clear request and follow it. Python's built-in robotparser can check a URL for you:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://quotes.toscrape.com/robots.txt")
rp.read()
print(rp.can_fetch("my-scraper", "https://quotes.toscrape.com/page/2/"))

This prints True because the sandbox has no robots.txt, and a missing file leaves the parser with nothing to block. On a real site, swap in that site's URLs.

Read the terms of service. Many sites restrict automated access or commercial reuse of their content. A robots.txt that allows a path doesn't override the terms. If you're scraping for a business, or you're unsure what's allowed, ask a lawyer.

Stay out of logged-in and personal data. Content behind a login usually comes with extra terms, and personal data brings privacy law into play. If you only need your own account's data, look for an export or an API first.

Keep the load low. Add delays, cap your page count, cache what you've fetched, and run outside the site's busy hours. A browser loads images, scripts, and fonts as well as the page, so Selenium costs the site more than a plain HTTP request does.

Treat a block as an answer. This guide doesn't cover getting past CAPTCHAs or bot detection, because those exist to tell you the owner doesn't want automated traffic. If you hit one, stop. Look for an official API or a licensed data feed, or contact the site owner and ask.

Fixing common Selenium scraping errors

These four errors cover most of what you'll see:

If a selector works in DevTools but fails in the script, check whether the content sits inside an iframe. Selenium needs an explicit driver.switch_to.frame(...) before it can see inside one.

FAQ

What is Selenium scraping?

It's using Selenium to control a real browser, load a web page, and extract data from it. Because the browser runs JavaScript, Selenium can read content that a plain HTTP request never receives.

Is Selenium good for web scraping?

It's a good fit for pages that need JavaScript or interaction, and a poor fit for static pages. A browser uses far more time and memory per page than requests with Beautiful Soup, so Selenium is the wrong tool for large crawls of simple HTML.

Is Selenium or Beautiful Soup better for scraping?

They do different jobs, so it depends on the page. Beautiful Soup parses HTML but can't run a browser, and Selenium runs a browser but has simpler parsing. You can combine them: load the page with Selenium, then pass driver.page_source to Beautiful Soup.

Is Selenium or Playwright better for scraping?

For new projects, Playwright is usually the easier start because it waits automatically before acting on elements. Selenium is a sensible choice if you already have Selenium code, need its language coverage, or want to use Selenium Grid.

Do I still need to download ChromeDriver?

No. Since Selenium 4.6, Selenium Manager comes with Selenium and sets up the driver automatically. A plain webdriver.Chrome() is enough on a normal setup.

Is web scraping with Selenium legal?

It depends on the site, the data, your location, and what you do with the results. Check robots.txt and the terms of service, avoid personal data, and get legal advice for commercial projects.

Why does my scraper work with a visible browser but fail in headless mode?

A common cause is window size. A small default window can trigger a mobile layout that hides elements, so set --window-size explicitly. Some sites also treat automated traffic differently. If that's what is happening, respect the block and don't try to work around it.

Key points

• Selenium scraping uses a real browser, so reserve it for pages that need JavaScript or interaction
• Check for an API first, then try requests and Beautiful Soup, and start a browser only if both fall short
• Use Selenium 4 syntax: find_element(By...), plus Selenium Manager instead of manual driver downloads
• Use explicit waits and EC.staleness_of for pagination, and don't mix them with implicit waits
• Set headless with --headless=new, and develop with the window visible
• Respect robots.txt and site terms, keep your request rate low, and stop when a site blocks you

A scraper is one more script to install, schedule, and repair when a site changes. If you'd rather start from an AI workflow someone else already got running, Taku is an AI-native desktop workspace where you can mirror a working setup, remix it for your own task, and keep it. Taku is in Beta, and the Mac app is available now.