How can I scroll a web page using selenium webdriver in python?

selenium

python

web scraping

web automation

webdriver

How can I scroll a web page using selenium webdriver in python?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

Selenium WebDriver in Python provides several ways to scroll a web page: executing JavaScript with driver.execute_script(), using keyboard actions via the ActionChains class, or scrolling to specific elements with scrollIntoView(). The most common approach is driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") to scroll to the bottom of the page. For infinite-scroll pages, use a loop that scrolls and waits for new content to load.

Setup

python

1from selenium import webdriver
2from selenium.webdriver.common.by import By
3from selenium.webdriver.common.keys import Keys
4from selenium.webdriver.common.action_chains import ActionChains
5import time
6
7driver = webdriver.Chrome()
8driver.get("https://example.com")

Scroll to Bottom of Page

python

1# Scroll to the very bottom
2driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
3
4# Scroll to the top
5driver.execute_script("window.scrollTo(0, 0);")
6
7# Scroll down by a specific pixel amount
8driver.execute_script("window.scrollBy(0, 500);")
9
10# Scroll up by a specific pixel amount
11driver.execute_script("window.scrollBy(0, -500);")

window.scrollTo(x, y) scrolls to an absolute position. window.scrollBy(x, y) scrolls relative to the current position. document.body.scrollHeight is the total height of the page content.

Scroll to a Specific Element

python

1# Find the element first
2element = driver.find_element(By.ID, "footer")
3
4# Method 1: JavaScript scrollIntoView
5driver.execute_script("arguments[0].scrollIntoView(true);", element)
6
7# Method 2: scrollIntoView with smooth scrolling
8driver.execute_script("arguments[0].scrollIntoView({behavior: 'smooth', block: 'center'});", element)
9
10# Method 3: ActionChains move_to_element
11actions = ActionChains(driver)
12actions.move_to_element(element).perform()

scrollIntoView(true) aligns the element to the top of the viewport. scrollIntoView({block: 'center'}) centers it vertically.

Scroll Using Keyboard Keys

python

1from selenium.webdriver.common.keys import Keys
2
3body = driver.find_element(By.TAG_NAME, "body")
4
5# Page Down
6body.send_keys(Keys.PAGE_DOWN)
7
8# Page Up
9body.send_keys(Keys.PAGE_UP)
10
11# Home (top of page)
12body.send_keys(Keys.HOME)
13
14# End (bottom of page)
15body.send_keys(Keys.END)
16
17# Arrow keys for small scrolls
18body.send_keys(Keys.ARROW_DOWN)
19body.send_keys(Keys.ARROW_DOWN)
20body.send_keys(Keys.ARROW_DOWN)

Infinite Scroll Handling

python

1def scroll_to_bottom_infinite(driver, pause_time=2, max_scrolls=50):
2    """Scroll an infinite-scroll page until no new content loads."""
3    last_height = driver.execute_script("return document.body.scrollHeight")
4    scrolls = 0
5
6    while scrolls < max_scrolls:
7        # Scroll to bottom
8        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
9
10        # Wait for new content to load
11        time.sleep(pause_time)
12
13        # Check if page height increased
14        new_height = driver.execute_script("return document.body.scrollHeight")
15        if new_height == last_height:
16            break  # No new content loaded
17
18        last_height = new_height
19        scrolls += 1
20
21    print(f"Scrolled {scrolls} times")
22
23# Usage
24driver.get("https://example.com/infinite-scroll-page")
25scroll_to_bottom_infinite(driver, pause_time=2)

The loop compares scrollHeight before and after each scroll. When the height stops increasing, all content has loaded.

Scroll Inside a Scrollable Container

python

1# Some pages have scrollable divs instead of page-level scroll
2container = driver.find_element(By.CSS_SELECTOR, ".scrollable-container")
3
4# Scroll the container down
5driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight;", container)
6
7# Scroll the container by a fixed amount
8driver.execute_script("arguments[0].scrollTop += 300;", container)
9
10# Scroll container to top
11driver.execute_script("arguments[0].scrollTop = 0;", container)

When content is inside a scrollable div (with overflow: auto or overflow: scroll), you must scroll that specific element, not the window.

Smooth Scroll with Custom Speed

python

1def smooth_scroll(driver, target_y, step=100, delay=0.05):
2    """Scroll smoothly to a target Y position."""
3    current_y = driver.execute_script("return window.pageYOffset;")
4
5    while current_y < target_y:
6        current_y = min(current_y + step, target_y)
7        driver.execute_script(f"window.scrollTo(0, {current_y});")
8        time.sleep(delay)
9
10# Scroll smoothly to 3000px from top
11smooth_scroll(driver, 3000, step=50, delay=0.02)

Wait for Element After Scrolling

python

1from selenium.webdriver.support.ui import WebDriverWait
2from selenium.webdriver.support import expected_conditions as EC
3
4# Scroll to load content, then wait for a specific element
5driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
6
7element = WebDriverWait(driver, 10).until(
8    EC.presence_of_element_located((By.CSS_SELECTOR, ".lazy-loaded-item"))
9)
10print(element.text)

Use WebDriverWait instead of time.sleep() for more reliable waits. It polls for the element and returns as soon as it appears, up to the timeout.

Common Pitfalls

Using time.sleep() instead of explicit waits: Hard-coded sleeps are unreliable — content may load faster or slower than expected. Use WebDriverWait with expected_conditions for robust waiting after scrolling.
Scrolling the window when content is in a scrollable div: If the target content is inside a container with its own scrollbar, window.scrollTo() has no effect on it. Find the container element and set its scrollTop property instead.
Infinite scroll loop without a max scroll limit: If the page generates content endlessly (e.g., social media feeds), a scroll loop without max_scrolls runs forever. Always include a maximum iteration count.
Not waiting for dynamic content after scrolling: Lazy-loaded images and AJAX content need time to render after scrolling. Scrolling and immediately reading the DOM may miss content that has not loaded yet.
Forgetting that scrollIntoView does not wait for visibility: scrollIntoView() scrolls the page but does not guarantee the element is visible or interactable. Use WebDriverWait with element_to_be_clickable before interacting with the element.

Summary

Use driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") to scroll to the bottom
Use scrollIntoView() or ActionChains.move_to_element() to scroll to a specific element
Handle infinite-scroll pages with a loop that compares scrollHeight before and after each scroll
Scroll inside containers by setting their scrollTop property, not window.scrollTo()
Use WebDriverWait instead of time.sleep() for reliable content detection after scrolling
Set max_scrolls limits to prevent infinite loops on endlessly loading pages