Introduction

If a page builds its content with JavaScript, a plain HTTP request plus HTML parsing often is not enough because the data may not exist in the initial HTML response. The practical solution is to either reproduce the underlying network calls directly or use a browser automation tool that can execute the page's JavaScript. Which approach is better depends on whether the site exposes a clean API behind the scenes.

First Check Whether You Need a Browser at All

Before reaching for Selenium or Playwright, inspect the page's network traffic in the browser developer tools. Many “dynamic” pages fetch JSON from an API endpoint after load. If you can call that endpoint directly, scraping becomes simpler and faster than controlling a full browser.

A plain requests example might then be enough:

1import requests
2
3response = requests.get("https://example.com/api/items")
4response.raise_for_status()
5data = response.json()
6print(data)

This is usually better than rendering a whole browser page if the real data source is already an HTTP API.

Use Browser Automation When the DOM Really Is Dynamic

If the content is assembled entirely in the browser and there is no simple API path to reuse, use a browser automation library.

Playwright is a strong modern option:

1from playwright.sync_api import sync_playwright
2
3with sync_playwright() as p:
4    browser = p.chromium.launch(headless=True)
5    page = browser.new_page()
6    page.goto("https://example.com")
7    page.wait_for_selector(".product-card")
8
9    items = page.locator(".product-card").all_text_contents()
10    print(items)
11
12    browser.close()

This works because Playwright drives a real browser engine, so JavaScript, DOM updates, and async rendering all happen normally.

Selenium Works Too

Selenium is also common and widely supported.

1from selenium import webdriver
2from selenium.webdriver.common.by import By
3from selenium.webdriver.support.ui import WebDriverWait
4from selenium.webdriver.support import expected_conditions as EC
5
6options = webdriver.ChromeOptions()
7options.add_argument("--headless=new")
8driver = webdriver.Chrome(options=options)
9
10driver.get("https://example.com")
11WebDriverWait(driver, 10).until(
12    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
13)
14
15items = driver.find_elements(By.CSS_SELECTOR, ".product-card")
16print([item.text for item in items])
17
18driver.quit()

The most important part is not the library name. It is waiting for the real content to appear instead of scraping the DOM immediately after navigation.

Wait for the Right Signal

Dynamic pages often render in stages. If you scrape too early, you will capture placeholders, loaders, or incomplete markup.

That is why you should wait for a meaningful selector, response, or state change rather than using a blind sleep whenever possible.

Good signals include:

• a specific element appears
• a loading spinner disappears
• a known API response completes
• the page URL or title changes after client-side navigation

Explicit waits are far more reliable than arbitrary delays.

Parsing the Rendered HTML Is Still an Option

Once the browser has rendered the page, you can still hand the final HTML to BeautifulSoup if you prefer that parsing style.

1from bs4 import BeautifulSoup
2
3html = page.content()
4soup = BeautifulSoup(html, "html.parser")
5print(soup.select_one(".product-card").get_text(strip=True))

This can be a clean hybrid approach: let the browser execute JavaScript, then parse the resulting HTML with familiar tools.

Consider Stability, Cost, and Legality

Browser automation is powerful but heavier than direct HTTP requests. It uses more CPU and memory, and it is easier to break when the site's frontend changes.

Also consider:

• rate limiting
• robots and terms of service
• authentication requirements
• anti-bot defenses
• whether scraping is actually allowed for the target site

A technically possible scraper is not automatically a scraper you should run carelessly.

Common Pitfalls

The most common mistake is trying to parse the initial HTML with requests and BeautifulSoup when the real data is injected later by JavaScript.

Another mistake is jumping straight to a browser tool without first checking whether the site already exposes a cleaner JSON endpoint.

Developers also use time.sleep() everywhere instead of waiting for a real rendering signal, which makes the scraper slower and less reliable.

Summary

• Dynamic content often requires either API-level scraping or browser automation.
• Check developer tools first to see whether the page fetches JSON behind the scenes.
• Use Playwright or Selenium when JavaScript execution is necessary.
• Wait for meaningful page state, not just arbitrary delays.
• Browser automation is powerful, but it is heavier and more fragile than direct HTTP scraping.