Practice Now
Web Scraping
Python
Beautiful Soup
HTML Parsing
Data Extraction

Beautiful Soup and extracting a div and its contents by ID

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
PythonJavaScript & Web

Introduction

One of the most common scraping tasks with Beautiful Soup is locating a specific div by its id and then extracting either its text or its nested HTML. Because an HTML id should be unique on a page, this is usually a clean and reliable way to target a section of the document.

Find The div By id

Beautiful Soup can search directly by id using find. You do not have to limit the tag name, but doing so can make the intent clearer.

python
1from bs4 import BeautifulSoup
2
3html = """
4<html>
5  <body>
6    <div id="header">Welcome</div>
7    <div id="content">
8      <p>This is the main content.</p>
9      <p>It has multiple paragraphs.</p>
10    </div>
11  </body>
12</html>
13"""
14
15soup = BeautifulSoup(html, "html.parser")
16content_div = soup.find("div", id="content")
17
18print(content_div)

If the element exists, content_div is a Beautiful Soup tag object that you can inspect further. If it does not exist, the result is None, so plan for that case before chaining more lookups.

Extract Text Versus Inner HTML

After you have the div, the next question is usually what kind of content you need.

If you want plain text:

python
if content_div is not None:
    print(content_div.get_text(separator=" ", strip=True))

That produces one text string with tags removed and whitespace normalized.

If you want the HTML inside the div, use decode_contents():

python
if content_div is not None:
    print(content_div.decode_contents())

This returns the inner markup without the outer div tag itself. That distinction matters when you want to preserve nested tags such as links, lists, or formatting.

Use CSS Selectors When They Read Better

Beautiful Soup also supports CSS selectors through select_one. For id lookups, the selector syntax is concise:

python
1from bs4 import BeautifulSoup
2
3soup = BeautifulSoup(html, "html.parser")
4content_div = soup.select_one("#content")
5
6if content_div is not None:
7    print(content_div.get_text(" ", strip=True))

find("div", id="content") and select_one("#content") are both valid. The better one is usually the one that makes the next developer understand the query fastest.

Extract Nested Elements After You Find The Section

Once you have the target div, you can treat it as a smaller document and search inside it.

python
1from bs4 import BeautifulSoup
2
3html = """
4<div id="product-list">
5  <div class="product">
6    <h3>Widget</h3>
7    <span class="price">$9.99</span>
8  </div>
9  <div class="product">
10    <h3>Gadget</h3>
11    <span class="price">$24.99</span>
12  </div>
13</div>
14"""
15
16soup = BeautifulSoup(html, "html.parser")
17product_list = soup.find("div", id="product-list")
18
19if product_list is not None:
20    for product in product_list.find_all("div", class_="product"):
21        name = product.find("h3").get_text(strip=True)
22        price = product.find("span", class_="price").get_text(strip=True)
23        print(name, price)

This pattern is useful because it reduces accidental matches elsewhere in the page. You first isolate the relevant section, then search within that scope.

Handle Real Pages Carefully

When scraping an actual site, the usual flow is requests plus Beautiful Soup:

python
1import requests
2from bs4 import BeautifulSoup
3
4response = requests.get("https://example.com", timeout=10)
5response.raise_for_status()
6
7soup = BeautifulSoup(response.text, "html.parser")
8main_content = soup.find("div", id="main-content")
9
10if main_content is not None:
11    print(main_content.get_text(" ", strip=True))

If the section is missing even though you can see it in the browser, the site may be rendering content with JavaScript after the initial HTML response. In that case, Beautiful Soup is parsing the server response correctly, but the data you want never existed in that HTML.

Common Pitfalls

  • Calling .text or .find_all() on the result without checking whether find() returned None.
  • Expecting JavaScript-rendered content to appear in static HTML fetched with requests.
  • Using get_text(strip=True) when you actually needed the nested HTML structure.
  • Assuming every page follows valid HTML rules and that every id is unique in practice.
  • Searching the entire document repeatedly instead of narrowing the search to the target div first.

Summary

  • Use find("div", id="...") or select_one("#...") to locate a div by id.
  • Use get_text() when you want plain text and decode_contents() when you want inner HTML.
  • After finding the target section, search inside it for nested elements.
  • Always handle the missing-element case and remember that Beautiful Soup only sees the HTML it was given.
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.