Practice Now
BeautifulSoup
Web Scraping
Python
HTML Parsing
Data Extraction

Extracting an attribute value with beautifulsoup

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
PythonJavaScript & Web

Introduction

BeautifulSoup is a Python library for parsing HTML and XML. Extracting attribute values — such as href from links, src from images, or class from any element — is one of its most common uses. You access attributes on a tag object using dictionary-style syntax (tag['href']), the .get() method (tag.get('href')), or the .attrs dictionary. Each approach handles missing attributes differently.

Setup

bash
pip install beautifulsoup4 lxml
python
1from bs4 import BeautifulSoup
2
3html = """
4<html>
5<body>
6  <a href="https://example.com" class="link primary" id="main-link">Example</a>
7  <img src="/images/photo.jpg" alt="A photo" width="300">
8  <div data-user-id="42" data-role="admin">User Info</div>
9</body>
10</html>
11"""
12
13soup = BeautifulSoup(html, "lxml")

Dictionary-Style Access

Access an attribute by name using square brackets. This raises KeyError if the attribute does not exist:

python
1link = soup.find("a")
2
3href = link["href"]
4print(href)  # https://example.com
5
6classes = link["class"]
7print(classes)  # ['link', 'primary'] — class is always a list
8
9link_id = link["id"]
10print(link_id)  # main-link

BeautifulSoup returns multi-valued attributes like class as a list, even if there is only one value.

The .get() Method

.get() returns None (or a default) instead of raising an error when the attribute is missing:

python
1link = soup.find("a")
2
3# Attribute exists
4print(link.get("href"))      # https://example.com
5
6# Attribute does not exist — returns None
7print(link.get("target"))    # None
8
9# Provide a default
10print(link.get("target", "_self"))  # _self

Use .get() when scraping pages where attributes may or may not be present.

The .attrs Dictionary

Every tag has an .attrs dictionary containing all its attributes:

python
1img = soup.find("img")
2print(img.attrs)
3# {'src': '/images/photo.jpg', 'alt': 'A photo', 'width': '300'}
4
5# Iterate over all attributes
6for attr, value in img.attrs.items():
7    print(f"{attr} = {value}")
8# src = /images/photo.jpg
9# alt = A photo
10# width = 300

Extracting Attributes from Multiple Elements

Use find_all() to get attributes from every matching element:

python
1html = """
2<ul>
3  <li><a href="/page1">Page 1</a></li>
4  <li><a href="/page2">Page 2</a></li>
5  <li><a href="/page3">Page 3</a></li>
6</ul>
7"""
8soup = BeautifulSoup(html, "lxml")
9
10# Get all hrefs
11hrefs = [a["href"] for a in soup.find_all("a")]
12print(hrefs)  # ['/page1', '/page2', '/page3']
13
14# With .get() to handle links without href
15hrefs_safe = [a.get("href") for a in soup.find_all("a") if a.get("href")]

Extracting data- Attributes

HTML5 data-* attributes work the same way:

python
1div = soup.find("div", attrs={"data-user-id": True})
2user_id = div["data-user-id"]
3print(user_id)  # 42
4
5role = div.get("data-role")
6print(role)  # admin

You can also search by data attribute value:

python
admin_div = soup.find("div", attrs={"data-role": "admin"})
print(admin_div["data-user-id"])  # 42

CSS Selector Approach

BeautifulSoup supports CSS selectors via .select(), which can filter by attribute:

python
1html = """
2<form>
3  <input type="text" name="username" value="alice">
4  <input type="password" name="password" value="">
5  <input type="hidden" name="csrf" value="abc123">
6</form>
7"""
8soup = BeautifulSoup(html, "lxml")
9
10# Select inputs by type
11hidden = soup.select('input[type="hidden"]')
12for inp in hidden:
13    print(inp["name"], inp["value"])
14# csrf abc123
15
16# Select elements with a specific attribute present
17all_with_value = soup.select("input[value]")

Checking if an Attribute Exists

python
1link = soup.find("a")
2
3# Using .has_attr()
4if link.has_attr("target"):
5    print(link["target"])
6else:
7    print("No target attribute")
8
9# Using 'in' operator
10if "href" in link.attrs:
11    print(link["href"])

Common Pitfalls

  • Using tag['attr'] on missing attributes: This raises KeyError and crashes the script. Use tag.get('attr') for optional attributes, especially when scraping inconsistent HTML.
  • Forgetting class returns a list: tag['class'] returns ['btn', 'primary'], not 'btn primary'. To get the string, use ' '.join(tag['class']). Comparing with == 'btn' fails because you are comparing a list to a string.
  • Attribute values are always strings: img['width'] returns '300' (a string), not 300 (an integer). Cast explicitly with int(img['width']) when you need a number.
  • Parsing with the wrong parser: The default html.parser may handle malformed HTML differently than lxml or html5lib. If attributes appear missing, try a different parser: BeautifulSoup(html, 'html5lib').
  • Scraping JavaScript-rendered content: BeautifulSoup parses static HTML only. If attributes are added by JavaScript after page load (e.g., React data-* attributes), BeautifulSoup sees the pre-rendered HTML. Use Selenium or Playwright to get the rendered DOM.

Summary

  • Use tag['attr'] for direct access (raises KeyError if missing)
  • Use tag.get('attr') for safe access (returns None if missing)
  • Use tag.attrs to get all attributes as a dictionary
  • class attributes always return a list — join with ' '.join() if you need a string
  • Use find_all() with list comprehensions to extract attributes from multiple elements
  • CSS selectors via soup.select() provide a powerful alternative for complex attribute queries
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.