Extracting an attribute value with beautifulsoup

BeautifulSoup

Web Scraping

Python

HTML Parsing

Data Extraction

Extracting an attribute value with beautifulsoup

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

BeautifulSoup is a Python library for parsing HTML and XML. Extracting attribute values — such as href from links, src from images, or class from any element — is one of its most common uses. You access attributes on a tag object using dictionary-style syntax (tag['href']), the .get() method (tag.get('href')), or the .attrs dictionary. Each approach handles missing attributes differently.

Setup

bash

pip install beautifulsoup4 lxml

python

1from bs4 import BeautifulSoup
2
3html = """
4<html>
5<body>
6  <a href="https://example.com" class="link primary" id="main-link">Example</a>
7  <img src="/images/photo.jpg" alt="A photo" width="300">
8  <div data-user-id="42" data-role="admin">User Info</div>
9</body>
10</html>
11"""
12
13soup = BeautifulSoup(html, "lxml")

Dictionary-Style Access

Access an attribute by name using square brackets. This raises KeyError if the attribute does not exist:

python

1link = soup.find("a")
2
3href = link["href"]
4print(href)  # https://example.com
5
6classes = link["class"]
7print(classes)  # ['link', 'primary'] — class is always a list
8
9link_id = link["id"]
10print(link_id)  # main-link

BeautifulSoup returns multi-valued attributes like class as a list, even if there is only one value.

The .get() Method

.get() returns None (or a default) instead of raising an error when the attribute is missing:

python

1link = soup.find("a")
2
3# Attribute exists
4print(link.get("href"))      # https://example.com
5
6# Attribute does not exist — returns None
7print(link.get("target"))    # None
8
9# Provide a default
10print(link.get("target", "_self"))  # _self

Use .get() when scraping pages where attributes may or may not be present.

The .attrs Dictionary

Every tag has an .attrs dictionary containing all its attributes:

python

1img = soup.find("img")
2print(img.attrs)
3# {'src': '/images/photo.jpg', 'alt': 'A photo', 'width': '300'}
4
5# Iterate over all attributes
6for attr, value in img.attrs.items():
7    print(f"{attr} = {value}")
8# src = /images/photo.jpg
9# alt = A photo
10# width = 300

Extracting Attributes from Multiple Elements

Use find_all() to get attributes from every matching element:

python

1html = """
2<ul>
3  <li><a href="/page1">Page 1</a></li>
4  <li><a href="/page2">Page 2</a></li>
5  <li><a href="/page3">Page 3</a></li>
6</ul>
7"""
8soup = BeautifulSoup(html, "lxml")
9
10# Get all hrefs
11hrefs = [a["href"] for a in soup.find_all("a")]
12print(hrefs)  # ['/page1', '/page2', '/page3']
13
14# With .get() to handle links without href
15hrefs_safe = [a.get("href") for a in soup.find_all("a") if a.get("href")]

Extracting data- Attributes

HTML5 data-* attributes work the same way:

python

1div = soup.find("div", attrs={"data-user-id": True})
2user_id = div["data-user-id"]
3print(user_id)  # 42
4
5role = div.get("data-role")
6print(role)  # admin

You can also search by data attribute value:

python

admin_div = soup.find("div", attrs={"data-role": "admin"})
print(admin_div["data-user-id"])  # 42

CSS Selector Approach

BeautifulSoup supports CSS selectors via .select(), which can filter by attribute:

python

1html = """
2<form>
3  <input type="text" name="username" value="alice">
4  <input type="password" name="password" value="">
5  <input type="hidden" name="csrf" value="abc123">
6</form>
7"""
8soup = BeautifulSoup(html, "lxml")
9
10# Select inputs by type
11hidden = soup.select('input[type="hidden"]')
12for inp in hidden:
13    print(inp["name"], inp["value"])
14# csrf abc123
15
16# Select elements with a specific attribute present
17all_with_value = soup.select("input[value]")

Checking if an Attribute Exists

python

1link = soup.find("a")
2
3# Using .has_attr()
4if link.has_attr("target"):
5    print(link["target"])
6else:
7    print("No target attribute")
8
9# Using 'in' operator
10if "href" in link.attrs:
11    print(link["href"])

Common Pitfalls

Using tag['attr'] on missing attributes: This raises KeyError and crashes the script. Use tag.get('attr') for optional attributes, especially when scraping inconsistent HTML.
Forgetting class returns a list: tag['class'] returns ['btn', 'primary'], not 'btn primary'. To get the string, use ' '.join(tag['class']). Comparing with == 'btn' fails because you are comparing a list to a string.
Attribute values are always strings: img['width'] returns '300' (a string), not 300 (an integer). Cast explicitly with int(img['width']) when you need a number.
Parsing with the wrong parser: The default html.parser may handle malformed HTML differently than lxml or html5lib. If attributes appear missing, try a different parser: BeautifulSoup(html, 'html5lib').
Scraping JavaScript-rendered content: BeautifulSoup parses static HTML only. If attributes are added by JavaScript after page load (e.g., React data-* attributes), BeautifulSoup sees the pre-rendered HTML. Use Selenium or Playwright to get the rendered DOM.

Summary

Use tag['attr'] for direct access (raises KeyError if missing)
Use tag.get('attr') for safe access (returns None if missing)
Use tag.attrs to get all attributes as a dictionary
class attributes always return a list — join with ' '.join() if you need a string
Use find_all() with list comprehensions to extract attributes from multiple elements
CSS selectors via soup.select() provide a powerful alternative for complex attribute queries