Practice Now
BeautifulSoup
web scraping
Python
href
duplicate question

BeautifulSoup getting href

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Python

Introduction

When scraping HTML with BeautifulSoup, extracting links usually means finding anchor tags and reading their href attributes. The two key points are to select only tags that actually have href, and to decide whether you want raw attribute values, absolute URLs, or only certain kinds of links.

The Basic Pattern

The most direct BeautifulSoup pattern is:

python
1from bs4 import BeautifulSoup
2
3html = """
4<html>
5  <body>
6    <a href="/about">About</a>
7    <a href="https://example.com/contact">Contact</a>
8    <a>No link here</a>
9  </body>
10</html>
11"""
12
13soup = BeautifulSoup(html, "html.parser")
14
15for tag in soup.find_all("a", href=True):
16    print(tag["href"])

Using href=True filters out anchor tags that do not have the attribute.

tag.get("href") Versus tag["href"]

Both patterns are common, but they behave slightly differently.

python
href = tag.get("href")

This returns None if the attribute is missing.

python
href = tag["href"]

This raises KeyError if the attribute is missing.

That is why many scrapers prefer find_all("a", href=True) together with tag["href"], or use tag.get("href") when the HTML is unreliable.

Here is a simple reusable helper.

python
1from bs4 import BeautifulSoup
2
3def extract_hrefs(html: str) -> list[str]:
4    soup = BeautifulSoup(html, "html.parser")
5    return [tag.get("href") for tag in soup.find_all("a", href=True)]
6
7html = """
8<a href="/a">A</a>
9<a href="/b">B</a>
10"""
11
12print(extract_hrefs(html))

This keeps the scraping logic short and easy to test.

Converting Relative URLs to Absolute URLs

Many pages contain relative links. If you want usable full URLs, combine BeautifulSoup with urllib.parse.urljoin.

python
1from bs4 import BeautifulSoup
2from urllib.parse import urljoin
3
4base_url = "https://example.com"
5html = '<a href="/docs/start">Docs</a>'
6
7soup = BeautifulSoup(html, "html.parser")
8links = [urljoin(base_url, tag["href"]) for tag in soup.find_all("a", href=True)]
9print(links)

Without this step, relative links such as /docs/start are not enough on their own.

Real pages often contain mailto:, fragment links, JavaScript pseudo-links, and tracking URLs. Filtering them early makes the result cleaner.

python
1from bs4 import BeautifulSoup
2
3html = """
4<a href="#top">Top</a>
5<a href="mailto:[email protected]">Mail</a>
6<a href="/products">Products</a>
7"""
8
9soup = BeautifulSoup(html, "html.parser")
10links = []
11for tag in soup.find_all("a", href=True):
12    href = tag["href"]
13    if href.startswith("#") or href.startswith("mailto:"):
14        continue
15    links.append(href)
16
17print(links)

This is especially useful when you are building a crawler rather than just collecting every attribute value mechanically.

Handling Duplicates

Pages often repeat the same link several times. If you only want unique results, use a set or preserve order with a small helper.

python
1from bs4 import BeautifulSoup
2
3html = """
4<a href="/about">About</a>
5<a href="/about">About again</a>
6<a href="/contact">Contact</a>
7"""
8
9soup = BeautifulSoup(html, "html.parser")
10links = [tag["href"] for tag in soup.find_all("a", href=True)]
11unique_links = list(dict.fromkeys(links))
12print(unique_links)

This preserves the first-seen order while removing duplicates.

Requests and Parsing Together

In real scraping code, BeautifulSoup is often paired with requests.

python
1import requests
2from bs4 import BeautifulSoup
3
4response = requests.get("https://example.com", timeout=10)
5response.raise_for_status()
6
7soup = BeautifulSoup(response.text, "html.parser")
8links = [tag["href"] for tag in soup.find_all("a", href=True)]
9print(links[:5])

This is the standard workflow for many lightweight scrapers.

Common Pitfalls

The biggest pitfall is calling find_all("a") and then indexing tag["href"] without checking whether the attribute exists. Some anchor tags do not actually contain links.

Another issue is forgetting about relative URLs. A scraper may appear to work while quietly collecting unusable partial paths.

Developers also ignore duplicate links and later wonder why the result list is much larger than expected.

Finally, do not assume static HTML always contains the links you see in the browser. Some pages render links with JavaScript, which BeautifulSoup alone will not execute.

Summary

  • Use soup.find_all("a", href=True) to extract anchor tags that actually have href.
  • Choose tag.get("href") or tag["href"] based on how defensive you want to be.
  • Convert relative links with urljoin when you need full URLs.
  • Filter fragments, mail links, and duplicates when your scraper needs cleaner output.
  • BeautifulSoup parses HTML well, but it does not execute JavaScript-rendered pages by itself.
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.