Introduction

The reliable way to extract text from HTML in Python is to parse the document and then select the text you actually want. In most cases, BeautifulSoup is the easiest tool to start with, while lxml is a strong choice when speed or XPath support matters.

Parse the HTML with BeautifulSoup

For a local HTML file, the basic workflow is straightforward:

1from bs4 import BeautifulSoup
2
3with open("page.html", "r", encoding="utf-8") as f:
4    html = f.read()
5
6soup = BeautifulSoup(html, "html.parser")
7text = soup.get_text(separator=" ", strip=True)
8
9print(text)

get_text walks the parsed tree and joins all text nodes together. The separator argument helps preserve readability instead of collapsing everything into one long string.

Remove Script and Style Content First

If you want visible page text rather than raw document text, remove tags such as script, style, and noscript before extracting:

1from bs4 import BeautifulSoup
2
3with open("page.html", "r", encoding="utf-8") as f:
4    soup = BeautifulSoup(f, "html.parser")
5
6for tag in soup(["script", "style", "noscript"]):
7    tag.decompose()
8
9text = soup.get_text(separator=" ", strip=True)
10print(text)

This avoids mixing JavaScript or CSS into the output.

Extract Only the Relevant Section

Often you do not want the entire page. If the main content lives inside an article or a known div, select that element first:

1article = soup.find("article")
2if article is not None:
3    text = article.get_text(separator="\n", strip=True)
4    print(text)

Targeted extraction is usually better for news articles, documentation pages, and blog posts because it avoids navigation menus, headers, and footers.

Use lxml for Faster Parsing

If performance matters or you prefer XPath, lxml works well:

1from lxml import html
2
3with open("page.html", "r", encoding="utf-8") as f:
4    tree = html.fromstring(f.read())
5
6text = tree.text_content()
7print(text)

lxml is fast and powerful, especially for larger parsing workloads.

Use Selectors When the Page Structure Is Known

If the HTML has predictable structure, selecting elements directly is often better than flattening the entire page:

titles = soup.select("h1, h2")
for title in titles:
    print(title.get_text(strip=True))

This works well for repeated content blocks such as product cards, search results, or article headings.

It also keeps the extracted text closer to the data you actually care about, which reduces the amount of cleanup needed later.

Normalize Whitespace After Extraction

HTML-derived text often contains repeated spaces and line breaks. A cleanup step makes the result easier to store or process:

1import re
2
3cleaned = re.sub(r"\s+", " ", text).strip()
4print(cleaned)

This is useful before feeding text into search indexes, NLP pipelines, or database records.

Common Pitfalls

The biggest mistake is trying to parse HTML with regular expressions alone. HTML nesting and malformed markup make regex-only approaches fragile quickly.

Another common issue is extracting the whole page when you only need the article body. That often pulls in unrelated text such as cookie banners and navigation labels.

People also forget about character encoding. If the file is not UTF-8, opening it with the wrong encoding can produce decoding errors or corrupted text.

Finally, remember that different parsers may handle broken HTML slightly differently. If the results look strange, try switching between html.parser, lxml, and html5lib if available.

Summary

• Use a parser such as BeautifulSoup or lxml instead of regex-only extraction.
• 'get_text is the simplest way to collect visible text from a parsed document.'
• Remove script and style tags before extracting text for cleaner output.
• Target specific elements when you need only the main content.
• Normalize whitespace after extraction so the result is easier to use downstream.