Practice Now
XPath
Python
XML Parsing
Programming Tutorial
Data Extraction

How to use XPath in Python?

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Python

Introduction

XPath is a query language for selecting nodes from XML and XML-like documents. In Python, it is most commonly used with lxml to extract data from XML feeds, HTML pages, and configuration files. The important part is not memorizing every XPath feature, but knowing how to combine a parser with a few reliable query patterns.

Choosing the Right Python Library

If you need real XPath support, lxml is the usual choice. The built-in xml.etree.ElementTree module supports only a limited subset of XPath, so many examples that work in lxml will not work there.

Install lxml with:

bash
python -m pip install lxml

For HTML, use lxml.html. For XML, use lxml.etree.

Basic XPath Patterns

A few patterns cover most day-to-day usage:

  • '/catalog/book selects an absolute path from the document root.'
  • '//book selects matching nodes anywhere in the document.'
  • '//book[@id='b2'] filters by attribute value.'
  • '//book/title/text() extracts text content.'
  • '//book[contains(@class, 'featured')] filters with a function.'

XPath queries usually return a list. That matters because even a query that matches one element still comes back as a sequence.

Working With HTML

For scraping or internal HTML processing, parse the document and then call .xpath().

python
1from lxml import html
2
3markup = """
4<html>
5  <body>
6    <ul id="products">
7      <li data-sku="A1">Keyboard</li>
8      <li data-sku="B2">Mouse</li>
9      <li data-sku="C3">Monitor</li>
10    </ul>
11  </body>
12</html>
13"""
14
15doc = html.fromstring(markup)
16
17names = doc.xpath("//ul[@id='products']/li/text()")
18skus = doc.xpath("//ul[@id='products']/li/@data-sku")
19second_name = doc.xpath("(//ul[@id='products']/li)[2]/text()")
20
21print(names)
22print(skus)
23print(second_name)

This example shows three common result types:

  • text node lists
  • attribute value lists
  • positional selection

If you expect a single value, you still need to handle the fact that XPath returns a sequence unless the expression itself returns a scalar.

Working With XML and Namespaces

Namespaces are where many XPath queries fail. If the XML uses a namespace, you must register a prefix in your query call, even if the original document uses a default namespace.

python
1from lxml import etree
2
3xml_text = """
4<feed xmlns="http://example.com/feed">
5  <item id="101">
6    <name>Alpha</name>
7  </item>
8  <item id="102">
9    <name>Beta</name>
10  </item>
11</feed>
12"""
13
14root = etree.fromstring(xml_text.encode("utf-8"))
15ns = {"f": "http://example.com/feed"}
16
17ids = root.xpath("//f:item/@id", namespaces=ns)
18names = root.xpath("//f:item/f:name/text()", namespaces=ns)
19
20print(ids)
21print(names)

Without the namespaces mapping, both queries return empty results, which often looks like a parser bug when it is really a namespace issue.

Returning Elements Versus Text

One useful distinction is whether you want nodes or values.

  • '//book/title returns element objects.'
  • '//book/title/text() returns strings.'
  • 'string(//book/title) returns a scalar string.'

If you need to inspect attributes or child nodes later, return elements. If you only need values, extract text directly and keep downstream code simpler.

When XPath Is a Good Fit

XPath is especially good when the document already has stable tree structure and you need concise selection logic. It is less attractive when the structure is inconsistent or when you need complex procedural cleanup after parsing.

A practical rule:

  • Use XPath for structural selection.
  • Use normal Python code for validation, conversion, and business rules.

That split keeps expressions readable and prevents giant XPath strings from becoming hard to maintain.

Common Pitfalls

  • Using xml.etree.ElementTree and expecting full XPath support. Use lxml when you need real XPath features.
  • Forgetting that many XPath queries return lists. Check for empty results before indexing.
  • Ignoring XML namespaces and then debugging empty matches. Register namespace prefixes explicitly.
  • Overwriting simple Python logic with very complex XPath expressions. Keep XPath focused on selection.
  • Treating scraped HTML as perfectly structured. Defensive result checking is still necessary.

Summary

  • 'lxml is the standard Python choice for full XPath support.'
  • Use .xpath() on parsed HTML or XML documents.
  • Learn a few reliable patterns such as //, attribute filters, and text().
  • Handle namespaces explicitly in XML queries.
  • Keep XPath for node selection and do the rest of the work in Python.
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.