Introduction

Getting JSON into a Python script is easy when the URL actually returns JSON, and much less direct when the browser page is ordinary HTML that only happens to display data. The first step is always to identify whether you are calling a JSON endpoint or scraping a page that embeds JSON somewhere inside its markup.

Start with the Real JSON Endpoint

If the server returns JSON directly, use requests and parse the response with response.json():

1import requests
2
3url = "https://api.example.com/items"
4response = requests.get(url, timeout=10)
5response.raise_for_status()
6
7data = response.json()
8print(type(data))
9print(data)

This is the cleanest case. requests handles the HTTP call, raise_for_status() fails fast on bad responses, and .json() converts the payload into Python objects such as dictionaries and lists.

If the endpoint needs query parameters or headers, pass them explicitly:

1import requests
2
3response = requests.get(
4    "https://api.example.com/search",
5    params={"q": "python", "limit": 5},
6    headers={"Accept": "application/json"},
7    timeout=10,
8)
9response.raise_for_status()
10data = response.json()
11print(data)

When the URL Returns HTML Instead

Many people say "webpage" when they really mean "data shown in the browser." Those are not the same thing. If requests.get(url).text contains HTML, then calling .json() will fail because the response body is not JSON.

In that case, inspect the page and find where the data comes from:

• a separate API request made by the page
• a script tag containing embedded JSON
• 'data-* attributes in the HTML'
• a JavaScript application that renders data after page load

If you can find the API request in the browser network tab, call that endpoint directly from Python instead of scraping the rendered page.

Extracting Embedded JSON from HTML

Some sites place JSON inside a script tag. A common example is application state stored for client-side rendering. You can extract it and decode it with the standard json module.

1import json
2import requests
3from bs4 import BeautifulSoup
4
5url = "https://example.com/page"
6html = requests.get(url, timeout=10).text
7
8soup = BeautifulSoup(html, "html.parser")
9script = soup.find("script", id="__NEXT_DATA__")
10
11if script is None:
12    raise RuntimeError("Embedded JSON not found")
13
14data = json.loads(script.string)
15print(data.keys())

The exact selector changes from site to site, but the pattern is stable: fetch HTML, isolate the JSON-containing element, then parse the text with json.loads.

Saving or Transforming the Result

Once the JSON is loaded into Python, treat it like any other dictionary or list. You can inspect fields, filter records, or save a cleaned result:

1import json
2
3with open("items.json", "w", encoding="utf-8") as f:
4    json.dump(data, f, indent=2, ensure_ascii=False)

That is useful when the script is part of a pipeline and you want a stable artifact for later processing.

Handle Errors Explicitly

Real sites fail in predictable ways: timeouts, rate limits, authentication requirements, HTML pages returned instead of JSON, or malformed embedded data. Good scripts make those failures obvious.

A compact defensive pattern is:

1import requests
2
3response = requests.get("https://api.example.com/items", timeout=10)
4response.raise_for_status()
5
6try:
7    data = response.json()
8except ValueError as exc:
9    raise RuntimeError("Response was not valid JSON") from exc

That gives you a clear error instead of silently continuing with bad assumptions.

Common Pitfalls

• Calling .json() on an HTML page just because the browser displays structured data.
• Scraping the rendered page when the browser is already calling a cleaner JSON API behind the scenes.
• Forgetting timeout and letting the script hang on slow responses.
• Ignoring status codes and trying to parse error pages as JSON.
• Assuming every site allows automated access without authentication, rate limiting, or anti-bot controls.

Summary

• First determine whether the target URL returns JSON or ordinary HTML.
• Use requests.get(...).json() when you have a real JSON endpoint.
• If the page is HTML, look for the underlying API call or embedded JSON in script tags.
• Parse embedded JSON with json.loads, not response.json().
• Add timeouts and status checks so failures are obvious and easier to debug.