Introduction

Making an HTTP request and turning the response into Python data is one of the most common tasks in backend work, scripting, and API integrations. In practice, the job is not just "download text and call json.loads", because you also need timeouts, status checks, and validation that the server actually returned JSON.

Making an HTTP request with requests

The most widely used library for this is requests. A basic GET request looks simple, but the important pieces are the timeout and status handling.

1import requests
2
3url = "https://jsonplaceholder.typicode.com/users/1"
4
5response = requests.get(url, timeout=10)
6response.raise_for_status()
7
8print(response.status_code)
9print(response.headers["Content-Type"])
10print(response.text[:80])

raise_for_status() is worth using by default. Without it, a 404 or 500 response still gives you a Response object, and your code may try to parse an error page as if it were valid data.

For query parameters and headers, pass dictionaries instead of building the URL by hand:

1import requests
2
3response = requests.get(
4    "https://httpbin.org/get",
5    params={"page": 2, "limit": 20},
6    headers={"Accept": "application/json"},
7    timeout=10,
8)
9response.raise_for_status()
10
11print(response.url)

This is safer and easier to maintain than manual string concatenation.

Parsing JSON safely

If the server returns JSON, use response.json() instead of json.loads(response.text). It is shorter and communicates intent clearly.

1import requests
2
3response = requests.get("https://jsonplaceholder.typicode.com/todos/1", timeout=10)
4response.raise_for_status()
5
6data = response.json()
7
8print(type(data))
9print(data["title"])
10print(data["completed"])

The parsed value is standard Python data, usually a dict or list. At that point you can treat it like any other object from Python code.

If you need to serialize Python data back into JSON for a request body, use the json= parameter:

1import requests
2
3payload = {
4    "name": "Ada",
5    "role": "admin",
6}
7
8response = requests.post(
9    "https://httpbin.org/post",
10    json=payload,
11    timeout=10,
12)
13response.raise_for_status()
14
15result = response.json()
16print(result["json"])

Using json=payload is better than calling json.dumps yourself in most cases because requests also sets the appropriate content type header.

Handling bad responses and invalid JSON

Real APIs fail in several ways:

• the server can be unreachable
• the status code can indicate an error
• the response body can be HTML instead of JSON
• the JSON can be valid but missing the fields you expect

A practical pattern is to catch transport errors separately from JSON parsing errors.

1import requests
2
3try:
4    response = requests.get("https://example.com/api/profile", timeout=5)
5    response.raise_for_status()
6    payload = response.json()
7except requests.Timeout:
8    print("The request timed out")
9except requests.HTTPError as exc:
10    print(f"HTTP error: {exc}")
11except requests.RequestException as exc:
12    print(f"Network error: {exc}")
13except ValueError:
14    print("Response was not valid JSON")
15else:
16    username = payload.get("username")
17    print(f"Loaded profile for {username}")

ValueError is the relevant exception here because response.json() raises it when the body cannot be decoded as JSON.

Working with nested JSON

Many APIs return nested objects. Use dictionary access carefully and prefer .get() when fields may be optional.

1user = {
2    "name": "Lin",
3    "address": {
4        "city": "Toronto",
5        "postal_code": "M5V"
6    }
7}
8
9city = user.get("address", {}).get("city")
10print(city)

That pattern avoids a KeyError if address is missing. For larger payloads, it is often worth validating the schema explicitly with a dataclass or a library such as Pydantic, but the core parsing step is still the same.

Common Pitfalls

• Forgetting a timeout. A request without one can hang much longer than expected.
• Calling response.json() before checking the status code and then debugging a confusing parse failure.
• Assuming every API always returns JSON. Some endpoints return HTML or plain text on errors.
• Building query strings manually instead of using params=.
• Accessing nested keys directly when the field may be optional.

Summary

• Use requests.get or requests.post with an explicit timeout.
• Call raise_for_status() so HTTP errors fail early.
• Parse JSON with response.json() instead of decoding text manually.
• Use json= when sending JSON in a request body.
• Handle network errors, HTTP errors, and invalid JSON separately.
• Treat nested API fields carefully, especially when keys may be missing.