App Engine
async programming
urlfetch
concurrency
cloud computing

Async urlfetch on App Engine

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

On legacy App Engine standard Python 2, urlfetch provided a platform-specific way to issue outbound HTTP requests, including asynchronous ones. The important detail is that this is an App Engine RPC-style async API, not modern Python async / await. If you are maintaining legacy code, urlfetch.create_rpc() and make_fetch_call() are the tools to know. If you are building a new App Engine Python 3 service, use standard HTTP libraries instead.

What Async urlfetch Actually Does

Synchronous urlfetch.fetch() blocks until the response arrives. The asynchronous version lets you start a request, do other work, and then retrieve the result later.

The usual flow is:

  1. create an RPC object
  2. start the fetch call
  3. continue doing other work
  4. call get_result() when you need the response

Here is the standard pattern:

python
1from google.appengine.api import urlfetch
2
3
4rpc = urlfetch.create_rpc(deadline=10)
5urlfetch.make_fetch_call(rpc, "https://example.com/api")
6
7# do other work here
8
9try:
10    result = rpc.get_result()
11    print(result.status_code)
12    print(result.content[:100])
13except urlfetch.DownloadError as exc:
14    print(f"Request failed: {exc}")

That is "asynchronous" in the sense that the request is in flight while your handler continues to execute.

Why People Used It

This approach was useful when one request handler needed to fan out to several remote services. Instead of waiting for one HTTP call to complete before starting the next, you could launch several RPCs and then collect the results.

Example:

python
1from google.appengine.api import urlfetch
2
3
4urls = [
5    "https://example.com/a",
6    "https://example.com/b",
7    "https://example.com/c",
8]
9
10rpcs = []
11for url in urls:
12    rpc = urlfetch.create_rpc(deadline=5)
13    urlfetch.make_fetch_call(rpc, url)
14    rpcs.append((url, rpc))
15
16for url, rpc in rpcs:
17    try:
18        response = rpc.get_result()
19        print(url, response.status_code)
20    except urlfetch.DownloadError:
21        print(url, "failed")

This is one of the few places where the legacy API still feels fairly elegant.

Time-Sensitive Reality: Legacy Runtime

There is an important date issue here. Google's current documentation places urlfetch under the legacy App Engine standard Python 2 runtime. The first-generation legacy runtimes passed their deprecation date on January 31, 2026, so as of March 11, 2026 this topic mainly matters for maintenance or migration work, not for new services.

That means the right advice depends on your runtime:

  • legacy Python 2 App Engine app: use urlfetch if you are maintaining existing code
  • modern App Engine Python 3 app: use normal HTTP clients such as requests, httpx, or aiohttp

Practical Limits and Caveats

Async urlfetch still runs within the lifecycle of a request handler. It is not a background job system. You still need to respect deadlines, outbound request limits, and the overall handler time budget.

Also note:

  • always set a deadline appropriate to the remote service
  • handle DownloadError
  • validate status codes instead of assuming 200
  • avoid blocking on get_result() immediately after starting the call, or you lose most of the benefit

Modern Equivalent Thinking

In Python 3 code, the conceptual replacement is not urlfetch.create_rpc(). It is usually either:

  • a synchronous client like requests for simple cases
  • a real async client plus async / await if your framework supports it

So if you are migrating old code, preserve the concurrency idea but replace the transport API.

Common Pitfalls

The biggest mistake is assuming urlfetch is the recommended approach for all App Engine Python apps. It is tied to the legacy runtime story.

Another mistake is calling get_result() immediately after make_fetch_call(), which turns the code back into a blocking workflow.

A third issue is ignoring deadlines and exception handling. Remote services fail, and the async API does not remove that responsibility.

Summary

  • Async urlfetch on App Engine uses create_rpc(), make_fetch_call(), and get_result().
  • It is an RPC-style concurrency feature for legacy App Engine Python 2 code.
  • It is useful when you need multiple outbound HTTP requests in parallel inside one handler.
  • As of March 11, 2026, this is mainly a legacy-maintenance topic because first-generation runtimes passed their January 31, 2026 deprecation date.
  • For modern App Engine Python services, prefer standard HTTP libraries instead of urlfetch.

Course illustration
Course illustration

All Rights Reserved.