Google App Engine
Google Cloud Storage
asynchronous processing
NDB functions
cloud computing

Does GAE GCS write have asynchronous version like the NDB functions

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

NDB in classic App Engine is well known for methods such as get_async and put_async, which return futures and fit naturally into tasklet-based code. Google Cloud Storage writes are different. In most App Engine setups, storage uploads are synchronous client calls, so there is no direct write_async equivalent that behaves like NDB futures. If you want non-blocking behavior, you usually add it at the application level.

Why NDB async feels different from GCS writes

NDB was designed around App Engine datastore RPCs and an event-loop style programming model. Its async methods are part of the library contract, so you can compose them as futures and yield on them later.

Cloud Storage clients are generally built around upload operations that stream bytes over HTTP. From your application code, an upload call usually blocks until the request has been sent and the server responds. That means the storage library itself does not give you the same tasklet-friendly future API that NDB does.

In other words, the difference is not just method naming. The two libraries are built around different execution models.

A normal synchronous upload

This is the basic shape of a Cloud Storage write in Python:

python
1from google.cloud import storage
2
3
4def write_report(bucket_name, blob_name, text):
5    client = storage.Client()
6    bucket = client.bucket(bucket_name)
7    blob = bucket.blob(blob_name)
8    blob.upload_from_string(text, content_type="text/plain")
9
10
11write_report("my-bucket", "reports/daily.txt", "job complete")

The upload_from_string call is straightforward, but it is synchronous from the caller's point of view. If you call it inside a web request, that request waits for the upload to finish.

How to make the workflow asynchronous

If the goal is "do not block the current request," the usual answer is to move the upload into background work rather than look for a hidden async method. In App Engine-style architectures, the most reliable pattern is to enqueue a task and let a worker perform the write.

python
1from concurrent.futures import ThreadPoolExecutor
2from google.cloud import storage
3
4executor = ThreadPoolExecutor(max_workers=4)
5
6
7def write_report(bucket_name, blob_name, text):
8    client = storage.Client()
9    bucket = client.bucket(bucket_name)
10    blob = bucket.blob(blob_name)
11    blob.upload_from_string(text, content_type="text/plain")
12
13
14def write_report_in_background(bucket_name, blob_name, text):
15    return executor.submit(write_report, bucket_name, blob_name, text)

This returns a future, but notice what changed: the future comes from your executor, not from the storage library itself. That can be good enough for in-process concurrency, but it is not as durable as a proper task queue. If the request ends or the instance shuts down, the background thread may never finish.

Better option for production workloads

For important uploads, queue the work instead of spinning off a best-effort thread. In App Engine terms, that might mean using a background endpoint, a task queue, or another worker process. The exact service can vary, but the design principle stays the same: storage writes that must complete should be handled by infrastructure meant for retriable background jobs.

This approach gives you:

  • Retry behavior when uploads fail
  • Isolation from request time limits
  • Better visibility into failed jobs
  • Less coupling between user-facing latency and storage I/O

That is usually closer to what developers actually want when they ask for an async storage write.

Common Pitfalls

The first pitfall is assuming Cloud Storage has an NDB-like put_async method somewhere in the client library. In most App Engine usage, it does not. The upload call is still a blocking network operation.

Another issue is confusing concurrency with durability. A thread pool or future can make the current code path return earlier, but it does not guarantee that the upload will finish if the process dies.

Developers also underestimate request deadlines and shutdown behavior. A write that looks fine in local testing may fail intermittently in production when instances scale down or requests are cut short.

Finally, avoid building request-handling logic that depends immediately on the result of a supposedly detached upload. If the caller needs to know whether the object was written, then the workflow is not really fire-and-forget and should be modeled explicitly.

Summary

  • Cloud Storage writes are usually synchronous from application code.
  • There is no direct NDB-style write_async equivalent in the usual client workflow.
  • If you need non-blocking behavior, create it at the application level.
  • For important uploads, prefer task queues or background workers over in-process threads.
  • Separate "return sooner" from "finish reliably" because they are not the same guarantee.

Course illustration
Course illustration

All Rights Reserved.