Python
Scala
Async API
Future
Concurrency

I would like to make/have a scala like 'future' async API for python

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

If you want a Scala-like Future API in Python, the real goal is usually composable asynchronous work with callbacks, mapping, and error propagation. Python already has the primitives for that in asyncio and concurrent.futures, but the API style is different, so building a thin wrapper can give you the same fluent feel without fighting the runtime.

Decide Which Python Concurrency Model You Want

Scala Future is often used for non-blocking workflows built around composition. In Python, there are two common foundations:

  • 'asyncio, which is best when your application already has an event loop'
  • 'concurrent.futures, which is best for thread-pool or process-pool execution'

If you want Scala-like chaining in modern Python applications, asyncio is usually the better base because cancellation and composition are built in.

Build a Small Wrapper Around asyncio.Future

A practical approach is to wrap an awaitable and expose methods like map, flat_map, and recover. The wrapper does not replace asyncio; it packages it into a more functional API.

python
1import asyncio
2from typing import Awaitable, Callable, TypeVar
3
4T = TypeVar("T")
5U = TypeVar("U")
6
7
8class PyFuture:
9    def __init__(self, awaitable: Awaitable[T]):
10        self._awaitable = asyncio.ensure_future(awaitable)
11
12    def map(self, fn: Callable[[T], U]) -> "PyFuture[U]":
13        async def run():
14            value = await self._awaitable
15            return fn(value)
16        return PyFuture(run())
17
18    def flat_map(self, fn: Callable[[T], "PyFuture[U]"]) -> "PyFuture[U]":
19        async def run():
20            value = await self._awaitable
21            next_future = fn(value)
22            return await next_future._awaitable
23        return PyFuture(run())
24
25    def recover(self, fn: Callable[[Exception], T]) -> "PyFuture[T]":
26        async def run():
27            try:
28                return await self._awaitable
29            except Exception as exc:
30                return fn(exc)
31        return PyFuture(run())
32
33    async def result(self) -> T:
34        return await self._awaitable

This gives you a familiar composition surface while still using the standard event loop underneath.

Compose Work in a Scala-Like Style

Once you have the wrapper, chaining reads naturally.

python
1async def fetch_user(user_id: int) -> dict:
2    await asyncio.sleep(0.1)
3    return {"id": user_id, "name": "Ada"}
4
5
6def to_upper_name(user: dict) -> str:
7    return user["name"].upper()
8
9
10async def main():
11    future = PyFuture(fetch_user(7)).map(to_upper_name)
12    print(await future.result())
13
14
15asyncio.run(main())

For dependent async steps, use flat_map.

python
1async def fetch_orders(user: dict) -> list[str]:
2    await asyncio.sleep(0.1)
3    return [f"order-for-{user['id']}"]
4
5
6def next_step(user: dict) -> PyFuture[list[str]]:
7    return PyFuture(fetch_orders(user))

That separates value transformation from asynchronous sequencing in the same way Scala users expect.

Keep Cancellation and Backpressure in Mind

A fluent API is not enough by itself. Real async systems also need cancellation, timeouts, and bounded concurrency. If your wrapper hides those concerns too deeply, it becomes harder to operate in production.

For example, timeouts should still be explicit:

python
1async def main():
2    future = PyFuture(fetch_user(7))
3    result = await asyncio.wait_for(future.result(), timeout=1.0)
4    print(result)

This is one place where staying close to asyncio pays off. Your Scala-like API should complement the runtime, not replace its operational tools.

When concurrent.futures Is the Better Base

If the work is blocking and you want background execution from threads or processes, you can wrap concurrent.futures.Future instead. That is common for CPU-heavy or legacy blocking code.

python
1from concurrent.futures import ThreadPoolExecutor
2
3
4def slow_square(x: int) -> int:
5    return x * x
6
7
8with ThreadPoolExecutor(max_workers=4) as pool:
9    future = pool.submit(slow_square, 12)
10    print(future.result())

In that world, you can still add a map-style wrapper, but remember that thread-based futures and asyncio futures are not interchangeable without adapters.

Common Pitfalls

One mistake is recreating a Scala-style API but forgetting Python's execution model. Threads, event loops, and coroutines have different tradeoffs, so the wrapper must stay honest about what it is built on.

Another issue is mixing blocking I/O into an asyncio-based future chain. That stalls the event loop and defeats the purpose of a non-blocking API.

It is also easy to over-design the abstraction. If all you need is await, asyncio.gather, and a few helper functions, adding a custom future type may increase complexity more than it helps.

Summary

  • A Scala-like future API in Python is best implemented as a thin wrapper over existing primitives.
  • 'asyncio is the best foundation for composable non-blocking workflows.'
  • Expose methods like map, flat_map, and recover without hiding cancellation and timeouts.
  • Use concurrent.futures only when the workload is thread-pool or process-pool based.
  • Keep the wrapper small so it works with Python's runtime instead of fighting it.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design