Design a Feed Generation Service

Question

Design a fault-tolerant feed generation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

For DoorDash, I would design the feed generation service around the fact that the feed is a high-QPS, user-facing, latency-sensitive system where partial freshness is usually acceptable, but outages or slow responses are not.

The first principle is **separation of concerns using SOLID**. Feed generation should not be one large service that fetches restaurants, promotions, delivery ETAs, user preferences, ranking scores, and experiments all inline. I would split responsibilities into clear components: candidate retrieval, feature hydration, ranking, filtering, feed assembly, and response formatting. This keeps the system maintainable because teams can change ranking logic, ads logic, or restaurant eligibility without rewriting the whole feed path.

The second principle is **availability over strict consistency**. A DoorDash home feed does not need perfect real-time consistency for every restaurant, promotion, or ETA. If a restaurant closed 10 seconds ago, the system should ideally remove it quickly, but it is worse for the entire feed to fail because one upstream inventory service is slow. I would use cached and eventually consistent data for most feed inputs, while applying last-mile validation for critical actions like checkout or cart submission.

The third principle is **CQRS**. Reads and writes have very different patterns. Feed reads are massive and latency-sensitive; writes come from merchant updates, menu changes, dasher availability, promotion changes, and user behavior events. I would maintain a write/event pipeline that builds read-optimized feed indexes, rather than querying normalized operational databases on every request.

The fourth principle is **thread safety and bounded concurrency**. Feed generation fans out to several data sources, so the service must use immutable request context, explicit timeouts, cancellation, and bounded thread pools or async execution. Without this, a slow dependency could exhaust worker threads and cause a cascading outage.

The fifth principle is **testability by design**. Each feed stage should be deterministic given its inputs. Ranking, filtering, deduplication, fallback behavior, and experiment assignment should be testable without standing up the entire DoorDash stack.

I would use a hybrid push/pull architecture.

On the write side, DoorDash events flow through Kafka: merchant open/close events, menu availability changes, promotion updates, delivery fee changes, estimated prep-time updates, dasher supply signals, user click/order events, and experiment configuration changes. These events are consumed by stream processors that update read models in Redis, Elasticsearch/OpenSearch, Cassandra/DynamoDB, or a specialized feature store.

The read path would look like this:

```text
Mobile/Web Client
  -> API Gateway
  -> Feed Generation Service
      -> User Context Service
      -> Geo/Address Service
      -> Candidate Store
      -> Feature Store
      -> Ranking Service
      -> Ads/Promo Service
      -> Feed Response Composer
  -> Client
```

The **Candidate Store** returns eligible restaurants, stores, grocery partners, convenience merchants, and sponsored placements near the user's delivery address. This should be geo-indexed by delivery region, merchant status, cuisine/category, price range, and availability.

The **Feature Store** provides precomputed features such as user cuisine affinity, prior orders, merchant conversion rate, delivery reliability, estimated delivery duration, affordability score, promo eligibility, and freshness timestamp.

The **Ranking Service** scores candidates using a model and business rules. For example, DoorDash may rank based on predicted conversion, delivery ETA, merchant quality, fees, customer preferences, DashPass eligibility, promotions, and marketplace balancing constraints.

The **Feed Composer** turns ranked candidates into modules: `Recommended for You`, `Fastest Near You`, `DashPass`, `Offers`, `Grocery`, `Convenience`, and `Recently Ordered`. This layer owns layout constraints, deduplication, pagination tokens, and backward-compatible response schemas.

I would avoid generating the full feed from scratch using live database joins. Instead, the online service should mostly read from precomputed, denormalized, cache-friendly structures and only perform lightweight request-time personalization.

The online request path should have a strict latency budget, for example p95 under 200-300 ms for the feed service itself, excluding client network latency. I would implement request fanout using async calls with deadlines propagated from the API gateway. Each dependency gets a budget, such as 20 ms for user context, 40 ms for candidate retrieval, 50 ms for feature hydration, 60 ms for ranking, and 20 ms for response composition.

The request context should be immutable and include `userId`, `consumerId`, `deliveryAddressId`, geohash, locale, app version, experiment assignments, and request deadline. Shared mutable state should be avoided. Caches should use thread-safe clients and explicit key versioning, for example:

```text
feed:candidates:{market}:{geohash}:{vertical}:{schemaVersion}
feature:user:{consumerId}:{featureVersion}
feature:merchant:{merchantId}:{featureVersion}
```

For pagination, I would use opaque cursor tokens containing the request context hash, ranking model version, experiment bucket, last seen score, and candidate offsets. This avoids inconsistent pagination when the underlying feed changes between page 1 and page 2.

For consistency, I would use event sourcing for important marketplace state changes. Merchant availability, menu item availability, promo changes, and store metadata updates are appended as events. Stream processors build read-optimized projections. This lets DoorDash replay events to rebuild feed indexes, debug why a merchant appeared in a feed, and recover from bad downstream projections.

For distributed transactions, I would avoid two-phase commit in the feed path. Feed generation should be read-heavy and tolerant of stale projections. If a merchant update and promo update arrive separately, the feed may briefly show slightly stale metadata, but the checkout path must revalidate availability, price, fees, and promo eligibility before order placement.

Backward compatibility matters because DoorDash clients may run old app versions. The response should be versioned, for example `FeedResponseV3`, with modules carrying typed payloads. New modules should be additive, and unknown module types should be ignored by older clients.

Security should include authenticated user context, authorization checks around address and consumer identity, PII minimization in logs, and privacy-safe feature generation. For example, logs should record feed module IDs, candidate IDs, model version, and latency, but not raw address text.

To handle millions of requests, I would horizontally scale the Feed Generation Service behind a load balancer. It should be stateless, with all durable state in read stores, caches, Kafka, and feature stores. Hot markets like New York, San Francisco, Los Angeles, and Toronto should be partitioned carefully because feed traffic and merchant density are not evenly distributed.

I would use multi-layer caching. The first layer is process-local cache for static configuration and experiment metadata. The second layer is Redis or Memcached for hot candidate sets and merchant features. The third layer is durable read storage such as Cassandra, DynamoDB, or OpenSearch for larger indexed queries. Cache entries should have short TTLs for availability-sensitive data and longer TTLs for stable metadata like cuisine tags.

Reliability mechanisms should include circuit breakers, dependency-specific timeouts, hedged reads for critical stores, bulkheads per downstream dependency, and graceful degradation. If the ranking model service is unavailable, the feed should fall back to a simpler heuristic ranker using ETA, distance, popularity, and availability. If promotions fail, the feed should still return restaurants without promo badges. If personalization fails, return a market-level popular feed.

Observability should be first-class. I would emit metrics for QPS, p50/p95/p99 latency, timeout rates by dependency, empty feed rate, fallback rate, cache hit rate, ranking model version, candidate count, module fill rate, and order conversion. Distributed traces should show time spent in candidate retrieval, feature hydration, ranking, and composition. Structured logs should support debugging questions like: why did this user see this merchant at this rank?

Testing should happen at multiple levels. Unit tests cover ranking rules, deduplication, pagination token parsing, fallback behavior, and schema compatibility. Integration tests run against test Kafka topics, Redis, and candidate stores to verify projection updates. End-to-end tests simulate a consumer opening the app, changing address, seeing a feed, selecting a merchant, and validating that checkout catches stale availability. Load tests should replay realistic traffic patterns by market and time of day, including lunch and dinner spikes. Chaos tests should inject ranking service latency, Kafka lag, Redis failures, and stale merchant projections.

The biggest trade-off is **freshness versus latency**. A fully live feed would query merchant state, menu availability, delivery estimates, promos, and personalization in real time, but that would be too slow and fragile at DoorDash scale. Precomputed projections and caches make the feed fast and available, but they introduce temporary staleness. I would accept this for browsing and enforce correctness at checkout.

Another trade-off is **personalization versus cacheability**. A completely personalized feed maximizes relevance but reduces cache hit rate. I would use a layered approach: cache market-level candidate pools and merchant features broadly, then apply lightweight user-specific ranking at request time.

There is also a **model complexity versus debuggability** trade-off. A sophisticated ML ranker may improve conversion, but engineers and operators still need to explain feed behavior. I would log model versions, feature availability, major ranking reasons, and fallback decisions so production issues can be investigated.

For **consistency versus availability**, I would choose eventual consistency in the feed and strong validation in transactional systems. The feed can be slightly stale; order placement cannot be.

For **developer experience versus platform complexity**, CQRS, event sourcing, feature stores, and fallback rankers add operational overhead. I would justify that complexity because DoorDash feed generation sits directly on revenue, marketplace health, and customer experience. The key is to keep interfaces clean: candidate retrieval, feature hydration, ranking, and composition should be independently testable and owned by clear teams.

Design a Feed Generation Service

DoorDash

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

1. Core Design Principles

2. Architecture for DoorDash Feed Generation

Submit Your Answer

DoorDash Software Engineer Interview Guide

Related Questions

Design a scalable Search System

Design a Video Streaming for Rippling

Design a Load Balancing Service

Design a Image Processing Service

Build a high-throughput Image Processing Pipeline