Loading...
Customers should be able to:
Restaurants should be able to:
Doordashers should be able to:
Let's assume we are catering to users in 10M area codes. This number will grow.
On average, each area code can have 100 restaurants.
Each restaurant can have 15 dishes that can be served.
Total records of dishes = 10M * 100 * 15 = 15B
Number of customers: 20M
If each customer on an average places 2 orders every day, number of orders in a day = 40M
The peak time of orders will vary usually based on the day of the week. For example, weekends might have more orders than on weekdays. Peak times could be somewhere around noon or at dinner time in each region.
In general, the searching of menus/restaurants will be read-heavy and the ordering functionality will be write-heavy. The potential of customers looking up past orders once they have been delivered and the food has been consumed is very less.
Some of the customer-facing APIs exposed by the various services:
Ordering food:
Profile management:
Sometimes the technical discussion could veer towards defining rest APIs, URIs, identifying HTTP methods, sample entities, sample payloads, request-response, etc.
The choice of the database usually depends on the amount of data that is being stored, the ease of scaling, partitioning, replication, and several other factors. Application owners may choose to use a mixture of different databases to accomplish certain use cases. For ACID (Atomicity, Consistency, Isolation, Durability) requirements, a relational database is always preferred over a NoSQL counterpart.
NoSQL and relational databases have their own prospects and constraints and the decision to use either should be clearly thought through based on functionalities.
As evident from the capacity estimation, the amount of data of restaurants, menu descriptions, user data, dasher data, etc is going to be huge, and hence, NoSQL/ Columnar database like Cassandra could be used. The structure of data, especially attributes might also vary between restaurants and it could be difficult to fit in the data into a relational schema.
Pictures (Restaurants, menu items) can be stored in an object storage service like Amazon S3.
Ordering is a transactional process and can be stored in Oracle/MySQL/Postgres.
UI Client
The application will be accessible via mobile, web, tablets, etc. Based on the actor, the interfaces presented will be different, or a combination of many. So, individual interfaces/pages will talk to the respective service for parts of the functionality. For example, the search can be performed by Restaurant Search Service, orders can be handled by Ordering Service, and so on.
Primarily, there will be four versions of the interface for the four actors, viz customer, restaurant, doordasher & admin.
Search Ecosystem
The most important and coveted functionality that has to be provided by the system is the capability of searching on menu items, cuisines, restaurants, among other things. In the food ordering journey, this functionality will be the point of entry for all the customers, unless they already have a favorite restaurant in their preferences from which they can select dishes. Thus a personalized discovery & search experience based on a customer’s past search and order history has to be provided. As is apparent, this particular part of the entire system will be read-heavy.
We can think of leveraging popular off-the-shelf search offerings from the market such as Elasticsearch or Apache Solr for quick lookup as per user search parameters. Both of these technologies are open-source, distributed, and based on Apache Lucene and have their own strengths and weaknesses.
We will need to have a queue in place to process asynchronous updates to the search cluster. When the Restaurant Profile Service (see below) creates/updates a restaurant/menu data by performing CRUD operations on the database, it can also post an event to the queue. This event could be any of the CRUD. We need a data indexer that listens to the queue for such events. The data indexer then picks up the event, runs a query against the database to formulate a document as per the correct format, and posts the data into the search cluster. We also need to have a Restaurant Search Service that executes queries on the search cluster based on user inputs and returns the result to be displayed on the user interface.
Elasticsearch has a Geo-distance query, which can be leveraged to return all the restaurant/menus that the user is searching for based on a defined radius from the location of the user. What this essentially means is, users will be shown only those restaurants that are reachable from the users’ addresses. Similarly, Apache Solr also has Spatial Search which caters to the use case of geographic search. As such, Elasticsearch is very fast in retrieving results and can be directly queried. In order to further reduce latency and improve user experience, we should use a cache, working alongside the Restaurant Search Service. More on caching later.
Ordering Service
This service will manage the menu selection, shopping cart management, and placement of food orders. It will process the payment using an External Payment Gateway and persist the result into an Orders database. Because order placement is transactional in nature, the best idea is to use a relational database.
Customers will be able to get the full receipt of their order using this service along with other details about the order. They can also cancel the order if they changed their mind for some reason. Customers can also view their past order histories.
Order Fulfillment Service
Few high-level functionalities that this service will handle are as below:
User Profile Management & Preferences Service
The actors in the system, namely, customers, restaurant staff, doordashers, and system admin will need a way to create their profile with personal information, contact, address, and will be assigned a role based on their profile. Individual actors will also have preferences based on their role. For example, customers may have set preferences for selecting from a fixed set of restaurants or zip codes or cuisines. Doordashers might have a preference for delivering only within their specific area codes, or choice of restaurants, etc. Similarly, actors will also have their own method of ordering or getting paid as appropriate. Notification preferences of actors will also vary. This service will manage the profiles and preferences of all such actors across the board.
Doordasher Dispatch Service
This service will be used to accomplish use-cases relating to a doordasher. A doordasher will be able to:
Restaurant Profile Service
This service will be managing the data related to restaurants, menus, offerings, etc. A restaurant or business can:
External Payment Gateway
This component can interface with popular payment gateways like Amazon Payments, We Pay, PayPal, ApplePay, or individual Credit Card providers like Amex, Visa, Mastercard, etc. The Order Service will interact with this component to ensure the payment is done at the time of confirming the order. The interaction should be synchronous in nature.
Notification Service
This service is responsible for delivering notifications to every actor with the system. The notifications could be sent out to the individual actors in the preferences they have set for receiving them. Some actors might prefer push notifications, some might receive text or emails. This service is supposed to abstract out the medium in which notifications are being sent. This means the underlying interactions with the mobile carrier, email service providers, etc will be abstracted. Actors could also receive In-App notifications. The responsibilities are notifying:
The complete component design with all the services will be like below:
Let's define some acronyms:
Once the customer places an order using the mobile/web client using the OS, the order processing workflow could be as follows:
The overall architecture could be microservices-based with heavy usage of the publisher-subscriber pattern, involving a queuing technology like Kafka, RabbitMQ, ActiveMQ, Amazon SNS, or Amazon MQ. Each microservice can talk with another one using this model of publishing a message and subscribing to channels or topics. This makes the services de-coupled from each other in the best possible way. Consequently, microservice A doesn’t need to know the endpoint of microservice B when it publishes a message to the queue. So, the publisher doesn’t need to know the consumers (subscribers) of its published messages. Similarly, the subscriber doesn’t know about the source of the message, i.e the publisher. The Pub/Sub system becomes a broker and serves as a contract between all the involved parties.
Also, it's imperative that each microservice interacts with their own database and doesn’t share with anyone else. This approach is motivated by the database-per-service paradigm. Microservice A doesn’t have direct access to microservice B’s database as they are separated and individually owned.
As described earlier, the data model proposes the idea of just one big fat schema containing every possible table. Microservices architecture is opposed to this concept. Functional partitioning is required in order to grant the ownership of each significant table or group of tables to one microservice. That discussion is out of scope here. However, as I have proposed that we could choose a mix of relational as well as non-relational databases for our data storage requirements, functional partitioning becomes more pertinent. The astute reader could take up, as an exercise, the ways the given schema could be partitioned to fit into the microservices architecture.
As the data grows, it becomes impossible to store all the data in just once instance of the database. Restaurant data can be partitioned based on:
Each of the above partitioning schemes has its own benefits and disadvantages. The strategy needs to be well thought out considering all possible side effects and performance improvements before implementing it.
Based on the recent orders in an area, or the most ordered items, or searched items, data could be cached so that the Restaurant Search Service could look up such information from a distributed cache, instead of hitting the search infrastructure, and immediately return few recommendations. Images of restaurants and dishes could be cached as well, instead of hitting the object storage all the time. The cache will hold the popular or most-ordered menu items/restaurants in a particular area and the search screen should show those options by default. The usage of cache definitely speeds up the viewing of search results as mentioned in this article.
Some of the popular cache technologies available in the market are Redis, hazelcast, and Memcached. Least Recently Used (LRU) or Least Frequently Used (LFU) algorithm, or a combination of both can be our cache eviction strategy for our use case. This article describes caching in more detail.
A content delivery network (CDN) could also be used as a cache in order to make local content available to users based on their geography. In our current application scenario, using a CDN will be an overkill.
Every service will have multiple instances running, in scenarios where the communication is over https POST/GET, etc load balancers should be placed in front of the individual services. Load balancing helps make sure that no one instance is inundated with a deluge of requests and the response time is faster overall. There are several load balancing techniques that could be used like Least Connection Method, Round Robin method, etc. More can be found here.
In case the service solely subscribes to a particular channel/topic in the queue, the responsibility of load balancing is on the queue itself. For example, in Kafka consumer consumption divides partitions over consumer instances within a consumer group. Each consumer in the consumer group is an exclusive consumer of a fair share of partitions. This is how Kafka does load balancing of consumers in a consumer group.
Explain any trade offs you have made and why you made certain tech choices...
It is imperative that we identify the various points of failure in the system and try to have replica/backup for each if any component dies. Each service should be horizontally scalable. The NoSQL infrastructure will also have multiple nodes. The search system can have multiple nodes set up. Queues can have partitions and replication as well. Each and every service/component can be individually scaled if needed. Autoscaling could be enabled to handle loads on individual components, for example, spinning up more instances in the face of a high peak load. Furthermore, in case any of the nodes go down, or any partition in the queueing infrastructure goes down, another instance can take up the job. The crashed node can then do a cleanup and restart using a process called self-healing.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?