Loading...
Functional:
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
User Registration
Search for activity to start
Start/stop activity
User interface
Statistics
Training videos
Non-Functional:
List non-functional requirements for the system...
Availability
Performance
Scailability
Useability
Estimate the scale of the system you are going to design...
10 million MAU, 1 Million DAU, with max throughput of 100,000 users in one hour session.
Each user can potentially watch videos during their training activity which can be about 1 GB per user per hour. Add another 100MB
for using the application.
1.1GB x 101,000 = max rate per hour = 101 TB
1.1GB X 101,000 = needed minimum per day = 101 TB * 2
1.1GB x 101,000 = needed per month = 101 TB * 4
We only add a minimal amount per day and per month since most of the data doesn't change...it's videos.
Define what APIs are expected from the system...
login(user_id, email, password) [POST]
search(search_term, selected_filters) [GET]
UpdateActivity(activity_id, category, start/stop, timestamp) [POST]
getStatistics(user_id) [GET]
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
User_table
user_id: string
email: string
password: string (hash)
index: user_id
Activity_table
user_id: string
id: string
category: string
start_time: string
end_time: string
name: string
index: name
Media_table
id: string
category: string
url: string
title: string
index: title
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
The Client uses the login API to attempt a sign in. The request is routed first to the Load balancer and then the API Gateway, if the credentials aren't there, the user is registered. The password is not directly stored in the database, instead bcrypt is used to hash the password and the hash is stored in the database. The user authentication is done in the API gateway, once the user authenticated they are routed to the User Interface which is the app. In the app the user can choose to do various activities. One activity would be to use the search API to search for an activity to start, such as "outdoor run". The search aPI checks the cache for the common term, if it's not found it enters the Load balancer. The load balancer distributes the request to the search queue, which goes to the search service and pulls the data from the next Load balancer and the Read DB.
The user Starts the activity using the updateActivity API with the "start" param along with the other params to post the request to the database. The request goes to the next load balancer, activity queue, activity service, another load balancer, and to the next available write database.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Indexes
We choose to use a noSQL since we want to prioritize writes and many noSQL databases use SS Tables and LSM Trees. LSM trees and SSTables are good for writes and provide performance/availability since they have an in-memory memtable. If the memtable fills up, they're written to SS Tables, both structures support logarithmic time for writes since the LSM trees use a balanced binary search tree and the SS Table are sorted so binary search works well. The noSQL database along with the LSM Trees and SS Tables provide Scailability since they compact tables in the background by minimizing the SS Tables and getting rid of tombstones and old data. NoSql database are generally faster than SQL databases. The fast writes support our indexes and support our non functional requirement of usability since it provides a good client experience.
Consistent Hashing
Since we have two write databases and 3 read databases it's important that we support consistent hashing since we want to be able to support our requirement of Scailability. An efficient sharding algorithm is used to distribute the load accross our 5 databases. Choosing the wrong hashing algorithm such as a modulo operator would impact our performance and provide a poor user experience. For example, if one of our servers crashes we may need to rebalance all 4 servers triggering lots of traffic over our network. Instead we utilize Consistent Hashing which uses a set number of 3 virtual nodes for every Database. The hashing algorithm spreads the load across all 3 severs in ranges where all Databases are responsible for data in one range across a circle. If one server does down, we look at the server before the server and after the server and only those two servers are rebalanced...because we're only looking at the range.
Explain any trade offs you have made and why you made certain tech choices...
We prioritize Writes over reads by using a database that suppors SS Tables and LSM Trees. This is because we want the user experience to be good when they use the app. Most clients want the app to work and to track their workouts/activities, they're likely to do this far more than searching for training videos.
Try to discuss as many failure scenarios/bottlenecks as possible.
The fault tolerance on our databases is high, however it's not that high in other parts of our application such as the services. We should at the very least have two of each service and caches and load balancers to support all of our non-functional requirements.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
We should add multiple layers of redundancy on each level, such as two API gateways, 2 loader balancers, 2 of each service, and 2 of each cache. Also we should implement an advertisement and payment service to start raking in capital. Finally, the API gateway should be extended to support monitoring to ensure high availability and support a good user experience. A Monitoring service is critical for ensuring our application keeps running smoothly.