Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
on average, each user posts 10 photos, total photo storage = (1.1mb) * 10 * 100m, round down to 1 PT
Define what APIs are expected from the system...
POST /v1/register
GET /v1/authenticate
POST /v1/post
POST /v1/uploadPhoto/{:filePath}
POST /v1/deletePhoto/{:id}
GET /v1/getFeed/{:id}
POST /v1/sendMessage
POST /v1/comment/{:id}
POST /v1/like/{:id}
GET /v1/pushNotification/{:id}
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
User Metadata DB:
UserDetails
-user_id: UUID (PK)
-username: string
-hashed_password, string
-bio: string
-created_at: datetime
-is_active: boolean
-last_login: datetime
-profile_pic: string, URL
-following_counts: int
-followers_counts: int
-is_celebrity: boolean. # >= 100,000 followers
Comments:
-comment_id: int
-comment: string
-user_id: foreign key referencing UserDetails
Follow:
-followee: foreign key referencing UserDetails
-follower: foreign key referencing UserDetails
Feed:
-feed_id: int
-user_id: user_id foreign key referencing UserDetails
-post_id: foreign key referencing Posts
-added_at: datetime
-seen: boolean, weather the user has seen the feed
Photos:
-photo_id: int
-photo_url: URL in S3
-thumbnial_url: URL in S3
-upload_date: datetime
Message:
-message_id: int
-content: string
-sent_date: string
-sent_to: user_id foreign key referencing UserDetails
-photo: foreign key referencing Photos
Posts:
-post_id: int
-photo_id: foreign key referencing Photos
-user_id: foreign key referencing UserDetails
-photo_url: URL to the photo in S3
-like_count: int
-is_private: boolean
-caption: string
-post_date: datetime
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Post photo (non-celebrity):
Register:
Client -> API Gateway -> Profile Service -> User metadata DB -> client
Login:
Client -> API Gateway -> Profile Service -> User metadata DB -> client
Comment:
Client -> API Gateway -> Interaction Service -> Interaction ID -> client
Like:
Client -> API Gateway -> Interaction Service -> Interaction ID -> client
Client -> API Gateway -> Post Service -> Photo Service Upload Photo -> Message Queue -> Object Store -> Post Service-> Post DB -> Feed Service Fanout -> message queue -> Notification Service -> Message Queue -> Client
Post (non-Celebrity):
Client -> API Gateway -> Post Service -> Photo Service Upload Photo -> Message Queue -> Object Store -> Photo Cache -> Post Service-> Post DB, when the followers login, it will call Feed Service to pull updates -> Fanout -> message queue -> Notification Service -> Message Queue -> Client
Post (Celebrity):
Client -> API Gateway -> Post Service -> Photo Service Upload Photo -> Message Queue -> Object Store -> Photo Cache -> Post Service-> Post DB -> Push updates Feed Service Fanout -> message queue -> Notification Service -> Message Queue -> Client
Instant Messaging:
Client webscoket request -> Message Service -> websocket connection established -> Dynamo DB,
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Photo Service will interact with the object store DB for upload/retrieval, and save the photos to cache (redis).
When a photo gets deleted, Photo Service will send a request to the interaction service to delete comments and likes in the interaction DB that's related to the photo thats being deleted
When Photo Service requests fail, it will retry a few times, if still fails, send an alert to the user and cloudwatch,then it will store photos in queues or cache, or store in local cache when the user is uploading photos, so when the service recovers it will work faster.
Feed Service will push or pull posts and fan out the followers.
Cloud Watch will monitor all of the micro services and DB system activities/performance and send alerts to developers if theres failures
CDN to store frequently accessed photos for fast delivery for global users
Interaction Service will write comments/likes/follower followers db to Interaction db/Graph DB, Interaction db for simple follower/followee query, Graph DB for complex traversal queries like people you may know, mutual followers
Notification Service:
Technology choice: AWS SNS or Firebase
Two queues: celebrity queue and regular queues.
Celebrity queue will have higher priority to push out notifications to users.
Regular queue is processed in paraelle with lower priority.
Explain any trade offs you have made and why you made certain tech choices...
CDN:
Pros: fast delivery, enhance system performance
Cons: more cost and dependent on S3
Message Queue:
Kafka:
Pros: high throughput and highly scalablity for data streaming, high fault tolerance
Cons: complexity, higher latency for small volume traffic compared to RabbitMQ
Notification Service:
Amazon SNS:
Pros: out of the box solution, lesser development effort
Cons: limited customization
Websocket:
Pros: real-time notifications for active users
Cons: complex to set up and requires maintenance
Post DB:
Dynamo DB:
Pros: high scalability, fast write/read
Cons: lower data consistency compared to relational DB, does not support complex queries
My SQL:
Pros: ACID high consistency, supports complex queries.
Cons: less scalability for high throughput, slower write/read
User Metadata DB: same as POST DB
Interaction DB:
Dynamo DB:
Pros: can store flat user data such as username, fast read/write, high scalability, less costly
Cons: does not support complex queries
Graph DB:
Pros: extramely fast for connected data like followee/follower, and complex queries like recommandations, people you may know that requires graph travelsal.
Cons: less scalability, more costly
Try to discuss as many failure scenarios/bottlenecks as possible.
Photo service failure
DB failure
API Gateway failure
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
Video uploading capability
Auth security
more worker DBs for fault tolerance and failover