Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
The app should be able to allow users to post items with comments, prices etc.
List non-functional requirements for the system...
Highly available and consistent, once a item is sold its gone.
Estimate the scale of the system you are going to design...
User data: name, age, username, password hash, location
user history: items bought or sold, comment history
chat history per user: if two users chat it should be stored
lets say 500 m users
100 m post items
the rest comment, buy, and chat
Define what APIs are expected from the system...
login
POST api/v1/login/{user_id}
post item
POST api/v1/post/{user_id, item metadata}
metadata includes title, description, picture(s)
this should also create an internal item_id
comment
POST api/v1/comment/{user_id,item_id}
chat
POST api/v1/chat{user_id,reciever_id}
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
For user data, an RDB works best.
Name, password hash, location, post history key
It makes more sense to partition the data on the post_id. For each post_id, we can make a separate RDB that include the user_id. That way, if someone wants to access someone's historical posts, they can query the user_id and grab all the necessary data.
For images, S3 makes the most sense as its the most flexible. Some users may post pictures or videos, we don't quite know.
Craiglist searches are based on location, we could implement a Quadtree (roughly partitioned on registered users per area) and make it a bit broader (you dont look at your specific cluster but rather yourself and your neighbors). We can create RDB's of posts and users based on this, as this will speed up search times.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Search is an important aspect of craigslist, and since all search is done through text it makes sense that Elastic Search stands as a great use case here. We can have a separate microservice scan the individual posts upon publishing and perform stemming, lemming, etc. to make Elastic Search more powerful.
We want the post to be searchable as soon as its public. Therefore, upon publication the post is immediately written to the RDB and then sent in a Kafka queue for stemming, lemming etc. before then being put in Elastic Search. We can incorporate retry mechanisms in kafka to ensure it goes through.
We can use a message queue for the initial posting as well (putting the post in the RDB). This way we can handle sudden increases in scale, while having a robust solution for downtimes.
Elastic Search will use Tf-IDF.
Explain any trade offs you have made and why you made certain tech choices...
Regarding users and their posts for items, it's a bit interesting to partition it. We could partition on the users and have the posts nested (something like MongoDB's nested json documents), but users will probably be looking at old posts rather than user data. Therefore it makes more sense to have the data partitioned on post_id.
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?