System requirements

Functional:

Collect user purchase history

Page Views Tracking

Click Tracking

Like or dislike for each product

Tracking products user click like and dislike

Track search history

Track user wishlist

Compare other users has purchased and like most common products. Recommend their other purchases to each other.

Non-Functional:

Scalability: plenty of data of user activities increasing in each user everyday. Database management should be scalable.

Maintainable: Multiple tables mapping are complex. Data table Sharding and decoupling must be important to data engineers narrow down the scope.

Capacity estimation

Estimate the scale of the system you are going to design...

API design

Supporting func:

purchased[] getPurchaseHistory(userId)

void addClickedItem(userId, itemId)

itemId[] getClickedItems(userId)

void addLikeOrDislike(userId, itemId, attitude)

Map>getLikeOrDislikeList(userId)

void addSearchHistory(userId, keyword)

itemId[] getSearchHistory(userId)

void addWishList(userId)

wishlist[] getWishList(userId)

userId[] getTopKUserPurchasedMostCommonItem(userId)

ItemId[] getTopKPurchasedItemsByTopKCommonUsers(userId:[])

RESTful API

// Combine all these API above in this API and ranking the different algorithm's by weight

GET /getRecommendations(userId, topK)

return itemId[]

// add new user activity in log, action is for action type eg: search, clicked, like/dislike, etc, keyword use for search action, attitude for like/dislike

POST /logUserActivity(userId, action, itemId, keyword, attitude)

// get user activity profiles

GET /userActivityProfile(userId)

Database design

Here are tables: the key end with "At" means the timestamp action took

Purchased:

[userId, itemId, purchasedAt]

Attitude:

[userId, itemId, attitude, createdAt]

Clicked:

[userId, itemId, clickedAt]

Search:

[userId, keyword, searchAt]

WishList:

[userId, itemId, wishAt]

Recommendation //list all recommended items for each user

[userId, itemId:[], confidence]

High-level design

1. Client: for external users
2. load balancer: distribute traffic to multiple API servers
3. API Sever Clusters: hosts UserActivityService and RecommendationService
4. UserActivityService: Use to Collect User Activities Data to store in ActivityHistoryDatabase. (POST /logUserActivity)
5. ActivityHistoryDatabase: Store tables Purchased, Attitude, Clicked, Search, WishList which mentioned in Database Design.
6. RecommendationService: Extract User Activities Data from ActivityHistoryDatabase then generate recommendation item list by using logic we mentioned in API Design. (GET /userActivityProfile), store recommendation item lists for each user in RecommendationDatabase as well. Then send back to client as well.
7. RecommendationDatabase: store Recommendation table mentioned in Database Design.

Request flows

*Please check my Sequence diagram!

1. Client sent request to load balancer(purchase, like/dislike,search, etc)
2. load balancer assign to appropriate API server to handle
3. UserActivityService received request for new activity(should listening all the time), write new activity event to ActivityHistoryDatabase. Then send a response back to UserActivityService.
4. Once UserActivityService received response from db, server trigger getUserActivityProfile in RecommendationService to get activity data from ActivityHistoryDatabase
5. Once getUserActivityProfile received response then call getRecommendations for top K recommendation
6. update recommendation list for user in RecommendationDatabase, return top K recommendation list to client as well.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?