Loading...
1) User Metadata - Realtional databse - information like user _id, user_name, created_on, status, etc.
2) Entity Database - entity id, embedding for that entity - a key value store
3) User Entity history Database - user_id, purchased_on, product_purchased, amount paid, - Mongo DB.
4) User Interaction database - user_id, liked_on, viewd_product, etc - Mongo DB
In order to keep it simple, let's use MongoDB for User Metadata, User Entity History Data and User Interaction Data.
5) Recommendation cache - Can be Redis
1) Embedding Model and embedding generator -Whenever an entity (product) is uploaded into the system, we should generate an embedding of the same. An embedding is a vector representation of the product, and will be used to compare similar products. We maintain these embeddings in a Entity database which holds entity-embedding for this product.
The embedding model can be generated offline - it's better to do this not on the fly but maybe a job scheduler which runs periodically to populate the embedding database.
2) The recommendation service will consist of two important parts - Retrieval of the last x products purchased by the user, and fetching y similar products for all those x products.
Ranking and Sorting those xy products to generate a list of recommendations.
3) Retrieving the last x products will be from the user-purchase history database , we can cache the top x in a cache, and query from there.
Fetching the similar entities for those , we will have to build kind of a neighbour index entry for each of the entities.
In short, we have to fetch the nearby vectors for a given vector. We could implement and put a vector database for this, which is essentially similar to a geohash index. Similar vectors will be living on the same partition.
1) Convert each vector to a geohash
2) Run linear scans on the bounding boxes for the above geohash to find at least z elements close to this vector.
3) Compute distance to our entity for which we need to fetch recommendations .
4) Sort the above vectors based on their distances relative to the input vector, and keep top y entries.
This relationship of entity- neighbouring vectors can actually be stored in a neighbour index entry, and can be read from there, instead of computing every time.
When a new entity is added, and let's say the neighbour-index entry will need to be changed for many entities - In order to do this easily, we maintain the distances for the vectors - entity in a max heap, so the new entity can be inserted into the head in O(N log n) time.
4) Ranking and Scoring the Entities generated in the above step - We have x different sorted lists, with y different neighbours, we need to merge these lists, the complexity will be o(y log x)
We can assign some scoring now to all these entities based on some entity interactions for these products ( which will be stored in the entity interaction database already). We can compute a scaling function which will be a function of various entity interactions and apply those to the entities. This will generate the final scores and then we need to sort them.