Loading...
Back of the envelope calculations
This sounds like a professional network for a particular industry, such as software engineers. Lets assume 10 million users then and 1 million DAU. Traffic will vary for events like recruiment drives, and lay offs from large companies.
Throughput
1 million DAU, average user makes 5 queries per day. 5 million queries daily, ~50 queries per second.
The only updates that will happen will be for creating new connections and update profiles. Thus there are much more reads than write, we can assume a 10:1 read to write ratio
45 read queries per second, 5 write queries
Storage
If we store 1 kb per user, we will have 10 million KB or 10 GB of storage needed. This is small and easy to handle in one database instance.
Database
We will use a graph database like Neo4J to efficiently traverse connections.
find_shortest_connection(user1, user2)
discover_connections(user1, user2)
suggest_connections(current_user)
search_users(current_user, search_criteria) - search_criteria can be many things name, profession, job title, users of experience. anything a user would want to search by
Database Design
user
user_id
first_name
last_name
connections: user_id[]
work_experience: experiences[]
education_experience: educations[]
is_looking_for_work: boolean
Graph traversal service
The api service needs to use the underlying graph database to find the shortest path. This can be achieved by using BFS on user1 and user2, then continuing until they find a match.
Far flugn connections may take a long time to process.
Suggested connection service
During off hours we want to constantly processing and creating suggestions for users. This will be an algorithm or machine learning model that we adapt over time as we collect data points and whether users accept the suggestions. We want to pre compute these results and push them to users either as a notification or a pop up when they open the app.
Search users service
We need to be able to search for users by other criteria. Filtering by fields will be easy, but because of the sheer volume of users we probably want to have a priortization mechanism as well to suggest users that they will be interested in following wehn the query is non specific. For example, recommending well known experts in the field. Or factoring in the user's geolocation.
Graph traversal service
Doing this effeciently is key. Firstly we can use a CDN to reduce latency and ensure that network latency is low. Each CDN will send traffic to a Load Balancer with a set of distributed services. Since first order traversals will be fast and 5th order traversals will be very slow we need a strategy that tracks the memory usage and gives it to the computer with the least amount of memory usage. This also needs to scale horizontally to traffic surges. There also needs to be a TTL cut off for requests to prevent two extermely distant users from breaking a whole server.
Memory will be the bottleneck here as in higher order connections we will be traversing a large large number of nodes at once.
Typically we will want to do a bidirectional BFS and stop when the 2 meet. But for exteremely popular celebrities we will only want to search from the less popular user, as one layer of connections from a celebrity will be massive.
Suggested connection service
This will use the graph database as inputs to train and create machine learning models to suggest connections for active users. This work can be done continuoulsy and suggestions either pushed an in app notification system or simply written to a notification table that will display when user opens the app. We want to keep track of how often suggestions are accepted to continously improve the algorithm.