Loading...
Functional:
Extended Requirements:
Assume 1 million DAU
load: average 1 post per user every day. 12 posts/sec. user gets timeline 5 times a day. 60 timelines/sec
storage: average post is 5MB. 5TB/day. 1535TB/year of storage.
bandwidth: 60MB/sec creating posts. 3GB/sec for retrieving timelines.
resources: 1440ms/sec of cpu processing power needed. 2 cores required.
createPost(userID, image, metadata)
likePost(userId, postID)
commentPost(userID, postID, comment)
createProfile(userName, name, email, profilePicture)
getTimeline(userId)
followUser(userID, followeeID)
unFollowUser(userID, followeeID)
A lot of storage is needed.
Needs to be scalable if we are not deleting posts
For user profiles we can user a graph based no sql Database like Neo4j to save user profiles and also provide links to followers.
User Table:
USER_ID_PK
USER_NAME_TXT
NAME_TXT
EMAIL_TXT
PROFILE_PIC_ID
save metadata in a column based DB like cassandra and save images in a cloud based storage like s3 that is scalable.
We can save post data also in a cassandra table since this data will be write heavy. We can use additional columns approach to store post comments.
POST_TABLE:
POST_ID_PK
POST_LIKES_NBR
COMMENTS_NBR
COMMENTS
flowchart TD
B[client] --> C{LB}
C --> D{API Gateway}
D --> E[Profile Service]
D --> F[Post Service]
D --> G[Timeline Service]
E --> H[(Profiles DB)]
H --> I[(Replica 1)]
H --> J[(Replica 2)]
E --> K[(Cache)]
K --> I
F --> O[(S3 storage)]
G --> P[(CDN)]
P --> O
P --> M
F --> L[(Post DB)]
L --> M[(Replica 1)]
L --> N[(Replica 2)]
G --> K
E --> Q[Notification queue]
F --> Q
Q --> B
When a user creates a profile it will be routed by the load balancer to the profile service. The profile service will save the data in a graph based DB which will have replicas. When retrieving the profiles we can cache the top 10% of profiles in our cache using the username as the key. when following other profiles we can just add an edge between user data in the neo4j database.
For creating posts the api gateway will redirect to our post service. Here post data + metadata will be saved to our cassandra DB + replicas. Images can be stored in our S3 storage.
For generating timelines we will be redirected to our timeline service. This can hit our cache to find users followers. It will also hit a CDN to retrieve posts. This will process the posts based on user and return timelines paginated to 10 posts at a time.
Our posts Database will be a column based No-SQL DB Cassandra. This DB can be sharded to allow for scaling be the number at the end of the post_ID. This DB will be responsible for housing info about posts including metadata, post likes and comments. comments can be added to the wide columns and if there a too many columns we can partition this data vertically as well.
Profiles will be stored in a graph based no Sql database like neo4j. for profile followers we can use edges between users in the database to represent connections between different user profiles. the timeline service will also use these edges to generate for the user their timeline.
We can make use of our notification queue which will ping the user whenever there is a new follow or interaction with their posts.
We use S3 storage for storing images instead of our own DB as it is easily scalable and we need a lot of storage.
We use a CDN to cache the most frequently seen posts to reduce the load on the DB.
We use a graph based DB instead of a sql database so we can have relationships between different users in the form of edges.
We use cassandra column based storage for post data since this will be write heavy operations updating post likes and comments.
We can use an api Gateway to route requests to the correct service.
We use a notification queue so that we can push notifications back to the client.
We can use a CDN and redis cache to reduce load on the DBs and reduce latency.
Our load balancer can fail so we can add multiple instances of it same goes for our api Gateway.
services can fail so we can add more services to get around this.
Databases may fail so we can have replicas just in case.
We can use kubernetes to bring back downed nodes.
Our cache may have a queue to access data so we can have multiple instances to reduce the load here.
We can have multiple databases in different regions to reduce latency.
We can use the graph based db in future to give users recommendations on who to follow
We can better our timeline service to also show recommended posts in future using the profile graph db also