System requirements

Functional:

1. Users should be able to upload photos
2. Users should be able to view other users photos
3. In theory we should be able to store all the photos

Non-Functional:

1. System should be highly available
2. System should be reliable (no photos lost)

Capacity estimation

Assume 100 requests per second * 200 kb (average photo size) * 0.1 (write ratio to read)=2mb per second

2mb * 3600 * 356 * 10 years=  63.072 TB for photos

Since the users table , and the follower, follow table are small in size we can ignore that.

API design

1. POST : post_picture(user_id,photo_path)
2. returns 200 if succesful
3. 400 if user_id does not exsists or invalid path
4. 500 if there is a server error
5. GET: get_feed(user_id)
6. returns a collection of image links if succesful sorted in chrnological order
7. returns 400 if user_id does not exsists
8. returns 500 if internal server error
9. POST: visit_picture(picture_id)
10. returns the image link (in larger size)
11. returns 400 if the image is not present
12. returns 500 if there is an error

Database design

Pictures:

1. picture_id (can be a predifined hash value)
2. user_id
3. meta_data
4. created_at

Users:

1. User_id
2. pass_word_hash
3. created_at

Follow:

1. user_id. (foreign_key to user_table)
2. follower_id (foreign key to user table)

New_Feed:

1. user_id
2. collection of photo_id as json

To Store the photos we will use a service such as S3

High-level design

Assumptions:

This is a heavy read application.

Scalability:

1. Since the bottleneck of our reads will be getting the pictures we can shard based on the picture_id. So a user can have multiple pictures going to different shards.
2. For photoId we will use hashing and a time stamp.The time stamp will allow us to sort the database (index it) based on time and help us with creating the news_feed.
3. To address the traffic on our services we will have a separate write service and a separate read service and run multiple instances of them
4. We will use a load balancer to distribute the load amongst our services.The load balancer will direct calls based on the load on each instance of services running
5. We will use a NOSQL database for our photo meta data as it is much easier to scale horizantally given that we will practically store the photos for ever.
6. S3 can easily be scaled to address scalability issues regarding storage of the actual photos.
7. To prevent our services being spammed we will use a rate limiter based on IP address and a bucket algorithm. It will block too many calls from one IP and also use the bucketing strategy to handle distributed attacks or abuses.
8. To help with read throughput we will use a write around cache to decrease direct reads from our db. While the cache size can be dynamic we can assume 20% cache size ,assuming 20% of the content is causing 80% of the reads

Fault Tolerance and Reliability

1. We will have multiple instances (2x) of the servers needed, to act as back up and aid in surges of requests.
2. All the persistent database will have back ups to prevent dataloss.
3. We will have a backup db and s3 for all our persistent data needs. This is different from replicas of the original database. This backup can be stored in a different region.
4. We can introduce canary testing and load testing in off peak times to address potential gaps in terms of system being able to perform under pressure.

Writes:

1. After redirection from the load balancer to a instance with minimal load, we will write the content of the write to our db and store the photo in S3. To create a photoid we will use a epoch time stamp + an md5 hash. This esnures uniqueness and that the photos can be sorted later on. To prevent datalosses we can write to a write ahead log and then copy the content to S3.
2. Return a succesful response or appropriate error message

News Feed (Reads)

1. Since the photoID are sorted we will use the followers table to prepopulate a news feed table

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?