Design Instagram - System Design

System Requirements

Functional:

Photo Upload
1. Users should be able to upload photos from their devices
2. Support various image formats such as JPEG, PNG, GIF etc.
3. Handle different image sizes effectively
Profile Management
1. Users can create and customize their profile with personal details
2. Allows users to be able to set their profile picture
3. Provide an option for users to add and edit their bio
Feed
1. Implement a personalized feed displaying photos from users that the current user follows
2. Support infinite scrolling for a seamless browsing experience
3. Utilize algorithms to personalize the feed based on user interactions, such as likes and comments
Likes and Comments
1. Users should be able to like photos posted by others
2. Users should be able to comment on a photos
3. Implement real time updates for likes and comments
Settings and Privact
1. Provide options for users to manage account visibility (public, private or custom)
2. Allow users to block other users and manage their block list
3. Enable users to control notification preferences (E.g. push notifications, email notifications)
Friends
1. Allow users to follow other users
2. Allow users to block certain users

Non-Functional:

Performance
1. Ensure low latency photo uploading and retrieval
2. Support a large number of concurrent users
3. Optimize feed generation algorithms for quick and efficient personalization
Scalability
1. Design the system to scale horizontally to accommodate a growing user base
2. Implement load balancing to distribute traffic evenly
Reliability
1. Ensure high availability with minimal downtime
2. Implement backup and recovery mechanisms for user data
Security
1. Secure authentication mechanisms for user logins
2. Implement encryption for data in transit and at rest
3. Protect against common web application vulnerabilities such as CSRF and XSS
User Experience
1. Design an intuitive and user friendly interface for both mobile and web platforms
2. Optimize the system for quick response times to user interactions
3. Provide clear and concise error messages for user guidance
Compatibility
1. Ensure compatibility with a variety of devices and browsers
2. Support the latest versions of major operating systems for mobile operations

Capacity estimation

We assume a total of 1 Billion users, with 500 million as daily active users
We assume that 100 million photos and 100 million videos are stored on Instagram everyday
We consider each photo is of size 5MB and each video is of size 100 MB
On average, each user sends on average 20 requests per day
For photos we would need, 100 * 10^6 * 5 * 10^6 bytes = 5 * 10^14 = 500 TB data everyday
For videos we would need, 100 * 10^6 * 100 * 10^6 = 10^13 = 1 PB data
Total storage space required is, 1500 TB everday
Across a month, we would need 45,000 TB data = 45 PB
Across a year, we would need 540 PB data
Factoring in replication, we would need 2000 PB data in a year

API Design

Photo Upload
1. Uploads a photo or video to the platform. Apart from the media file, also takes in a caption of the post. The result of the API is a confirmation message along with metadata of the uploaded photo or video
Profile Management
1. Responsible for managing user profile and settings, this API utilizes the user authentication token and profile information as input. User receives a success message, signaling that their profile has been successfully updated
Feed Retrieval
1. Retrieves the feed of a user based on the people the user follows again by using the user authentication service
Like
1. Used to express appreciation for a post, parameters would include the post ID, user ID which likes the photo. Increments the number of likes of the post by 1
Comment
1. Allows the user to post a comment on a post. Accepts the user authentication token as a parameter, user ID, the post ID on which the comment is being made. The user receives a success message along with a comment being added to the post and its comment count increased by 1
Settings and Privacy
1. This allows the user to update their privacy settings. As a parameter it would accept the changed parameter, all existing parameters would be a part of the body and be optional. As a response, it would return success and the privacy settings would be altered
User Search
1. This allows the user to search for other users. Takes in the user authentication token and the input parameters and does a search on the user table based on that information
User Follow / Un-follow
1. Allows a user to follow or un-follow another user. Takes in the user authentication token along with the user that we wish to follow or un-follow. Returns a success message
Trending
1. This provides a curated list of trending photos on the platform. It takes in the user authentication token as input and returns the currently most popular content
Notification
1. This delivers real time notification to users. Using the user authentication token, it returns all the unread notifications to the user

Database design

The tables and their preferred stores are listed below.

User Profile and Photo Metadata Database
1. Here we need strong consistency to maintain up to data and accurate information. Availability can be achieved by replicating the database, but it might impact write latencies
2. Employ techniques like database clustering and replication but be mindful of potential trade offs in terms of write latencies
Activity Feed and Notification Database
1. Here eventual consistency is acceptable, we do not see a post the moment it is made, non relational databases like Cassandra or MongoDB will allow for better scalability and availability in scenarios where real time consistency is not critical
Authentication and Session Management Database
1. High availability is crucial for authentication and session management. Immediate consistency might be more important to ensure that users are authenticated correctly
2. We can use an in-memory database like Redis for authentication and session management to ensure fast response times and high availability
Comments
1. We can use a non-relational database like Cassandra to store comments. The focus here would be on strong availability and partition tolerance rather than consistency, making a non-relational database like Cassandra a great choice
Photos & Videos
1. Media objects can be stores in an object store like S3

High-Level Design

Key components are given below.

User Interface
1. Responsibility - Provides the graphical interface for the user to interact with the platform
2. Components - Various UI elements for different features, responsive design for mobile and web
Front End Service
1. Responsibility - Handles user requests from the User Interface and coordinate with the back-end service to return results to the user
2. Components - Front end logic, API logic, state management
Authentication Service
1. Responsibility - Manages user authentication and authorization. Generates and verifies user tokens for secure access to the back-end service
2. Components - Authentication logic, token generation, user sessions
User Service
1. Responsibility - Manages user related operations, including profile creation, updating and searching for users. Also takes care of follow and un-follow functionality
2. Components - User profile management, follow and un-follow, user search
Content Service
1. Responsibility - Manages the upload and retrieval of photos and videos and posting and fetching comments on a post, contents of a feed
2. Components - Object Storage Integration, Content Management, Upload/Download Logic
Activity Feed Service
1. Responsibility - Generates user feed based on their interactions and follows. Utilizes a ranking algorithm to rank posts based on engagement
2. Components - Feed Generation, Ranking System
Notification Service
1. Responsibility - Manages the responsibility of sending real time notifications to users for actions like likes, comments and new followers
2. Components - Notification Delivery
Privacy and Settings Service
1. Handles user privacy settings and preferences, including account visibility, notification service and blocking users
2. Components - Privacy and settings management
Analytics Service
1. Responsibility - Collects and analyzes user interactions and platform usage for insights into user behavior, engagement, and system performance
2. Components - Analytics

graph TD subgraph UserInterface UI(User Interface) end subgraph Frontend_Services FS(Frontend Services) end subgraph Authentication_Service AS(Authentication Service) end subgraph User_Service US(User Service) end subgraph Content_Service CS(Content Service) end subgraph Activity_Feed_Service AFS(Activity Feed Service) end subgraph Notification_Service NS(Notification Service) end subgraph Privacy_Settings_Service PSS(Privacy and Settings Service) end subgraph Analytics_Service AS2(Analytics Service) end subgraph External_Integrations EI(External Integrations) end subgraph User_Database UD(User Database) end subgraph Analytics_Database AD(Analytics Database) end subgraph Photo_Video_Database PVD(Photo and Video Database) end subgraph User_Caching UC(User Caching) end subgraph Analytics_Caching AC(Analytics Caching) end subgraph PhotoVideo_Caching PVC(Photo and Video Caching) end subgraph Database_Systems DB(Database Systems) end subgraph Caching_Layer CL(Caching Layer) end subgraph Load Balancer LB(Load Balancer) SC(Scaling Components) end subgraph a SEC(Security Components) end UI --> FS FS --> NS FS --> PSS FS --> AS2 FS --> EI FS --> SEC FS --> AS AS --> FS FS --> LB LB --> US CL --> CS LB --> AFS CS --> DB AS2 --> AC AC --> AD LB --> CL LB --> UC LB --> AC LB --> PVC UD --> UC AD --> AC PVD --> PVC

Request Flows

sequenceDiagram participant User participant Frontend participant Backend participant Database participant Storage User ->> Frontend: Request to upload photo Frontend ->> Backend: Submit photo data and metadata Backend -->> Database: Save photo information Backend ->> Storage: Store photo file Backend -->> Frontend: Acknowledge upload completion Frontend -->> User: Photo successfully uploaded Database --x Backend: Retrieve user's feed data Backend -->> Frontend: Send feed data to display Frontend -->> User: Display user's personalized feed

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade Offs/Tech Choices

For Databases, we have chosen different databases depending on different storage requirements and CAP theorem.

For user data, we have chosen a RDBMS, since we need high consistency.

For photos and videos, we have chosen Amazon S3 to get high scalability, high availability, and partition tolerance.

For comments, Activity Feed, and Notifications, we have chosen no-SQL databases, since they need to be eventually consistent, with high availability.

For Session management, we have used Redis to have low latency with frequent authentication and token generation services.

Failure Scenarios/Bottlenecks

Future Improvements

We could have included an additional database for storing the metadata for each post in a different database and could have included more services that run in the background, that generate feed for each user and store them.