System requirements

Functional:

1. Photo Uploading:
2. Users should be able to upload photos from their devices.
3. Support various image formats such as JPEG, PNG, and GIF.
4. Handle different sizes of images efficiently.
5. Profile Management:
6. Users can create and customize profiles with personal details.
7. Allow users to upload and set profile pictures.
8. Provide an option for users to add and edit a bio.
9. Feed:
10. Implement a personalized feed displaying photos from users that the current user follows.
11. Utilize algorithms to personalize the feed based on user interactions, such as likes and comments.
12. Support infinite scrolling for a seamless browsing experience.
13. Like and Comment:
14. Users should be able to like photos posted by others.
15. Allow users to leave comments on photos.
16. Implement real-time updates for likes and comments.
17. Settings and Privacy:
18. Provide options for users to manage account visibility (public, private, or custom).
19. Allow users to block other users and manage their block list.
20. Enable users to control notification preferences (e.g., push notifications, email notifications).

Non-Functional:

1. Performance:
2. Ensure low-latency photo uploading and retrieval.
3. Support a large number of concurrent users.
4. Optimize feed generation algorithms for quick and efficient personalization.
5. Scalability:
6. Design the system to scale horizontally to accommodate a growing user base.
7. Implement load balancing to distribute traffic evenly.
8. Reliability:
9. Ensure high availability with minimal downtime.
10. Implement backup and recovery mechanisms for user data.
11. Security:
12. Use secure authentication mechanisms for user logins.
13. Implement encryption for data in transit and at rest.
14. Protect against common web application vulnerabilities such as CSRF and XSS.
15. User Experience:
16. Design an intuitive and user-friendly interface for both mobile and web platforms.
17. Optimize the system for quick response times to user interactions.
18. Provide clear and concise error messages for user guidance.
19. Compatibility:
20. Ensure compatibility with a variety of devices and browsers.
21. Support the latest versions of major operating systems for mobile applications.

Capacity estimation

Let us assume,

• Assume 1 billion users, with 500 million as daily active users.
• Assume 60 million photos and 35 million videos are shared on Instagram per day.
• Consider 3 MB as the maximum size of each photo and 150 MB as the maximum size of each video uploaded on Instagram.
• On average, each user sends 20 requests (of any type) per day to our service.

Total Storage Required Per Day

Photos: 60 million photos/day * 3 MB = 180 TeraBytes / day

Videos: 35 million videos/day * 150 MB = 5250 TB / day

Total content size = 180 + 5250 = 5430 TB

The Total Space Required for a Year:

5430 TB/day * 365 (days a year) = 1981950 TB = 1981.95 PetaBytes

API design

1. Photo Uploading API:

• Description: This API empowers users to upload photos to the platform. It accepts the User authentication token, the image file, and an optional caption as input. The output is a confirmation message along with the metadata of the uploaded photo.

2. Profile Management API:

• Description: Responsible for managing user profile details and settings, this API utilizes the User authentication token and profile information (e.g., bio, profile picture) as input. Users receive a confirmation message as output, signaling the successful update of their profile details.

3. Feed Retrieval API:

• Description: This API retrieves a personalized feed for the user, leveraging their User authentication token as input. The output consists of a list of photos with metadata, tailored to the user's preferences, ensuring an engaging and relevant content experience.

4. Like API:

• Description: Dedicated to facilitating user engagement, the Like API enables users to express appreciation for photos. Input parameters include the User authentication token and the photo ID, with the output being a confirmation message and an updated like count.

5. Comment API:

• Description: Enabling user interaction through comments, this API accepts the User authentication token, photo ID, and comment text as input. The output comprises a confirmation message and an updated comment count, fostering a dynamic and interactive user environment.

6. Settings and Privacy API:

• Description: This API empowers users to manage their privacy settings and preferences. Input parameters encompass the User authentication token and privacy settings, with the output providing a confirmation message and reflecting the updated privacy preferences.

7. User Search API:

• Description: The User Search API facilitates user discovery by allowing users to search for others. It requires the User authentication token and a search query as input. The output includes a list of user profiles matching the provided search query.

8. User Follow/Unfollow API:

• Description: Supporting social connections, this API allows users to follow or unfollow others. Input parameters comprise the User authentication token, target user ID, and the action (follow/unfollow). The output consists of a confirmation message and the updated follower/following counts.

9. Trending API:

• Description: This API provides a curated list of trending photos on the platform. It optionally takes the User authentication token as input and outputs a list of trending photos along with their metadata, ensuring users stay informed about popular content.

10. Notification API:

• Description: The Notification API delivers real-time notifications to users. Utilizing the User authentication token as input, it retrieves the list of unread notifications for the user. The output includes this list, enhancing user engagement through timely updates.

Database design

For the tables required in this design, refer to the class diagram, the list of classes is not exhaustive but this is a good number of tables to start with.

Let us now discuss the different Databases and CAP Theorem:

1. User Profile and Photo Metadata Database (e.g., MySQL):
2. Implication: For user-profiles and photo metadata, strong consistency may be preferred to maintain accurate and up-to-date information. Availability can be achieved by replicating the database, but it might impact write latencies.
3. Design Consideration: Employ techniques like database clustering and replication, but be mindful of potential trade-offs in terms of write latencies.
4. Activity Feed and Notifications Database (e.g., Cassandra or MongoDB):
5. Implication: For the activity feed and notifications, eventual consistency may be acceptable, allowing for higher availability and improved response times.
6. Design Consideration: Use NoSQL databases that support eventual consistency, allowing for better scalability and availability in scenarios where real-time consistency is not critical.
7. Authentication and Session Management Database (e.g., Redis):
8. Implication: High availability is crucial for authentication and session management. Immediate consistency might be more important to ensure that users are authenticated correctly.
9. Design Consideration: Consider using in-memory databases like Redis for authentication and session management to ensure fast response times and high availability.
10. Comments:
11. Storage System: NoSQL database (e.g., MongoDB, Cassandra)
12. CAP Theorem Implications: Comments can be stored in a NoSQL database, providing scalability, flexibility, and eventual consistency. The focus might be on availability and partition tolerance rather than strong consistency, making NoSQL a suitable choice for accommodating a large volume of comments.
13. Photos:
14. Storage System: Object Storage (e.g., Amazon S3, Google Cloud Storage)
15. CAP Theorem Implications: Photos can be stored in object storage, leveraging the high availability and partition tolerance offered by cloud-based solutions. Eventual consistency is often acceptable for photos, making object storage a scalable and reliable option.
16. Videos:
17. Storage System: Object Storage (e.g., Amazon S3, specialized video hosting services)
18. CAP Theorem Implications: Similar to photos, videos can be stored in object storage, benefiting from the high availability and partition tolerance of cloud-based solutions. Eventual consistency is often sufficient for video storage.

Let us now discuss concepts like partitioning, sharding

1. Data Partitioning:

• For Partitioning the data, we can use 1 of the below strategies
• Hash-based Partitioning: Assigns each data item to a partition based on a hash function applied to a chosen attribute. This evenly distributes data across partitions.
• Range-based Partitioning: Divides data based on ranges of values in a chosen attribute. Commonly used for time-based data where entries are time-stamped.
• Additionally, we will also need to partition the data regionally and geographically, Partitioning by Country or Continent and creating data centers in multiple regions will help us achieve reduced latency and high availability.

2. Sharding:

Sharding is a database architecture strategy that involves horizontally dividing a large dataset across multiple independent database instances or shards. Each shard operates as a separate database, managing its subset of the data. Sharding is employed to distribute the load, improve performance, and handle large volumes of data in a scalable manner.

Below are a few ways we can apply sharding to our data

1. User Data Sharding:
2. Scenario: As the user base grows, sharding can be applied to distribute user profiles across multiple shards.
3. Implementation: Shard based on user IDs or usernames.
4. Each shard is responsible for managing a subset of user profiles.
5. Content Sharding:
6. Scenario: Sharding can be applied to manage photos and videos efficiently.
7. Implementation: Shard based on content IDs or timestamps.
8. Each shard is responsible for storing and serving a subset of media content.
9. Activity Feed Sharding:
10. Scenario: Sharding can optimize the generation of activity feeds for users.
11. Implementation: Shard based on user IDs or a combination of user and timestamp.
12. Each shard handles the activity feed for a subset of users.

High-level design

In a high-level design for an Instagram-like system, we can identify various components that collectively address the end-to-end functionality of the platform. Here are the key components:

1. User Interface (UI):
2. Responsibility: Provides the graphical interface for users to interact with the platform, including features like uploading photos, viewing feeds, and managing profiles.
3. Components: Various UI elements for different features, responsive design for mobile and web.
4. Frontend Services:
5. Responsibility: Handles user requests from the UI and communicates with backend services. Responsible for rendering dynamic content and managing user interactions.
6. Components: Frontend logic, API communication, state management.
7. Authentication Service:
8. Responsibility: Manages user authentication and authorization. Generates and verifies authentication tokens for secure access to user accounts.
9. Components: Authentication logic, token generation, user sessions.
10. User Service:
11. Responsibility: Manages user-related operations, including profile creation, updating, and searching for users. Handles follow/unfollow functionality.
12. Components: User profile management, search logic, follow/unfollow operations.
13. Content Service:
14. Responsibility: Manages the storage and retrieval of photos, videos, and comments. Handles photo uploading, commenting, and displaying content in feeds.
15. Components: Storage integration (e.g., Object Storage), upload/download logic, comment management.
16. Activity Feed Service:
17. Responsibility: Generates personalized activity feeds for users based on their interactions and follows. Utilizes algorithms to prioritize and order content.
18. Components: Feed generation logic, content ranking, personalization algorithms.
19. Notification Service:
20. Responsibility: Manages the delivery of real-time notifications to users for actions like likes, comments, and new followers.
21. Components: Notification logic, delivery mechanisms, unread notification tracking.
22. Privacy and Settings Service:
23. Responsibility: Handles user privacy settings and preferences, including account visibility, notification preferences, and blocking users.
24. Components: Privacy logic, settings management, user blocking.
25. Analytics Service:
26. Responsibility: Collects and analyzes user interactions and platform usage for insights into user behavior, engagement, and system performance.
27. Components: Data collection, analytics algorithms, reporting.
28. External Integrations:
29. Responsibility: Interfaces with external services for tasks such as sending emails, handling user authentication (e.g., OAuth), or integrating with third-party APIs.
30. Components: Email services, OAuth integration, API connectors.
31. Database Systems:
32. Responsibility: Stores and manages data in various databases, including user profiles, content, sessions, and settings.
33. Components: User database, content database, session database.
34. Caching Layer:
35. Responsibility: Improves performance by caching frequently accessed data, such as user profiles, feed content, and trending photos.
36. Components: Cache management, data pre-fetching.
37. Load Balancer and Scaling Components:
38. Responsibility: Distributes incoming requests across multiple server instances to ensure load balancing and scalability.
39. Components: Load balancer, auto-scaling mechanisms.
40. Security Components:
41. Responsibility: Implements security measures, including encryption, secure communication, and protection against common web vulnerabilities.
42. Components: SSL/TLS, encryption modules, security policies.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

1. User Interface (UI):

• Description: The UI provides a graphical interface for users to interact with the platform. It includes various components for uploading photos, managing profiles, and viewing feeds.
• Functionality: Renders dynamic content based on user interactions.
• Sends requests to Frontend Services to fetch or update data.

2. Frontend Services (FS):

• Description: Frontend Services handle user requests from the UI and communicate with backend services. They are responsible for rendering dynamic content and managing user interactions.
• Functionality: Communicates with Authentication Service for user login and session management.
• Interacts with User Service for user-related operations.
• Utilizes Content Service for photo and video management.
• Engages with Activity Feed Service for generating personalized feeds.
• Sends notifications through Notification Service.
• Manages user privacy settings through Privacy Settings Service.
• Gathers analytics data through Analytics Service.

3. Authentication Service (AS):

• Description: Manages user authentication and authorization, ensuring secure access to user accounts.
• Functionality: Handles user login, generating and verifying authentication tokens.
• Manages user sessions and ensures secure communication.

4. User Service (US):

• Description: Manages user-related operations, including profile creation, updating, and searching for users. Handles follow/unfollow functionality.
• Functionality: Manages user profiles and interactions.
• Handles follow/unfollow operations and user searches.

5. Content Service (CS):

• Description: Manages the storage and retrieval of photos, videos, and comments.
• Functionality:Handles photo and video uploading and downloading.
• Manages comments and interactions on content.

6. Activity Feed Service (AFS):

• Description: Generates personalized activity feeds for users based on their interactions and follows.
• Functionality: Utilizes algorithms to prioritize and order content in activity feeds.
• Provides personalized recommendations and trending content.

7. Notification Service (NS):

• Description: Manages the delivery of real-time notifications to users for actions like likes, comments, and new followers.
• Functionality: Sends real-time notifications to users.
• Tracks unread notifications for users.

8. Privacy Settings Service (PSS):

• Description: Handles user privacy settings and preferences, including account visibility, notification preferences, and blocking users.
• Functionality: Manages user privacy settings and preferences.
• Supports user blocking functionality.

9. Analytics Service (AS2):

• Description: Collects and analyzes user interactions and platform usage for insights into user behavior, engagement, and system performance.
• Functionality: Gathers analytics data on user interactions, content popularity, and system performance.
• Generates reports and insights for decision-making.

10. External Integrations (EI):

• Description: Interfaces with external services for tasks such as sending emails, handling user authentication (e.g., OAuth), or integrating with third-party APIs.
• Functionality: Manages external integrations for features like email notifications and social media logins.

11. Database Systems (DB):

• Description: Stores and manages data in various databases, including user profiles, content, sessions, and settings.
• Functionality: The user Database stores user profiles and authentication data.
• Analytics Database stores data for analytics and reporting.
• Photo and Video Database stores media content.

12. Caching Layer (CL):

• Description: Improves performance by caching frequently accessed data, such as user profiles, feed content, and trending photos.
• Functionality: Caches data to reduce the load on the backend databases.
• Implements data pre-fetching strategies.

13. Load Balancer (LB):

• Description: Distributes incoming requests across multiple server instances to ensure load balancing and scalability.
• Functionality: Routes user requests to available server instances.
• Distributes traffic evenly to prevent overload on specific servers.
• Enhances fault tolerance by rerouting traffic in case of server failures.

14. Scaling Components (SC):

• Description: Manages the automatic scaling of resources based on demand to handle varying workloads.
• Functionality: Automatically adjusts the number of server instances based on traffic.
• Ensures optimal resource allocation during peak and off-peak periods.

15. Security Components (SEC):

• Description: Implements security measures, including encryption, secure communication, and protection against common web vulnerabilities.
• Functionality: Enforces SSL/TLS for secure communication.
• Implements encryption for sensitive data.
• Implements security policies to prevent unauthorized access.

Load Balancers, Caching, and CDN:

• Load Balancers (LB):
• Ensures even distribution of traffic among server instances.
• Enhances fault tolerance by rerouting traffic during server failures.
• Supports scalability by distributing requests across multiple servers.
• Caching (CL):
• Improves performance by reducing the load on backend databases.
• Stores frequently accessed data to minimize response times. This can be used to store contents from social media influencers and important personalities.
• Content Delivery Network (CDN):
• Distributes static assets (images, videos) to edge servers for faster content delivery.
• Reduces latency for users by serving content from geographically distributed servers.
• Enhances the overall user experience by accelerating the loading of media content.

Trade-offs / Tech choices

For Databases, we have chosen different databases depending on different storage requirements and CAP theorem.

For user data, we have chosen a RDBMS, since we need high consistency.

For photos and videos, we have chosen Amazon S3 to get high scalability, high availability, and partition tolerance.

For comments, Activity Feed, and Notifications, we have chosen no-SQL databases, since they need to be eventually consistent, with high availability.

For Session management, we have used Redis to have low latency with frequent authentication and token generation services.

Future improvements

We could have included an additional database for storing the metadata for each post in a different database and could have included more services that run in the background, that generate feed for each user and store them.