System requirements

Functional:

User Management:

Users should be able to register, log in, and manage their profiles (e.g., update profile picture, bio, and privacy settings).
Users should have the ability to follow and unfollow other users.
Users should be able to manage their account settings (e.g., password change, email update).

Photo Upload and Management:

Users should be able to upload photos with metadata (e.g., captions, tags).
The platform should allow users to organize photos into albums or collections.
Users should be able to edit or delete photos after uploading.
Photos should be automatically resized and optimized for various devices.

Newsfeed:

Users should be able to see a feed of photos from the users they follow.
The feed should show recent uploads in real time or with slight delay.
Users should be able to like, comment, and share photos in their feed.

Search Functionality:

Users should be able to search for photos using hashtags, tags, or usernames.
Users should be able to browse trending photos, hashtags, or accounts.

Notifications:

Users should receive notifications when someone likes or comments on their photos or when they are followed.
Users should be notified about new followers and photo interactions.

Privacy and Security:

Users should be able to set their profiles or photos to private or public.
The platform should allow users to block or report other users for inappropriate behavior.
Photos should be secured and stored with proper encryption.

Engagement:

Users should be able to like, comment, and share photos posted by others.
The platform should allow users to tag others in photos and comments.

Non-Functional:

Performance:

The system should be able to handle a high volume of photo uploads and interactions, ensuring low-latency access and display of photos.
Photo upload times should be under 5 seconds for users with good internet connections, even with large files.

Scalability:

The platform should scale horizontally to accommodate a growing number of users and uploaded photos.
It should be able to scale to handle millions of daily active users and hundreds of millions of photos.

Availability:

The system should be highly available with an uptime of 99.99%, ensuring users can upload, view, and interact with photos at any time.
It should have mechanisms in place for failover and recovery.

Security:

User data, including passwords, email addresses, and photos, should be securely stored with end-to-end encryption.
The system should implement OAuth or other secure authentication mechanisms.
It should protect against common web vulnerabilities, such as SQL injection and cross-site scripting (XSS).

Data Retention:

The platform should store photos for a defined period based on the user's privacy settings and account activity.
Users should have the ability to delete their photos or accounts, and the system should ensure complete data removal upon request.

Usability:

The platform should be easy to use, with intuitive interfaces for uploading, sharing, and interacting with photos.
It should provide mobile-friendly access and a responsive design for multiple devices.

Reliability:

The system should be reliable under heavy load, with appropriate caching and content delivery network (CDN) support for quick photo access.
Data should be backed up regularly, with proper disaster recovery procedures in place.

Compliance:

The system should comply with GDPR or other relevant privacy laws for user data protection, providing users with control over their personal data.
The platform should have measures for handling user content in accordance with community guidelines and legal regulations.

Capacity estimation

1. Active Users

Assumptions:
- The platform is expected to have 10 million active users.
- Each user interacts with the platform at least once per day (either uploading photos, browsing the feed, or interacting with posts).
Daily Active Users (DAU): 10,000,000 users
Peak Concurrent Users: 10% of DAU (1,000,000 concurrent users) during peak hours.

2. Photo Uploads

Assumptions:
- Each user uploads an average of 1 photo per day.
- Photo file size: An average of 2 MB per photo.
- Photos per day: 10 million photos/day
- Peak Uploads: Assume 10% of the uploads happen during peak hours.
Total Photo Uploads per Day: 10 million photos
Peak Uploads per Hour: 10% of 10 million = 1 million photos/hour

3. Photo Storage

Assumptions:
- Each photo is 2 MB in size.
- Users keep all their uploaded photos for a long time (5 years for data retention).
Storage per Year:
- Total Storage per Day: 10 million photos/day * 2 MB/photo = 20 TB/day.
- Total Storage per Year: 20 TB/day * 365 days = 7,300 TB/year (7.3 petabytes).
Peak Storage Requirements: Assume the platform will need to scale to handle sudden spikes in uploads, so include redundancy and backup storage (30% extra).
- Total Storage Estimate (including redundancy): 7.3 petabytes * 1.3 = 9.5 petabytes per year.

4. Network Bandwidth

Assumptions:
- Each photo is accessed 3 times on average per day (view, like, comment, etc.).
- Video and high-quality media could contribute to network traffic as well.
Daily Data Access:
- Total Data Access per Day: 10 million photos * 3 accesses/photo = 30 million accesses/day.
- Data Transfer per Day: 30 million accesses * 2 MB/photo = 60 TB/day.
Peak Data Transfer:
- Assume 10% of the data access happens during peak hours: 6 TB/hour.

5. System Resources and Processing

Assumptions:
- API Requests: Each user makes an average of 20 API requests per day (browsing feed, liking photos, commenting, etc.).
- Backend Queries: Assume each request involves interacting with databases and external services like caching, authentication, etc.
API Requests per Day:
- Total Requests per Day: 10 million users * 20 requests/day = 200 million API requests/day.
- Peak API Requests per Hour: 200 million requests/day ÷ 24 hours = 8.3 million requests/hour.
Compute Power: The platform will need significant compute resources for:
- Handling API requests, photo processing (compression, resizing), and media streaming.
- Auto-scaling can be used to dynamically allocate compute resources based on usage patterns.

6. Database Storage

Assumptions:
- User Data: Basic user profile data (name, email, password) takes up around 1 KB per user.
- Meeting Data: Additional metadata about the photos, such as comments, likes, and shares.
- PostgreSQL or NoSQL could be used for user data, photo metadata, and other structured data.
User Database Storage:
- Total User Data per Year: 10 million users * 1 KB/user * 365 days = 3.65 GB/year for user profiles.
Metadata (Post/Comments/Interactions):
- Assume 500 KB per photo for metadata (comments, likes, shares, etc.).
- Total Metadata Storage per Year: 10 million photos/day * 500 KB/photo * 365 days = 1.83 petabytes/year.

API design

1. User Authentication APIs

POST /login
- Description: Authenticates a user by validating their email and password. Returns a JWT token for session management.
POST /register
- Description: Registers a new user by creating a profile with details such as email, password, and username.
POST /logout
- Description: Logs out the user by invalidating their JWT token, ending the session.
POST /password-reset
- Description: Sends a password reset link to the user's email for resetting their password.

2. Photo Upload and Management APIs

POST /photos
- Description: Allows a user to upload a new photo, including metadata like caption, tags, and location. The photo is optimized and stored.
GET /photos/{photo_id}
- Description: Retrieves the details of a specific photo by its unique photo_id.
DELETE /photos/{photo_id}
- Description: Deletes a specific photo by its photo_id and removes it from storage.
PUT /photos/{photo_id}
- Description: Edits metadata of an existing photo (e.g., caption, tags) but keeps the photo itself unchanged.

3. User Profile and Management APIs

GET /users/{user_id}
- Description: Retrieves a user's profile information, including their bio, followers count, and photos.
PUT /users/{user_id}
- Description: Updates a user's profile, including changes to their bio, profile picture, and other personal details.
POST /users/follow
- Description: Allows a user to follow another user. Accepts follower_id and followee_id.
POST /users/unfollow
- Description: Allows a user to unfollow another user.

4. Photo Interaction APIs (Likes, Comments)

POST /photos/{photo_id}/like
- Description: Allows a user to like a specific photo by its photo_id.
DELETE /photos/{photo_id}/like
- Description: Allows a user to remove their like from a specific photo.
POST /photos/{photo_id}/comments
- Description: Allows a user to post a comment on a photo.
DELETE /photos/{photo_id}/comments/{comment_id}
- Description: Allows a user to delete their comment from a photo.

5. Newsfeed and Search APIs

GET /feed
- Description: Returns the user's feed, showing the most recent photos from the people they follow, ordered by relevance (likes, recency, etc.).
GET /search/photos
- Description: Allows users to search for photos based on keywords or hashtags.
GET /search/users
- Description: Allows users to search for other users by their username.

6. Notifications APIs

GET /notifications
- Description: Retrieves the list of notifications for the authenticated user (likes, comments, new followers).
POST /notifications/mark-read
- Description: Marks notifications as read, so they are no longer shown as unread in the user interface.

7. Privacy and Security APIs

POST /photos/{photo_id}/privacy
- Description: Allows a user to set the privacy of a photo (e.g., public or private).
POST /users/{user_id}/block
- Description: Blocks a user, preventing them from following or interacting with the blocked user.
POST /users/{user_id}/report
- Description: Allows a user to report inappropriate content or behavior (e.g., bullying or harassment).

8. Social Integration and Calendar Sync APIs

POST /social/sync
- Description: Syncs the user's photos with external social media accounts (e.g., Facebook, Twitter) or calendar services.
GET /social/friends
- Description: Retrieves a list of the user's friends or followers from other integrated social platforms.

9. Analytics APIs

GET /analytics/user-engagement
- Description: Retrieves engagement data for the authenticated user, such as photo likes, comments, and shares over time.
GET /analytics/trending
- Description: Returns a list of trending photos, hashtags, or users based on popularity metrics.

Database design

1. Users Database

Database Details:
- Table Name: users
- Columns:
  - user_id (Primary Key, INT)
  - username (VARCHAR, unique)
  - email (VARCHAR, unique)
  - password_hash (VARCHAR)
  - profile_picture_url (VARCHAR, nullable)
  - bio (TEXT, nullable)
  - followers_count (INT)
  - following_count (INT)
  - created_at (DATETIME)
  - updated_at (DATETIME)
Purpose:
- Stores user account information, including usernames, passwords (hashed), and basic profile details like bio and profile picture. It tracks user statistics like followers and following counts.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL is a robust relational database system that provides strong ACID compliance and supports complex queries and transactions. It is ideal for managing structured user data and maintaining referential integrity across relationships like users, followers, and posts.

2. Photos Database

Database Details:
- Table Name: photos
- Columns:
  - photo_id (Primary Key, INT)
  - user_id (Foreign Key, INT, references users)
  - caption (TEXT, nullable)
  - location (VARCHAR, nullable)
  - photo_url (VARCHAR)
  - created_at (DATETIME)
  - updated_at (DATETIME)
Purpose:
- Stores metadata about each photo, such as caption, location, and URL (stored in cloud storage like AWS S3). It also tracks the user who uploaded the photo.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL is ideal for managing relational data like photos linked to users. It also supports indexing and querying large datasets efficiently, which is crucial for fast retrieval of photo details for millions of users.

3. Comments Database

Database Details:
- Table Name: comments
- Columns:
  - comment_id (Primary Key, INT)
  - photo_id (Foreign Key, INT, references photos)
  - user_id (Foreign Key, INT, references users)
  - text (TEXT)
  - created_at (DATETIME)
Purpose:
- Stores comments made by users on photos, including the text content and the user who made the comment. Each comment is linked to a specific photo.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL provides efficient querying and indexing for relationships between comments, users, and photos. It also ensures referential integrity, making it easy to retrieve and manage comment data for a given photo.

4. Likes Database

Database Details:
- Table Name: likes
- Columns:
  - like_id (Primary Key, INT)
  - photo_id (Foreign Key, INT, references photos)
  - user_id (Foreign Key, INT, references users)
  - created_at (DATETIME)
Purpose:
- Stores like interactions between users and photos. It tracks which user liked which photo and when.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL is well-suited for many-to-many relationships like likes, where each user can like multiple photos, and each photo can be liked by multiple users. This ensures fast retrieval of like counts and efficient queries.

5. Follows Database

Database Details:
- Table Name: follows
- Columns:
  - follower_id (Foreign Key, INT, references users)
  - followee_id (Foreign Key, INT, references users)
  - created_at (DATETIME)
Purpose:
- Stores follower-following relationships between users. It helps to track which users are following others and enables the implementation of the Newsfeed feature.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL is suitable for managing the many-to-many relationship between users and their followers/followees. This enables efficient queries for retrieving a user's followers or followees, which is crucial for features like feed generation and social interaction.

6. Notifications Database

Database Details:
- Table Name: notifications
- Columns:
  - notification_id (Primary Key, INT)
  - user_id (Foreign Key, INT, references users)
  - type (VARCHAR, e.g., like, comment, follow)
  - message (TEXT)
  - is_read (BOOLEAN)
  - created_at (DATETIME)
Purpose:
- Stores notifications for users, including interactions like new followers, likes on photos, or comments on posts.
Type of Database Chosen:
- PostgreSQL (Relational Database)
Justification for Choosing It:
- PostgreSQL is optimal for storing structured notification data and allows for efficient queries, such as fetching unread notifications or filtering notifications based on their type (like or comment). Additionally, it ensures transactional consistency when processing notifications.

7. Search Database (Hashtags, Metadata)

Database Details:
- Table Name: search_index
- Columns:
  - index_id (Primary Key, INT)
  - photo_id (Foreign Key, INT, references photos)
  - hashtags (TEXT, array or list)
  - tags (TEXT, array or list)
  - created_at (DATETIME)
Purpose:
- Stores indexed metadata about photos, including hashtags and tags, to support fast and efficient searches.
Type of Database Chosen:
- Elasticsearch (NoSQL, Search Engine)
Justification for Choosing It:
- Elasticsearch is highly optimized for search operations and provides powerful features like full-text search, ranking, and relevance scoring. It allows for efficient searching of photos based on hashtags, keywords, or tags.

8. Media Storage

Database Details:
- Table Name: media_storage
- Columns:
  - media_id (Primary Key, INT)
  - photo_id (Foreign Key, INT, references photos)
  - file_url (VARCHAR)
  - file_type (VARCHAR, e.g., image/jpeg)
  - created_at (DATETIME)
Purpose:
- Stores metadata about the photos and videos stored in cloud storage, such as file URLs and file types.
Type of Database Chosen:
- AWS S3 (Object Storage)
Justification for Choosing It:
- AWS S3 provides scalable, durable, and cost-effective storage for large media files (photos and videos). It is designed for high availability and is easy to integrate with cloud services.

erDiagram
    USER }|..|{ PHOTO : uploads
    USER ||--o{ FOLLOW : follows
    USER ||--o{ NOTIFICATION : receives
    USER ||--o{ COMMENT : writes
    USER ||--o{ LIKE : likes
    PHOTO ||--o{ COMMENT : receives
    PHOTO ||--o{ LIKE : receives
    PHOTO ||--o{ HASHTAG : tagged with
    COMMENT ||--o{ USER : written by
    COMMENT ||--o{ PHOTO : on
    LIKE ||--o{ USER : given by
    LIKE ||--o{ PHOTO : on
    HASHTAG ||--|{ PHOTO : "tagged in"
    FOLLOW ||--o{ USER : follows
    NOTIFICATION ||--o{ USER : notifies
    PHOTO ||--o{ MEDIA_STORAGE : stored in
    PHOTO ||--o{ SEARCH_INDEX : indexed by

High-level design

1. Client (Web/Mobile)

The Client is the user interface of the Instagram-like Photo Sharing Platform. It allows users to interact with the platform, upload photos, browse the newsfeed, like and comment on posts, and follow other users. The client can be a web application or a mobile app. It communicates with the backend services through the API Gateway to fetch data, interact with photos, and manage user profiles.

Responsibilities:
- Provide an intuitive UI for users to upload photos, view their feed, and interact with other users' content.
- Handle real-time updates, such as likes, comments, and notifications.
- Send API requests to the backend services and display responses to users.

2. Load Balancer

The Load Balancer distributes incoming traffic to multiple instances of the API Gateway, ensuring that the system can handle high traffic efficiently. It prevents any single server from becoming a bottleneck and improves fault tolerance by rerouting traffic to healthy servers if one becomes unavailable.

Responsibilities:
- Distribute user requests to various API Gateway instances based on load and availability.
- Improve system resilience and prevent downtime by automatically rerouting traffic in case of failure.

3. API Gateway

The API Gateway acts as the central entry point for client requests. It routes requests to the appropriate backend services, handles authentication, validates requests, and aggregates responses. The API Gateway also enforces security measures like rate limiting and manages cross-origin requests.

Responsibilities:
- Route incoming client requests to the appropriate backend service (e.g., Photo Service, User Service, etc.).
- Perform authentication and authorization using JWT tokens.
- Manage and throttle requests to avoid overloading services.

4. Authentication Service

The Authentication Service is responsible for managing user registration, login, and session handling. When a user logs in with their credentials, the service verifies them against the database and returns a JWT token that the client uses for future requests to authenticate the user. It also handles password resets and user sessions.

Responsibilities:
- Authenticate users by validating their credentials and issuing JWT tokens for session management.
- Handle user registration, password changes, and secure account access.
- Ensure the system is protected against unauthorized access.

5. Photo Service

The Photo Service manages the core functionality of uploading, storing, and interacting with photos. Users can upload new photos, edit captions, tag photos with hashtags, and associate metadata with each photo. The service also handles comments and likes for each photo and stores them in the relevant database.

Responsibilities:
- Handle photo uploads and ensure photos are processed (resized, optimized) before storing them in cloud storage (AWS S3).
- Store photo metadata (e.g., captions, hashtags) in the database (PostgreSQL).
- Allow users to edit, like, comment, and share photos.
- Serve media files from cloud storage with the help of a CDN (CloudFront) to improve loading times.

6. Notification Service

The Notification Service is responsible for sending notifications to users when they receive likes, comments, or when they are followed by other users. It ensures that users are kept up to date with activities related to their photos. Notifications are cached in Redis to ensure low-latency access.

Responsibilities:
- Send notifications to users for various interactions, such as new followers, likes, comments, etc.
- Cache frequently accessed notifications in Redis for fast delivery.
- Allow users to mark notifications as read and manage notification settings.

7. Search Service

The Search Service enables users to search for photos, hashtags, and users. It uses Elasticsearch to index photo metadata, hashtags, and user data, allowing users to perform fast, efficient searches. This is especially important for an image-heavy platform where users often need to search for specific content quickly.

Responsibilities:
- Index photos, hashtags, and user profiles for fast searching.
- Provide search functionalities based on keywords, hashtags, and user profiles.
- Return search results ranked by relevance and popularity.

8. User Service

The User Service manages user profiles, including personal information, privacy settings, and user interactions (e.g., following other users). It tracks user activities and relationships, such as who they follow and who follows them. It also handles user preferences and account settings.

Responsibilities:
- Manage user profiles, including bio, email, and other personal information.
- Handle follow/unfollow actions, maintaining the list of followers and following for each user.
- Manage user preferences, privacy settings, and account-related changes.
- Store session data in Redis for quick access to user session details.

9. Analytics Service

The Analytics Service tracks user interactions and engagement, providing insights into how users are interacting with photos and other content. It collects data about likes, comments, shares, and views, aggregates it in real-time with Kafka, and stores it in DynamoDB for real-time metrics. Long-term metrics are stored in PostgreSQL.

Responsibilities:
- Track user engagement metrics like likes, shares, and comments.
- Aggregate and process engagement data in real-time using Kafka.
- Provide insights into popular photos, hashtags, and user activity over time.
- Store real-time analytics in DynamoDB and historical data in PostgreSQL for later analysis.

10. Cloud Storage (AWS S3 and CloudFront)

AWS S3 is used to store photos and other media files, while CloudFront CDN is used to deliver media to users quickly. S3 provides scalable, durable storage, while CloudFront ensures low-latency, high-performance delivery of media content worldwide.

Responsibilities:
- Store photos and media files securely and reliably in S3.
- Distribute media files globally using CloudFront to reduce load on the main server and improve performance.

11. Caching Service (Redis)

Redis is used for caching frequently accessed data such as user sessions, notifications, and feed data. By caching this data, the system can reduce the load on the databases and deliver faster responses to the client.

Responsibilities:
- Cache user session data for fast access to authentication information.
- Cache notifications and feed data to reduce the number of database queries required.
- Improve system performance by reducing latency in retrieving frequently accessed data.

Databases and Storage:

PostgreSQL: Stores relational data like user profiles, photo metadata, comments, likes, and follow relationships.
Elasticsearch: Provides fast search capabilities for photos, users, and hashtags.
AWS S3: Stores photos and media files.
CloudFront CDN: Delivers media files with low latency to users.
Redis: Caches frequently accessed data for fast retrieval and low-latency access.
DynamoDB: Stores real-time engagement metrics and analytics data.

flowchart TD
    A[Client Web/Mobile] --> LB[Load Balancer]
    LB --> API_GATEWAY[Application Programming Interface API Gateway]
    API_GATEWAY --> AuthService[Authentication Service]
    API_GATEWAY --> PhotoService[Photo Service]
    API_GATEWAY --> NotificationService[Notification Service]
    API_GATEWAY --> SearchService[Search Service]
    API_GATEWAY --> UserService[User Service]
    API_GATEWAY --> AnalyticsService[Analytics Service]
    PhotoService --> DB_Photos[PostgreSQL Photos]
    PhotoService --> DB_Likes[PostgreSQL Likes]
    PhotoService --> DB_Comments[PostgreSQL Comments]
    PhotoService --> DB_Hashtags[PostgreSQL Hashtags]
    NotificationService --> DB_Notifications[PostgreSQL Notifications]
    SearchService --> Elasticsearch[Elasticsearch Search Index]
    UserService --> DB_Users[PostgreSQL Users]
    AnalyticsService --> DB_Analytics[DynamoDB Real-Time Metrics]
    AnalyticsService --> DB_HistoricalMetrics[PostgreSQL Historical Metrics]
    PhotoService --> S3[AWS S3 Media Storage]
    PhotoService --> CloudFront[CloudFront CDN Media Delivery]
    UserService --> Redis[Redis User Sessions]
    NotificationService --> Redis_Notification[Redis Notification Cache]

Request flows

1. User Login Request

Client sends login credentials to API Gateway.
API Gateway forwards credentials to Authentication Service.
Authentication Service validates credentials using PostgreSQL (User Data) and returns a JWT token.
API Gateway sends the JWT token back to the Client.

2. Photo Upload Request

Client sends photo data and caption to API Gateway.
API Gateway forwards the request to Photo Service.
Photo Service uploads the photo to AWS S3, stores metadata in PostgreSQL (Photos Data).
Photo Service sends a notification to the Notification Service.
API Gateway returns upload confirmation to Client.

3. Comment on Photo Request

Client sends comment data to API Gateway.
API Gateway forwards the request to Photo Service.
Photo Service stores comment in PostgreSQL (Comments Data).
Photo Service sends a comment notification via Notification Service.
API Gateway confirms comment submission to Client.

4. Like Photo Request

Client sends like request for a photo to API Gateway.
API Gateway forwards the request to Photo Service.
Photo Service stores like in PostgreSQL (Likes Data).
Photo Service sends a like notification to the Notification Service.
API Gateway returns like confirmation to Client.

5. Follow User Request

Client sends follow request to API Gateway.
API Gateway forwards the request to User Service.
User Service stores follow data in PostgreSQL (Followers Data).
User Service sends a follow notification via Notification Service.
API Gateway confirms follow action to Client.

6. Search Photos Request

Client sends a search request to API Gateway.
API Gateway forwards the search query to Search Service.
Search Service queries Elasticsearch and returns search results to the API Gateway.
API Gateway sends search results to Client.

7. Fetch Analytics Data Request

Client sends request for analytics data to API Gateway.
API Gateway forwards the request to Analytics Service.
Analytics Service fetches engagement data from PostgreSQL and DynamoDB.
Analytics Service returns aggregated data to API Gateway.
API Gateway sends the analytics data to Client.

8. Logout Request

Client sends logout request to API Gateway.
API Gateway forwards the request to Authentication Service.
Authentication Service invalidates the JWT token.
API Gateway confirms logout to Client.

sequenceDiagram
    participant A as Client (Web/Mobile)
    participant B as API Gateway
    participant C as Authentication Service
    participant D as Photo Service
    participant E as Notification Service
    participant F as User Service
    participant G as Search Service
    participant H as Analytics Service
    participant I as PostgreSQL (User Data)
    participant J as PostgreSQL (Photo Data)
    participant K as PostgreSQL (Comments Data)
    participant L as PostgreSQL (Likes Data)
    participant M as PostgreSQL (Followers Data)
    participant N as Elasticsearch (Search Index)
    participant O as AWS S3 (Media Storage)
    participant P as CloudFront (CDN)
    participant Q as Redis (Cache)

    %% User Login
    A->>+B: Login Request (email, password)
    B->>+C: Authenticate User
    C->>I: Verify Credentials (email, password)
    I-->>C: Return User Data
    C-->>B: Return JWT Token
    B-->>A: Send JWT Token

    %% Photo Upload
    A->>+B: Upload Photo Request (photo data, caption)
    B->>+D: Process Upload
    D->>O: Store Media (AWS S3)
    O-->>D: Return Media URL
    D->>J: Store Photo Metadata (caption, user_id)
    J-->>D: Return Photo Data
    D->>P: Serve Media via CDN
    D->>E: Send Photo Upload Notification
    E->>+Q: Cache Notification
    Q-->>E: Return Cache Status
    E-->>B: Send Notification
    B-->>A: Return Photo Upload Confirmation

    %% Comment on Photo
    A->>+B: Comment on Photo Request (photo_id, comment)
    B->>+D: Process Comment
    D->>K: Store Comment Data (photo_id, user_id, text)
    K-->>D: Return Comment Data
    D->>E: Send Comment Notification
    E->>+Q: Cache Notification
    Q-->>E: Return Cache Status
    E-->>B: Send Notification
    B-->>A: Return Comment Confirmation

    %% Like a Photo
    A->>+B: Like Photo Request (photo_id)
    B->>+D: Process Like
    D->>L: Store Like Data (photo_id, user_id)
    L-->>D: Return Like Data
    D->>E: Send Like Notification
    E->>+Q: Cache Notification
    Q-->>E: Return Cache Status
    E-->>B: Send Notification
    B-->>A: Return Like Confirmation

    %% Follow a User
    A->>+B: Follow User Request (user_id)
    B->>+F: Process Follow
    F->>M: Store Follow Data (follower_id, followee_id)
    M-->>F: Return Follow Data
    F->>E: Send Follow Notification
    E->>+Q: Cache Notification
    Q-->>E: Return Cache Status
    E-->>B: Send Notification
    B-->>A: Return Follow Confirmation

    %% Search for Photos
    A->>+B: Search Photos Request (keywords)
    B->>+G: Process Search
    G->>+N: Query Search Index
    N-->>G: Return Search Results
    G-->>B: Return Search Results
    B-->>A: Display Search Results

    %% Fetch Analytics Data
    A->>+B: Request Analytics Data
    B->>+H: Fetch Real-Time Analytics
    H->>I: Query User Engagement Data
    I-->>H: Return Engagement Data
    H->>J: Query Historical Data
    J-->>H: Return Historical Metrics
    H-->>B: Return Analytics Data
    B-->>A: Display Analytics Data

    %% Logout
    A->>+B: Logout Request
    B->>+C: Invalidate JWT Token
    C-->>B: Token Invalidated
    B-->>A: Logout Confirmation

Detailed component design

1. Authentication Service

Component Functionality:

Inputs: User credentials (email, password) for login, user registration details (email, password, username), JWT token for validation on subsequent requests.
Outputs: JWT tokens for authenticated sessions, error messages on failed login or registration.
Purpose: Manages user login, registration, and session management by issuing and verifying JWT tokens for secure access to other system components.

Responsibilities:

Handle user authentication during login and user registration.
Generate and validate JWT tokens for session management.
Implement password hashing and reset functionality.

Data Flow:

The Client sends login/registration requests via HTTP to the API Gateway.
The API Gateway forwards requests to the Authentication Service.
The Authentication Service queries the PostgreSQL User Database to validate user credentials.
On successful authentication, the Authentication Service generates a JWT token and returns it to the Client.

Technology Stack:

Database: PostgreSQL (for storing user credentials and session information).
Framework: Spring Boot (Java-based framework) or Express.js (Node.js).
Authentication: JWT, bcrypt for password hashing.
Cache: Redis (for storing active sessions).

Scaling and Performance:

Horizontal scaling with a load balancer to distribute incoming traffic.
Redis can be used to cache JWT tokens for faster token validation and session handling.

Security:

Uses HTTPS to ensure secure communication between client and server.
Passwords are hashed with bcrypt.
JWT tokens are encrypted to ensure secure authentication.

Monitoring and Analytics:

Collect metrics on failed login attempts and session expirations.
Logs failed login attempts and unusual activity for security monitoring.

2. Photo Service

Component Functionality:

Inputs: Photo files, metadata (caption, tags), photo ID for edits, user ID.
Outputs: Media URL, photo metadata, notifications, error messages.
Purpose: Manages photo uploads, metadata storage, photo edits, and generates notifications for actions related to photos (e.g., likes, comments).

Responsibilities:

Upload and process photos (resize, optimize).
Store photo metadata (captions, user associations) in the database.
Deliver media using AWS S3 and CloudFront CDN.
Handle notifications related to new uploads or actions on photos (e.g., likes, comments).

Data Flow:

The Client sends a photo upload request via the API Gateway to the Photo Service.
Photo Service uploads the photo to AWS S3 and stores the URL in the PostgreSQL Photos Database.
After processing the photo, it is served via CloudFront for optimized media delivery.
The Photo Service sends notifications through the Notification Service.

Technology Stack:

Database: PostgreSQL (for storing photo metadata).
Storage: AWS S3 (for media storage).
CDN: CloudFront (for media delivery).
Framework: Spring Boot or Express.js.

Scaling and Performance:

AWS S3 automatically scales to handle large volumes of media storage.
CloudFront CDN handles media delivery globally, reducing load on origin servers and improving user experience.
Horizontal scaling of the Photo Service behind a load balancer ensures high availability during peak traffic.

Security:

Photos are stored and delivered securely using HTTPS.
AWS IAM is used for managing access control and secure S3 buckets.

Monitoring and Analytics:

Monitor the number of photo uploads and their successful delivery via AWS CloudWatch.
Track errors in photo uploads or failed media delivery.

3. Notification Service

Component Functionality:

Inputs: User interactions (like, comment, follow), user preferences for notification channels (email, push).
Outputs: Notifications (like, comment, follow alerts), delivery status (success/failure).
Purpose: Sends real-time notifications to users about interactions on their posts (e.g., likes, comments, follows).

Responsibilities:

Send notifications when a user receives a like, comment, or follow.
Allow users to set preferences for notification channels (push, email, etc.).
Cache notifications in Redis to ensure fast delivery.

Data Flow:

The Photo Service or User Service triggers notifications when a user performs actions like liking a photo or following another user.
The Notification Service processes the event and sends notifications to the relevant users via the chosen channel (email, push).
Notifications are cached in Redis for quick access.

Technology Stack:

Cache: Redis (for caching notifications).
Messaging: Push notifications (e.g., Firebase for push notifications).
Email Service: SendGrid or Amazon SES for email notifications.

Scaling and Performance:

Redis scales horizontally for caching and fast access to notifications.
Horizontal scaling of the Notification Service can be used to manage high notification volumes.

Security:

All notifications are sent over HTTPS.
User preferences are stored securely with encryption.

Monitoring and Analytics:

Track the delivery success rate of notifications.
Monitor user engagement and the number of notifications sent using AWS CloudWatch.

4. User Service

Component Functionality:

Inputs: User registration data, profile updates, follow/unfollow requests.
Outputs: User profiles, follow status, error messages.
Purpose: Manages user accounts, profiles, and social relationships (follows/unfollows).

Responsibilities:

Handle user registration and profile management (bio, profile picture).
Implement follow/unfollow logic and maintain relationships between users.
Fetch user profiles and related data for display.

Data Flow:

The Client sends a registration or profile update request via the API Gateway.
The API Gateway forwards it to the User Service, which updates the PostgreSQL User Database.
Follow/unfollow requests are stored in PostgreSQL (Followers Data).

Technology Stack:

Database: PostgreSQL (for user profiles and follower relationships).
Framework: Spring Boot or Express.js.

Scaling and Performance:

PostgreSQL is horizontally scalable with read replicas to handle high traffic.
Load balancing is applied to scale the User Service horizontally.

Security:

OAuth or JWT are used to authenticate and authorize user actions.
HTTPS ensures secure communication between the Client and API Gateway.

Monitoring and Analytics:

Collect metrics on user registration and profile updates.
Monitor user interactions (e.g., follows, account changes) using AWS CloudWatch.

5. Search Service

Component Functionality:

Inputs: Search queries (keywords, hashtags).
Outputs: Search results (photos, users, hashtags).
Purpose: Allows users to search for photos, users, and hashtags based on keywords or tags.

Responsibilities:

Index photos, hashtags, and user profiles.
Provide fast and efficient search results based on user queries.

Data Flow:

The Client sends a search query via the API Gateway.
The API Gateway forwards the query to the Search Service, which queries Elasticsearch.
Elasticsearch returns the search results (photos, users, hashtags) to the API Gateway, which sends them back to the Client.

Technology Stack:

Search Engine: Elasticsearch.
Framework: Spring Boot or Express.js.

Scaling and Performance:

Elasticsearch scales horizontally with sharding and replication, handling large-scale search queries efficiently.
Query caching in Redis can further improve search performance.

Security:

All search queries and responses are encrypted using HTTPS.

Monitoring and Analytics:

Track search query performance and most popular queries.
Log failed search queries or system errors for debugging.

6. Cloud Storage and CDN (AWS S3 & CloudFront)

Component Functionality:

Inputs: Photo files (for storage), media URLs (for delivery).
Outputs: Media URLs, error messages.
Purpose: Store media files (photos, videos) and serve them efficiently using a CDN.

Responsibilities:

Store media files securely in AWS S3.
Serve media content globally using CloudFront CDN.

Data Flow:

The Photo Service uploads media to AWS S3.
Media URLs are generated and sent back to the Photo Service.
The Photo Service serves the media using CloudFront to optimize delivery speed.

Technology Stack:

Storage: AWS S3.
CDN: CloudFront.

Scaling and Performance:

AWS S3 automatically scales with increasing storage needs.
CloudFront handles global media delivery efficiently, reducing latency.

Security:

Media files are stored securely in S3 with proper access control using AWS IAM.
All media is delivered over HTTPS.

Monitoring and Analytics:

Track media storage usage and access patterns via AWS CloudWatch.

Trade offs/Tech choices

PostgreSQL vs NoSQL: Chose PostgreSQL for user data, photo metadata, likes, and comments to maintain strong data consistency and support complex queries. NoSQL (like DynamoDB) is used for real-time analytics due to scalability.

S3 for Media Storage vs DB: Used S3 for storing photos instead of a database due to better scalability and cost-efficiency for large file storage.

Redis for Caching vs DB: Redis caches notifications and session data for faster access, reducing load on the database and improving performance.

Microservices vs Monolithic: Chose microservices for better scalability and isolation of components, though more complex than a monolithic approach.

Elasticsearch vs Relational DB for Search: Elasticsearch is used for efficient, fast search across unstructured data (hashtags, captions), whereas a relational DB would struggle with complex search queries.

WebSockets vs HTTP for Notifications: Chose HTTP polling over WebSockets to simplify implementation and improve scalability, though WebSockets are better for real-time communication.

AWS vs Self-Hosting: AWS provides managed infrastructure (S3, EC2, CloudFront), reducing operational overhead and offering scalable storage and CDN options.

Monitoring with CloudWatch: Used AWS CloudWatch for monitoring and logging instead of building custom solutions, as it integrates well with AWS services and scales automatically.

Failure scenarios/bottlenecks

Database Overload: High traffic can overwhelm PostgreSQL, leading to slow queries.

Mitigation: Use read replicas and sharding for better load distribution.

Network Latency: Poor network conditions could degrade the performance of WebRTC or media delivery.

Mitigation: Implement adaptive bitrate streaming and multi-path to adjust media quality dynamically.

Cache Invalidation: Stale or inconsistent data in Redis can cause users to see outdated notifications or feed data.

Mitigation: Implement cache expiration and cache eviction strategies.

Search Slowdown: Elasticsearch might slow down with a high volume of search queries or complex queries.

Mitigation: Use sharding and index optimization to scale and speed up searches.

Notification Delays: High volume of notifications can overwhelm the Notification Service.

Mitigation: Implement rate limiting, backpressure, and retry mechanisms.

Media Delivery Failures: S3/CloudFront may face issues due to high load or server failures.

Mitigation: Use multi-region S3 buckets and CloudFront caching to reduce load and improve availability.

API Gateway Bottleneck: High traffic can overload the API Gateway, slowing down request processing.

Mitigation: Use auto-scaling for API Gateway instances and distribute traffic with load balancers.

Authentication Failures: Overload on the Authentication Service may lead to delays in token validation.

Mitigation: Use Redis for token caching and auto-scaling for horizontal scaling of authentication instances.

Future improvements

Improvement: Implement machine learning for personalized feed ranking.
- Mitigation: Use microservices for better isolation and scalability to handle increased processing loads.
Improvement: Implement edge caching for media delivery closer to users.
- Mitigation: Use multi-region S3 buckets and advanced CDN configurations for better availability and performance.
Improvement: Move to a serverless architecture for dynamic scaling during traffic spikes.
- Mitigation: Use AWS Lambda with auto-scaling to dynamically handle increased load without manual intervention.
Improvement: Implement advanced data partitioning strategies in PostgreSQL.
- Mitigation: Use read replicas and partitioned tables to efficiently manage high write and read workloads.
Improvement: Use WebSockets for real-time notifications.
- Mitigation: Implement scalable WebSocket clusters to avoid overload during high traffic.

These improvements enhance scalability, reduce latency, and ensure system resilience during failure scenarios.

Solutions

alchemy1135 - 03/01/2024

0

iridescent_luminous693 - 01/01/2025

0

utopia4715 - 04/25/2024

1

Solutions

alchemy1135 - 03/01/2024

0

iridescent_luminous693 - 01/01/2025

0

utopia4715 - 04/25/2024

1