System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

1. User can upload a video.
2. User can watch a video.
3. User can search and be presented with a list of videos.
4. User can like/dislike videos.
5. User can comment on videos.
6. User can subscribe to other accounts.
7. User can create a profile.

Non-Functional:

List non-functional requirements for the system...

• Availability: Ensuring high availability is critical. Users should experience minimal downtime to maintain trust and usability.
• Scalability: The system needs to be able to scale horizontally, accommodating an increasing number of users, video uploads, and interactions without degradation in performance.
• Performance: Quick loading times for video playback and fast processing of uploads are crucial for user satisfaction. Users shouldn't experience buffering or extended delays.

Capacity estimation

Estimate the scale of the system you are going to design...

Monthly Storage Calculation

• For 500 videos/month: [ 500 \text{ videos} \times 125 \text{ MB/video} = 62,500 \text{ MB} = 63 \text{ GB/month} ]

Yearly Storage Calculation

• For storage over a year: [ 63 \text{ GB/month} \times 12 \text{ months} = 756 \text{ GB/year} ]

Thus, your total expected storage requirement would be 756 GB/year in blob storage for video content.

• Initial DAU Estimate: Approximately 2,000 DAU based on 10,000 users with 20% engagement.
• Post-Growth DAU Estimate: Approaching around 2,100 DAU now based on 10,500 users and sustained engagement.
• Future Potential DAU: If user engagement improves to 30%, you might project approximately 3,150 DAU.

API design

Define what APIs are expected from the system...

1. Video Upload
   • POST /videos: Upload a video.
   • Request Body: Should include metadata about the video (title, description, etc.), while the video file itself would be handled via a multipart/form-data upload.
2. Watch a Video
   • GET /videos/{video_id}: Watch a video.
   • Response: Returns video metadata (title, description, video link) and potentially streaming options.
3. Search Videos
   • GET /videos/search?query={search_query}: Search for videos based on a query.
   • Response: Returns a list of videos that match the search criteria. This will be paginated, we can use cursor-based pagination here.
4. Like a Video
   • POST /videos/{video_id}/like: Like a video.
   • Request Body: Can include user ID if necessary, but could also be part of the session.
5. Dislike a Video
   • POST /videos/{video_id}/dislike: Dislike a video.
   • Request Body: Similar to like, user ID handling.
6. Comment on a Video
   • POST /videos/{video_id}/comments: Comment on a video.
   • Request Body: Should include the user ID and the comment text.
7. Subscribe to an Account
   • POST /subscriptions: Subscribe to an account.
   • Request Body: Should include both the subscriber ID and the account ID to subscribe to.
8. Create a Profile
   • POST /profiles: Create a user profile.
   • Request Body: Should include profile details (name, email, etc.).

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

Structured Data

1. Users: This would contain user profiles, possibly with fields like user ID, name, email, profile picture, and any other relevant user information.
2. Videos: This would include fields such as video ID, title, description, upload date, and importantly, a link to the actual video file stored in blob storage.

Unstructured Data

1. Video Interactions: This could capture likes, dislikes, and comments. You could have fields that reference the user, the video, and the action taken (e.g., like, dislike, comment).
2. Subscribers/Subscriptions: This would involve users following other users. It could contain user IDs for the subscriber and the subscribed account.

SQL Database Design

Users Table

• Table Name: Users
• Fields:
  • id: Primary Key (Unique identifier for each user).
  • email: User's email (Indexed for fast look-up).
  • username: User's username (Indexed for fast look-up).
  • first_name: User's first name.
  • last_name: User's last name.

CREATE TABLE Users ( id SERIAL PRIMARY KEY, email VARCHAR(255) UNIQUE NOT NULL, username VARCHAR(255) UNIQUE NOT NULL, first_name VARCHAR(100), last_name VARCHAR(100)); CREATE INDEX idx_email ON Users (email);CREATE INDEX idx_username ON Users (username);

Videos Table

• Table Name: Videos
• Fields:
  • id: Primary Key (Unique identifier for each video).
  • video_name: Title of the video.
  • upload_date: The date the video was uploaded.
  • length: Duration of the video.
  • file_size: Size of the video file.
  • description: Description of the video.
  • video_link: A link pointing to the video file in blob storage.

CREATE TABLE Videos ( id SERIAL PRIMARY KEY, video_name VARCHAR(255) NOT NULL, upload_date TIMESTAMP NOT NULL, length INTERVAL NOT NULL, file_size INT NOT NULL, description TEXT, video_link VARCHAR(255) NOT NULL);

NoSQL Database Design

For the NoSQL part, storing video interactions, subscriptions, and user comments adds scalability and flexibility.

Video Interactions

• Key-Value Structure:
  • Key: video_id
  • Value: {
    • comments: array of comment IDs.
    • like_count: number of likes.
    • dislike_count: number of dislikes. }

{ "video_id_1": { "comments": ["comment_id_1", "comment_id_2"], "like_count": 100, "dislike_count": 5 }, "video_id_2": { "comments": ["comment_id_3", "comment_id_4"], "like_count": 50, "dislike_count": 2 } }

User Subscriptions

• Key-Value Structure for User Subscriptions:
• Key: user_id
• Value: {
  • subscribed_to: array of user IDs the user is subscribed to.
  • subscribers: number of users subscribed to them.

{ "user_id_1": { "subscribed_to": ["user_id_2", "user_id_3"], "subscribers_count": 250 }, "user_id_2": { "subscribed_to": ["user_id_1"], "subscribers_count": 150 } }

Summary of Design

• SQL database handles structured data (users and videos) allowing for ACID transactions and complex querying.
• NoSQL database efficiently manages unstructured data (interactions, comments, and subscriptions) allowing for scalability and rapid access patterns.

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

High-Level Design Overview

1. Load Balancer:
   • Function: Distributes incoming user requests across multiple application servers ensuring high availability and fault tolerance.
   • Benefit: Improves responsiveness by minimizing the load on each server and provides redundancy.
2. Application Servers:
   • Function: Handles business logic, interacts with the databases, and processes user requests.
   • Interaction with Databases:
     • SQL database for user and video metadata.
     • NoSQL database for video interactions (likes, comments).
3. Video Playback:
   • Video Watch Service:
     • Users request to watch a video—this service fetches the video link from the SQL database.
     • The link is used to retrieve the video file directly from blob storage.
   • Interaction with Blob Storage: Allows for seamless streaming of video content, optimizing performance and minimizing buffering.
4. Interaction Service:
   • Function: Fetches interaction data (likes, dislikes, comments) related to the specific video.
   • Eventual Consistency Model: Accepts slight delays in displaying like/dislike counts and comments, thus improving performance for users.
5. Video Interaction:
   • Handling User Actions:
     • When a user likes or dislikes a video, they submit an interaction request to the interaction service.
     • This service processes the interaction and stores it in the NoSQL database.
6. Video Upload:
   • Upload Service:
     • Processes video uploads from users and stores the video file in blob storage.
     • Metadata is saved to the SQL database, ensuring video indexing and searchability.

flowchart TD A[User Request] -->|Routes via| B[Load Balancer] B -->|Routes Request| C[App Server] C -->|Fetch Video Link| D[SQL DB] C -->|Fetch Interactions| E[Interaction Service] D -->|Video Link| F[Blob Storage] E -->|Comments/Likes/Dislikes| G[NoSQL DB] F -->|Streams Video| H[Client] G -->|Returns Interactions| H

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Video Upload Design

1. Queue for Video Upload Processing:
   • Implementation: Use a message queuing service (like RabbitMQ, Amazon SQS, or similar) to decouple the uploading process from video processing.
   • Process Flow:
     • When a user uploads a video, the upload service sends the video metadata to the queue.
     • A dedicated worker service (consuming from the queue) processes the video, making necessary transformations (e.g., encoding, generating thumbnails).
     • Once processed, the video is stored in blob storage and the metadata is saved in the SQL database.

Video Caching with CDN

1. Content Delivery Network (CDN) for Popular Videos:
   • Implementation: Automatically cache newly uploaded videos or those that gain popularity after a configurable threshold (e.g., a certain number of views or likes).
   • Benefits:
     • Reduces latency for users accessing popular videos, enhancing load speeds and streaming performance.
     • Offloads traffic from your original server, decreasing the load during high demand.

Batched Like Requests

1. Batching Like Requests:
   • Implementation: Collect multiple like/dislike actions per user session and send them in a batch to the interaction service.
   • Benefits:
     • Prevents a single video from becoming a "hot spot" in the database, which can lead to performance degradation.
     • Reduces write operations to the database, allowing for higher throughput.
   • Process Flow:
     • Users submit likes/dislikes during their session.
     • On a set interval or action threshold, the service processes these requests in bulk, updating the NoSQL database.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Structured Data with SQL

• • Why SQL?Ease of Querying: SQL databases provide powerful querying capabilities with support for joins, indexing, and complex queries, which make it easy to retrieve user and video metadata.
  • ACID Transactions: Ensures data consistency and reliability, particularly important for user data and video metadata where integrity is critical.
• Trade-offs:
  • Scalability: As traffic increases, SQL databases can face challenges in horizontal scaling without complex distribution strategies.
  • Schema Rigidity: SQL databases have a defined schema, which can make evolving the data models more challenging over time compared to NoSQL solutions.

2. Unstructured Data with NoSQL

• Why NoSQL?
  • Scalability: NoSQL databases are designed to handle large volumes of unstructured data and can scale horizontally across multiple servers.
  • Flexibility: You can easily handle changing data structures, such as interactions that may evolve over time (adding new fields or metrics).
• Trade-offs:
  • Eventual Consistency: Many NoSQL databases prioritize availability and partition tolerance over immediate consistency, which could lead to scenarios where users see slightly outdated interaction counts.
  • Complex Queries: NoSQL databases may not support complex queries in the same way SQL does, requiring careful planning of data retrieval strategies.

3. Queue for Video Processing

• Why Use a Queue?
  • Asynchronous Processing: Enables decoupling of the upload process and the video processing workload, effectively smoothing out peaks in upload traffic.
  • Scalability and Efficiency: Allows for multiple workers to consume messages from the queue, utilizing resources efficiently without overloading any single service.
• Trade-offs:
  • Additional Point of Failure: Introducing a message queue means adding complexity, as it becomes another component that can fail (network issues, service outages).
  • Latency: While queues improve processing efficiency, they may introduce slight delays in processing completion as videos wait in the queue.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

1. Database Downtime

Scenario:

• Both the SQL and NoSQL databases could experience downtime due to hardware failures, network issues, or maintenance.

Impact:

• User requests for video data, interactions, and profiles would fail, resulting in service outages and a poor user experience.

2. Message Queue Downtime

Scenario:

• The message queue (e.g., RabbitMQ, AWS SQS) could go down, causing video upload requests to get lost or delayed.

Impact:

• Users trying to upload videos would see errors, resulting in frustration and potential loss of content if retries are not handled properly.

3. Bottleneck from Super Popular Videos

Scenario:

• A video that suddenly becomes popular receives an overwhelming number of reads and writes, putting significant pressure on the database.

Impact:

• Database performance might degrade, leading to slow response times or timeouts for users trying to watch the video or interact with it.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

Mitigation Strategies:

• Caching with CDN: Store popular videos in a CDN to offload streaming traffic from the origin server and reduce database reads.
• Rate Limiting: Implement rate limiting on how frequently certain actions (like/dislike) can be performed by users to manage spikes.
• Sharded Database Architecture: Consider sharding your database so that you can distribute the load evenly across multiple database instances.
• Read Replicas: Use read replicas of your SQL database to handle increased read loads without straining the primary database.