Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
1) Allow users to view a video stream
3) Allow users to create account
4) Allow users to adjust the stream quality
List non-functional requirements for the system...
1) Sysetm should be highly available
2) System should have low latency
3) System should be scalable
Estimate the scale of the system you are going to design...
Lets assume there around 10 million total users and in case of a highly famous match there are around 5 million active users users
so let's assume qps =5 * 1000,000
Define what APIs are expected from the system...
POST /v1/User Allow user to access streaming platform
request Body ->
{
"firstName:"",
"lastName":"",
"phoneNo":"",
}
DElete /v1/User - Allow user to delete his account
request body ->
{
userId:"userId"
}
Put /v1/User Api is used to update the user details
{
userId:""
firstName:"",
}
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
For storing the user Details and metadata about a stream or videos we will be using relational database as relational database is read heavy and all the metadata will also be pushed to elastic search which uses inverted index to seach videos by name or cateory for storing video metadata we will be using video table with following cateory video_id (PK) ,video_name,videoLength, actors and for users will have following attributes
userId(PK), userName, lastName, phoneNo
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Each user interacts with an API Gateway, which handles rate limiting and authentication before directing the request to the appropriate service—either the streaming service for viewing streams or the user service for creating or updating account details. When a user opts to view a stream, the system employs streaming protocols like MPEG-DASH that support adaptive bitrate streaming, initially fetching video chunks based on the user's internet connectivity and dynamically adjusting the quality if the user changes it via the UI. Streamers upload their content through the streaming service, which transcodes the video into various chunks and publishes these chunks to an event bus powered by Kafka, chosen for its high throughput, message replay capabilities, and durable disk storage across multiple brokers to ensure high availability and scalability. The transcoded video chunks are then stored in blob storage solutions such as AWS S3 or Azure Blob Storage, leveraging cross-region replication to guarantee their availability. To enhance performance and reduce latency, these chunks are cached in a Content Delivery Network (CDN) instead of being fetched directly from the application servers. Additionally, stream metadata is stored in Elasticsearch to enable efficient searching by category, ensuring users can quickly find and access the streams they desire. This architecture not only ensures seamless and scalable video delivery but also maintains robust data management and high availability across the system.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Each user interacts with an API Gateway, which handles rate limiting and authentication before directing the request to the appropriate service—either the streaming service for viewing streams or the user service for creating or updating account details. When a user opts to view a stream, the system employs streaming protocols like MPEG-DASH that support adaptive bitrate streaming, initially fetching video chunks based on the user's internet connectivity and dynamically adjusting the quality if the user changes it via the UI. Streamers upload their content through the streaming service, which transcodes the video into various chunks and publishes these chunks to an event bus powered by Kafka, chosen for its high throughput, message replay capabilities, and durable disk storage across multiple brokers to ensure high availability and scalability. The transcoded video chunks are then stored in blob storage solutions such as AWS S3 or Azure Blob Storage, leveraging cross-region replication to guarantee their availability. To enhance performance and reduce latency, these chunks are cached in a Content Delivery Network (CDN) instead of being fetched directly from the application servers. Additionally, stream metadata is stored in Elasticsearch to enable efficient searching by category, ensuring users can quickly find and access the streams they desire. This architecture not only ensures seamless and scalable video delivery but also maintains robust data management and high availability across the system.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Each user interacts with an API Gateway, which handles rate limiting and authentication before directing the request to the appropriate service—either the streaming service for viewing streams or the user service for creating or updating account details. When a user opts to view a stream, the system employs streaming protocols like MPEG-DASH that support adaptive bitrate streaming, initially fetching video chunks based on the user's internet connectivity and dynamically adjusting the quality if the user changes it via the UI. Streamers upload their content through the streaming service, which transcodes the video into various chunks and publishes these chunks to an event bus powered by Kafka, chosen for its high throughput, message replay capabilities, and durable disk storage across multiple brokers to ensure high availability and scalability. The transcoded video chunks are then stored in blob storage solutions such as AWS S3 or Azure Blob Storage, leveraging cross-region replication to guarantee their availability. To enhance performance and reduce latency, these chunks are cached in a Content Delivery Network (CDN) instead of being fetched directly from the application servers. Additionally, stream metadata is stored in Elasticsearch to enable efficient searching by category, ensuring users can quickly find and access the streams they desire. This architecture not only ensures seamless and scalable video delivery but also maintains robust data management and high availability across the system.
Explain any trade offs you have made and why you made certain tech choices...
1) Using kafka helps us with availability and scalability
2) Storing the chunks in different regions guarantees availability
Try to discuss as many failure scenarios/bottlenecks as possible.
1) Blob store in a region can crash or go down but then the event can be fetched from other region
2) Kafka node can go down as kafka persists data on disk so this will not be problem as other broker can easliy read this data
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
1) Use techniques like collaborative filtering to provide recommendations to user.