Loading...
Assumptions:
We can leverage key-value based DB, let say, DDB here as the queries are going to be simpler, there is no such ACID demand, no tight-coupling of columns is there.
Upload the video ---> API_gateway --> Video_Downloader(s3) --> processor --> S3
* API_gateway-> It helps in authN/ authZ and throtlling of the service
Video Downloader -> It downloads the video to a particular S3 location makes a call to processor to act upon(async fashion, like via aws lambda or something equivalent)
Processor -> The processor will act upon the uploaded video, it converts the video to a more compatible format(whether for speed or storage, depending upon we want low latency or save on storage or some trade-off in between), segregate the video into chunks of 10 sec's each, and update the s3 locations with chunks.
During this, the video_downloader will also update the video table as such, it will have statuses and update those like:
* video_downloader-> before starting, makes status as pending
-> then, uploading
-> then, when processor is acting upon, it makes a call to downloader to update the status to processing
-> then, when processor is done, it makes a call to dowloader to update the status to active
For live-updates on the status, it can open web-socket with the customer's client or we can simply rely on push operation(whenever there is status change), as there aren't much states for the status, push approach is also fine as web-sockets will keep the connection open.
Play the video ---> API_gateway --> Video Feeder --> S3
* The video-feeder will search in the video table to get all the chunks corresponding to the video_id, to fasten up, it can leverage redis to keep the chunk-details for a video_id for subsequent fetches. It takes upon the current timestamp and fetches the video for the particular timstamp from S3 location and then responds back to the customer.
On every t-2th second, the browser, or some monitoring service(client-side service), will make a call the playTheVideo for the next t+1th chunk.
* Search for video-> Api_gateway --> Feed Service(can leverage a cache, redis) --> Ranking Service
|
---> ElastiSearch
we can leverage our Elasticsearch stack to search for a particular keyword provided by the customer
it would be like, for a particular search, the feed-service first of check its cache, if not found, it makes a call to elasticSearch to get the matched records(video_ids)
Then, it leverages the ranking_service to rank these video_ids and provide top: 20 records(based upon some pre-defined criteria), On getting back the top:20 items back from ranking_service, feed service can now responds backs to the customer with the response.. Ranking service can leverage something like heap to get the top 20 items from a list of videos_ids
Since, we want faster latency for read, search operations. we can categorize our backend servers and use L7 LB to segregate read/write/search requests.
Upload the video ---> API_gateway --> Video_Downloader(s3) --> processor --> S3
* API_gateway-> It helps in authN/ authZ and throtlling of the service
Video Downloader -> It downloads the video to a particular S3 location makes a call to processor to act upon(async fashion, like via aws lambda or something equivalent)
Processor -> The processor will act upon the uploaded video, it converts the video to a more compatible format(whether for speed or storage, depending upon we want low latency or save on storage or some trade-off in between), segregate the video into chunks of 10 sec's each, and update the s3 locations with chunks.
During this, the video_downloader will also update the video table as such, it will have statuses and update those like:
* video_downloader-> before starting, makes status as pending
-> then, uploading
-> then, when processor is acting upon, it makes a call to downloader to update the status to processing
-> then, when processor is done, it makes a call to dowloader to update the status to active
For live-updates on the status, it can open web-socket with the customer's client or we can simply rely on push operation(whenever there is status change), as there aren't much states for the status, push approach is also fine as web-sockets will keep the connection open.
Play the video ---> API_gateway --> Video Feeder --> S3
* The video-feeder will search in the video table to get all the chunks corresponding to the video_id, to fasten up, it can leverage redis to keep the chunk-details for a video_id for subsequent fetches. It takes upon the current timestamp and fetches the video for the particular timstamp from S3 location and then responds back to the customer.
On every t-2th second, the browser, or some monitoring service(client-side service), will make a call the playTheVideo for the next t+1th chunk.
* Search for video-> Api_gateway --> Feed Service(can leverage a cache, redis) --> Ranking Service
|
---> ElastiSearch
we can leverage our Elasticsearch stack to search for a particular keyword provided by the customer
it would be like, for a particular search, the feed-service first of check its cache, if not found, it makes a call to elasticSearch to get the matched records(video_ids)
Then, it leverages the ranking_service to rank these video_ids and provide top: 20 records(based upon some pre-defined criteria), On getting back the top:20 items back from ranking_service, feed service can now responds backs to the customer with the response.. Ranking service can leverage something like heap to get the top 20 items from a list of videos_ids
Upload the video ---> API_gateway --> Video_Downloader(s3) --> processor --> S3
* API_gateway-> It helps in authN/ authZ and throtlling of the service
Video Downloader -> It downloads the video to a particular S3 location makes a call to processor to act upon(async fashion, like via aws lambda or something equivalent)
Processor -> The processor will act upon the uploaded video, it converts the video to a more compatible format(whether for speed or storage, depending upon we want low latency or save on storage or some trade-off in between), segregate the video into chunks of 10 sec's each, and update the s3 locations with chunks.
During this, the video_downloader will also update the video table as such, it will have statuses and update those like:
* video_downloader-> before starting, makes status as pending
-> then, uploading
-> then, when processor is acting upon, it makes a call to downloader to update the status to processing
-> then, when processor is done, it makes a call to dowloader to update the status to active
For live-updates on the status, it can open web-socket with the customer's client or we can simply rely on push operation(whenever there is status change), as there aren't much states for the status, push approach is also fine as web-sockets will keep the connection open.
Play the video ---> API_gateway --> Video Feeder --> S3
* The video-feeder will search in the video table to get all the chunks corresponding to the video_id, to fasten up, it can leverage redis to keep the chunk-details for a video_id for subsequent fetches. It takes upon the current timestamp and fetches the video for the particular timstamp from S3 location and then responds back to the customer.
On every t-2th second, the browser, or some monitoring service(client-side service), will make a call the playTheVideo for the next t+1th chunk.
* Search for video-> Api_gateway --> Feed Service(can leverage a cache, redis) --> Ranking Service
|
---> ElastiSearch
we can leverage our Elasticsearch stack to search for a particular keyword provided by the customer
it would be like, for a particular search, the feed-service first of check its cache, if not found, it makes a call to elasticSearch to get the matched records(video_ids)
Then, it leverages the ranking_service to rank these video_ids and provide top: 20 records(based upon some pre-defined criteria), On getting back the top:20 items back from ranking_service, feed service can now responds backs to the customer with the response.. Ranking service can leverage something like heap to get the top 20 items from a list of videos_ids
Since, we want faster latency for read, search operations. we can categorize our backend servers and use L7 LB to segregate read/write/search requests.
The Ranking system currently works upon the the pre-defined fix ranking-algorithm, this can be discussed laterwards on how to improve it.
We can also add additional support of analytics service to build upon the recommendation service lateron for the customers.