Requirements

Functional Requirements:

• the systems should be able to precisely count the views of each video.
• the system should be able to determine illegitimate view.
• the system should be able to update view counts in real time.

Non-Functional Requirements:

• Availability
  • high availability: 99.99% up time.
• Scalability
  • able to support horizontal scaling to handle high traffic volume spike with minimal failure
• Consistency
  • Little inconsistency is acceptable.
• Latency
  • Low response time: real time response < 500 ms

API Design

External API

1.Retrieve Video

• METHOD: GET
• Endpoint: /videos/{video_id}
• Path Variables:
  • video_id: str: id of video
• Response
  • Success:
    • status: ENUM: SUCCESS
    • status_code: int: 200
    • video: blob: the video itself
    • video_metadata: object: the metadata of the video
  • Video not found
    • status: ENUM: ERROR
    • status_code: int: 404
    • message: str: video not found. It might be deleted or limited access
  • Bad service
    • status: ENUM: ERROR
    • status_code: int: 503
    • message: str: log that can be used for root cause analysis

Internal API

1.Increase View

• METHOD: PATCH
• Endpoint: /increase_view/{video_id}
• Path Variables:
  • video_id: str: id of video
• Body:
  • views: int: number of view increase
• Response
  • Success
    • status: ENUM: SUCCESS
    • status_code: 200
  • Video not found
    • status: ENUM: ERROR
    • status_code: int: 404
    • message: str: Video not found. It is possible that it might be deleted or limited
  • Access denied
    • status: ENUM: ERROR
    • status_code: int: 403
    • message: str: no permission allowViewUpdate for update video

High-Level Design

Database Design

• System Nature
  • in this system we only needs to map the video with its view
  • hence there is only 2 columns that needs to be kept together
• Choices
  • SQL database
    • database such as mysql can be used to deal this type of problem by storing video_id, and view count in table Views
    • video_id will act as primary key while views_count will behave like one of the usual column
  • noSQL database
    • in this use case we can also apply noSQL database, to be more specific key-value database.
    • We only need to keep video_id as key and views_count as value.
    • So key-value database such as dynamodb from AWS is more suitable for this problem
• Decision
  • In this case, the design would benefit from using key-value database.
  • While the structure of data is consistent across the system's life sql database is known for its complexity for scaling. Action like sharding is far more complex than its noSQL database
  • And as there is only 2 columns we can safely use key-value database.

Detailed Component Design

View Verification Algorithm

• To get rid of the fraud views we can filter them by asking
  • Does the view last longer than 1 minute
  • Is it a part of spam
• for the first question, we can easily detected it by measuring time for each view.
• for the second question, it may be done through api gateway rate limit functionality
• We can use Captcha periodically to detect bots.
• If there are too many request from the same source ip, we will consider it as fraud view and doesn't count it.

Real Time Update

• Views are aggregrated by kafka architecture.
• Kafka producer will handle the incoming view then send it to kafka topic
• Kafka subscriber will receive the incoming view and aggregate (Count) them and sending the request to api gateway to access view increase service
• After the view increase if the cache hit, cache will launch push notification to update views on web ui (cache miss, database will launch push notification)
• WebUI show real time update using Web Socket

Cache Management

• Cache algorithm
  • Cache retain its fast read/write speed by limiting its size
  • hence, it needs algorithm to terminate item in case its size is overload
  • LRU
    • Least Recently Used cache
    • This algorithm will prioritize the most recent item
    • hence, it will delete item with the oldest latest used timestamp
  • LFU
    • Least Frequently Used cache
    • This algorithm will prioritize the most used item
    • hence, it will delete item with the least amount of usage
  • in our case, we may use the combination of both cache
  • The first layer will be LFU cache and in case there are multiple least frequently used items
  • We will use LRU cache

Traffic Handling

• Scaling up our service is one of the solution for this problem
• To detect that the current traffic is too much for current capacity to handle, we need to focus on metrics such
  • request/seconds
  • cpu utilization
  • memory utilization
• In case this metric exceeds its predefine value
• we will spawn new instance of service
• To execute this, we deploy our service on container orchestration system like kubernetes or AWS ECS as they have built in auto scaling that can integrate with your cloud service of choice