System requirements

Stakeholder: Ad click producer, Aggregation info reader

Functional:

• Ad click producer writes ad click event to a event queue
• The system read from the event queue and aggregate.
• Data scientists are able to query the system for the aggregation information.
• Data scientists are able to query the popular ads within a specified timeframe

Non-Functional:

• Scalability: The system is able to handle large volume of Ad Click events: expected to handle 50k ad click events per second.
• Fault tolerant: The single instance failure should not affect the overall system behavior
• Security: Only the authorized users are able to read the aggregation information
• Latency: The report should be generated within seconds.
• Correctness: Duplicate ad click events should be detected and the reports should be correct in terms of the given timeframe.

Capacity estimation

The system should be able to support 11574 QPS, the peak estimation is about 5 times which is 57870 QPS.

Storage estimation: each click takes 0.1KB, so it is 100 GB per day, which is 3TB per month.

API design

Aggregate info reviewer:

• GET /view_ad
  • Request: int ad_id
  • Response: Status, int count, Datetime aggregate at
• GET /popular
  • Request: time start, time end
  • Response: Status, list of ad_ids

Database design

There are 2 different types of data need to be stored: The raw click event and the aggregated report.

Raw ad click event: The storage requirement of this system is heavy write, write is mostly append only.

• PrimaryKey int event_id
• ForeignKey int ad_id
• DateTime click_on

Ad click aggregation: The aggregation data can be stored in an OLAP database to support analitical queries better.

• PrimaryKey record_id
• ForeignKey int ad_id
• int num_clicks
• DateTime start
• DateTime end

High-level design

There are some key components in the system:

• MessageQueue: This MessageQueue is used to receive the raw click events. Considering the high volume of the click events, the MessageQueue should be distributed.
• Event writer: The event writer is the main consumer of the MessageQueue, it reads from the MessageQueue and perform the initial processing of the data. After the preprocess, it write the raw event to the database.
• Data processor: We make this a separate component because the algorithm to identify bot behavior and talking to the DB to deduplicate events can be independently evolved. It can be a seperate service because the metrics of the performance of the data processor is also meaningful.
• Raw Data Event storage: As described in the previous section, it stores the raw click event. It should handle heavy write, so NoSQL is more suitable for this case.
• Data aggregator: An independent service to asynchronously read from the raw data event storage and generate aggregated information to write to the aggregation storage.
• Aggregation storage: stores the aggregation date within a time frame for a givent ad_id.
• Monitoring service: There are different pieces in the system to monitor: The queue size of MQ, the consume rate of MQ, resource usage of each component
• Report service: exposes the report API to build up the analytical query to query DB
• API Gateway to authorize, authenticate, rate limit on querying the report.

Request flows

Process raw event

• Raw events are produced by different sources
• Event writer pulls batches of events from the MessageQueue.
• Event writer sends the events to Data processor to validate and deduplicate
• Event writer then write the preprocessed events to the Raw Data Event Storage

Data aggregation: a separate process

• Data aggregator reads data from Raw data event storage, process the data and write to the aggregation storage

Report query flow:

• User uses API through API gateway
• API gateway does authentication, rate limit and load balance to different instances.
• The report service build up the query and send to aggregation storage
• report service generate report and response

Detailed component design

• Data Aggregator: considering the large volume of event size, it is possible to redesign this component into a data processing pipeline and process the data streams read from raw data event storage. This increases the system throughput, and it is easy to plug in new nodes in between to support new features. Also different individual nodes can scale separately.
• Event writer: event writer is the consumer of the Message Queue, so when the queue size increases, the event write can scale indepentantly by adding more instances.
• Report service can support different type of queries from different sources, so scale up the service is needed. For the real-time dashboard, it is ok to implement the websocket to update the dashboard easier.

Trade offs/Tech choices

• complexity vs feature richness: when choosing the storage for aggregation storage, both NoSQL and OLAP fits our requirements. This is a tradeoff between feature richness and complexity. OLAP is more complicated, but it supports more types of query. Some of the queries might be replaced by ML models, so they're not necessary. NoSQL might support only the limited features as described in the requirements, but it is easy to maintain and easy to design.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?