Requirements

Functional Requirements:

• Track the ad clicks in real time, record and serve total hits in real time
• Record metadata about the click (e.g. IP, user agent, userId if available) for analytics
• Out of scope
  • Serving of ads, ad campaign managment

Non-Functional Requirements:

• System is high available (99.99%)
• Eventual consistency is okay
• Minimum latency for a click to make its way into click count (<10 sec)
• System is fault tolerant and highly scalable

API Design

// Register a click

POST /api/v1/clicks

{

impressionId,

userId, // optional

sourceIP,

userAgent,

...

}

// Ad campaign management API is omitted

// Get ad clicks

GET /api/v1/clicks?adIds=[]&window={}&startTime={}&pageSize={}&pageToken={}

// Analytics

GET /api/v1/clicks?query={}

High-Level Design

Capacity estimation

• Assume 100k clicks per second, that is 100k * 86400 = 1B clicks per day.
• Assume each click carries 200 bytes of metadata, that is 200GB raw data per day.
• If we keep all raw data for 1 year, that is 72TB per year.
• Assume we have 1000 vendors looking at real-time ads at any given time. Assume small fan-out (e.g. up to 10 ads per vendor)

Aggregation table size

• Say we are storing per second aggregation for 10k ads, and that each row including metadata is 100 bytes. That is 10k * 100 = 1MB / sec, which amounts to 10GB per day and 3.6TB per year. This is a manageable amount of data. Partitioning can help (e.g partition using merchant id)

On the ingestion path, when an ad is clicked, the APIGateway sends data to an IngestionService, which validates the request and forward to IngestionQueue. From there, the PersistenceService takes the messages and store them in blob storage, partitioned by time. This is our raw data.

The Real time aggregation service (Flink or Spark Streaming) performs aggregation, and stores the aggregation result in Aggregation Database. When new results are available, it pushes the result onto notification queue. The NotificationService picks it up, and updates VendorClient via SSE.

The BatchProcess service periodically queries raw data, redo the aggregation, and generate higher level aggregations for analytics (e.g. per hour, per day/week/month). It will also be able to process data that came outside the watermark.

On the query path, vendor client might query for pre-aggregated data, in which case QueryService can simple serve from AggregationDatabase and/or Aggregation Cache. If the query is complicated and needs to perform adhoc aggregation, it can leverage the same aggregation service used in real-time path, although the priority will be lower and latency more.

Detailed Component Design

Data schema

Merchant table

• id
• name
• ...

Campaign table

• id
• merchant_id
• name
• ...

Impression table

• id
• campaign_id
• created_at

Click table

• id
• impression_id
• user_id // optional
• source_ip
• user_agent

Real-time aggregation

• If we only need click count per ad, we can simply use a redis cache, where the key is the ad id, and increment count when the an impression of the ad is clicked
• Choosing a real-time processing framework gives us more flexibility in the types of aggregations we can do, at the cost of higher latency, and operational complexity. Given that we need to support analytics in real-time or non-real time cases, we choose to introduce a real-time processing framework like Flink or Spark Streaming to accomplish this task

Real-time updates to the client

• Since the update is real-time unidirectional (from server to merchant client), SSE is the better chose over websocket or polling.
• We will need a pool of notification servers, where the merchant client have a persistent connection to the server. When RTAggService updates results for an ad, it pushes a message to the notification queue.
• NotificationService can be put onto a consistent hash ring, partitioned by merchant id. The notification queue can also be partitioned using merchant id, and each notification server can subscribe to the right partition for the merchant.

Deduping clicks

• This can be done during aggregation.

Traffic spikes

• If there is a large merchant, we might consider further partition by merchant_id+ad_id, for the queues and database.
• If server is experiencing downtime, or there is a network issue, consider batching the request on the client side, and flush the data when service recovers. In addition, the server can also instruct the client to slow down or batch requests.

Reliability, scalability

• IngestionService, PersistenceService, QueryService are stateless and can be scaled horizontally
• RTAggService and BatchProcess services needs to be configured for high availability (e.g. Flink/Spark cluster)
• AggDB - no sql DB like DynamoDB or Cassandra. They can be partitioned by merchant since we are not very likely to perform cross-merchant queries. Any non-cloud DB solutions need to be configured for high-availability and failover.
• The 2 queues can be Kafka queue or a managed cloud solution. Partitioning of the queue based on merchant id.