Requirements

Functional Requirements:

• Capture page views, clicks, conversions, and user actions on websites via embedded scripts.
• Measure page load times, response times, and site speed metrics.
• Track traffic sources, user demographics, sessions, and audience segmentation.
• Manage client accounts, websites, and subscription plans.

Scale:

• 1B users and 100M QPS at peak

Non-Functional Requirements:

• High Availability to ingest events quickly and high availability to show the current statistics so availability > consistency for this use case. The latency < 200 ms to see the statistics, and less than 10ms to fire an ingestion event.
• Fault tolerant system, i.e. events are not lost in the system and it is eventually consistent

Capacity Estimations

1B users and 100M QPS -> so about 100M events to be ingested per second.

Our read QPS is defined by the number of websites. I am taking a rough number like 1M websites that would be curated on this platform. So our read qps would be in order of not more than 10M assuming 10 read requests per website from the website admin

Our event ingestion system needs to have 3 months of active storage being kept and the rest in cold storage

• so about 100M per second is 10 * 10^8 * 10^5 = 10^13
• multiply the above with 10^13 * 100B per event -> so about 1PB of data.
• If we want to store for a year it would be 4PB of data

API Design

REST API

POST /event {

User // all user data

website_id

event

ts

..metadata

}

GET /website/:website_id/analytics?analysis_type=&ts_start=&ts_end=&page_size=&cursor=

-> response depends on the analytics type itself

analysis type is what we can show aggregated analytics of what type like conversions etc, or even performance analytics as a type.

High-Level Design

So let's walk through the cycle of how click processing happened when a website is clicked or there's some page view event or something like conversion event this goes through something like a processing service let's call this just a processing service and in here we can just check first the head is cash which can help us dedoop any like bad links or any bad impressions or like retries that might be happening from the website front end this will help us like not write multiple events to the event stream and it's actually only just one minute so we do this by keeping like a lock with some timeout so that next time when it happens like there's a double click or something like that or double event coming in the processing service we see that the key already with that particular like event with some.Something nature that defines it and that would make sure that we do this this then goes to like the events event service event stream which is basically then consumed by the flink service the flink is basically a stream processing system which takes into the entire stream or stream of data and when we coming from any streams are like Kafka or kinesis or something like that to consume it from time to time on a window based system and Aggregate this information on timestamp basis and we can have multiple different like timestamps aggregations like minute-based aggregations or like our base aggregations in our case if we are looking at like aggregations per second which is what we have accounted for in our capacity estimations we're looking at per second base?

Aggregations happening for given website and this is then written back to like the whole updatabase which is storing all the per second based aggregations onto a read optimized database and this read optimized database can be something like a Time series database or something like that which helps us query via the analytics are read service now?

Let's go to the read path so on the analytics read path we are having an Analytics admin panel which is coating the analytics read service which has a cache to look up to and read optimized database to look after that's what I have put up in the diagram so what what it does is it takes the query and then figures out what way to be written for the analytics olam DB to query which which partitions because it's like a Time series DB that we have chosen so which partitions based on the TS start in dsn that this again passed to the backend and get that data from the read optimized database inside the cache we store stuff like partial queries and partial database like partial query results that we have gotten so something like if someone else query like for one day and then the query for like seven days we already have like one days?

Data so we will get it from the cache instead of querying and from the up database this is what the analytics read service like queries and gives back to the Android experiment that's how it like is going to help us write to the read from the analytics.

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.