Requirements

Functional Requirements:

• Track users traffic
• Track website performance
• Ability to analyze and visualize data

Non-Functional Requirements:

• System should be able to process large amount of data
• No need for realtime data collection

API Design

POST /{website_id}/logs

Input : { timestamp, page, ... }

GET /{website_id}/logs/{timeframe}

POST /{website_id}/metrics/{timeframe}

GET /{website_id}/metrics/{timeframe}

GET /{website_id}/dashboards/{id}

High-Level Design

Traffic is spread across many webservers using a load-balancer

Data collection (logs services) is separated from data visualization (dashboard webapp)

Logs are processed asynchronously using a queue

Detailed Component Design

The key point in this design is to be able to ingest very large amount of data efficienlty.

That's why we use a queue system to process the logs asynchronously. We can add more logs processing workers when needed.

The database technology for logs is NoSQL. Ideally we should have flexible schema so different sites can track different types of data.

Exemple of a database record:

• Timestamp
• Page
• Timetaken
• User agent
• Device (pc, mobile?)

We can also choose a time series oriented database that can scale well for this kind of purpose.

The logs data can be partionned using the timestamp as the key.

We don't need to keep data forever, we can have a purge system for data older than 1 year for instance.