Loading...
- System collect perf data from servers
- Server CPU, memory, disk
- Server request handling time
- User can see data in a time graph
- User can set up criteria for alerts
- System sends email to on alerts
Available
- Scale: collect data from 10K servers
- Response time: User can visualize data at most 10 s delay
- 10K servers, each server reports every 5s
- Per day, 10K * 17K req. 170M req/day
- Each req - 1KB
- Per day, 170GB
- Per 2 years, 124TB. 248 TB to be safe.
- Server sends report_data(server_id, timestamp, data) every 5s
data is JSON format of perf data and service response time data
- Client sends get_data(server_id, start_time, end_time, type)
-> returns JSON output
- configure_alerts(user_id, tenant_id, threshold, email_addresses)
Data:
- Write Path: perf data. LSM-based time series DB.
- Read Path: get_data() for visualization. Time based DB also supports efficient time based query.
To further improve write and read throughput, we should partition data. Tenant ID would be a good partition key because users are typically only interested in (and allowed to) access data from servers in their own tenant.
Data Model:
ReportedData
- Timestamp
- server_id
- CPU used
- CPU capaciy
- Memory capacity
- Disk I/O
Write throughput is key. Write goes to Message Queue to handle in case Write Database cannot catch up to the writes.
Performance Data are duplicated in Write DB (source of truth, optimized for write throughput) and Cache (so that real time data can be retrieved with small response time)
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
An interesting question is duplication of data.
Write DB is the source of truth, so we need it.
Cache is necessary to make response time for real-time small.
Error Cases:
- Servers may not stop sending data, making the whole system unaware of what is going on with servers. We need to have default alerts who notify admin if servers stop sending updates.
- Duplicate all services. All services are critical in this system. As such, all systems should have some redundancy. All systems picked support redundant configurations.
Because all subsystems are critical, we should have monitoring & alerting for our subsystems.
Data in Write DB should be replicated three ways (in the same data center, in another data center in the same region, and in a faraway datacenter for disaster recovery).
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?