Loading...
- System collect perf data from servers
- Server CPU, memory, disk
- Server request handling time
- User can see data in a time graph
- User can set up criteria for alerts
- System sends email to on alerts
Available
- Scale: collect data from 10K servers
- Response time: User can visualize data at most 10 s delay
- 10K servers, each server reports every 5s
- Per day, 10K * 17K req. 170M req/day
- Each req - 1KB
- Per day, 170GB
- Per 2 years, 124TB. 248 TB to be safe.
- Server sends report_data(server_id, timestamp, data) every 5s
data is JSON format of perf data and service response time data
- Client sends get_data(server_id, start_time, end_time, type)
-> returns JSON output
- configure_alerts(user_id, tenant_id, threshold, email_addresses)
Data:
- Write Path: perf data. LSM-based DB like Cassandra
- Read Path: get_data() for visualization. Document DB like MongoDB.
Data Model:
ReportedData
- Timestamp
- server_id
- CPU used
- CPU capaciy
- Memory capacity
- Disk I/O
Write throughput is key. Write goes to Message Queue to handle in case Write Database cannot catch up to the writes.
Performance Data are duplicated in Write DB (source of truth, optimized for write throughput), Cache (so that real time data can be retrieved with small response time), and Read DB (Document DB with secondary indices for fast query and protect Cassandra from read accesses). This may look redundant, but I think it is important to separate a subsystem optimized for writes, and one optimized for reads.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
An interesting question is duplication of data.
Write DB is the source of truth, so we need it.
Cache is necessary to make response time for real-time small.
Read DB is needed in case queried data have left cache, but we don't want to burden Write DB for read and search indices.
Error Cases:
- Servers may not stop sending data, making the whole system unaware of what is going on with servers. We need to have default alerts who notify admin if servers stop sending updates.
- Duplicate all services. All services are critical in this system. As such, all systems should have some redundancy. All systems picked support redundant configurations.
Because all subsystems are critical, we should have monitoring & alerting for our subsystems.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?