Loading...
For example, each crawler can complete crawling a url within 5s, so 10k crawlers can produces ~ 2k PUT requests/s on our data system
For data storage, let's assume an average article size of about 1MB including texts, metadata and its images. 2k PUT requests/s ~ 2GB/s data. If the data is permanently stored, we will have 2GB/s * 24 * 60 * 60 = 180000GB ~ 180TB/day
For storing task & its statuses, each task records ~ 2kb(id - 128bits, status ~ 2bits , start_time ~ 10bytes, end_time ~ 10bytes, url ~ 1kb ) 2k PUT requests/s ~ 2k * 2kb ~ 4MB/s
After complete task crawlers fetch new urls to be crawled ~ 2k GET requests/s.
Define what APIs are expected from the system...
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
Task schedulers fetch tasks from task database. Then, it will place tasks based on strategy such as round-robin or weighted by checking CPU usage of each crawler.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?