Loading...
1) Crawl the web
2.) same url not be crawled again
3.) follow robots.txt guideline
Estimate the scale of the system you are going to design...
POST /api/crawl/startPOST /api/crawl/startGET /api/crawl/statusPOST /api/urls/addGET /api/data/crawledAmazon S3 for storing webcontent and metadata db like sql for storing the urls and their crawl status
BFS Will be applicable here as we are more interested in going in breadth for higher priority urls.
one Queue is reuired for keeping urls which are to be crawled and ohter queue will be basically for priority wise . Deduplication Service for avoiding crawling the urls which are already crawled which wil be reffering the metadata database sql one for checking so we can store the md sum for those urls in the sql database
client -> load balancer -> frontier queue -> deduplicationservice -> meta data database for storing hash ofthe urls which are crawled -> connected with s3 for storing the content there
client -> load balancer -> frontier queue -> deduplicationservice -> meta data database for storing hash ofthe urls which are crawled -> connected with s3 for storing the content there
BFS Will be applicable here as we are more interested in going in breadth for higher priority urls.
one Queue is reuired for keeping urls which are to be crawled and ohter queue will be basically for priority wise . Deduplication Service for avoiding crawling the urls which are already crawled which wil be reffering the metadata database sql one for checking so we can store the md sum for those urls in the sql database
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?