Loading...
POST /seed/urls body:{urls:JSON} this is used by admins if they want to start with multiple wikipedia urls and not just the home page one
POST /limits body:{depthLimit:number} - also used by admins if they want to set a depth limit so the crawler doesnt go too deep in an interation
GET /metrics - returns the metrics for admins
GET /status/{jobId} - returns status of crawling jobId
POST /crawl/start body:{maxDepth: number, crawlPage: string} starts a new crawl job
POST /crawl/stop - stops gracefully the current jobs by not allowing new ones to start and letting the current ones finish
GET /urls?status={queued|crawling|done|failed}&limit=100&offset=0 - lists urls filtered by status, it limits them, adds offset - paging them
GET /urls/{urlId} - lists a url status
DELETE /urls/{urlId} - deletes a url
the admin will provide some seed urls such as home page and more frequently visited articles or pages.
the URL will go through the URL Frontier, which inside has a Prioritizer which computes the priority of each article/page and maps it to a front queue based on the resulting number;
the front queues will contain the articles grouped by priority.
front queue selector will then choose a queue with a bias on the higher priority ones and it will distribute it to the worker threads which are HTML downloaders.
the HTML downloaders will handle politeness and in case of many calls it will add some delays to their calls. also they could use a rate limiter like the token bucket one.
HTML downloaders will first check the cached DNS for the IPs, and in case they dont find the specific url, they will make a call to DNS.
Content loader will then be hit, it will check if this content wasnt already processed under a different URL by leveraging bloom filters. the content is hashed and saved in content DB and in cache along with the entire content. when checking if we havent already processed this content, we check the current hashed content against the DB, if we find it, then we stop the current crawl process and continue with the next URL, if not, we continue.
Link extractor will extract all the links found in the page content;
URL filter will check if the found URLs are excluded by robots.txt or by admins, it will also check if they are malformed and exclude them;
afterwards it will reach the URL finder which makes sure the same URL wasnt already processed, again by using bloom filters it would hash the url and check it against the cache and in case it isnt find against the db and will update the cache. in case it is a new url it will forward it to the frontier and start processing it.
for detecting changes and recrawling, we could use Wikipedia's recent changes feed and call a change detector which will check in the url db when was the URL lastly crawled and recrawl if it was crawled before the newest updates by pushing the url again to frontier. there wont be any duplicates because we'll update the url lastCrawledDate and wont add a new entry (upsert instead of create)
to also be more sure, we could have a cronjob called recrawl scheduler which will check in the db when were the urls lastly crawled and if it exceeded a set CrawlInterval, then repush to the url frontier the url
the prioritizer scores each URL based on multiple weights:
page type: main article, forums, etc;
freshness; article quality;
tasks will be distributed by the front queue selector, it will act also as a load balancer and will distribute evenly the tasks among the worker threads.
we could use kubernetes to manage the pods and nodes health, this will be done automatically, in case of high spikes or failing nodes, the number of pods increases accordingly
wikipedia recent changes feed is helpful for tracking changes, we could check against the db the date of last crawl and if it had more recent changes, we will recrawl it.
as a double check we could also use a scheduler which checks the db and if a set interval has passed, we recrawl the url.
for deduplication, we will use bloom filter, we will hash both the content and the URLs and check before processing again, if we find matches, we drop the specific crawl