hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ashish vyas <mailashishv...@gmail.com>
Subject Difference between Nutch crawl giving depth='N' and crawling in loop N times with depth='1'
Date Thu, 12 Jul 2012 08:43:59 GMT
Background of my problem: I am running Nutch1.4 on Hadoop0.20.203. There
are series of MapReduce jobs that i am performing on Nutch segments to get
final output. But waiting for whole crawl to happen before running
mapreduce causes solution to run for longer time. I am now triggering
MapReduce jobs on segments as soon as they are dumped. I am running crawl
in a loop('N=depth' times ) by giving depth=1.I am getting some urls
getting lost when i crawl with depth 1 in a loop N times vs crawl giving
depth N.

Please find below pseudo code:

*Case 1*: Nutch crawl on Hadoop giving depth=3.

// Create the list object to store arguments which we are going to pass to
NUTCH

List nutchArgsList = new ArrayList();

nutchArgsList.add("-depth");

nutchArgsList.add(Integer.toString(3));

<...other nutch args...>

ToolRunner.run(nutchConf, new Crawl(), nutchArgsList.toArray(new
String[nutchArgsList.size()]));

*Case 2*: Crawling in loop 3 times with depth='1'

for(int depthRun=0;depthRun< 3;depthRun++) {

// Create the list object to store arguments which we are going to pass to
NUTCH

List nutchArgsList = new ArrayList();

nutchArgsList.add("-depth");

nutchArgsList.add(Integer.toString(1)); //*NOTE* i have given depth as 1
here

<...other nutch args...>

ToolRunner.run(nutchConf, new Crawl(), nutchArgsList.toArray(new
String[nutchArgsList.size()]));

}

I am getting some urls getting lost(db unfetched) when i crawling in loop
as many times as depth.

I have tried this on standalone Nutch where i run with depth 3 vs running 3
times over same urls with depth 1. I have compared the crawldb and urls
difference is only 12. But when i do the same on Hadoop using toolrunner i
am getting 1000 urls as db_unfetched.

As far i understood till now,Nutch triggers crawl in a loop as many times
as depth value. Please suggest.

Also please let me know why difference is huge when i do this on Hadoop
using toolrunner vs doing the same on standalone Nutch.

Thanks in advandce.


Regards:

Ashish V

Mime
View raw message