yes i'v just tested nohup and it works :) thx to all > Date: Mon, 7 Dec 2009 19:26:42 +0100 > Subject: Re: recrawl.sh stopped at depth 7/10 without error > From: millebii@gmail.com > To: nutch-user@lucene.apache.org > > Another an alternative to crontab, I use nohup command to get my jobs running. > > 2009/12/7, BELLINI ADAM <mbellil@msn.com>: > > > > thx fuad for the info...yes i was just closing my laptop without exting the > > ssh session. > > but now i hv it running form my cron and it didnt stop :) > > thx again > > > > > > > > > >> From: fuad@efendi.ca > >> To: nutch-user@lucene.apache.org > >> Subject: RE: recrawl.sh stopped at depth 7/10 without error > >> Date: Mon, 7 Dec 2009 12:58:48 -0500 > >> > >> >crawl.log 2>&1 & > >> > >> You forgot 2>&1... output for errors... > >> > >> Also, you need to close _politely_ the SSH session by executing "exit". > >> Without it, it pipe is broken, OS will kill the process. > >> > >> > >> Fuad Efendi > >> +1 416-993-2060 > >> http://www.tokenizer.ca > >> Data Mining, Vertical Search > >> > >> > >> > -----Original Message----- > >> > From: BELLINI ADAM [mailto:mbellil@msn.com] > >> > Sent: December-07-09 12:01 PM > >> > To: nutch-user@lucene.apache.org > >> > Subject: RE: recrawl.sh stopped at depth 7/10 without error > >> > > >> > > >> > > >> > > >> > hi, > >> > > >> > mabe i found my probleme, it's not nutch mistake, i beleived when > >> > running > >> > the crawl command as background process when closing my console it will > >> > not stop the process, but it seems that it realy kill the process > >> > > >> > > >> > i launched the porcess like this : ./bin/nutch crawl urls -dir crawl > >> > depth > >> > -10 > crawl.log & > >> > > >> > but even with the '&' caractere when closing my console it kills the > >> > process. > >> > > >> > thx > >> > > >> > > Date: Mon, 7 Dec 2009 19:00:37 +0800 > >> > > Subject: Re: recrawl.sh stopped at depth 7/10 without error > >> > > From: yeahyf@gmail.com > >> > > To: nutch-user@lucene.apache.org > >> > > > >> > > I sill want to know the reason. > >> > > > >> > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > >> > > > >> > > > > >> > > > hi, > >> > > > > >> > > > anay idea guys ?? > >> > > > > >> > > > > >> > > > > >> > > > thanx > >> > > > > >> > > > > From: mbellil@msn.com > >> > > > > To: nutch-user@lucene.apache.org > >> > > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > >> > > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > >> > > > > > >> > > > > > >> > > > > > >> > > > > hi, > >> > > > > > >> > > > > this is the main loop of my recrawl.sh > >> > > > > > >> > > > > > >> > > > > do > >> > > > > > >> > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > >> > > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments > >> > $topN \ > >> > > > > -adddays $adddays > >> > > > > if [ $? -ne 0 ] > >> > > > > then > >> > > > > echo "runbot: Stopping at depth $depth. No more URLs to > >> > > > > fetch." > >> > > > > break > >> > > > > fi > >> > > > > segment=`ls -d $crawl/segments/* | tail -1` > >> > > > > > >> > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > >> > > > > if [ $? -ne 0 ] > >> > > > > then > >> > > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > >> > > > > echo "runbot: Deleting segment $segment." > >> > > > > rm $RMARGS $segment > >> > > > > continue > >> > > > > fi > >> > > > > > >> > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > >> > > > > > >> > > > > done > >> > > > > > >> > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > >> > > > > > >> > > > > > >> > > > > > >> > > > > in my log file i never find the message "----- Merge Segments > >> > > > > (Step > >> > 3 of > >> > > > $steps) -----" ! so it breaks the loop and stops the process. > >> > > > > > >> > > > > i dont understand why it stops at depth 7 without any errors ! > >> > > > > > >> > > > > > >> > > > > > From: mbellil@msn.com > >> > > > > > To: nutch-user@lucene.apache.org > >> > > > > > Subject: recrawl.sh stopped at depth 7/10 without error > >> > > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > hi, > >> > > > > > > >> > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 > >> > without > >> > > > any error ! but when run the bin/crawl with the same crawl-urlfilter > >> > and the > >> > > > same seeds file it finishs softly in 1h50 > >> > > > > > > >> > > > > > i checked the hadoop.log, and dont find any error there...i just > >> > find > >> > > > the last url it was parsing > >> > > > > > do fetching or crawling has a timeout ? > >> > > > > > my recrawl takes 2 hours before it stops. i set the time fetch > >> > interval > >> > > > 24 hours and i'm running the generate with adddays = 1 > >> > > > > > > >> > > > > > best regards > >> > > > > > > >> > > > > > _________________________________________________________________ > >> > > > > > Eligible CDN College & University students can upgrade to > >> > > > > > Windows > >> > 7 > >> > > > before Jan 3 for only $39.99. Upgrade now! > >> > > > > > http://go.microsoft.com/?linkid=9691819 > >> > > > > > >> > > > > _________________________________________________________________ > >> > > > > Eligible CDN College & University students can upgrade to Windows > >> > > > > 7 > >> > > > before Jan 3 for only $39.99. Upgrade now! > >> > > > > http://go.microsoft.com/?linkid=9691819 > >> > > > > >> > > > _________________________________________________________________ > >> > > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on > >> > Windows 7 > >> > > > now > >> > > > http://go.microsoft.com/?linkid=9691818 > >> > > >> > _________________________________________________________________ > >> > Ready. Set. Get a great deal on Windows 7. See fantastic deals on > >> > Windows > >> > 7 now > >> > http://go.microsoft.com/?linkid=9691818 > >> > >> > > > > _________________________________________________________________ > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > > now > > http://go.microsoft.com/?linkid=9691818 > > > -- > -MilleBii- _________________________________________________________________ Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now http://go.microsoft.com/?linkid=9691818
Another an alternative to crontab, I use nohup command to get my jobs running. 2009/12/7, BELLINI ADAM <mbellil@msn.com>: > > thx fuad for the info...yes i was just closing my laptop without exting the > ssh session. > but now i hv it running form my cron and it didnt stop :) > thx again > > > > >> From: fuad@efendi.ca >> To: nutch-user@lucene.apache.org >> Subject: RE: recrawl.sh stopped at depth 7/10 without error >> Date: Mon, 7 Dec 2009 12:58:48 -0500 >> >> >crawl.log 2>&1 & >> >> You forgot 2>&1... output for errors... >> >> Also, you need to close _politely_ the SSH session by executing "exit". >> Without it, it pipe is broken, OS will kill the process. >> >> >> Fuad Efendi >> +1 416-993-2060 >> http://www.tokenizer.ca >> Data Mining, Vertical Search >> >> >> > -----Original Message----- >> > From: BELLINI ADAM [mailto:mbellil@msn.com] >> > Sent: December-07-09 12:01 PM >> > To: nutch-user@lucene.apache.org >> > Subject: RE: recrawl.sh stopped at depth 7/10 without error >> > >> > >> > >> > >> > hi, >> > >> > mabe i found my probleme, it's not nutch mistake, i beleived when >> > running >> > the crawl command as background process when closing my console it will >> > not stop the process, but it seems that it realy kill the process >> > >> > >> > i launched the porcess like this : ./bin/nutch crawl urls -dir crawl >> > depth >> > -10 > crawl.log & >> > >> > but even with the '&' caractere when closing my console it kills the >> > process. >> > >> > thx >> > >> > > Date: Mon, 7 Dec 2009 19:00:37 +0800 >> > > Subject: Re: recrawl.sh stopped at depth 7/10 without error >> > > From: yeahyf@gmail.com >> > > To: nutch-user@lucene.apache.org >> > > >> > > I sill want to know the reason. >> > > >> > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> >> > > >> > > > >> > > > hi, >> > > > >> > > > anay idea guys ?? >> > > > >> > > > >> > > > >> > > > thanx >> > > > >> > > > > From: mbellil@msn.com >> > > > > To: nutch-user@lucene.apache.org >> > > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error >> > > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 >> > > > > >> > > > > >> > > > > >> > > > > hi, >> > > > > >> > > > > this is the main loop of my recrawl.sh >> > > > > >> > > > > >> > > > > do >> > > > > >> > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" >> > > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments >> > $topN \ >> > > > > -adddays $adddays >> > > > > if [ $? -ne 0 ] >> > > > > then >> > > > > echo "runbot: Stopping at depth $depth. No more URLs to >> > > > > fetch." >> > > > > break >> > > > > fi >> > > > > segment=`ls -d $crawl/segments/* | tail -1` >> > > > > >> > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads >> > > > > if [ $? -ne 0 ] >> > > > > then >> > > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." >> > > > > echo "runbot: Deleting segment $segment." >> > > > > rm $RMARGS $segment >> > > > > continue >> > > > > fi >> > > > > >> > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment >> > > > > >> > > > > done >> > > > > >> > > > > echo "----- Merge Segments (Step 3 of $steps) -----" >> > > > > >> > > > > >> > > > > >> > > > > in my log file i never find the message "----- Merge Segments >> > > > > (Step >> > 3 of >> > > > $steps) -----" ! so it breaks the loop and stops the process. >> > > > > >> > > > > i dont understand why it stops at depth 7 without any errors ! >> > > > > >> > > > > >> > > > > > From: mbellil@msn.com >> > > > > > To: nutch-user@lucene.apache.org >> > > > > > Subject: recrawl.sh stopped at depth 7/10 without error >> > > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 >> > > > > > >> > > > > > >> > > > > > >> > > > > > hi, >> > > > > > >> > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 >> > without >> > > > any error ! but when run the bin/crawl with the same crawl-urlfilter >> > and the >> > > > same seeds file it finishs softly in 1h50 >> > > > > > >> > > > > > i checked the hadoop.log, and dont find any error there...i just >> > find >> > > > the last url it was parsing >> > > > > > do fetching or crawling has a timeout ? >> > > > > > my recrawl takes 2 hours before it stops. i set the time fetch >> > interval >> > > > 24 hours and i'm running the generate with adddays = 1 >> > > > > > >> > > > > > best regards >> > > > > > >> > > > > > _________________________________________________________________ >> > > > > > Eligible CDN College & University students can upgrade to >> > > > > > Windows >> > 7 >> > > > before Jan 3 for only $39.99. Upgrade now! >> > > > > > http://go.microsoft.com/?linkid=9691819 >> > > > > >> > > > > _________________________________________________________________ >> > > > > Eligible CDN College & University students can upgrade to Windows >> > > > > 7 >> > > > before Jan 3 for only $39.99. Upgrade now! >> > > > > http://go.microsoft.com/?linkid=9691819 >> > > > >> > > > _________________________________________________________________ >> > > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on >> > Windows 7 >> > > > now >> > > > http://go.microsoft.com/?linkid=9691818 >> > >> > _________________________________________________________________ >> > Ready. Set. Get a great deal on Windows 7. See fantastic deals on >> > Windows >> > 7 now >> > http://go.microsoft.com/?linkid=9691818 >> >> > > _________________________________________________________________ > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > now > http://go.microsoft.com/?linkid=9691818 -- -MilleBii-
thx fuad for the info...yes i was just closing my laptop without exting the ssh session. but now i hv it running form my cron and it didnt stop :) thx again > From: fuad@efendi.ca > To: nutch-user@lucene.apache.org > Subject: RE: recrawl.sh stopped at depth 7/10 without error > Date: Mon, 7 Dec 2009 12:58:48 -0500 > > >crawl.log 2>&1 & > > You forgot 2>&1... output for errors... > > Also, you need to close _politely_ the SSH session by executing "exit". > Without it, it pipe is broken, OS will kill the process. > > > Fuad Efendi > +1 416-993-2060 > http://www.tokenizer.ca > Data Mining, Vertical Search > > > > -----Original Message----- > > From: BELLINI ADAM [mailto:mbellil@msn.com] > > Sent: December-07-09 12:01 PM > > To: nutch-user@lucene.apache.org > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > > > > > > > > hi, > > > > mabe i found my probleme, it's not nutch mistake, i beleived when running > > the crawl command as background process when closing my console it will > > not stop the process, but it seems that it realy kill the process > > > > > > i launched the porcess like this : ./bin/nutch crawl urls -dir crawl depth > > -10 > crawl.log & > > > > but even with the '&' caractere when closing my console it kills the > > process. > > > > thx > > > > > Date: Mon, 7 Dec 2009 19:00:37 +0800 > > > Subject: Re: recrawl.sh stopped at depth 7/10 without error > > > From: yeahyf@gmail.com > > > To: nutch-user@lucene.apache.org > > > > > > I sill want to know the reason. > > > > > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > > > > > > > > > > hi, > > > > > > > > anay idea guys ?? > > > > > > > > > > > > > > > > thanx > > > > > > > > > From: mbellil@msn.com > > > > > To: nutch-user@lucene.apache.org > > > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > > > > > > > > > > > > > hi, > > > > > > > > > > this is the main loop of my recrawl.sh > > > > > > > > > > > > > > > do > > > > > > > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments > > $topN \ > > > > > -adddays $adddays > > > > > if [ $? -ne 0 ] > > > > > then > > > > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > > > > break > > > > > fi > > > > > segment=`ls -d $crawl/segments/* | tail -1` > > > > > > > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > > > > if [ $? -ne 0 ] > > > > > then > > > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > > > > echo "runbot: Deleting segment $segment." > > > > > rm $RMARGS $segment > > > > > continue > > > > > fi > > > > > > > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > > > > > > > done > > > > > > > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > > > > > > > > > > > > > in my log file i never find the message "----- Merge Segments (Step > > 3 of > > > > $steps) -----" ! so it breaks the loop and stops the process. > > > > > > > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > > > > > > > > > > From: mbellil@msn.com > > > > > > To: nutch-user@lucene.apache.org > > > > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > > > > > > > > > > > > > hi, > > > > > > > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 > > without > > > > any error ! but when run the bin/crawl with the same crawl-urlfilter > > and the > > > > same seeds file it finishs softly in 1h50 > > > > > > > > > > > > i checked the hadoop.log, and dont find any error there...i just > > find > > > > the last url it was parsing > > > > > > do fetching or crawling has a timeout ? > > > > > > my recrawl takes 2 hours before it stops. i set the time fetch > > interval > > > > 24 hours and i'm running the generate with adddays = 1 > > > > > > > > > > > > best regards > > > > > > > > > > > > _________________________________________________________________ > > > > > > Eligible CDN College & University students can upgrade to Windows > > 7 > > > > before Jan 3 for only $39.99. Upgrade now! > > > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > > > > > _________________________________________________________________ > > > > > Eligible CDN College & University students can upgrade to Windows 7 > > > > before Jan 3 for only $39.99. Upgrade now! > > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > > > _________________________________________________________________ > > > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on > > Windows 7 > > > > now > > > > http://go.microsoft.com/?linkid=9691818 > > > > _________________________________________________________________ > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows > > 7 now > > http://go.microsoft.com/?linkid=9691818 > > _________________________________________________________________ Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now http://go.microsoft.com/?linkid=9691818
>crawl.log 2>&1 & You forgot 2>&1... output for errors... Also, you need to close _politely_ the SSH session by executing "exit". Without it, it pipe is broken, OS will kill the process. Fuad Efendi +1 416-993-2060 http://www.tokenizer.ca Data Mining, Vertical Search > -----Original Message----- > From: BELLINI ADAM [mailto:mbellil@msn.com] > Sent: December-07-09 12:01 PM > To: nutch-user@lucene.apache.org > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > > > hi, > > mabe i found my probleme, it's not nutch mistake, i beleived when running > the crawl command as background process when closing my console it will > not stop the process, but it seems that it realy kill the process > > > i launched the porcess like this : ./bin/nutch crawl urls -dir crawl depth > -10 > crawl.log & > > but even with the '&' caractere when closing my console it kills the > process. > > thx > > > Date: Mon, 7 Dec 2009 19:00:37 +0800 > > Subject: Re: recrawl.sh stopped at depth 7/10 without error > > From: yeahyf@gmail.com > > To: nutch-user@lucene.apache.org > > > > I sill want to know the reason. > > > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > > > > > > > hi, > > > > > > anay idea guys ?? > > > > > > > > > > > > thanx > > > > > > > From: mbellil@msn.com > > > > To: nutch-user@lucene.apache.org > > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > > > > > > > > > hi, > > > > > > > > this is the main loop of my recrawl.sh > > > > > > > > > > > > do > > > > > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments > $topN \ > > > > -adddays $adddays > > > > if [ $? -ne 0 ] > > > > then > > > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > > > break > > > > fi > > > > segment=`ls -d $crawl/segments/* | tail -1` > > > > > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > > > if [ $? -ne 0 ] > > > > then > > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > > > echo "runbot: Deleting segment $segment." > > > > rm $RMARGS $segment > > > > continue > > > > fi > > > > > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > > > > > done > > > > > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > > > > > > > > > in my log file i never find the message "----- Merge Segments (Step > 3 of > > > $steps) -----" ! so it breaks the loop and stops the process. > > > > > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > > > > > > > From: mbellil@msn.com > > > > > To: nutch-user@lucene.apache.org > > > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > > > > > > > > > hi, > > > > > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 > without > > > any error ! but when run the bin/crawl with the same crawl-urlfilter > and the > > > same seeds file it finishs softly in 1h50 > > > > > > > > > > i checked the hadoop.log, and dont find any error there...i just > find > > > the last url it was parsing > > > > > do fetching or crawling has a timeout ? > > > > > my recrawl takes 2 hours before it stops. i set the time fetch > interval > > > 24 hours and i'm running the generate with adddays = 1 > > > > > > > > > > best regards > > > > > > > > > > _________________________________________________________________ > > > > > Eligible CDN College & University students can upgrade to Windows > 7 > > > before Jan 3 for only $39.99. Upgrade now! > > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > > > _________________________________________________________________ > > > > Eligible CDN College & University students can upgrade to Windows 7 > > > before Jan 3 for only $39.99. Upgrade now! > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > _________________________________________________________________ > > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on > Windows 7 > > > now > > > http://go.microsoft.com/?linkid=9691818 > > _________________________________________________________________ > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows > 7 now > http://go.microsoft.com/?linkid=9691818
Hi! Did anybody added the search with "or" operator in the nutch1.0 successfully? i found a patch for the 0.9 version, but doesn't work. thanks. -- View this message in context: http://old.nabble.com/OR-support-tp26680899p26680899.html Sent from the Nutch - User mailing list archive at Nabble.com.
i fixed it by putting it in crontab and now i can sleep without thinking of it :) thx u very much > Date: Mon, 7 Dec 2009 12:03:25 -0500 > From: ptomblin@gmail.com > To: nutch-user@lucene.apache.org > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > Try starting it with nohup. 'man nohup' for details. > > -- Sent from my Palm Prº > BELLINI ADAM wrote: > > > > > > > > hi, > > > > mabe i found my probleme, it's not nutch mistake, i beleived when running the crawl command as background process when closing my console it will not stop the process, but it seems that it realy kill the process > > > > > > i launched the porcess like this : ./bin/nutch crawl urls -dir crawl depth -10 > crawl.log & > > > > but even with the '&' caractere when closing my console it kills the process. > > > > thx > > > > > Date: Mon, 7 Dec 2009 19:00:37 +0800 > > > Subject: Re: recrawl.sh stopped at depth 7/10 without error > > > From: yeahyf@gmail.com > > > To: nutch-user@lucene.apache.org > > > > > > I sill want to know the reason. > > > > > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > > > > > > > > > > hi, > > > > > > > > anay idea guys ?? > > > > > > > > > > > > > > > > thanx > > > > > > > > > From: mbellil@msn.com > > > > > To: nutch-user@lucene.apache.org > > > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > > > > > > > > > > > > > hi, > > > > > > > > > > this is the main loop of my recrawl.sh > > > > > > > > > > > > > > > do > > > > > > > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments $topN \ > > > > > -adddays $adddays > > > > > if [ $? -ne 0 ] > > > > > then > > > > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > > > > break > > > > > fi > > > > > segment=`ls -d $crawl/segments/* | tail -1` > > > > > > > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > > > > if [ $? -ne 0 ] > > > > > then > > > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > > > > echo "runbot: Deleting segment $segment." > > > > > rm $RMARGS $segment > > > > > continue > > > > > fi > > > > > > > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > > > > > > > done > > > > > > > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > > > > > > > > > > > > > in my log file i never find the message "----- Merge Segments (Step 3 of > > > > $steps) -----" ! so it breaks the loop and stops the process. > > > > > > > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > > > > > > > > > > From: mbellil@msn.com > > > > > > To: nutch-user@lucene.apache.org > > > > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > > > > > > > > > > > > > hi, > > > > > > > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 without > > > > any error ! but when run the bin/crawl with the same crawl-urlfilter and the > > > > same seeds file it finishs softly in 1h50 > > > > > > > > > > > > i checked the hadoop.log, and dont find any error there...i just find > > > > the last url it was parsing > > > > > > do fetching or crawling has a timeout ? > > > > > > my recrawl takes 2 hours before it stops. i set the time fetch interval > > > > 24 hours and i'm running the generate with adddays = 1 > > > > > > > > > > > > best regards > > > > > > > > > > > > _________________________________________________________________ > > > > > > Eligible CDN College & University students can upgrade to Windows 7 > > > > before Jan 3 for only $39.99. Upgrade now! > > > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > > > > > _________________________________________________________________ > > > > > Eligible CDN College & University students can upgrade to Windows 7 > > > > before Jan 3 for only $39.99. Upgrade now! > > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > > > _________________________________________________________________ > > > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > > > > now > > > > http://go.microsoft.com/?linkid=9691818 > > > > _________________________________________________________________ > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now > > http://go.microsoft.com/?linkid=9691818 _________________________________________________________________ Windows Live: Friends get your Flickr, Yelp, and Digg updates when they e-mail you. http://go.microsoft.com/?linkid=9691817
Try starting it with nohup.  'man nohup' for details. -- Sent from my Palm Prē BELLINI ADAM wrote: hi, mabe i found my probleme, it's not nutch mistake, i beleived when running the crawl command as background process when closing my console it will not stop the process, but it seems that it realy kill the process i launched the porcess like this : ./bin/nutch crawl urls -dir crawl depth -10 > crawl.log & but even with the '&' caractere when closing my console it kills the process. thx > Date: Mon, 7 Dec 2009 19:00:37 +0800 > Subject: Re: recrawl.sh stopped at depth 7/10 without error > From: yeahyf@gmail.com > To: nutch-user@lucene.apache.org > > I sill want to know the reason. > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > > > > hi, > > > > anay idea guys ?? > > > > > > > > thanx > > > > > From: mbellil@msn.com > > > To: nutch-user@lucene.apache.org > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > > > > > hi, > > > > > > this is the main loop of my recrawl.sh > > > > > > > > > do > > > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments $topN \ > > > -adddays $adddays > > > if [ $? -ne 0 ] > > > then > > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > > break > > > fi > > > segment=`ls -d $crawl/segments/* | tail -1` > > > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > > if [ $? -ne 0 ] > > > then > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > > echo "runbot: Deleting segment $segment." > > > rm $RMARGS $segment > > > continue > > > fi > > > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > > > done > > > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > > > > > in my log file i never find the message "----- Merge Segments (Step 3 of > > $steps) -----" ! so it breaks the loop and stops the process. > > > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > > > > From: mbellil@msn.com > > > > To: nutch-user@lucene.apache.org > > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > > > > > hi, > > > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 without > > any error ! but when run the bin/crawl with the same crawl-urlfilter and the > > same seeds file it finishs softly in 1h50 > > > > > > > > i checked the hadoop.log, and dont find any error there...i just find > > the last url it was parsing > > > > do fetching or crawling has a timeout ? > > > > my recrawl takes 2 hours before it stops. i set the time fetch interval > > 24 hours and i'm running the generate with adddays = 1 > > > > > > > > best regards > > > > > > > > _________________________________________________________________ > > > > Eligible CDN College & University students can upgrade to Windows 7 > > before Jan 3 for only $39.99. Upgrade now! > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > _________________________________________________________________ > > > Eligible CDN College & University students can upgrade to Windows 7 > > before Jan 3 for only $39.99. Upgrade now! > > > http://go.microsoft.com/?linkid=9691819 > > > > _________________________________________________________________ > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > > now > > http://go.microsoft.com/?linkid=9691818 _________________________________________________________________ Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now http://go.microsoft.com/?linkid=9691818
hi, mabe i found my probleme, it's not nutch mistake, i beleived when running the crawl command as background process when closing my console it will not stop the process, but it seems that it realy kill the process i launched the porcess like this : ./bin/nutch crawl urls -dir crawl depth -10 > crawl.log & but even with the '&' caractere when closing my console it kills the process. thx > Date: Mon, 7 Dec 2009 19:00:37 +0800 > Subject: Re: recrawl.sh stopped at depth 7/10 without error > From: yeahyf@gmail.com > To: nutch-user@lucene.apache.org > > I sill want to know the reason. > > 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > > > > hi, > > > > anay idea guys ?? > > > > > > > > thanx > > > > > From: mbellil@msn.com > > > To: nutch-user@lucene.apache.org > > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > > > > > hi, > > > > > > this is the main loop of my recrawl.sh > > > > > > > > > do > > > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments $topN \ > > > -adddays $adddays > > > if [ $? -ne 0 ] > > > then > > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > > break > > > fi > > > segment=`ls -d $crawl/segments/* | tail -1` > > > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > > if [ $? -ne 0 ] > > > then > > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > > echo "runbot: Deleting segment $segment." > > > rm $RMARGS $segment > > > continue > > > fi > > > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > > > done > > > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > > > > > in my log file i never find the message "----- Merge Segments (Step 3 of > > $steps) -----" ! so it breaks the loop and stops the process. > > > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > > > > From: mbellil@msn.com > > > > To: nutch-user@lucene.apache.org > > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > > > > > hi, > > > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 without > > any error ! but when run the bin/crawl with the same crawl-urlfilter and the > > same seeds file it finishs softly in 1h50 > > > > > > > > i checked the hadoop.log, and dont find any error there...i just find > > the last url it was parsing > > > > do fetching or crawling has a timeout ? > > > > my recrawl takes 2 hours before it stops. i set the time fetch interval > > 24 hours and i'm running the generate with adddays = 1 > > > > > > > > best regards > > > > > > > > _________________________________________________________________ > > > > Eligible CDN College & University students can upgrade to Windows 7 > > before Jan 3 for only $39.99. Upgrade now! > > > > http://go.microsoft.com/?linkid=9691819 > > > > > > _________________________________________________________________ > > > Eligible CDN College & University students can upgrade to Windows 7 > > before Jan 3 for only $39.99. Upgrade now! > > > http://go.microsoft.com/?linkid=9691819 > > > > _________________________________________________________________ > > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > > now > > http://go.microsoft.com/?linkid=9691818 _________________________________________________________________ Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now http://go.microsoft.com/?linkid=9691818
Is there any readymade plug-in for office 2007 documents available or I have to write it by my own? -----Original Message----- From: yangfeng [mailto:yeahyf@gmail.com] Sent: Monday, December 07, 2009 4:35 PM To: nutch-user@lucene.apache.org Subject: Re: How to successfully crawl and index office 2007 documents in Nutch 1.0 docx should be parsed,A plugin can be used to Parsed docx file. you get some help info from parse-html plugin and so on. 2009/12/4 Rupesh Mankar <rupesh_mankar@persistent.co.in> > Hi, > > I am new to Nutch. I want to crawl and search office 2007 documents (.docx, > .pptx etc) from Nutch. But when I try to crawl, crawler throws following > error: > > fetching http://10.88.45.140:8081/tutorial/Office-2007-document.docx > Error parsing: http://10.88.45.140:8081/tutorial/Office-2007-document.docx: > org.apache.nutch.parse.ParseException: parser not found for > contentType=application/zip url= > http://10.88.45.140:8081/tutorial/Office-2007-document.docx > at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:74) > at > org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:766) > at > org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:552) > > When I add zip plugin in nutch-site.xml under plugin.includes, crawling > becomes successful but nothing gets search. > > How can we successfully crawl and search contents of office 2007 documents? > > Thanks, > Rupesh > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is > the property of Persistent Systems Ltd. It is intended only for the use of > the individual or entity to which it is addressed. If you are not the > intended recipient, you are not authorized to read, retain, copy, print, > distribute or use this message. If you have received this communication in > error, please notify the sender and delete all copies of this message. > Persistent Systems Ltd. does not accept any liability for virus infected > mails. > DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
yangfeng wrote: > I have completed the plugin for parsing the wml(wiredless mark language). I > hope to add it to lucene, what i do? > The best long-term option would be to submit this work to the Tika project - see http://lucene.apache.org/tika/. If you already implemented this as a Nutch plugin, please creata a JIRA issue in Nutch, and attach the patch. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
Hello fellow Nutch users, In a few days we'll start crawling a long list of Thai websites. With previous crawls we noticed there were A LOT of poorly formatted html pages and the crawler would sometimes fetch links that contain html code (ex: http://www.website.com/news/index.php</ul> ). How can we regex those URLs so that the html code (</ul>) is removed? Would we use the regex-normalizer.xml file? If so, what would the code look like? Thanks in advance, Kirk Gillock Isara Charity Foundation Nong Khai, Thailand http://www.isara.org
I have completed the plugin for parsing the wml(wiredless mark language). I hope to add it to lucene, what i do?
docx should be parsed,A plugin can be used to Parsed docx file. you get some help info from parse-html plugin and so on. 2009/12/4 Rupesh Mankar <rupesh_mankar@persistent.co.in> > Hi, > > I am new to Nutch. I want to crawl and search office 2007 documents (.docx, > .pptx etc) from Nutch. But when I try to crawl, crawler throws following > error: > > fetching http://10.88.45.140:8081/tutorial/Office-2007-document.docx > Error parsing: http://10.88.45.140:8081/tutorial/Office-2007-document.docx: > org.apache.nutch.parse.ParseException: parser not found for > contentType=application/zip url= > http://10.88.45.140:8081/tutorial/Office-2007-document.docx > at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:74) > at > org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:766) > at > org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:552) > > When I add zip plugin in nutch-site.xml under plugin.includes, crawling > becomes successful but nothing gets search. > > How can we successfully crawl and search contents of office 2007 documents? > > Thanks, > Rupesh > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is > the property of Persistent Systems Ltd. It is intended only for the use of > the individual or entity to which it is addressed. If you are not the > intended recipient, you are not authorized to read, retain, copy, print, > distribute or use this message. If you have received this communication in > error, please notify the sender and delete all copies of this message. > Persistent Systems Ltd. does not accept any liability for virus infected > mails. >
I sill want to know the reason. 2009/12/2 BELLINI ADAM <mbellil@msn.com> > > hi, > > anay idea guys ?? > > > > thanx > > > From: mbellil@msn.com > > To: nutch-user@lucene.apache.org > > Subject: RE: recrawl.sh stopped at depth 7/10 without error > > Date: Fri, 27 Nov 2009 20:11:12 +0000 > > > > > > > > hi, > > > > this is the main loop of my recrawl.sh > > > > > > do > > > > echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---" > > $NUTCH_HOME/bin/nutch generate $crawl/crawldb $crawl/segments $topN \ > > -adddays $adddays > > if [ $? -ne 0 ] > > then > > echo "runbot: Stopping at depth $depth. No more URLs to fetch." > > break > > fi > > segment=`ls -d $crawl/segments/* | tail -1` > > > > $NUTCH_HOME/bin/nutch fetch $segment -threads $threads > > if [ $? -ne 0 ] > > then > > echo "runbot: fetch $segment at depth `expr $i + 1` failed." > > echo "runbot: Deleting segment $segment." > > rm $RMARGS $segment > > continue > > fi > > > > $NUTCH_HOME/bin/nutch updatedb $crawl/crawldb $segment > > > > done > > > > echo "----- Merge Segments (Step 3 of $steps) -----" > > > > > > > > in my log file i never find the message "----- Merge Segments (Step 3 of > $steps) -----" ! so it breaks the loop and stops the process. > > > > i dont understand why it stops at depth 7 without any errors ! > > > > > > > From: mbellil@msn.com > > > To: nutch-user@lucene.apache.org > > > Subject: recrawl.sh stopped at depth 7/10 without error > > > Date: Wed, 25 Nov 2009 15:43:33 +0000 > > > > > > > > > > > > hi, > > > > > > i'm running recrawl.sh and it stops every time at depth 7/10 without > any error ! but when run the bin/crawl with the same crawl-urlfilter and the > same seeds file it finishs softly in 1h50 > > > > > > i checked the hadoop.log, and dont find any error there...i just find > the last url it was parsing > > > do fetching or crawling has a timeout ? > > > my recrawl takes 2 hours before it stops. i set the time fetch interval > 24 hours and i'm running the generate with adddays = 1 > > > > > > best regards > > > > > > _________________________________________________________________ > > > Eligible CDN College & University students can upgrade to Windows 7 > before Jan 3 for only $39.99. Upgrade now! > > > http://go.microsoft.com/?linkid=9691819 > > > > _________________________________________________________________ > > Eligible CDN College & University students can upgrade to Windows 7 > before Jan 3 for only $39.99. Upgrade now! > > http://go.microsoft.com/?linkid=9691819 > > _________________________________________________________________ > Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 > now > http://go.microsoft.com/?linkid=9691818
you should add property below£º
<property>
<name>hadoop.job.ugi</name>
<value>rider,iamsolomon</value>
</property>
it's ok!
2009/12/1 Mischa Tuffield <mischa.tuffield@garlik.com>
> Hello Brian,
>
> Getting a response from another newbie here, so I could be wrong (do excuse
> if I am).
>
> If you are attempting to run a search index from the filesystem you need to
> have the following in your nutch-site.xml :
>
> <property>
> <name>fs.default.name</name>
> <value>file:///</value>
> </property>
>
> The fs.default.name is require by the nutch-site.xml when you build your
> .war file for deployment to tomcat. This should be accompanied by the below
> config, which should point to the direct where your index has been copied
> to, in my case it looks something like below :
>
> <property>
> <name>searcher.dir</name>
> <value>/home/nutch/nutch/service/crawl</value>
> <description>
> Path to root of crawl. This directory is searched (in
> order) for either the file search-servers.txt, containing a list of
> distributed search servers, or the directory "index" containing
> merged indexes, or the directory "segments" containing segment
> indexes.
> </description>
> </property>
>
> Regarding your second question :
>
> bin/nutch readdb yourcrawldir/crawldb -dump -format csv
>
> Gives you a nice flat file serialisation of your crawl database.
>
> I hope this helps,
>
> Mischa
> On 1 Dec 2009, at 08:44, brian wrote:
>
> > also, I would like to know how to extract flat text files of the crawl
> data.
>
> ___________________________________
> Mischa Tuffield
> Email: mischa.tuffield@garlik.com
> Homepage - http://mmt.me.uk/
> Garlik Limited, 2 Sheen Road, Richmond, TW9 1AE, UK
> +44(0)20 8973 2465 http://www.garlik.com/
> Registered in England and Wales 535 7233 VAT # 849 0517 11
> Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD
>
>
New and longer run ... I get plenty of : failed with: java.lang.OutOfMemoryError: Java heap space Fetching still goes on, not sure if this the expected behavior. 2009/12/6 MilleBii <millebii@gmail.com> > Works fine and my memory problem had to do with the fact that I had too > many threads... > > 2009/12/5 MilleBii <millebii@gmail.com> > >> Thx again Julien, >> >> Yes I'm going to buy myself the Hadoop book, because I thought I could do >> without but I realize that I need to make good use of hadooop. >> >> Didn't know you could split fetching & parsing: so I suppose you just >> issue nutch fetch <segment> -noParsing, followed by nutch parse <segment>. I >> will try on my next run. >> >> >> >> 2009/12/5 Julien Nioche <lists.digitalpebble@gmail.com> >> >> HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and >>> does NOT affect the memory used for the map/ reduce jobs. Maybe you >>> should >>> invest a bit of time reading about Hadoop first? >>> >>> As for your memory problem it could be due to the parsing and not the >>> fetching. If you don't already do so I suggest that you separate the >>> fetching from the parsing. First that will tell you which part fails + if >>> it >>> does fail in the parsing then you would not need to refetch the content >>> >>> J. >>> >>> 2009/12/5 MilleBii <millebii@gmail.com> >>> >>> > My fetch cycle failed on the following initial error : >>> > >>> > java.io.IOException: Task process exit with nonzero status of 65. >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425) >>> > >>> > Than it makes a second attempt and after 3 hours I bump on that error >>> > (altough I had double HADOOP_HEAPSIZE): >>> > >>> > java.lang.OutOfMemoryError: GC overhead limit exceeded >>> > >>> > >>> > Any idea what the initial error is or could be ? >>> > For the second one, I'm going to reduce number of threads... but I'm >>> > wondering if there could be a memory leak ? And I don't how to trace >>> that. >>> > >>> > -- >>> > -MilleBii- >>> > >>> >>> >>> >>> -- >>> DigitalPebble Ltd >>> http://www.digitalpebble.com >>> >> >> >> >> -- >> -MilleBii- >> > > > > -- > -MilleBii- > -- -MilleBii-
Hi - this is my first post to the nutch mailing list, please let me know if I commit any list protocol errors. I'm currently using Nutch 1.0 with the Powerpoint plugin enabled and can verify that Nutch is indeed pulling in the entire file for passing off to the parser (i.e., I've set the content limit to -1 to get the full file). However it appears that most Powerpoint files with any complexity (they use a template, have tables, images, etc.) do not get indexed at all. In one case I created a new file with one "title" slide and the title text was recognized but the subtitle text directly underneath was not. My question is whether I'm missing something that has already been covered (like for example, http://issues.apache.org/jira/browse/NUTCH-463, though I don't see any logs indicating issues in my crawl) or that this is a known defect in the existing Powerpoint plugin? It goes without saying that I'd very much like to be able to completely index Powerpoint slides as this is going to be the most common document type on my site. Thanks, Joe This message is confidential to Prodea Systems, Inc unless otherwise indicated or apparent from its nature. This message is directed to the intended recipient only, who may be readily determined by the sender of this message and its contents. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a)any dissemination or copying of this message is strictly prohibited; and(b)immediately notify the sender by return message and destroy any copies of this message in any form(electronic, paper or otherwise) that you have.The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications. The authority of the individual sending this message to legally bind Prodea Systems is neither apparent nor implied,and must be independently verified.
Works fine and my memory problem had to do with the fact that I had too many threads... 2009/12/5 MilleBii <millebii@gmail.com> > Thx again Julien, > > Yes I'm going to buy myself the Hadoop book, because I thought I could do > without but I realize that I need to make good use of hadooop. > > Didn't know you could split fetching & parsing: so I suppose you just > issue nutch fetch <segment> -noParsing, followed by nutch parse <segment>. I > will try on my next run. > > > > 2009/12/5 Julien Nioche <lists.digitalpebble@gmail.com> > > HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and >> does NOT affect the memory used for the map/ reduce jobs. Maybe you should >> invest a bit of time reading about Hadoop first? >> >> As for your memory problem it could be due to the parsing and not the >> fetching. If you don't already do so I suggest that you separate the >> fetching from the parsing. First that will tell you which part fails + if >> it >> does fail in the parsing then you would not need to refetch the content >> >> J. >> >> 2009/12/5 MilleBii <millebii@gmail.com> >> >> > My fetch cycle failed on the following initial error : >> > >> > java.io.IOException: Task process exit with nonzero status of 65. >> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425) >> > >> > Than it makes a second attempt and after 3 hours I bump on that error >> > (altough I had double HADOOP_HEAPSIZE): >> > >> > java.lang.OutOfMemoryError: GC overhead limit exceeded >> > >> > >> > Any idea what the initial error is or could be ? >> > For the second one, I'm going to reduce number of threads... but I'm >> > wondering if there could be a memory leak ? And I don't how to trace >> that. >> > >> > -- >> > -MilleBii- >> > >> >> >> >> -- >> DigitalPebble Ltd >> http://www.digitalpebble.com >> > > > > -- > -MilleBii- > -- -MilleBii-
Currently the depth queue is hardcoded to 50... however one needs to keep #Threads x Depth below a certain number otherwise fetcher spends its life in managing the queues and CPU becomes the limiting factor. This is what was creating my L shape kind of bandwidth usage. I had to patch it by hand, however I suggest we create a conf item to enable different combination #Threads x Depth. Or we could also set it the opposite way which is to define a "max.fetchqueue.totalSize" and derive the depth as totalSize/#Threads. And of course max.fetchqueue.totalSize is very dependent on the system specs, but this is what you want to control. -- -MilleBii-
Hi,
Just upgraded to the latest version of Nutch with Hadoop 0.20.
I'm getting the following exception in the namenode log and DFS doesn't
start:
2009-12-06 15:48:32,523 ERROR namenode.NameNode -
java.lang.SecurityException: sealing violation: can't seal package
org.mortbay.util: already loaded
at java.net.URLClassLoader.defineClass(URLClassLoader.java:235)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:220)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
Any help will be appreciated ... quite stuck with this.
Thanks,
Eran
hi, u have to make your segments smaller than that, just cut every segment in small pieces > Subject: Indexing with solrindexer -> OutOfMemoryError > From: felizimm@gmx.de > To: nutch-user@lucene.apache.org > Date: Sun, 6 Dec 2009 01:35:04 +0100 > > Hi, > > when trying to index four segments (~5 GB) with solrindexer, I get this > error in hadoop.log. There is no error in the logs of Tomcat, where I > deployed Solr. I crawled with "crawl"-command. > > I`ve read that increasing the hadoop heap space will change nothing. > What can I do? > > Thanks for help! > Felix. > > > 2009-12-06 00:21:51,061 WARN mapred.LocalJobRunner - job_local_0001 > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2882) > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) > at java.lang.StringBuffer.append(StringBuffer.java:320) > at java.io.StringWriter.write(StringWriter.java:60) > at org.apache.solr.common.util.XML.escape(XML.java:180) > at org.apache.solr.common.util.XML.escapeCharData(XML.java:78) > at org.apache.solr.common.util.XML.writeXML(XML.java:148) > at > org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:117) > at > org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:169) > at > org.apache.solr.client.solrj.request.UpdateRequest.getContentStreams(UpdateRequest.java:160) > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:191) > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183) > at > org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217) > at > org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48) > at > org.apache.nutch.indexer.solr.SolrWriter.write(SolrWriter.java:58) > at org.apache.nutch.indexer.IndexerOutputFormat > $1.write(IndexerOutputFormat.java:54) > at org.apache.nutch.indexer.IndexerOutputFormat > $1.write(IndexerOutputFormat.java:44) > at org.apache.hadoop.mapred.ReduceTask > $3.collect(ReduceTask.java:410) > at > org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158) > at > org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) > at org.apache.hadoop.mapred.LocalJobRunner > $Job.run(LocalJobRunner.java:170) > 2009-12-06 00:21:51,650 FATAL solr.SolrIndexer - SolrIndexer: > java.io.IOException: Job failed! > at > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > at > org.apache.nutch.indexer.solr.SolrIndexer.indexSolr(SolrIndexer.java:73) > at > org.apache.nutch.indexer.solr.SolrIndexer.run(SolrIndexer.java:95) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at > org.apache.nutch.indexer.solr.SolrIndexer.main(SolrIndexer.java:104) > > > _________________________________________________________________ Windows Live: Make it easier for your friends to see what you’re up to on Facebook. http://go.microsoft.com/?linkid=9691816
Hi,
when trying to index four segments (~5 GB) with solrindexer, I get this
error in hadoop.log. There is no error in the logs of Tomcat, where I
deployed Solr. I crawled with "crawl"-command.
I`ve read that increasing the hadoop heap space will change nothing.
What can I do?
Thanks for help!
Felix.
2009-12-06 00:21:51,061 WARN mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuffer.append(StringBuffer.java:320)
at java.io.StringWriter.write(StringWriter.java:60)
at org.apache.solr.common.util.XML.escape(XML.java:180)
at org.apache.solr.common.util.XML.escapeCharData(XML.java:78)
at org.apache.solr.common.util.XML.writeXML(XML.java:148)
at
org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:117)
at
org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:169)
at
org.apache.solr.client.solrj.request.UpdateRequest.getContentStreams(UpdateRequest.java:160)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:191)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183)
at
org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217)
at
org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
at
org.apache.nutch.indexer.solr.SolrWriter.write(SolrWriter.java:58)
at org.apache.nutch.indexer.IndexerOutputFormat
$1.write(IndexerOutputFormat.java:54)
at org.apache.nutch.indexer.IndexerOutputFormat
$1.write(IndexerOutputFormat.java:44)
at org.apache.hadoop.mapred.ReduceTask
$3.collect(ReduceTask.java:410)
at
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158)
at
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
at org.apache.hadoop.mapred.LocalJobRunner
$Job.run(LocalJobRunner.java:170)
2009-12-06 00:21:51,650 FATAL solr.SolrIndexer - SolrIndexer:
java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at
org.apache.nutch.indexer.solr.SolrIndexer.indexSolr(SolrIndexer.java:73)
at
org.apache.nutch.indexer.solr.SolrIndexer.run(SolrIndexer.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.nutch.indexer.solr.SolrIndexer.main(SolrIndexer.java:104)
Thx again Julien, Yes I'm going to buy myself the Hadoop book, because I thought I could do without but I realize that I need to make good use of hadooop. Didn't know you could split fetching & parsing: so I suppose you just issue nutch fetch <segment> -noParsing, followed by nutch parse <segment>. I will try on my next run. 2009/12/5 Julien Nioche <lists.digitalpebble@gmail.com> > HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and > does NOT affect the memory used for the map/ reduce jobs. Maybe you should > invest a bit of time reading about Hadoop first? > > As for your memory problem it could be due to the parsing and not the > fetching. If you don't already do so I suggest that you separate the > fetching from the parsing. First that will tell you which part fails + if > it > does fail in the parsing then you would not need to refetch the content > > J. > > 2009/12/5 MilleBii <millebii@gmail.com> > > > My fetch cycle failed on the following initial error : > > > > java.io.IOException: Task process exit with nonzero status of 65. > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425) > > > > Than it makes a second attempt and after 3 hours I bump on that error > > (altough I had double HADOOP_HEAPSIZE): > > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > > > Any idea what the initial error is or could be ? > > For the second one, I'm going to reduce number of threads... but I'm > > wondering if there could be a memory leak ? And I don't how to trace > that. > > > > -- > > -MilleBii- > > > > > > -- > DigitalPebble Ltd > http://www.digitalpebble.com > -- -MilleBii-
HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and does NOT affect the memory used for the map/ reduce jobs. Maybe you should invest a bit of time reading about Hadoop first? As for your memory problem it could be due to the parsing and not the fetching. If you don't already do so I suggest that you separate the fetching from the parsing. First that will tell you which part fails + if it does fail in the parsing then you would not need to refetch the content J. 2009/12/5 MilleBii <millebii@gmail.com> > My fetch cycle failed on the following initial error : > > java.io.IOException: Task process exit with nonzero status of 65. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425) > > Than it makes a second attempt and after 3 hours I bump on that error > (altough I had double HADOOP_HEAPSIZE): > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > Any idea what the initial error is or could be ? > For the second one, I'm going to reduce number of threads... but I'm > wondering if there could be a memory leak ? And I don't how to trace that. > > -- > -MilleBii- > -- DigitalPebble Ltd http://www.digitalpebble.com
My fetch cycle failed on the following initial error : java.io.IOException: Task process exit with nonzero status of 65. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425) Than it makes a second attempt and after 3 hours I bump on that error (altough I had double HADOOP_HEAPSIZE): java.lang.OutOfMemoryError: GC overhead limit exceeded Any idea what the initial error is or could be ? For the second one, I'm going to reduce number of threads... but I'm wondering if there could be a memory leak ? And I don't how to trace that. -- -MilleBii-
Thx, a bit too complex for me right now. I don't -yet- fully understand this map/reduce technique. But I'll keep the idea for a future development. 2009/12/4 Dennis Kubes <kubes@apache.org> > Sorry, segments, not indexes. > > > Dennis Kubes wrote: > >> You would need to write a custom MapReduce job to run through the indexes >> and only keeps the ones identified by your plugin. Be sure to update the >> CrawlDb with the extracted urls before you drop the content from the >> segments. >> >> Dennis >> >> MilleBii wrote: >> >>> Hi guys, >>> >>> I'm looking if I can optimize the size occupied on disk by my segments. >>> I have implemented a topical-scoring plugin... this means I know at that >>> steps if I should keep that page content or not. >>> Is there a way to drop some pages content after parsing it, but of course >>> keep the links because I want to follow the graph ? >>> >>> PS: Prune is no option to me because it only cleans up the indexes, not >>> the >>> segments and my indexer does that clean-up very well. >>> >>> -- -MilleBii-
Hi, I'm developing my own set of tools, plugins and some minor code changes to Nutch. I still want to get updates from the main Nutch repository, but I would like to keep my own SVN for tracking my local code changes. I'm using normal shell SVN (I have no expirence with GIT) to track my changes. My question is - can I create a branch from the main repository to my own repository, which will only track my changes and keep getting updates from Nutch main repository with easy merge? Thanks, Eran
I did it many times. But I am still receiving these mails. prashant ullegaddi wrote: > Take a look at it: > > http://lucene.apache.org/nutch/mailing_lists.html > > or > probably sending a blank mail to: > nutch-user-unsubscribe@lucene.apache.orgshould also > work. > > Thanks, > Prashant. > > On Fri, Dec 4, 2009 at 8:30 PM, M S Ram <msram@cse.iitk.ac.in> wrote: > > >> Same here. Please remove my ID also from the mailing list. >> >> Thanks, >> MSR >> >> rengan xu wrote: >> >> >>> To whom it may concern, >>> >>> Hello! Because I will use this E-mail for special purpose. I will use >>> another E-mail to subscribe in nutch-user. So I want to unsubscribe from >>> nutch-user. >>> >>> Thank you! >>> >>> >>> >>> >>> >> >> > > >
Hi, I am using nutch to crawl the data from the web. Now I want to extract the images using nutch. Can somebody please suggest me some way how to do that or sugeest me some url? Regards, Manish Bawne Software Engineer Biz Integra Systems Pvt Ltd http://www.bizhandel.com -- View this message in context: http://old.nabble.com/Nutch-image-extraction-tp26653542p26653542.html Sent from the Nutch - User mailing list archive at Nabble.com.
Sorry, segments, not indexes. Dennis Kubes wrote: > You would need to write a custom MapReduce job to run through the > indexes and only keeps the ones identified by your plugin. Be sure to > update the CrawlDb with the extracted urls before you drop the content > from the segments. > > Dennis > > MilleBii wrote: >> Hi guys, >> >> I'm looking if I can optimize the size occupied on disk by my segments. >> I have implemented a topical-scoring plugin... this means I know at that >> steps if I should keep that page content or not. >> Is there a way to drop some pages content after parsing it, but of course >> keep the links because I want to follow the graph ? >> >> PS: Prune is no option to me because it only cleans up the indexes, >> not the >> segments and my indexer does that clean-up very well. >>
You would need to write a custom MapReduce job to run through the indexes and only keeps the ones identified by your plugin. Be sure to update the CrawlDb with the extracted urls before you drop the content from the segments. Dennis MilleBii wrote: > Hi guys, > > I'm looking if I can optimize the size occupied on disk by my segments. > I have implemented a topical-scoring plugin... this means I know at that > steps if I should keep that page content or not. > Is there a way to drop some pages content after parsing it, but of course > keep the links because I want to follow the graph ? > > PS: Prune is no option to me because it only cleans up the indexes, not the > segments and my indexer does that clean-up very well. >
Hi guys, I'm looking if I can optimize the size occupied on disk by my segments. I have implemented a topical-scoring plugin... this means I know at that steps if I should keep that page content or not. Is there a way to drop some pages content after parsing it, but of course keep the links because I want to follow the graph ? PS: Prune is no option to me because it only cleans up the indexes, not the segments and my indexer does that clean-up very well. -- -MilleBii-
Sounds like Nutch for crawling to gather the data, custom tools to read the gathered data, call the KV store, construct SolrInputDocuments, and index those to Solr. If you want Solr and not Lucene, which is a bigger question that I can't answer without knowing the details. Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch ----- Original Message ---- > From: Mr Hadoop <mrhadoop@gmail.com> > To: nutch-user@lucene.apache.org > Sent: Fri, December 4, 2009 2:51:47 PM > Subject: What is the best choice: nutch/lucene or nutch/solr? > > I am going over mailing list and still didn't find an answer. > > For a project, I need to crawl the web, index it and merge that content with > another site's content which is stored inside the key-value storage system. > > What is the best approach to merge these two contents in to a lucene index, > solr index or keep the index separate but merge during the search query > results?
I am going over mailing list and still didn't find an answer. For a project, I need to crawl the web, index it and merge that content with another site's content which is stored inside the key-value storage system. What is the best approach to merge these two contents in to a lucene index, solr index or keep the index separate but merge during the search query results?
Hi, Does anyone know what packages I have to install in Suse to get Nutch running? I have another installation with nutch where everything is fine. So I copied the hole installation. It's also an Suse linux but it is in 64 bit and I don’t installed it. But the same problem. At the moment I installed the following packages: Tomcat 6 Openjdk devel 1.6 Sun java devel 1.6 Ant 1.7 Now it is enough for today. Hope someone can help. Tom -----Original Message----- From: MilleBii [mailto:millebii@gmail.com] Sent: Freitag, 4. Dezember 2009 17:31 To: nutch-user@lucene.apache.org Subject: Re: Problems with a new Installation of Nutch I don't know that hadoop uses tomcat... But I think it uses Jetty instead. The nodes communicate via http: so you need some kind of web server... And for monitorin its the best way 2009/12/4, Tom Landvoigt <tom.landvoigt@linklift.de>: > Hi, > > I don't have tomcat on this system because I don't want to use the > websearch. But if it is necessary for hadoop what I don’t think I will > install it. > > nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls / > Found 1 items > -rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt > nutch@ip-10-224-113-210:/nutch/search> > > I get the normal answer but the file is empty. > > -----Original Message----- > From: MilleBii [mailto:millebii@gmail.com] > Sent: Freitag, 4. Dezember 2009 15:06 > To: nutch-user@lucene.apache.org > Subject: Re: Problems with a new Installation of Nutch > > Did you check with the web interface ? It gives a lot of info you can > even browse the file system. > > Try hadoop fs -ls to see what it gives you ? > > 2009/12/4, Tom Landvoigt <tom.landvoigt@linklift.de>: >> Hallo, >> >> >> >> I hope someone can help me. >> >> >> >> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I >> can't put data in the hdfs. >> >> >> >> I formatted the namenode and start the hdfs with start all. >> >> >> >> All java processes start properly, but when I want to make hadoop fs >> -put something / I get these logs: >> >> >> >> >> >> >> >> nutch@bla:/nutch/search> ./bin/hadoop fs -put >> /tmp/hadoop-nutch-tasktracker.pid blub >> >> put: Protocol not available >> >> >> >> DATA NODE LOG on the master: >> >> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:16,483 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e >> >> 2009-12-04 12:50:16,614 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:16,882 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4 >> >> 2009-12-04 12:50:16,883 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:17,827 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1 >> >> 2009-12-04 12:50:17,849 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50075 >> >> 2009-12-04 12:50:18,485 INFO util.Container - Started >> org.mortbay.jetty.Server@36527f >> >> 2009-12-04 12:54:20,745 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,746 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,747 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,747 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> >> >> NAME NODE LOG >> >> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:12,488 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451 >> >> 2009-12-04 12:50:12,565 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:12,891 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@1570945 >> >> 2009-12-04 12:50:12,891 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:13,569 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5 >> >> 2009-12-04 12:50:13,582 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50070 >> >> 2009-12-04 12:50:13,613 INFO util.Container - Started >> org.mortbay.jetty.Server@173ec72 >> >> >> >> SECONDARY NAMENODE LOG >> >> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:20,365 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a >> >> 2009-12-04 12:50:20,454 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:21,396 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7 >> >> 2009-12-04 12:50:21,396 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No >> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving >> files and default/dynamic servlets only >> >> 2009-12-04 12:50:22,206 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@383118 >> >> 2009-12-04 12:50:22,785 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50090 >> >> 2009-12-04 12:50:22,787 INFO util.Container - Started >> org.mortbay.jetty.Server@297ffb >> >> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint >> Period :3600 secs (60 min) >> >> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size >> Trigger :67108864 bytes (65536 KB) >> >> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint >> done. New Image Size: 1056 >> >> >> >> HADOOP LOG >> >> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception: >> java.io.IOException: Unable to create new block. >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D >> FSClient.java:2722) >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j >> ava:1996) >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli >> ent.java:2183) >> >> >> >> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block >> blk_5506837520665828594_1002 bad datanode[0] nodes == null >> >> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block >> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" - >> Aborting... >> >> >> >> DATA NODE LOG on the slave >> >> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:49:50,288 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e >> >> 2009-12-04 12:49:50,357 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:49:50,555 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0 >> >> 2009-12-04 12:49:50,555 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:49:50,816 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@118278a >> >> 2009-12-04 12:49:50,820 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50075 >> >> 2009-12-04 12:49:50,849 INFO util.Container - Started >> org.mortbay.jetty.Server@b02928 >> >> >> >> HADOOP SITE XML >> >> <property> >> >> <name>fs.default.name</name> >> >> <value>hdfs://(yes here is the right ip):9000</value> >> >> <description> >> >> The name of the default file system. Either the literal string >> >> "local" or a host:port for NDFS. >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege) >> zu finden ist. --> >> >> <property> >> >> <name>mapred.job.tracker</name> >> >> <value>hdfs://(here to):9001</value> >> >> <description> >> >> The host and port that the MapReduce job tracker runs at. If >> >> "local", then jobs are run in-process as a single map and >> >> reduce task. >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen--> >> >> <property> >> >> <name>mapred.tasktracker.map.tasks.maximum</name> >> >> <value>2</value> >> >> <description> >> >> define mapred.map tasks to be number of slave hosts >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen--> >> >> <property> >> >> <name>mapred.tasktracker.reduce.tasks.maximum</name> >> >> <value>2</value> >> >> <description> >> >> define mapred.reduce tasks to be number of slave hosts >> >> </description> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.child.java.opts</name> >> >> <value>-Xmx1500m</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.jobtracker.restart.recover</name> >> >> <value>true</value> >> >> </property> >> >> >> >> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein >> auf der Festplatte jeder Instanz speichert. --> >> >> <property> >> >> <name>dfs.name.dir</name> >> >> <value>/nutch/filesystem/name</value> >> >> </property> >> >> >> >> <property> >> >> <name>dfs.data.dir</name> >> >> <value>/nutch/filesystem/data</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.system.dir</name> >> >> <value>/nutch/filesystem/mapreduce/system</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.local.dir</name> >> >> <value>/nutch/filesystem/mapreduce/local</value> >> >> </property> >> >> >> >> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden >> sein muessen damit sie erreichbar ist. Am Anfang 1 --> >> >> <property> >> >> <name>dfs.replication</name> >> >> <value>2</value> >> >> </property> >> >> >> >> I hope someone can help me. >> >> >> >> Thanks >> >> >> >> Tom >> >> > > > -- > -MilleBii- > -- -MilleBii-
I don't know that hadoop uses tomcat... But I think it uses Jetty instead. The nodes communicate via http: so you need some kind of web server... And for monitorin its the best way 2009/12/4, Tom Landvoigt <tom.landvoigt@linklift.de>: > Hi, > > I don't have tomcat on this system because I don't want to use the > websearch. But if it is necessary for hadoop what I don’t think I will > install it. > > nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls / > Found 1 items > -rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt > nutch@ip-10-224-113-210:/nutch/search> > > I get the normal answer but the file is empty. > > -----Original Message----- > From: MilleBii [mailto:millebii@gmail.com] > Sent: Freitag, 4. Dezember 2009 15:06 > To: nutch-user@lucene.apache.org > Subject: Re: Problems with a new Installation of Nutch > > Did you check with the web interface ? It gives a lot of info you can > even browse the file system. > > Try hadoop fs -ls to see what it gives you ? > > 2009/12/4, Tom Landvoigt <tom.landvoigt@linklift.de>: >> Hallo, >> >> >> >> I hope someone can help me. >> >> >> >> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I >> can't put data in the hdfs. >> >> >> >> I formatted the namenode and start the hdfs with start all. >> >> >> >> All java processes start properly, but when I want to make hadoop fs >> -put something / I get these logs: >> >> >> >> >> >> >> >> nutch@bla:/nutch/search> ./bin/hadoop fs -put >> /tmp/hadoop-nutch-tasktracker.pid blub >> >> put: Protocol not available >> >> >> >> DATA NODE LOG on the master: >> >> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:16,483 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e >> >> 2009-12-04 12:50:16,614 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:16,882 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4 >> >> 2009-12-04 12:50:16,883 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:17,827 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1 >> >> 2009-12-04 12:50:17,849 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50075 >> >> 2009-12-04 12:50:18,485 INFO util.Container - Started >> org.mortbay.jetty.Server@36527f >> >> 2009-12-04 12:54:20,745 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,746 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,747 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> 2009-12-04 12:54:20,747 ERROR datanode.DataNode - >> DatanodeRegistration(10.224.113.210:50010, >> storageID=DS-1135263253-10.224.113.210-50010-1259926637370, >> infoPort=50075, ipcPort=50020):DataXceiver >> >> java.io.EOFException >> >> at java.io.DataInputStream.readShort(DataInputStream.java:315) >> >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: >> 79) >> >> at java.lang.Thread.run(Thread.java:636) >> >> >> >> NAME NODE LOG >> >> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:12,488 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451 >> >> 2009-12-04 12:50:12,565 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:12,891 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@1570945 >> >> 2009-12-04 12:50:12,891 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:13,569 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5 >> >> 2009-12-04 12:50:13,582 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50070 >> >> 2009-12-04 12:50:13,613 INFO util.Container - Started >> org.mortbay.jetty.Server@173ec72 >> >> >> >> SECONDARY NAMENODE LOG >> >> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:50:20,365 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a >> >> 2009-12-04 12:50:20,454 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:50:21,396 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7 >> >> 2009-12-04 12:50:21,396 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No >> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving >> files and default/dynamic servlets only >> >> 2009-12-04 12:50:22,206 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@383118 >> >> 2009-12-04 12:50:22,785 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50090 >> >> 2009-12-04 12:50:22,787 INFO util.Container - Started >> org.mortbay.jetty.Server@297ffb >> >> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint >> Period :3600 secs (60 min) >> >> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size >> Trigger :67108864 bytes (65536 KB) >> >> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint >> done. New Image Size: 1056 >> >> >> >> HADOOP LOG >> >> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception: >> java.io.IOException: Unable to create new block. >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D >> FSClient.java:2722) >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j >> ava:1996) >> >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli >> ent.java:2183) >> >> >> >> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block >> blk_5506837520665828594_1002 bad datanode[0] nodes == null >> >> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block >> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" - >> Aborting... >> >> >> >> DATA NODE LOG on the slave >> >> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4 >> >> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource >> aliases >> >> 2009-12-04 12:49:50,288 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e >> >> 2009-12-04 12:49:50,357 INFO util.Container - Started >> WebApplicationContext[/static,/static] >> >> 2009-12-04 12:49:50,555 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0 >> >> 2009-12-04 12:49:50,555 INFO util.Container - Started >> WebApplicationContext[/logs,/logs] >> >> 2009-12-04 12:49:50,816 INFO util.Container - Started >> org.mortbay.jetty.servlet.WebApplicationHandler@118278a >> >> 2009-12-04 12:49:50,820 INFO util.Container - Started >> WebApplicationContext[/,/] >> >> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started >> SocketListener on 0.0.0.0:50075 >> >> 2009-12-04 12:49:50,849 INFO util.Container - Started >> org.mortbay.jetty.Server@b02928 >> >> >> >> HADOOP SITE XML >> >> <property> >> >> <name>fs.default.name</name> >> >> <value>hdfs://(yes here is the right ip):9000</value> >> >> <description> >> >> The name of the default file system. Either the literal string >> >> "local" or a host:port for NDFS. >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege) >> zu finden ist. --> >> >> <property> >> >> <name>mapred.job.tracker</name> >> >> <value>hdfs://(here to):9001</value> >> >> <description> >> >> The host and port that the MapReduce job tracker runs at. If >> >> "local", then jobs are run in-process as a single map and >> >> reduce task. >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen--> >> >> <property> >> >> <name>mapred.tasktracker.map.tasks.maximum</name> >> >> <value>2</value> >> >> <description> >> >> define mapred.map tasks to be number of slave hosts >> >> </description> >> >> </property> >> >> >> >> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen--> >> >> <property> >> >> <name>mapred.tasktracker.reduce.tasks.maximum</name> >> >> <value>2</value> >> >> <description> >> >> define mapred.reduce tasks to be number of slave hosts >> >> </description> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.child.java.opts</name> >> >> <value>-Xmx1500m</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.jobtracker.restart.recover</name> >> >> <value>true</value> >> >> </property> >> >> >> >> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein >> auf der Festplatte jeder Instanz speichert. --> >> >> <property> >> >> <name>dfs.name.dir</name> >> >> <value>/nutch/filesystem/name</value> >> >> </property> >> >> >> >> <property> >> >> <name>dfs.data.dir</name> >> >> <value>/nutch/filesystem/data</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.system.dir</name> >> >> <value>/nutch/filesystem/mapreduce/system</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapred.local.dir</name> >> >> <value>/nutch/filesystem/mapreduce/local</value> >> >> </property> >> >> >> >> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden >> sein muessen damit sie erreichbar ist. Am Anfang 1 --> >> >> <property> >> >> <name>dfs.replication</name> >> >> <value>2</value> >> >> </property> >> >> >> >> I hope someone can help me. >> >> >> >> Thanks >> >> >> >> Tom >> >> > > > -- > -MilleBii- > -- -MilleBii-
Running:
bin/nutch readdb crawldb -url <url> I got the following exception.
Also, how do I force a recrawl in Nutch 1.0?
Exception in thread "main" java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartiti
oner.java:32)
at
org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFo
rmat.java:104)
at
org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:380)
at
org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386)
at
org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511)
Vijaya Peters
SRA International, Inc.
4350 Fair Lakes Court North
Room 4004
Fairfax, VA 22033
Tel: 703-502-1184
www.sra.com
Named to FORTUNE's "100 Best Companies to Work For" list for 10
consecutive years
P Please consider the environment before printing this e-mail
This electronic message transmission contains information from SRA
International, Inc. which may be confidential, privileged or
proprietary. The information is intended for the use of the individual
or entity named above. If you are not the intended recipient, be aware
that any disclosure, copying, distribution, or use of the contents of
this information is strictly prohibited. If you have received this
electronic information in error, please notify us immediately by
telephone at 866-584-2143.
-----Original Message-----
From: reinhard schwab [mailto:reinhard.schwab@aon.at]
Sent: Friday, December 04, 2009 8:32 AM
To: nutch-user@lucene.apache.org
Subject: Re: How to force recrawl of everything
Peters, Vijaya schrieb:
> I am using Nutch 1.0. I want to perform a 'clean' crawl.
>
>
>
> I see the force option in this patch: NUTCH-601v1.0.patch
>
<https://issues.apache.org/jira/secure/attachment/12375717/NUTCH-601v1.0
> .patch>
>
> Do I have to make those code changes, or does Nutch 1.0 have another
way
> to do this?
>
>
>
> Also, everytime I do another crawl, I see the same file being fetched
> over and over again. Is it appending the same url over and over to the
>
which file?
you can check the crawl date of this file with
reinhard@thord:>bin/nutch readdb <crawldb> -url <url>
> fetch list?
>
>
>
> Thanks,
>
> - Vijaya
>
>
>
>
>
> Vijaya Peters
> SRA International, Inc.
> 4350 Fair Lakes Court North
> Room 4004
> Fairfax, VA 22033
> Tel: 703-502-1184
>
> www.sra.com <http://www.sra.com/>
> Named to FORTUNE's "100 Best Companies to Work For" list for 10
> consecutive years
>
> P Please consider the environment before printing this e-mail
>
> This electronic message transmission contains information from SRA
> International, Inc. which may be confidential, privileged or
> proprietary. The information is intended for the use of the
individual
> or entity named above. If you are not the intended recipient, be
aware
> that any disclosure, copying, distribution, or use of the contents of
> this information is strictly prohibited. If you have received this
> electronic information in error, please notify us immediately by
> telephone at 866-584-2143.
>
>
>
>
>
Hi, I don't have tomcat on this system because I don't want to use the websearch. But if it is necessary for hadoop what I don’t think I will install it. nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls / Found 1 items -rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt nutch@ip-10-224-113-210:/nutch/search> I get the normal answer but the file is empty. -----Original Message----- From: MilleBii [mailto:millebii@gmail.com] Sent: Freitag, 4. Dezember 2009 15:06 To: nutch-user@lucene.apache.org Subject: Re: Problems with a new Installation of Nutch Did you check with the web interface ? It gives a lot of info you can even browse the file system. Try hadoop fs -ls to see what it gives you ? 2009/12/4, Tom Landvoigt <tom.landvoigt@linklift.de>: > Hallo, > > > > I hope someone can help me. > > > > I installed nutch on 2 Amazon EC2 computers. Everything is fine but I > can't put data in the hdfs. > > > > I formatted the namenode and start the hdfs with start all. > > > > All java processes start properly, but when I want to make hadoop fs > -put something / I get these logs: > > > > > > > > nutch@bla:/nutch/search> ./bin/hadoop fs -put > /tmp/hadoop-nutch-tasktracker.pid blub > > put: Protocol not available > > > > DATA NODE LOG on the master: > > 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4 > > 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource > aliases > > 2009-12-04 12:50:16,483 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e > > 2009-12-04 12:50:16,614 INFO util.Container - Started > WebApplicationContext[/static,/static] > > 2009-12-04 12:50:16,882 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4 > > 2009-12-04 12:50:16,883 INFO util.Container - Started > WebApplicationContext[/logs,/logs] > > 2009-12-04 12:50:17,827 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1 > > 2009-12-04 12:50:17,849 INFO util.Container - Started > WebApplicationContext[/,/] > > 2009-12-04 12:50:18,485 INFO http.SocketListener - Started > SocketListener on 0.0.0.0:50075 > > 2009-12-04 12:50:18,485 INFO util.Container - Started > org.mortbay.jetty.Server@36527f > > 2009-12-04 12:54:20,745 ERROR datanode.DataNode - > DatanodeRegistration(10.224.113.210:50010, > storageID=DS-1135263253-10.224.113.210-50010-1259926637370, > infoPort=50075, ipcPort=50020):DataXceiver > > java.io.EOFException > > at java.io.DataInputStream.readShort(DataInputStream.java:315) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: > 79) > > at java.lang.Thread.run(Thread.java:636) > > 2009-12-04 12:54:20,746 ERROR datanode.DataNode - > DatanodeRegistration(10.224.113.210:50010, > storageID=DS-1135263253-10.224.113.210-50010-1259926637370, > infoPort=50075, ipcPort=50020):DataXceiver > > java.io.EOFException > > at java.io.DataInputStream.readShort(DataInputStream.java:315) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: > 79) > > at java.lang.Thread.run(Thread.java:636) > > 2009-12-04 12:54:20,747 ERROR datanode.DataNode - > DatanodeRegistration(10.224.113.210:50010, > storageID=DS-1135263253-10.224.113.210-50010-1259926637370, > infoPort=50075, ipcPort=50020):DataXceiver > > java.io.EOFException > > at java.io.DataInputStream.readShort(DataInputStream.java:315) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: > 79) > > at java.lang.Thread.run(Thread.java:636) > > 2009-12-04 12:54:20,747 ERROR datanode.DataNode - > DatanodeRegistration(10.224.113.210:50010, > storageID=DS-1135263253-10.224.113.210-50010-1259926637370, > infoPort=50075, ipcPort=50020):DataXceiver > > java.io.EOFException > > at java.io.DataInputStream.readShort(DataInputStream.java:315) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java: > 79) > > at java.lang.Thread.run(Thread.java:636) > > > > NAME NODE LOG > > 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4 > > 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource > aliases > > 2009-12-04 12:50:12,488 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@19fe451 > > 2009-12-04 12:50:12,565 INFO util.Container - Started > WebApplicationContext[/static,/static] > > 2009-12-04 12:50:12,891 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@1570945 > > 2009-12-04 12:50:12,891 INFO util.Container - Started > WebApplicationContext[/logs,/logs] > > 2009-12-04 12:50:13,569 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@11410e5 > > 2009-12-04 12:50:13,582 INFO util.Container - Started > WebApplicationContext[/,/] > > 2009-12-04 12:50:13,613 INFO http.SocketListener - Started > SocketListener on 0.0.0.0:50070 > > 2009-12-04 12:50:13,613 INFO util.Container - Started > org.mortbay.jetty.Server@173ec72 > > > > SECONDARY NAMENODE LOG > > 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4 > > 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource > aliases > > 2009-12-04 12:50:20,365 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@174d93a > > 2009-12-04 12:50:20,454 INFO util.Container - Started > WebApplicationContext[/static,/static] > > 2009-12-04 12:50:21,396 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7 > > 2009-12-04 12:50:21,396 INFO util.Container - Started > WebApplicationContext[/logs,/logs] > > 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No > WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving > files and default/dynamic servlets only > > 2009-12-04 12:50:22,206 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@383118 > > 2009-12-04 12:50:22,785 INFO util.Container - Started > WebApplicationContext[/,/] > > 2009-12-04 12:50:22,787 INFO http.SocketListener - Started > SocketListener on 0.0.0.0:50090 > > 2009-12-04 12:50:22,787 INFO util.Container - Started > org.mortbay.jetty.Server@297ffb > > 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint > Period :3600 secs (60 min) > > 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size > Trigger :67108864 bytes (65536 KB) > > 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint > done. New Image Size: 1056 > > > > HADOOP LOG > > 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception: > java.io.IOException: Unable to create new block. > > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D > FSClient.java:2722) > > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j > ava:1996) > > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli > ent.java:2183) > > > > 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block > blk_5506837520665828594_1002 bad datanode[0] nodes == null > > 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block > locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" - > Aborting... > > > > DATA NODE LOG on the slave > > 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4 > > 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource > aliases > > 2009-12-04 12:49:50,288 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e > > 2009-12-04 12:49:50,357 INFO util.Container - Started > WebApplicationContext[/static,/static] > > 2009-12-04 12:49:50,555 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@2016b0 > > 2009-12-04 12:49:50,555 INFO util.Container - Started > WebApplicationContext[/logs,/logs] > > 2009-12-04 12:49:50,816 INFO util.Container - Started > org.mortbay.jetty.servlet.WebApplicationHandler@118278a > > 2009-12-04 12:49:50,820 INFO util.Container - Started > WebApplicationContext[/,/] > > 2009-12-04 12:49:50,849 INFO http.SocketListener - Started > SocketListener on 0.0.0.0:50075 > > 2009-12-04 12:49:50,849 INFO util.Container - Started > org.mortbay.jetty.Server@b02928 > > > > HADOOP SITE XML > > <property> > > <name>fs.default.name</name> > > <value>hdfs://(yes here is the right ip):9000</value> > > <description> > > The name of the default file system. Either the literal string > > "local" or a host:port for NDFS. > > </description> > > </property> > > > > <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege) > zu finden ist. --> > > <property> > > <name>mapred.job.tracker</name> > > <value>hdfs://(here to):9001</value> > > <description> > > The host and port that the MapReduce job tracker runs at. If > > "local", then jobs are run in-process as a single map and > > reduce task. > > </description> > > </property> > > > > <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen--> > > <property> > > <name>mapred.tasktracker.map.tasks.maximum</name> > > <value>2</value> > > <description> > > define mapred.map tasks to be number of slave hosts > > </description> > > </property> > > > > <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen--> > > <property> > > <name>mapred.tasktracker.reduce.tasks.maximum</name> > > <value>2</value> > > <description> > > define mapred.reduce tasks to be number of slave hosts > > </description> > > </property> > > > > <property> > > <name>mapred.child.java.opts</name> > > <value>-Xmx1500m</value> > > </property> > > > > <property> > > <name>mapred.jobtracker.restart.recover</name> > > <value>true</value> > > </property> > > > > <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein > auf der Festplatte jeder Instanz speichert. --> > > <property> > > <name>dfs.name.dir</name> > > <value>/nutch/filesystem/name</value> > > </property> > > > > <property> > > <name>dfs.data.dir</name> > > <value>/nutch/filesystem/data</value> > > </property> > > > > <property> > > <name>mapred.system.dir</name> > > <value>/nutch/filesystem/mapreduce/system</value> > > </property> > > > > <property> > > <name>mapred.local.dir</name> > > <value>/nutch/filesystem/mapreduce/local</value> > > </property> > > > > <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden > sein muessen damit sie erreichbar ist. Am Anfang 1 --> > > <property> > > <name>dfs.replication</name> > > <value>2</value> > > </property> > > > > I hope someone can help me. > > > > Thanks > > > > Tom > > -- -MilleBii-
Well three is a charm.. I need to move these to a different email as well.. Please if you could.. Could we also remove this email address as well.. Thanks ray -----Original Message----- From: M S Ram [mailto:msram@cse.iitk.ac.in] Sent: Friday, December 04, 2009 10:01 AM To: nutch-user@lucene.apache.org Subject: Re: unsubscribe from nutch-user Same here. Please remove my ID also from the mailing list. Thanks, MSR rengan xu wrote: > To whom it may concern, > > Hello! Because I will use this E-mail for special purpose. I will use > another E-mail to subscribe in nutch-user. So I want to unsubscribe from > nutch-user. > > Thank you! > > >
Take a look at it: http://lucene.apache.org/nutch/mailing_lists.html or probably sending a blank mail to: nutch-user-unsubscribe@lucene.apache.orgshould also work. Thanks, Prashant. On Fri, Dec 4, 2009 at 8:30 PM, M S Ram <msram@cse.iitk.ac.in> wrote: > Same here. Please remove my ID also from the mailing list. > > Thanks, > MSR > > rengan xu wrote: > >> To whom it may concern, >> >> Hello! Because I will use this E-mail for special purpose. I will use >> another E-mail to subscribe in nutch-user. So I want to unsubscribe from >> nutch-user. >> >> Thank you! >> >> >> >> > > > -- Thanks, Prashant Ullegaddi, Search and Information Extraction Lab, IIIT-Hyderabad, INDIA.