nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinci <vinci.w...@polyu.edu.hk>
Subject RE: Recrawling without deleting crawl directory
Date Sun, 23 Mar 2008 12:01:45 GMT

Hi,

Seems you need to mention what is "modified document". Which case would it
be?
Case 1: you dump the crawled page from nutch segment and do what you like on
them
If this is the case, you need to think of which action you want:
I. modified the document and then ask nutch to crawl the modified directory?
II. modified the document, write back to segment (the crawl DB), then do the
indexing?
Case 2: Keep track the document update
For this case, when you keep on doing re-crawl based on the same crawl DB
(what you need to tune the the day of re-crawl), then nutch will do the
update for you.
Hope it help :)



Jean-Christophe Alleman wrote:
> 
> 
> 
> Hi,
> 
> I have nothing said. This works fine ! It's morning and I'm still not woke
> up :-D
> 
> I just want to know if it was possible to re index modified documents ? Or
> re index documents which are already in database ?
> 
> Thank's in advance !
> 
> Jisay
> 
> 
>> 
>> Hi Susam Pal and thank's for your help !
>> 
>> The solution you give to me doesn't work... I have still an error with
>> Hadoop... And if I download an older version of the API, will this patch
>> work ? I have Nutch-0.9 and I don't know if I compile with an oder Hadoop
>> API, this patch will work. But if it will work where can I find an older
>> version of Hadoop API ?
>> 
>> Thank's in advance for your help,
>> 
>> Jisay
>> 
>> 
>>>
>>> I am not sure but it seems that this is because of an older version of
>>> Hadoop. I don't have older versions of Nutch or Hadoop with me to
>>> confirm this. Just try omitting the second argument in:
>>> fs.listPaths(indexes, HadoopFSUtil.getPassAllFilter()) and see if it
>>> compiles?
>>>
>>> I guess, fs.listPaths(indexes) should work since I can find such a
>>> method (though it is deprecated now) in the latest Hadoop API.
>>>
>>> Regards,
>>> Susam pal
>>>
>>> On Tue, Mar 18, 2008 at 9:09 PM, Jean-Christophe Alleman
>>>  wrote:
>>>>
>>>>
>>>>
>>>> Thank's for your reply Susam Pal !
>>>>
>>>> I have run ant and I have an error I can't resolve... Look at this :
>>>>
>>>> debian:~/nutch-0.9# ant
>>>> Buildfile: build.xml
>>>>
>>>> init:
>>>> [unjar] Expanding: /root/nutch-0.9/lib/hadoop-0.12.2-core.jar into
>>>> /root/nutch-0.9/build/hadoop
>>>> [untar] Expanding: /root/nutch-0.9/build/hadoop/bin.tgz into
>>>> /root/nutch-0.9/bin
>>>> [unjar] Expanding: /root/nutch-0.9/lib/hadoop-0.12.2-core.jar into
>>>> /root/nutch-0.9/build
>>>>
>>>> compile-core:
>>>> [javac] Compiling 133 source files to /root/nutch-0.9/build/classes
>>>> [javac] /root/nutch-0.9/src/java/org/apache/nutch/crawl/Crawl.java:150:
>>>> cannot find symbol
>>>> [javac] symbol : variable HadoopFSUtil
>>>> [javac] location: class org.apache.nutch.crawl.Crawl
>>>> [javac] merger.merge(fs.listPaths(indexes,
>>>> HadoopFSUtil.getPassAllFilter()),
>>>> [javac] ^
>>>> [javac] Note: Some input files use or override a deprecated API.
>>>> [javac] Note: Recompile with -Xlint:deprecation for details.
>>>> [javac] Note: Some input files use unchecked or unsafe operations.
>>>> [javac] Note: Recompile with -Xlint:unchecked for details.
>>>> [javac] 1 error
>>>>
>>>> BUILD FAILED
>>>> /root/nutch-0.9/build.xml:106: Compile failed; see the compiler error
>>>> output for details.
>>>>
>>>> Total time: 8 seconds
>>>>
>>>> I have already corrected 3errors but I can't correct this one... I
>>>> don't know what's HadoopFSUtil and so I can't correct the error... Help
>>>> me please,
>>>>
>>>> Thank's for your help !
>>>>
>>>> Jisay
>>>>
>>>>
>>>>
>>>>>
>>>>> The patch was generated for Nutch 1.0 development version which is
>>>>> currently in trunk. So, it is unable to patch your older version
>>>>> cleanly.
>>>>>
>>>>> I also see that you are using NUTCH-601v0.3.patch. However,
>>>>> NUTCH-601v1.0.patch is the recommended patch. If this patch fails, you
>>>>> can make the modifications manually. This patch is extremely simple
>>>>> and if you just open the patch using a text editor, you would find
>>>>> that 3 lines have been removed from the original source code
>>>>> (indicated by leading minus signs) and 11 new lines have been added
>>>>> (indicated by plus signs). You have to make these changes manually to
>>>>> your Nutch 0.9 source code directory.
>>>>>
>>>>> Once you make the changes, just build your project again with ant and
>>>>> you would be ready for recrawl.
>>>>>
>>>>> Regards,
>>>>> Susam Pal
>>>>>
>>>>> On Tue, Mar 18, 2008 at 7:12 PM, Jean-Christophe Alleman
>>>>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Hi, I'm interested by this patch but I can't patch it. I have some
>>>>>> problems when I try to patch...
>>>>>>
>>>>>> Here is what I do :
>>>>>>
>>>>>> debian:~/patch# patch -p0> can't find file to patch at input line
5
>>>>
>>>>>> Perhaps you used the wrong -p or --strip option?
>>>>>> The text leading up to this was:
>>>>>> --------------------------
>>>>>> |Index: src/java/org/apache/nutch/crawl/Crawl.java
>>>>>> |===================================================================
>>>>>> |--- src/java/org/apache/nutch/crawl/Crawl.java (revision 628119)
>>>>>> |+++ src/java/org/apache/nutch/crawl/Crawl.java (working copy)
>>>>>> --------------------------
>>>>>> File to patch:
>>>>>> /root/nutch-0.9/src/java/org/apache/nutch/crawl/Crawl.java
>>>>>> patching file
>>>>>> /root/nutch-0.9/src/java/org/apache/nutch/crawl/Crawl.java
>>>>>> Reversed (or previously applied) patch detected! Assume -R? [n] y
>>>>>> Hunk #2 FAILED at 100.
>>>>>> Hunk #3 FAILED at 131.
>>>>>> 2 out of 3 hunks FAILED -- saving rejects to file
>>>>>> /root/nutch-0.9/src/java/org/apache/nutch/crawl/Crawl.java.rej
>>>>>>
>>>>>> Can you please help me ! It's first time I patch. Please help me
!
>>>>>>
>>>>>> Thank's in advance,
>>>>>>
>>>>>> Jisay
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> The recrawl patch in https://issues.apache.org/jira/browse/NUTCH-601
>>>>>>> got committed today. So if you check out the latest trunk, you
can
>>>>>>> recrawl without deleting the crawl directory.
>>>>>>>
>>>>>>> However, if you are using an older version, you may use the script
>>>>>>> at:
>>>>>>> http://wiki.apache.org/nutch/Crawl
>>>>>>>
>>>>>>> Regards,
>>>>>>> Susam Pal
>>>>>>>
>>>>>>> On Fri, Mar 14, 2008 at 3:48 AM, Bradford Stephens
>>>>
>>>>>>> wrote:
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> A coworker and I are experimenting with Nutch in anticipation
of a
>>>>>>>> pretty large rollout at our company. However, we seem to
be stuck
>>>>>>>> on
>>>>>>>> something -- after the crawler is finished, we can't manually
>>>>>>>> re-crawl
>>>>>>>> into the same directory/index! It says "Directory already
exists"
>>>>>>>> when
>>>>>>>> we try to initiate a new crawl. Any ideas?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Bradford
>>>>>>>>
>>>>>>
>>>>>> _________________________________________________________________
>>>>>> Changez votre Live en un clic !
>>>>>> http://get.live.com
>>>>
>>>> _________________________________________________________________
>>>> Windows Live: une foule de solutions orginales pour partager vos
>>>> souvenirs !
>>>> http://get.live.com
>> 
>> _________________________________________________________________
>> Emmenez vos amis avec vous, grâce à Messenger sur votre GSM.
>> http://get.live.com
> 
> _________________________________________________________________
> Vous partez ? Emmenez vos amis avec vous !
> http://www.windowslivemobile.msn.com/nl-be
> 

-- 
View this message in context: http://www.nabble.com/Recrawling-without-deleting-crawl-directory-tp16039970p16235138.html
Sent from the Nutch - User mailing list archive at Nabble.com.


Mime
View raw message