hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jibjoice <sudarat_...@hotmail.com>
Subject Re: Error when i index by nutchwax-0.10.0
Date Mon, 26 Nov 2007 02:08:37 GMT

now i use hadoop-0.9.2 and lucene-core-2.1.0.jar but it's not work.


pvvpr wrote:
> 
> Dedup tries to acquire lock to the file system from IndexReader.java of 
> lucene. Latest versions of hadoop dont support this. I think this error is 
> because of that. What you can try is comment out acquire lock
> functionality 
> in lucene IndexReader, compile lucene and replace the old lucene jar with
> new 
> one. It might work (worked for me), but this may be a problem when you run 
> concurrent jobs.
> 
> - Prasad.
> 
> On Friday 23 November 2007 03:10, jibjoice wrote:
>> when i use this command "${HADOOP_HOME}/bin/hadoop jar
>> ${NUTCHWAX_HOME}/nutchwax.jar all /tmp/inputs /tmp/outputs test"
>> i have error :
>>
>> - LinkDb: done
>> -  indexing [Lorg.apache.hadoop.fs.Path;@66e64686
>> - Indexer: starting
>> - Indexer: linkdb: outputs/linkdb
>> - parsing file:/nutch/search/conf/hadoop-default.xml
>> - parsing file:/nutch/search/conf/nutch-default.xml
>> - parsing file:/tmp/hadoop-unjar50228/wax-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - adding segment: /user/nutch/outputs/segments/25501123143257-test
>> - Running job: job_0004
>> -  map 0% reduce 0%
>> -  map 5% reduce 0%
>> -  map 15% reduce 0%
>> -  map 27% reduce 0%
>> -  map 37% reduce 0%
>> -  map 47% reduce 0%
>> -  map 57% reduce 0%
>> -  map 72% reduce 0%
>> -  map 82% reduce 0%
>> -  map 94% reduce 0%
>> -  map 97% reduce 0%
>> -  map 100% reduce 0%
>> -  map 100% reduce 37%
>> -  map 100% reduce 79%
>> -  map 100% reduce 87%
>> -  map 100% reduce 100%
>> - Job complete: job_0004
>> - Indexer: done
>> - dedup outputs/index
>> - Dedup: starting
>> - parsing file:/nutch/search/conf/hadoop-default.xml
>> - parsing file:/nutch/search/conf/nutch-default.xml
>> - parsing file:/tmp/hadoop-unjar50228/wax-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - parsing file:/nutch/search/conf/mapred-default.xml
>> - Dedup: adding indexes in: outputs/indexes
>> - Running job: job_0005
>> -  map 0% reduce 0%
>> -  map 100% reduce 100%
>> Exception in thread "main" java.io.IOException: Job failed!
>>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:399)
>>         at
>> org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:433)
>>         at org.archive.access.nutch.Nutchwax.doDedup(Nutchwax.java:257)
>>         at org.archive.access.nutch.Nutchwax.doAll(Nutchwax.java:156)
>>         at org.archive.access.nutch.Nutchwax.doJob(Nutchwax.java:389)
>>         at org.archive.access.nutch.Nutchwax.main(Nutchwax.java:674)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
>>9) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>>l.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>>
>> why and how i solve it?
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Error-when-i-index-by-nutchwax-0.10.0-tf4860227.html#a13942649
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message