hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Leibert <...@leibert.de>
Subject contrib/Indexing
Date Thu, 16 Jul 2009 19:19:05 GMT
Hi,
I'm running into trouble when trying to create a sharded lucene index with
the contrib package.

Everything works when using small input files / few input paths, but I am
running into trouble when
trying to create a larger index. I get the exception that's attached to this
mail. Running 200 map tasks / 75 (75 shards) reduce tasks.

thanks,
florian

09/07/16 12:16:15 INFO mapred.JobClient: Task Id :
attempt_200907160011_2592_m_000339_0, Status : FAILED
java.io.FileNotFoundException: _0.frq
    at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
    at org.apache.lucene.store.Directory.openInput(Directory.java:105)
    at
org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:372)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4215)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
    at
org.apache.lucene.index.IndexWriter.resolveExternalSegments(IndexWriter.java:3109)
    at
org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:3011)
    at
org.apache.hadoop.contrib.index.mapred.IntermediateForm.process(IntermediateForm.java:135)
    at
org.apache.hadoop.contrib.index.mapred.IndexUpdateCombiner.reduce(IndexUpdateCombiner.java:56)
    at
org.apache.hadoop.contrib.index.mapred.IndexUpdateCombiner.reduce(IndexUpdateCombiner.java:38)
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:1106)
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:979)
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:832)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message