lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rishisinghal <singhal.ri...@gmail.com>
Subject Re: Indexer crashes with "hit exception during merge"
Date Thu, 13 Aug 2009 12:21:44 GMT

I tried creating the index in different disks but still i see the issue :-(

I tried to index documents in other disks also and got the same exception. 

I also tried
$ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
-segment _61
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene...', so assertions are enabled
Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]

Checking only these segments: _61:
No problems were detected with this index.

Regards,
Rishi


Shai Erera wrote:
> 
> I noticed the exception is "Caused by: java.io.FileNotFoundException:
> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
> 
> I searched for i/o error (errno:5) and found some information which
> associates it w/ a more native IO problem, like corrupt file due to system
> crash etc.
> 
> Did you experience that? Can you try to create the index somewhere else,
> or
> on another drive?
> 
> Shai
> 
> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
> <singhal.rishi@gmail.com>wrote:
> 
>>
>> It is a local file system.
>>
>> We are using lucene 2.4 and java 1.5
>>
>> Regards,
>> Rishi
>>
>>
>> Shai Erera wrote:
>> >
>> > Is that a local file system, or a network share?
>> >
>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>> > <singhal.rishi@gmail.com>wrote:
>> >
>> >>
>> >> >>Is there any chance that two writers are open on this directory?
>> >> No, thats not true.
>> >>
>> >> >>something external to Lucene is removing files from the directory.
>> >> No this also has rare chances as I am the owner of these files and
>> other
>> >> then me no one can delete the, :-)
>> >>
>> >> Here are all the files in the index directory:
>> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>> >>
>> >> I see _37 present.
>> >>
>> >> Regards,
>> >> Rishi
>> >>
>> >>
>> >> Michael McCandless-2 wrote:
>> >> >
>> >> > Is there any chance that two writers are open on this directory? 
>> Or,
>> >> > something external to Lucene is removing files from the directory.
>> >> >
>> >> > It looks like there were at least two missing files (_37
>> >> >
>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
>> rishisinghal<singhal.rishi@gmail.com
>> >
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am trying to index documents and when all is complete and
>> optimize
>> >> is
>> >> >> called I get
>> >> >>
>> >> >> IFD [main]: setInfoStream
>> >> >>
>> >>
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>> >> >> IW 0 [main]: setInfoStream:
>> >> >>
>> >>
>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>> >> >> autoCommit=false mergePolicy=org
>> >> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>> >> >>
>> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
>> >> >> ramBufferS
>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>> >> >> maxFieldLength=10000
>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>> >> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>> >> >> ocs=0 numBufDelTerms=0
>> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
pending]
>> >> >> IW 0 [main]: CMS: now merge
>> >> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
>> >> _ay:c225->_0
>> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> >> >> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
>> >> >> IW 0 [main]: CMS:   no more merges pending; now return
>> >> >> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
>> >> >> IW 0 [Lucene Merge Thread #73]: now merge
>> >> >>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
_b0:c5->_0
>> >> into
>> >> >> _b5 [optimize]
>> >> >>  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
>> >> >>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>> >> _ay:c225->_0
>> >> >> _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> >> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge
>> >> >> Exception in thread "Lucene Merge Thread #73"
>> >> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> >> java.io.FileNotFoundException: /SY
>> >> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
>> >> >> Caused by: java.io.FileNotFoundException:
>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>> >> >>        at
>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
>> >> >>        at
>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>> >> >>        at
>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>> >> >>        at
>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
>> >> >>        at
>> >> >>
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
>> >> >> Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
>> >> >> jspMain
>> >> >> SEVERE: Indexing not completebackground merge hit exception:
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
_b5 [
>> >> >> optimize]
>> >> >> java.io.IOException: background merge hit exception: _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
_b5 [optimize
>> >> >> ]
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
>> >> >>        at
>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
>> >> >> Source)
>> >> >>        at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
>> Source)
>> >> >> Caused by: java.io.FileNotFoundException:
>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>> >> >>        at
>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
>> >> >>        at
>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>> >> >>        at
>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>> >> >>        at
>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
>> >> >>        at
>> >> >>
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
>> >> >> IW 0 [main]: now flush at close
>> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>> >> >> ocs=0 numBufDelTerms=0
>> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: CMS: now merge
>> >> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: CMS:   no more merges pending; now return
>> >> >> IW 0 [main]: now call final commit()
>> >> >> IW 0 [main]: startCommit(): start sizeInBytes=0
>> >> >> IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
>> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
>> >> >> 95
>> >> >> IW 0 [main]: now sync _37.cfs
>> >> >> IW 0 [main]: hit exception while closing
>> >> >> Exception in thread "main" java.io.FileNotFoundException:
>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>> >> >>        at
>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>> >> >>        at
>> >> org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
>> >> >>        at
>> >> >>
>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
>> >> >>        at
>> >> >>
>> >>
>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
>> >> >>        at
>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
>> >> >>        at
>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
>> >> >> Source)
>> >> >>        at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
>> Source)
>> >> >>
>> >> >>
>> >> >>
>> >> >> I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
>> >> >> different
>> >> >> combinations of these to see if that helps but was unsuccessful.
>> >> >>
>> >> >>
>> >> >> Regards,
>> >> >> Rishi
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
>> >> >> Sent from the Lucene - Java Users mailing list archive at
>> Nabble.com.
>> >> >>
>> >> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> >>
>> >> >>
>> >> >
>> >> >
>> ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> >
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
>> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message