Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 54597 invoked from network); 13 Aug 2009 10:07:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 Aug 2009 10:07:39 -0000 Received: (qmail 49323 invoked by uid 500); 13 Aug 2009 10:07:44 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 49252 invoked by uid 500); 13 Aug 2009 10:07:43 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 49242 invoked by uid 99); 13 Aug 2009 10:07:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 10:07:43 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 10:07:34 +0000 Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1MbXDF-0003cM-LC for java-user@lucene.apache.org; Thu, 13 Aug 2009 03:07:13 -0700 Message-ID: <24952295.post@talk.nabble.com> Date: Thu, 13 Aug 2009 03:07:13 -0700 (PDT) From: rishisinghal To: java-user@lucene.apache.org Subject: Re: Indexer crashes with "hit exception during merge" In-Reply-To: <9ac0c6aa0908130249x15226100w5183cd591f43181d@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: singhal.rishi@gmail.com References: <24951667.post@talk.nabble.com> <9ac0c6aa0908130249x15226100w5183cd591f43181d@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org >>Is there any chance that two writers are open on this directory? =20 No, thats not true. >>something external to Lucene is removing files from the directory. No this also has rare chances as I am the owner of these files and other then me no one can delete the, :-) Here are all the files in the index directory: segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1 _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1 _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1 _b2.tis;1 _b3.fdt;1 _b4.fdt;1 I see _37 present. Regards, Rishi Michael McCandless-2 wrote: >=20 > Is there any chance that two writers are open on this directory? Or, > something external to Lucene is removing files from the directory. >=20 > It looks like there were at least two missing files (_37 >=20 > On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal > wrote: >> >> Hi, >> >> I am trying to index documents and when all is complete and optimize is >> called I get >> >> IFD [main]: setInfoStream >> deletionPolicy=3Dorg.apache.lucene.index.KeepOnlyLastCommitDeletionPolic= y@4fced0 >> IW 0 [main]: setInfoStream: >> dir=3Dorg.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_160= 0 >> autoCommit=3Dfalse mergePolicy=3Dorg >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49 >> mergeScheduler=3Dorg.apache.lucene.index.ConcurrentMergeScheduler@6038e4 >> ramBufferS >> izeMB=3D16.0 maxBufferedDocs=3D5 maxBuffereDeleteTerms=3D-1 >> maxFieldLength=3D10000 >> index=3D_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 >> =C2=A0_ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: =C2=A0 flush: segment=3Dnull docStoreSegment=3Dnull docStor= eOffset=3D0 >> flushDocs=3Dfalse flushDeletes=3Dtrue flushDocStores=3Dfalse numD >> ocs=3D0 numBufDelTerms=3D0 >> IW 0 [main]: =C2=A0 index before flush _37:c465->_0 _4c:c165->_0 _61:c24= 5->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0 >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending] >> IW 0 [main]: CMS: now merge >> IW 0 [main]: CMS: =C2=A0 index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: CMS: =C2=A0 consider merge _61:c245->_0 _9e:c485->_0 _ay:c2= 25->_0 >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >> IW 0 [main]: CMS: =C2=A0 =C2=A0 launch new thread [Lucene Merge Thread #= 73] >> IW 0 [main]: CMS: =C2=A0 no more merges pending; now return >> IW 0 [Lucene Merge Thread #73]: CMS: =C2=A0 merge thread: start >> IW 0 [Lucene Merge Thread #73]: now merge >> =C2=A0merge=3D_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->= _0 into >> _b5 [optimize] >> =C2=A0merge=3Dorg.apache.lucene.index.MergePolicy$OneMerge@3725db >> =C2=A0index=3D_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c2= 25->_0 >> _az:c5->_0 _b0:c5->_0 >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0 >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >> IW 0 [Lucene Merge Thread #73]: hit exception during merge >> Exception in thread "Lucene Merge Thread #73" >> org.apache.lucene.index.MergePolicy$MergeException: >> java.io.FileNotFoundException: /SY >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co= ncurrentMergeScheduler.java:309) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr= entMergeScheduler.java:286) >> Caused by: java.io.FileNotFoundException: >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.open(Native Metho= d) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.(RandomAcce= ssFile.java:231) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDir= ectory.java:552) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java= :582) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.jav= a:70) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge= Scheduler.java:205) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr= entMergeScheduler.java:260) >> Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP >> jspMain >> SEVERE: Indexing not completebackground merge hit exception: _61:c245->_= 0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [ >> optimize] >> java.io.IOException: background merge hit exception: _61:c245->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize >> ] >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.hp.openvms.lucene.index.FileIndexerJSP= .jspMain(Unknown >> Source) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.hp.openvms.lucene.index.FileIndexer.ma= in(Unknown Source) >> Caused by: java.io.FileNotFoundException: >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.open(Native Metho= d) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.(RandomAcce= ssFile.java:231) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDir= ectory.java:552) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java= :582) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.jav= a:70) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge= Scheduler.java:205) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr= entMergeScheduler.java:260) >> IW 0 [main]: now flush at close >> IW 0 [main]: =C2=A0 flush: segment=3Dnull docStoreSegment=3Dnull docStor= eOffset=3D0 >> flushDocs=3Dfalse flushDeletes=3Dtrue flushDocStores=3Dfalse numD >> ocs=3D0 numBufDelTerms=3D0 >> IW 0 [main]: =C2=A0 index before flush _37:c465->_0 _4c:c165->_0 _61:c24= 5->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: CMS: now merge >> IW 0 [main]: CMS: =C2=A0 index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >> IW 0 [main]: CMS: =C2=A0 no more merges pending; now return >> IW 0 [main]: now call final commit() >> IW 0 [main]: startCommit(): start sizeInBytes=3D0 >> IW 0 [main]: startCommit index=3D_37:c465->_0 _4c:c165->_0 _61:c245->_0 >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=3D11 >> 95 >> IW 0 [main]: now sync _37.cfs >> IW 0 [main]: hit exception while closing >> Exception in thread "main" java.io.FileNotFoundException: >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5)) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.open(Native Metho= d) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.RandomAccessFile.(RandomAcce= ssFile.java:231) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.lucene.store.FSDirectory.sync(F= SDirectory.java:456) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.hp.openvms.lucene.index.FileIndexerJSP= .jspMain(Unknown >> Source) >> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.hp.openvms.lucene.index.FileIndexer.ma= in(Unknown Source) >> >> >> >> I have setMaxBufferedDocs =3D 5 and setMergeFactor=3D5. I was trying >> different >> combinations of these to see if that helps but was unsuccessful. >> >> >> Regards, >> Rishi >> -- >> View this message in context: >> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge= %22-tp24951667p24951667.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org >=20 >=20 >=20 --=20 View this message in context: http://www.nabble.com/Indexer-crashes-with-%2= 2hit-exception-during-merge%22-tp24951667p24952295.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org