Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 9147 invoked from network); 28 Aug 2009 05:26:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 28 Aug 2009 05:26:57 -0000 Received: (qmail 74766 invoked by uid 500); 28 Aug 2009 05:26:55 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 74675 invoked by uid 500); 28 Aug 2009 05:26:54 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 74647 invoked by uid 99); 28 Aug 2009 05:26:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 05:26:54 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=MSGID_MULTIPLE_AT,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [64.13.253.17] (HELO guavus.com) (64.13.253.17) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 05:26:42 +0000 Received: (qmail 7922 invoked from network); 27 Aug 2009 22:26:20 -0700 Received: from isp-net67.spectranet.com (HELO station151) (@125.63.71.67) by guavus.com with SMTP; 27 Aug 2009 22:26:19 -0700 From: "Sumanta Bhowmik" To: Cc: References: <-61520930041827677@unknownmsgid> <9ac0c6aa0908200237kcb7d0f3q73d37b5fdec27f2d@mail.gmail.com> <-3936163573164025315@unknownmsgid> <9ac0c6aa0908210413w92a8280rd8a1d9deb0548bb8@mail.gmail.com> <4608543773190021384@unknownmsgid> <9ac0c6aa0908270200l6064dc0fgdce08d0501441db0@mail.gmail.com> In-Reply-To: <9ac0c6aa0908270200l6064dc0fgdce08d0501441db0@mail.gmail.com> Subject: RE: Merge Exception in Lucene 2.4 Date: Fri, 28 Aug 2009 10:56:11 +0530 Message-ID: <01f401ca27a0$130a4050$391ec0f0$@bhowmik@guavus.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acom9OpnOBKBpGsZTrqEOTG+LE6maQApB9YA Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org Hi I am not sure if the -Xint option works as the performance goes down = like anything. Since I am indexing streaming data -Xint is not an option. Is = it possible that we do the merge synchronously so that I can catch the exception, clean the index and start afresh. Even if it is asynchronous = if there is a way to know the status of the merge operation? Sumanta -----Original Message----- From: Michael McCandless [mailto:lucene@mikemccandless.com]=20 Sent: Thursday, August 27, 2009 2:30 PM To: Sumanta Bhowmik Cc: java-user@lucene.apache.org Subject: Re: Merge Exception in Lucene 2.4 Was this a smallish contained test case? Can you post the sources? Can you run your next test with IndexWriter's infoStream turned on, and post the resluting output? Was this with jdk1.6u16? That version fixes the original hotspot bug from LUCENE-1282... (and your exception looks different from the one in LUCENE-1282). Mike On Thu, Aug 27, 2009 at 12:46 AM, Sumanta Bhowmik wrote: > Hi > > I ran a long running test and now got this exception. > > Exception in thread "Lucene Merge Thread #39" > org.apache.lucene.index.MergePolicy$MergeException: = java.io.IOException: > read past EOF > =A0 =A0 =A0 =A0at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Con= cur > rentMergeScheduler.java:309) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntM > ergeScheduler.java:286) > Caused by: java.io.IOException: read past EOF > =A0 =A0 =A0 =A0at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java= :15 > 1) > =A0 =A0 =A0 =A0at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.ja= va: > 38) > =A0 =A0 =A0 =A0at = org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTer= mPo > sitions.java:73) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPosi= tio > ns.java:69) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:6= 67) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:58= 6) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:5= 47) > =A0 =A0 =A0 =A0at > = org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:500) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4226) > =A0 =A0 =A0 =A0at = org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeS= che > duler.java:205) > =A0 =A0 =A0 =A0at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntM > ergeScheduler.java:260) > > > > I am running it on Linux version 2.6.26-79.fc9 , gcc version 4.1.2 20071124 > (Red Hat 4.1.2-42) > > This exception occurs after we have indexed more than 100 gb of data. = I will > try running with -Xint to see if I can avoid this exception. > > Regards > Sumanta > > -----Original Message----- > From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com] > Sent: Monday, August 24, 2009 1:10 PM > To: java-user@lucene.apache.org > Subject: RE: Merge Exception in Lucene 2.4 > > I will be running a long running test with jdk1.6u16. If the problem = still > appears I will try to create a test case to show the issue. > > Sumanta > > -----Original Message----- > From: Michael McCandless [mailto:lucene@mikemccandless.com] > Sent: Friday, August 21, 2009 4:44 PM > To: java-user@lucene.apache.org > Subject: Re: Merge Exception in Lucene 2.4 > > That code looks fine... > > What OS/filesystem are you using? > > Can you make a small test case that shows the issue? > > Mike > > On Thu, Aug 20, 2009 at 7:41 AM, Sumanta > Bhowmik wrote: >> We put all the IndexWriters in an array which is defined by >> >> final Directory[] finalDir; >> >> We also declare an indexer >> As >> private volatile static Indexer indexer; >> >> final Indexer finalIndexer =3D indexer; >> >> Next we call the merge in a new thread : >> >> Thread thread =3D new Thread(){ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void run() >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0logger.debug("starts merging w/o >> optimization"); >> >> finalIndexer.getWriter().addIndexesNoOptimize(finalDir); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0logger.debug("ends merging w/o >> optimization"); >> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} = catch (CorruptIndexException e) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0logger.notice("",e); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} = catch (IOException e) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0logger.notice("",e); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> >> =A0 =A0 =A0 =A0} >> } >> >> Sumanta >> >> >> >> >> -----Original Message----- >> From: Michael McCandless [mailto:lucene@mikemccandless.com] >> Sent: Thursday, August 20, 2009 3:07 PM >> To: java-user@lucene.apache.org >> Subject: Re: Merge Exception in Lucene 2.4 >> >> You should definitely upgrade to the latest JDK 1.6 to get the fix = for >> the JRE bug in LUCENE-1282, but, I don't think you are hitting that >> bug (read past EOF during merge is a different exception). >> >> Can you describe more detail on how you merge 6 IndexWriters? >> >> Mike >> >> On Thu, Aug 20, 2009 at 5:21 AM, Sumanta >> Bhowmik wrote: >>> I checked at http://issues.apache.org/jira/browse/LUCENE-1282 >>> SegmentMerger.java has this code >>> >>> TermFreqVector[] vectors =3D reader.getTermFreqVectors(docNum); >>> termVectorsWriter.addAllDocVectors(vectors); >>> >>> so this issue appears inspite of this fix. >>> >>> I am using java version "1.6.0_07". Is it fixed in jdk6u10 and above >>> (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D6707044) ? >>> >>> Regards >>> Sumanta >>> >>> >>> -----Original Message----- >>> From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com] >>> Sent: Thursday, August 20, 2009 1:15 PM >>> To: java-user@lucene.apache.org >>> Subject: Merge Exception in Lucene 2.4 >>> >>> Hi >>> >>> >>> >>> I am getting this issue in Lucene2.4 when I try to merge multiple >>> IndexWriters(generally 6) >>> >>> >>> >>> sh-3.2# Exception in thread "Lucene Merge Thread #5" >>> org.apache.lucene.index.MergePolicy$MergeException: = java.io.IOException: >>> read past EOF >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Con= cur >>> rentMergeScheduler.java:309) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntM >>> ergeScheduler.java:286) >>> >>> Caused by: java.io.IOException: read past EOF >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.j= ava >>> :135) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.ja= va: >>> 228) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:= 184 >>> ) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.ja= va: >>> 204) >>> >>> =A0 =A0 =A0 =A0at >>> = org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260) >>> >>> =A0 =A0 =A0 =A0at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeS= che >>> duler.java:205) >>> >>> =A0 =A0 =A0 =A0at >>> >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntM >>> ergeScheduler.java:260) >>> >>> >>> >>> >>> >>> Is this a known issue and has any fix been provided for it ? Would >>> appreciate any help. >>> >>> >>> >>> Regards >>> >>> Sumanta >>> >>> >>> >>> >>> >>> >>> >>> >>> __________ NOD32 4349 (20090819) Information __________ >>> >>> This message was checked by NOD32 antivirus system. >>> http://www.eset.com >>> >>> >>> >>> = --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>> For additional commands, e-mail: java-user-help@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> >> __________ NOD32 4350 (20090820) Information __________ >> >> This message was checked by NOD32 antivirus system. >> http://www.eset.com >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > > > __________ NOD32 4354 (20090821) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > > > __________ NOD32 4361 (20090823) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org __________ NOD32 4371 (20090826) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org