lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Merge Exception in Lucene 2.4
Date Thu, 27 Aug 2009 09:00:04 GMT
Was this a smallish contained test case?  Can you post the sources?

Can you run your next test with IndexWriter's infoStream turned on,
and post the resluting output?

Was this with jdk1.6u16?  That version fixes the original hotspot bug
from LUCENE-1282... (and your exception looks different from the one
in LUCENE-1282).

Mike

On Thu, Aug 27, 2009 at 12:46 AM, Sumanta
Bhowmik<sumanta.bhowmik@guavus.com> wrote:
> Hi
>
> I ran a long running test and now got this exception.
>
> Exception in thread "Lucene Merge Thread #39"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> read past EOF
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Concur
> rentMergeScheduler.java:309)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
> ergeScheduler.java:286)
> Caused by: java.io.IOException: read past EOF
>        at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:15
> 1)
>        at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:
> 38)
>        at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
>        at
> org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTermPo
> sitions.java:73)
>        at
> org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPositio
> ns.java:69)
>        at
> org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:667)
>        at
> org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:586)
>        at
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:547)
>        at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:500)
>        at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
>        at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4226)
>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeSche
> duler.java:205)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
> ergeScheduler.java:260)
>
>
>
> I am running it on Linux version 2.6.26-79.fc9 , gcc version 4.1.2 20071124
> (Red Hat 4.1.2-42)
>
> This exception occurs after we have indexed more than 100 gb of data. I will
> try running with -Xint to see if I can avoid this exception.
>
> Regards
> Sumanta
>
> -----Original Message-----
> From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com]
> Sent: Monday, August 24, 2009 1:10 PM
> To: java-user@lucene.apache.org
> Subject: RE: Merge Exception in Lucene 2.4
>
> I will be running a long running test with jdk1.6u16. If the problem still
> appears I will try to create a test case to show the issue.
>
> Sumanta
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Friday, August 21, 2009 4:44 PM
> To: java-user@lucene.apache.org
> Subject: Re: Merge Exception in Lucene 2.4
>
> That code looks fine...
>
> What OS/filesystem are you using?
>
> Can you make a small test case that shows the issue?
>
> Mike
>
> On Thu, Aug 20, 2009 at 7:41 AM, Sumanta
> Bhowmik<sumanta.bhowmik@guavus.com> wrote:
>> We put all the IndexWriters in an array which is defined by
>>
>> final Directory[] finalDir;
>>
>> We also declare an indexer
>> As
>> private volatile static Indexer indexer;
>>
>> final Indexer finalIndexer = indexer;
>>
>> Next we call the merge in a new thread :
>>
>> Thread thread = new Thread(){
>>                        public void run()
>>                        {
>>                                try {
>>                                        logger.debug("starts merging
w/o
>> optimization");
>>
>> finalIndexer.getWriter().addIndexesNoOptimize(finalDir);
>>                                        logger.debug("ends merging
w/o
>> optimization");
>>
>>                                } catch (CorruptIndexException e)
{
>>                                        logger.notice("",e);
>>                                } catch (IOException e) {
>>                                        logger.notice("",e);
>>                                }
>>
>>        }
>> }
>>
>> Sumanta
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Thursday, August 20, 2009 3:07 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: Merge Exception in Lucene 2.4
>>
>> You should definitely upgrade to the latest JDK 1.6 to get the fix for
>> the JRE bug in LUCENE-1282, but, I don't think you are hitting that
>> bug (read past EOF during merge is a different exception).
>>
>> Can you describe more detail on how you merge 6 IndexWriters?
>>
>> Mike
>>
>> On Thu, Aug 20, 2009 at 5:21 AM, Sumanta
>> Bhowmik<sumanta.bhowmik@guavus.com> wrote:
>>> I checked at http://issues.apache.org/jira/browse/LUCENE-1282
>>> SegmentMerger.java has this code
>>>
>>> TermFreqVector[] vectors = reader.getTermFreqVectors(docNum);
>>> termVectorsWriter.addAllDocVectors(vectors);
>>>
>>> so this issue appears inspite of this fix.
>>>
>>> I am using java version "1.6.0_07". Is it fixed in jdk6u10 and above
>>> (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6707044) ?
>>>
>>> Regards
>>> Sumanta
>>>
>>>
>>> -----Original Message-----
>>> From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com]
>>> Sent: Thursday, August 20, 2009 1:15 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Merge Exception in Lucene 2.4
>>>
>>> Hi
>>>
>>>
>>>
>>> I am getting this issue in Lucene2.4 when I try to merge multiple
>>> IndexWriters(generally 6)
>>>
>>>
>>>
>>> sh-3.2# Exception in thread "Lucene Merge Thread #5"
>>> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
>>> read past EOF
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Concur
>>> rentMergeScheduler.java:309)
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
>>> ergeScheduler.java:286)
>>>
>>> Caused by: java.io.IOException: read past EOF
>>>
>>>        at
>>>
>>
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java
>>> :135)
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:
>>> 228)
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184
>>> )
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.java:
>>> 204)
>>>
>>>        at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>>>
>>>        at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeSche
>>> duler.java:205)
>>>
>>>        at
>>>
>>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
>>> ergeScheduler.java:260)
>>>
>>>
>>>
>>>
>>>
>>> Is this a known issue and has any fix been provided for it ? Would
>>> appreciate any help.
>>>
>>>
>>>
>>> Regards
>>>
>>> Sumanta
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> __________ NOD32 4349 (20090819) Information __________
>>>
>>> This message was checked by NOD32 antivirus system.
>>> http://www.eset.com
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> __________ NOD32 4350 (20090820) Information __________
>>
>> This message was checked by NOD32 antivirus system.
>> http://www.eset.com
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> __________ NOD32 4354 (20090821) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> __________ NOD32 4361 (20090823) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message