lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumanta Bhowmik" <sumanta.bhow...@guavus.com>
Subject RE: Merge Exception in Lucene 2.4
Date Thu, 27 Aug 2009 04:46:08 GMT
Hi

I ran a long running test and now got this exception.

Exception in thread "Lucene Merge Thread #39"
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
read past EOF
        at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Concur
rentMergeScheduler.java:309)
        at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
ergeScheduler.java:286)
Caused by: java.io.IOException: read past EOF
        at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:15
1)
        at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:
38)
        at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
        at
org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTermPo
sitions.java:73)
        at
org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPositio
ns.java:69)
        at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:667)
        at
org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:586)
        at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:547)
        at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:500)
        at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
        at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4226)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
        at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeSche
duler.java:205)
        at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
ergeScheduler.java:260)



I am running it on Linux version 2.6.26-79.fc9 , gcc version 4.1.2 20071124
(Red Hat 4.1.2-42)

This exception occurs after we have indexed more than 100 gb of data. I will
try running with -Xint to see if I can avoid this exception. 

Regards
Sumanta

-----Original Message-----
From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com] 
Sent: Monday, August 24, 2009 1:10 PM
To: java-user@lucene.apache.org
Subject: RE: Merge Exception in Lucene 2.4

I will be running a long running test with jdk1.6u16. If the problem still
appears I will try to create a test case to show the issue.

Sumanta

-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com] 
Sent: Friday, August 21, 2009 4:44 PM
To: java-user@lucene.apache.org
Subject: Re: Merge Exception in Lucene 2.4

That code looks fine...

What OS/filesystem are you using?

Can you make a small test case that shows the issue?

Mike

On Thu, Aug 20, 2009 at 7:41 AM, Sumanta
Bhowmik<sumanta.bhowmik@guavus.com> wrote:
> We put all the IndexWriters in an array which is defined by
>
> final Directory[] finalDir;
>
> We also declare an indexer
> As
> private volatile static Indexer indexer;
>
> final Indexer finalIndexer = indexer;
>
> Next we call the merge in a new thread :
>
> Thread thread = new Thread(){
>                        public void run()
>                        {
>                                try {
>                                        logger.debug("starts merging
w/o
> optimization");
>
> finalIndexer.getWriter().addIndexesNoOptimize(finalDir);
>                                        logger.debug("ends merging
w/o
> optimization");
>
>                                } catch (CorruptIndexException e) {
>                                        logger.notice("",e);
>                                } catch (IOException e) {
>                                        logger.notice("",e);
>                                }
>
>        }
> }
>
> Sumanta
>
>
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Thursday, August 20, 2009 3:07 PM
> To: java-user@lucene.apache.org
> Subject: Re: Merge Exception in Lucene 2.4
>
> You should definitely upgrade to the latest JDK 1.6 to get the fix for
> the JRE bug in LUCENE-1282, but, I don't think you are hitting that
> bug (read past EOF during merge is a different exception).
>
> Can you describe more detail on how you merge 6 IndexWriters?
>
> Mike
>
> On Thu, Aug 20, 2009 at 5:21 AM, Sumanta
> Bhowmik<sumanta.bhowmik@guavus.com> wrote:
>> I checked at http://issues.apache.org/jira/browse/LUCENE-1282
>> SegmentMerger.java has this code
>>
>> TermFreqVector[] vectors = reader.getTermFreqVectors(docNum);
>> termVectorsWriter.addAllDocVectors(vectors);
>>
>> so this issue appears inspite of this fix.
>>
>> I am using java version "1.6.0_07". Is it fixed in jdk6u10 and above
>> (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6707044) ?
>>
>> Regards
>> Sumanta
>>
>>
>> -----Original Message-----
>> From: Sumanta Bhowmik [mailto:sumanta.bhowmik@guavus.com]
>> Sent: Thursday, August 20, 2009 1:15 PM
>> To: java-user@lucene.apache.org
>> Subject: Merge Exception in Lucene 2.4
>>
>> Hi
>>
>>
>>
>> I am getting this issue in Lucene2.4 when I try to merge multiple
>> IndexWriters(generally 6)
>>
>>
>>
>> sh-3.2# Exception in thread "Lucene Merge Thread #5"
>> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
>> read past EOF
>>
>>        at
>>
>
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Concur
>> rentMergeScheduler.java:309)
>>
>>        at
>>
>
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
>> ergeScheduler.java:286)
>>
>> Caused by: java.io.IOException: read past EOF
>>
>>        at
>>
>
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java
>> :135)
>>
>>        at
>>
>
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:
>> 228)
>>
>>        at
>>
>
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184
>> )
>>
>>        at
>>
>
org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.java:
>> 204)
>>
>>        at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>>
>>        at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>>
>>        at
>>
>
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeSche
>> duler.java:205)
>>
>>        at
>>
>
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentM
>> ergeScheduler.java:260)
>>
>>
>>
>>
>>
>> Is this a known issue and has any fix been provided for it ? Would
>> appreciate any help.
>>
>>
>>
>> Regards
>>
>> Sumanta
>>
>>
>>
>>
>>
>>
>>
>>
>> __________ NOD32 4349 (20090819) Information __________
>>
>> This message was checked by NOD32 antivirus system.
>> http://www.eset.com
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> __________ NOD32 4350 (20090820) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


__________ NOD32 4354 (20090821) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


__________ NOD32 4361 (20090823) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message