lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Help to solve an issue when upgrading Lucene-Oracle integration to lucene 2.3.1
Date Tue, 06 May 2008 22:00:58 GMT

Hi Marcelo,

Hmmm something is not right.

Somehow the byte slices, which DocumentsWriter uses to hold the  
postings in RAM, became corrupt.

Is this easily reproduced?

Mike

Marcelo Ochoa wrote:
> Hi Lucene experts:
>   I am working upgrading Lucene-Oracle integration project to latest
> Lucene 2.3.1 code.
>  After correcting a minor issue on OJVMDirectory file implementation I
> have the integration running with latest 2.3.1 code.
>  But it only works with small indexes, I think index which are lower
> than the Memory threshold.
>  If I performs an index in a big table, I got this exception:
>  Exception in thread "Root Thread"  
> java.lang.ArrayIndexOutOfBoundsException
>         at org.apache.lucene.index.DocumentsWriter 
> $ByteSliceReader.nextSlice(DocumentsWriter.java)
>         at org.apache.lucene.index.DocumentsWriter 
> $ByteSliceReader.readByte(DocumentsWriter.java)
>         at org.apache.lucene.store.IndexInput.readVInt 
> (IndexInput.java)
>         at org.apache.lucene.index.DocumentsWriter.appendPostings 
> (DocumentsWriter.java)
>         at org.apache.lucene.index.DocumentsWriter.writeSegment 
> (DocumentsWriter.java:2011)
>         at org.apache.lucene.index.DocumentsWriter.flush 
> (DocumentsWriter.java:548)
>         at org.apache.lucene.index.IndexWriter.doFlush 
> (IndexWriter.java:2497)
>         at org.apache.lucene.index.IndexWriter.flush 
> (IndexWriter.java:2397)
>         at org.apache.lucene.index.IndexWriter.addDocument 
> (IndexWriter.java)
>         at org.apache.lucene.index.IndexWriter.addDocument 
> (IndexWriter.java)
>         at org.apache.lucene.indexer.TableIndexer.index 
> (TableIndexer.java)
>         at  
> org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate 
> (LuceneDomainIndex.java:477)
>
>   Something is wrong at nextSlice method :(
>   I added some System.out info and, before the exception is thrown,
> index and arrays have this information:
>
> .nextSlice (previous)
> limit: 11393 buffer.length: 32768
> level: 0 nextLevelArray.length: 10
> bufferUpto: 147 pool.buffers.length: 366
> .nextSlice (current)
> limit: 6189 buffer.length: 32768
> level: 0 nextLevelArray.length: 10
> bufferUpto: 8192 pool.buffers.length: 366
>
>    As you can see bufferUpto variable have 8192 value and pool.buffers
> is an array of 366 elements, this cause the exception  nextSlice()
> method is:
>     public void nextSlice() {
>
>       // Skip to our next slice
>       System.out.println(".nextSlice");
>       System.out.println("limit: "+limit+" buffer.length:  
> "+buffer.length);
>       final int nextIndex = ((buffer[limit]&0xff)<<24) +
> ((buffer[1+limit]&0xff)<<16) + ((buffer[2+limit]&0xff)<<8) +
> (buffer[3+limit]&0xff);
>
>       System.out.println("level: "+level+" nextLevelArray.length:
> "+nextLevelArray.length);
>       level = nextLevelArray[level];
>       final int newSize = levelSizeArray[level];
>
>       bufferUpto = nextIndex / BYTE_BLOCK_SIZE;
>       bufferOffset = bufferUpto * BYTE_BLOCK_SIZE;
>
>       System.out.println("bufferUpto: "+bufferUpto+"
> pool.buffers.length: "+pool.buffers.length);
>       buffer = pool.buffers[bufferUpto];
>       upto = nextIndex & BYTE_BLOCK_MASK;
>
>       if (nextIndex + newSize >= endIndex) {
>         // We are advancing to the final slice
>         assert endIndex - nextIndex > 0;
>         limit = endIndex - bufferOffset;
>       } else {
>         // This is not the final slice (subtract 4 for the
>         // forwarding address at the end of this new slice)
>         limit = upto+newSize-4;
>       }
>     }
>
>     IndexWriter InfoStream information is:
>   RAM: now flush @ usedMB=53.016 allocMB=53.016 triggerMB=53
> IW 0 [Root Thread]:   flush: segment=_0 docStoreSegment=_0
> docStoreOffset=0 flushDocs=true flushDeletes=false
> flushDocStores=false numDocs=85
> 564 numBufDelTerms=0
> IW 0 [Root Thread]:   index before flush
> flush postings as segment _0 numDocs=85564
> ..... nextSlice output here.....
> docWriter: now abort
> IW 1 [Root Thread]: hit exception flushing segment _0
> docWriter: now abort
> IFD [Root Thread]: now checkpoint "segments_1" [0 segments ;  
> isCommit = false]
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.tii"
> IFD [Root Thread]: delete "_0.tii"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fdx"
> IFD [Root Thread]: delete "_0.fdx"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fnm"
> IFD [Root Thread]: delete "_0.fnm"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fdt"
> IFD [Root Thread]: delete "_0.fdt"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.prx"
> IFD [Root Thread]: delete "_0.prx"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.frq"
> IFD [Root Thread]: delete "_0.frq"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.tis"
> IFD [Root Thread]: delete "_0.tis"
> IW 1 [Root Thread]: now flush at close
> IW 1 [Root Thread]:   flush: segment=null docStoreSegment=null
> docStoreOffset=0 flushDocs=false flushDeletes=false
> flushDocStores=false numDocs=0 numBufDelTerms=0
> IW 1 [Root Thread]:   index before flush
> IW 1 [Root Thread]: close: wrote segments file "segments_2"
> IFD [Root Thread]: now checkpoint "segments_2" [0 segments ;  
> isCommit = true]
> IFD [Root Thread]: deleteCommits: now remove commit "segments_1"
> IFD [Root Thread]: delete "segments_1"
> IW 1 [Root Thread]: at close:
> Exception in thread "Root Thread"  
> java.lang.ArrayIndexOutOfBoundsException
>         at org.apache.lucene.index.DocumentsWriter 
> $ByteSliceReader.nextSlice(DocumentsWriter.java)
>         at org.apache.lucene.index.DocumentsWriter 
> $ByteSliceReader.readByte(DocumentsWriter.java)
>         at org.apache.lucene.store.IndexInput.readVInt 
> (IndexInput.java)
>         at org.apache.lucene.index.DocumentsWriter.appendPostings 
> (DocumentsWriter.java)
>         at org.apache.lucene.index.DocumentsWriter.writeSegment 
> (DocumentsWriter.java:2011)
>         at org.apache.lucene.index.DocumentsWriter.flush 
> (DocumentsWriter.java:548)
>         at org.apache.lucene.index.IndexWriter.doFlush 
> (IndexWriter.java:2497)
>         at org.apache.lucene.index.IndexWriter.flush 
> (IndexWriter.java:2397)
>         at org.apache.lucene.index.IndexWriter.addDocument 
> (IndexWriter.java)
>         at org.apache.lucene.index.IndexWriter.addDocument 
> (IndexWriter.java)
>         at org.apache.lucene.indexer.TableIndexer.index 
> (TableIndexer.java)
>         at  
> org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate 
> (LuceneDomainIndex.java:477)
>
>   Any help to find the incompatibility issue will be great.
>   Best regards, Marcelo.
>
> PD: If anybody needs the complete log I can send it my email.
> -- 
> Marcelo F. Ochoa
> http://marceloochoa.blogspot.com/
> http://marcelo.ochoa.googlepages.com/home
> ______________
> Do you Know DBPrism? Look @ DB Prism's Web Site
> http://www.dbprism.com.ar/index.html
> More info?
> Chapter 17 of the book "Programming the Oracle Database using Java &
> Web Services"
> http://www.amazon.com/gp/product/1555583296/
> Chapter 21 of the book "Professional XML Databases" - Wrox Press
> http://www.amazon.com/gp/product/1861003587/
> Chapter 8 of the book "Oracle & Open Source" - O'Reilly
> http://www.oreilly.com/catalog/oracleopen/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message