lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Ochoa" <marcelo.oc...@gmail.com>
Subject Help to solve an issue when upgrading Lucene-Oracle integration to lucene 2.3.1
Date Tue, 06 May 2008 21:26:17 GMT
Hi Lucene experts:
  I am working upgrading Lucene-Oracle integration project to latest
Lucene 2.3.1 code.
 After correcting a minor issue on OJVMDirectory file implementation I
have the integration running with latest 2.3.1 code.
 But it only works with small indexes, I think index which are lower
than the Memory threshold.
 If I performs an index in a big table, I got this exception:
 Exception in thread "Root Thread" java.lang.ArrayIndexOutOfBoundsException
        at org.apache.lucene.index.DocumentsWriter$ByteSliceReader.nextSlice(DocumentsWriter.java)
        at org.apache.lucene.index.DocumentsWriter$ByteSliceReader.readByte(DocumentsWriter.java)
        at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java)
        at org.apache.lucene.index.DocumentsWriter.appendPostings(DocumentsWriter.java)
        at org.apache.lucene.index.DocumentsWriter.writeSegment(DocumentsWriter.java:2011)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:548)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2497)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2397)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java)
        at org.apache.lucene.indexer.TableIndexer.index(TableIndexer.java)
        at org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate(LuceneDomainIndex.java:477)

  Something is wrong at nextSlice method :(
  I added some System.out info and, before the exception is thrown,
index and arrays have this information:

.nextSlice (previous)
limit: 11393 buffer.length: 32768
level: 0 nextLevelArray.length: 10
bufferUpto: 147 pool.buffers.length: 366
.nextSlice (current)
limit: 6189 buffer.length: 32768
level: 0 nextLevelArray.length: 10
bufferUpto: 8192 pool.buffers.length: 366

   As you can see bufferUpto variable have 8192 value and pool.buffers
is an array of 366 elements, this cause the exception  nextSlice()
method is:
    public void nextSlice() {

      // Skip to our next slice
      System.out.println(".nextSlice");
      System.out.println("limit: "+limit+" buffer.length: "+buffer.length);
      final int nextIndex = ((buffer[limit]&0xff)<<24) +
((buffer[1+limit]&0xff)<<16) + ((buffer[2+limit]&0xff)<<8) +
(buffer[3+limit]&0xff);

      System.out.println("level: "+level+" nextLevelArray.length:
"+nextLevelArray.length);
      level = nextLevelArray[level];
      final int newSize = levelSizeArray[level];

      bufferUpto = nextIndex / BYTE_BLOCK_SIZE;
      bufferOffset = bufferUpto * BYTE_BLOCK_SIZE;

      System.out.println("bufferUpto: "+bufferUpto+"
pool.buffers.length: "+pool.buffers.length);
      buffer = pool.buffers[bufferUpto];
      upto = nextIndex & BYTE_BLOCK_MASK;

      if (nextIndex + newSize >= endIndex) {
        // We are advancing to the final slice
        assert endIndex - nextIndex > 0;
        limit = endIndex - bufferOffset;
      } else {
        // This is not the final slice (subtract 4 for the
        // forwarding address at the end of this new slice)
        limit = upto+newSize-4;
      }
    }

    IndexWriter InfoStream information is:
  RAM: now flush @ usedMB=53.016 allocMB=53.016 triggerMB=53
IW 0 [Root Thread]:   flush: segment=_0 docStoreSegment=_0
docStoreOffset=0 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=85
564 numBufDelTerms=0
IW 0 [Root Thread]:   index before flush
flush postings as segment _0 numDocs=85564
..... nextSlice output here.....
docWriter: now abort
IW 1 [Root Thread]: hit exception flushing segment _0
docWriter: now abort
IFD [Root Thread]: now checkpoint "segments_1" [0 segments ; isCommit = false]
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.tii"
IFD [Root Thread]: delete "_0.tii"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fdx"
IFD [Root Thread]: delete "_0.fdx"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fnm"
IFD [Root Thread]: delete "_0.fnm"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fdt"
IFD [Root Thread]: delete "_0.fdt"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.prx"
IFD [Root Thread]: delete "_0.prx"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.frq"
IFD [Root Thread]: delete "_0.frq"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.tis"
IFD [Root Thread]: delete "_0.tis"
IW 1 [Root Thread]: now flush at close
IW 1 [Root Thread]:   flush: segment=null docStoreSegment=null
docStoreOffset=0 flushDocs=false flushDeletes=false
flushDocStores=false numDocs=0 numBufDelTerms=0
IW 1 [Root Thread]:   index before flush
IW 1 [Root Thread]: close: wrote segments file "segments_2"
IFD [Root Thread]: now checkpoint "segments_2" [0 segments ; isCommit = true]
IFD [Root Thread]: deleteCommits: now remove commit "segments_1"
IFD [Root Thread]: delete "segments_1"
IW 1 [Root Thread]: at close:
Exception in thread "Root Thread" java.lang.ArrayIndexOutOfBoundsException
        at org.apache.lucene.index.DocumentsWriter$ByteSliceReader.nextSlice(DocumentsWriter.java)
        at org.apache.lucene.index.DocumentsWriter$ByteSliceReader.readByte(DocumentsWriter.java)
        at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java)
        at org.apache.lucene.index.DocumentsWriter.appendPostings(DocumentsWriter.java)
        at org.apache.lucene.index.DocumentsWriter.writeSegment(DocumentsWriter.java:2011)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:548)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2497)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2397)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java)
        at org.apache.lucene.indexer.TableIndexer.index(TableIndexer.java)
        at org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate(LuceneDomainIndex.java:477)

  Any help to find the incompatibility issue will be great.
  Best regards, Marcelo.

PD: If anybody needs the complete log I can send it my email.
-- 
Marcelo F. Ochoa
http://marceloochoa.blogspot.com/
http://marcelo.ochoa.googlepages.com/home
______________
Do you Know DBPrism? Look @ DB Prism's Web Site
http://www.dbprism.com.ar/index.html
More info?
Chapter 17 of the book "Programming the Oracle Database using Java &
Web Services"
http://www.amazon.com/gp/product/1555583296/
Chapter 21 of the book "Professional XML Databases" - Wrox Press
http://www.amazon.com/gp/product/1861003587/
Chapter 8 of the book "Oracle & Open Source" - O'Reilly
http://www.oreilly.com/catalog/oracleopen/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message