Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of marcelo.ochoa@gmail.com
 designates 209.85.221.201 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YCCEpoUHZZXK3Qc7v86VZgOdmI7Xaet9xw1kt/7FEqiNz1blptVYFpxTdhP91ST8MN
         xnIWE2i3oxMFJQZHJEHRYyeiHLj5ZfisnNjndrJLrkVlbt/VNyexo6CWayuDtXCJHd3J
         nM9te9ABEE2A5f3+7dxCJ76gccOaFxGPj/AOs=
MIME-Version: 1.0
Date: Fri, 12 Mar 2010 20:25:14 -0300
Message-ID: <126142c1003121525v24499625u1589bbef4c0792e7@mail.gmail.com>
Subject: Different behavior of Directory.fieldLength()
From: Marcelo Ochoa <marcelo.ochoa@gmail.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi:
  During some test of Lucene Domain Index
(http://docs.google.com/View?id=ddgw7sjp_54fgj9kg) with big data
sources we found an exception caused for calling
Directory.fieldLength() method on non existing file.
  FSDirectory implements this method as:
  /** Returns the length in bytes of a file in the directory. */
  public long fileLength(String name) {
    ensureOpen();
    File file = new File(directory, name);
    return file.length();
  }

  According to JDK1.5 calling to File constructor causes a file
creation without throwing an exception:
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#File(java.lang.String,
java.lang.String)
  But either RAMDirectory nor OJVMDirectory do this:
RAMDirectory:
  /** Returns the length in bytes of a file in the directory.
   * @throws IOException if the file does not exist
   */
  public final long fileLength(String name) throws IOException {
    ensureOpen();
    RAMFile file;
    synchronized (this) {
      file = (RAMFile)fileMap.get(name);
    }
    if (file==null)
      throw new FileNotFoundException(name);
    return file.getLength();
  }

  If OJVMDirectory throws an exception if a file doesn't exist it
causes that the IndexWriter fail to do the job, here the stack trace:
IW 3 [Root Thread]: DW:   RAM: now flush @ usedMB=15.001
allocMB=15.001 deletesMB=0 triggerMB=15
IW 3 [Root Thread]:   flush: segment=_0 docStoreSegment=_0
docStoreOffset=0 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=109169 numBufDelTerms=0
IW 3 [Root Thread]:   index before flush
IW 3 [Root Thread]: DW: flush postings as segment _0 numDocs=109169
*** 2010-03-11 17:27:15.696
IW 3 [Root Thread]: DW: docWriter: now abort
IW 3 [Root Thread]: hit exception flushing segment _0
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.tii"
IFD [Root Thread]: delete "_0.tii"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fnm"
IFD [Root Thread]: delete "_0.fnm"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fdx"
IFD [Root Thread]: delete "_0.fdx"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.fdt"
IFD [Root Thread]: delete "_0.fdt"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.prx"
IFD [Root Thread]: delete "_0.prx"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.nrm"
IFD [Root Thread]: delete "_0.nrm"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.frq"
IFD [Root Thread]: delete "_0.frq"
IFD [Root Thread]: refresh [prefix=_0]: removing newly created
unreferenced file "_0.tis"
IFD [Root Thread]: delete "_0.tis"
Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex
ODCIIndexCreate
SEVERE: failed to create index: cannot verify file: _0.fdx. Reason:
Exhausted Resultset
Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex
ODCIIndexCreate
FINER: THROW
java.io.IOException: cannot verify file: _0.fdx. Reason: Exhausted Resultset
        at org.apache.lucene.store.OJVMDirectory.fileLength(OJVMDirectory.java:633)
        at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:271)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:593)
        at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4311)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4209)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4200)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2497)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2451)
        at org.apache.lucene.indexer.TableIndexer.index(TableIndexer.java:374)
        at org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate(LuceneDomainIndex.java:568)
IW 3 [Root Thread]: now flush at close
IW 3 [Root Thread]:   flush: segment=null docStoreSegment=null
docStoreOffset=0 flushDocs=false flushDeletes=true
flushDocStores=false numDocs=0 numBufDelTerms=0
IW 3 [Root Thread]:   index before flush
IW 3 [Root Thread]: CMS: now merge
IW 3 [Root Thread]: CMS:   index:
IW 3 [Root Thread]: CMS:   no more merges pending; now return
IW 3 [Root Thread]: now call final commit()
IW 3 [Root Thread]: startCommit(): start sizeInBytes=0
IW 3 [Root Thread]: startCommit index= changeCount=1
IW 3 [Root Thread]: done all syncs
IW 3 [Root Thread]: commit: pendingCommit != null
IW 3 [Root Thread]: commit: wrote segments file "segments_2"
IFD [Root Thread]: now checkpoint "segments_2" [0 segments ; isCommit = true]
IFD [Root Thread]: deleteCommits: now decRef commit "segments_1"
IFD [Root Thread]: delete "segments_1"
IW 3 [Root Thread]: commit: done
IW 3 [Root Thread]: at close:

   Which is the correct behavior for this method?
   We changed OJVMDirectory.fileLength() method to returns 0 if no
file exists instead of throwing an exception and IndexWriter works
properly, but I am not complete sure if this is the correct
implementation. Here OJVMDirectory implementation:
    /**
     * Returns the length of a file in the directory.
     */
    public long fileLength(String name) throws IOException {
        PreparedStatement stmt = null;
        ResultSet rs = null;
        try {
            stmt =
              conn.prepareStatement(StringUtils.replace(getFileLengthStmt,"%IDX%",this.prefix));
            stmt.setString(1, name);
            rs = stmt.executeQuery();
            if (rs.next())
              return rs.getLong(1);
            else
              return 0;
        } catch (SQLException e) {
            throw new IOException("cannot verify file: " + name +
                                ". Reason: " + e.getMessage());
        } finally {
            OJVMUtil.closeDbResources(stmt, rs);
        }
    }

   Best regards, Marcelo

-- 
Marcelo F. Ochoa
http://marceloochoa.blogspot.com/
http://mochoa.sites.exa.unicen.edu.ar/
______________
Want to integrate Lucene and Oracle?
http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
Is Oracle 11g REST ready?
http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org