lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Different behavior of Directory.fieldLength()
Date Sat, 13 Mar 2010 09:53:11 GMT
That is not true, the API says:
"Creates a new File *instance* from a parent pathname string and a child pathname string."

Please note "instance", so it will never create the file on disk. New File() just creates
a file instance but no file on disk. You can check this with a simple test.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Marcelo Ochoa [mailto:marcelo.ochoa@gmail.com]
> Sent: Saturday, March 13, 2010 12:25 AM
> To: java-dev@lucene.apache.org
> Subject: Different behavior of Directory.fieldLength()
> 
> Hi:
>   During some test of Lucene Domain Index
> (http://docs.google.com/View?id=ddgw7sjp_54fgj9kg) with big data
> sources we found an exception caused for calling
> Directory.fieldLength() method on non existing file.
>   FSDirectory implements this method as:
>   /** Returns the length in bytes of a file in the directory. */
>   public long fileLength(String name) {
>     ensureOpen();
>     File file = new File(directory, name);
>     return file.length();
>   }
> 
>   According to JDK1.5 calling to File constructor causes a file
> creation without throwing an exception:
> http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#File(java.lan
> g.String,
> java.lang.String)
>   But either RAMDirectory nor OJVMDirectory do this:
> RAMDirectory:
>   /** Returns the length in bytes of a file in the directory.
>    * @throws IOException if the file does not exist
>    */
>   public final long fileLength(String name) throws IOException {
>     ensureOpen();
>     RAMFile file;
>     synchronized (this) {
>       file = (RAMFile)fileMap.get(name);
>     }
>     if (file==null)
>       throw new FileNotFoundException(name);
>     return file.getLength();
>   }
> 
>   If OJVMDirectory throws an exception if a file doesn't exist it
> causes that the IndexWriter fail to do the job, here the stack trace:
> IW 3 [Root Thread]: DW:   RAM: now flush @ usedMB=15.001
> allocMB=15.001 deletesMB=0 triggerMB=15
> IW 3 [Root Thread]:   flush: segment=_0 docStoreSegment=_0
> docStoreOffset=0 flushDocs=true flushDeletes=false
> flushDocStores=false numDocs=109169 numBufDelTerms=0
> IW 3 [Root Thread]:   index before flush
> IW 3 [Root Thread]: DW: flush postings as segment _0 numDocs=109169
> *** 2010-03-11 17:27:15.696
> IW 3 [Root Thread]: DW: docWriter: now abort
> IW 3 [Root Thread]: hit exception flushing segment _0
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.tii"
> IFD [Root Thread]: delete "_0.tii"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fnm"
> IFD [Root Thread]: delete "_0.fnm"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fdx"
> IFD [Root Thread]: delete "_0.fdx"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.fdt"
> IFD [Root Thread]: delete "_0.fdt"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.prx"
> IFD [Root Thread]: delete "_0.prx"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.nrm"
> IFD [Root Thread]: delete "_0.nrm"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.frq"
> IFD [Root Thread]: delete "_0.frq"
> IFD [Root Thread]: refresh [prefix=_0]: removing newly created
> unreferenced file "_0.tis"
> IFD [Root Thread]: delete "_0.tis"
> Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex
> ODCIIndexCreate
> SEVERE: failed to create index: cannot verify file: _0.fdx. Reason:
> Exhausted Resultset
> Mar 11, 2010 5:27:15 PM org.apache.lucene.indexer.LuceneDomainIndex
> ODCIIndexCreate
> FINER: THROW
> java.io.IOException: cannot verify file: _0.fdx. Reason: Exhausted
> Resultset
>         at
> org.apache.lucene.store.OJVMDirectory.fileLength(OJVMDirectory.java:633
> )
>         at
> org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:271)
>         at
> org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:593)
>         at
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:43
> 11)
>         at
> org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4209)
>         at
> org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4200)
>         at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2497)
>         at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2451)
>         at
> org.apache.lucene.indexer.TableIndexer.index(TableIndexer.java:374)
>         at
> org.apache.lucene.indexer.LuceneDomainIndex.ODCIIndexCreate(LuceneDomai
> nIndex.java:568)
> IW 3 [Root Thread]: now flush at close
> IW 3 [Root Thread]:   flush: segment=null docStoreSegment=null
> docStoreOffset=0 flushDocs=false flushDeletes=true
> flushDocStores=false numDocs=0 numBufDelTerms=0
> IW 3 [Root Thread]:   index before flush
> IW 3 [Root Thread]: CMS: now merge
> IW 3 [Root Thread]: CMS:   index:
> IW 3 [Root Thread]: CMS:   no more merges pending; now return
> IW 3 [Root Thread]: now call final commit()
> IW 3 [Root Thread]: startCommit(): start sizeInBytes=0
> IW 3 [Root Thread]: startCommit index= changeCount=1
> IW 3 [Root Thread]: done all syncs
> IW 3 [Root Thread]: commit: pendingCommit != null
> IW 3 [Root Thread]: commit: wrote segments file "segments_2"
> IFD [Root Thread]: now checkpoint "segments_2" [0 segments ; isCommit =
> true]
> IFD [Root Thread]: deleteCommits: now decRef commit "segments_1"
> IFD [Root Thread]: delete "segments_1"
> IW 3 [Root Thread]: commit: done
> IW 3 [Root Thread]: at close:
> 
>    Which is the correct behavior for this method?
>    We changed OJVMDirectory.fileLength() method to returns 0 if no
> file exists instead of throwing an exception and IndexWriter works
> properly, but I am not complete sure if this is the correct
> implementation. Here OJVMDirectory implementation:
>     /**
>      * Returns the length of a file in the directory.
>      */
>     public long fileLength(String name) throws IOException {
>         PreparedStatement stmt = null;
>         ResultSet rs = null;
>         try {
>             stmt =
> 
> conn.prepareStatement(StringUtils.replace(getFileLengthStmt,"%IDX%",thi
> s.prefix));
>             stmt.setString(1, name);
>             rs = stmt.executeQuery();
>             if (rs.next())
>               return rs.getLong(1);
>             else
>               return 0;
>         } catch (SQLException e) {
>             throw new IOException("cannot verify file: " + name +
>                                 ". Reason: " + e.getMessage());
>         } finally {
>             OJVMUtil.closeDbResources(stmt, rs);
>         }
>     }
> 
>    Best regards, Marcelo
> 
> --
> Marcelo F. Ochoa
> http://marceloochoa.blogspot.com/
> http://mochoa.sites.exa.unicen.edu.ar/
> ______________
> Want to integrate Lucene and Oracle?
> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-
> oracle-jvm.html
> Is Oracle 11g REST ready?
> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message