lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Tucker" <m...@jivesoftware.com>
Subject FSDirectory patch for file renaming
Date Sun, 16 Feb 2003 22:35:49 GMT
Hello all,

For quite some time, we've been experiencing intermittent indexing failures
that look similar to the following:

java.io.IOException: couldn't rename segments.new to segments 
     at
org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
String)V(Unknown Source) 
     at
org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
y)V(Unknown Source)

This is caused by File.renameTo failing. From previous run-ins with that
method, it's actually fairly flaky, especially under various Windows JVMs.
The fix is pretty easy -- if File.renameTo fails, manually copy the file
from the old to new destination.

Old code (FSDirectory.renameFile): 

------------------------------
    if (!old.renameTo(nu))
      throw new IOException("couldn't rename " + from + " to " + to);
------------------------------

New code ("buf" variable is declared at top of file as null):

------------------------------
    // Rename the old file to the new one. Unfortunately, the renameTo()
method doesn't
    // work reliably under some JVMs. Therefore, if the rename fails, we
manually rename by
    // copying the old file to the new one.
    if (!old.renameTo(nu)) {
      try {
        java.io.InputStream in = new FileInputStream(old);
        java.io.OutputStream out = new FileOutputStream(nu);
        int len;
        // See if the buf needs to be initialized. Initialization is only
done on-demand
        // since many VM's will never run into the renameTo bug and hence
shouldn't waste
        // 1K of mem for no reason.
        if (buf == null) {
          buf = new byte[1024];
        }
        while ((len = in.read(buf)) >= 0) {
           out.write(buf, 0, len);
        }
        in.close();
        out.close();

        // Delete the old file.
        old.delete();
      }
      catch (IOException ioe) {
        throw new IOException("couldn't rename " + from + " to " + to);
      }
    }
------------------------------

Advantages to this fix:

 * Makes indexing more reliable, especially under VM's where the renameTo
method doesn't always work.
 * Uses no additional memory or resources when File.renameTo works normally.

Disadvantages:

 * It always sucks to add workarounds to bugs in other libraries, but what
is one to do?
 * We've run into File.renameTo issues on multiple machines and VM versions,
but it would be good if others could confirm they've had problems as well. 

I'd suggest this fix as a good candidate for Lucene 1.3 unless others see
problems with it.

Regards,
Matt


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message