lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: FSDirectory patch for file renaming
Date Mon, 17 Feb 2003 20:18:33 GMT
Hello Matt,

Is there any way you can make this reproducable consistenly?
You said renamaTo is 'fairly flaky' - can you make something that will
always fail?
I have never seen that fail, but I rarely use (Lucene on) Winblows.

Adding your patch would not be a big problem, but I'd be much happier
if we could find out what's causing this to fail.
Does this ever fail in a single-threaded application? (yes, Lucene's
locks and the fact renameFile method in FSDirectory is synchronized
should prevent multiple threads from stepping at each other toes,
but...)

Do other developers have suggestions and/or opionions regarding this?

Thanks,
Otis


--- Matt Tucker <matt@jivesoftware.com> wrote:
> Hello all,
> 
> For quite some time, we've been experiencing intermittent indexing
> failures
> that look similar to the following:
> 
> java.io.IOException: couldn't rename segments.new to segments 
>      at
>
org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
> String)V(Unknown Source) 
>      at
>
org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
> y)V(Unknown Source)
> 
> This is caused by File.renameTo failing. From previous run-ins with
> that
> method, it's actually fairly flaky, especially under various Windows
> JVMs.
> The fix is pretty easy -- if File.renameTo fails, manually copy the
> file
> from the old to new destination.
> 
> Old code (FSDirectory.renameFile): 
> 
> ------------------------------
>     if (!old.renameTo(nu))
>       throw new IOException("couldn't rename " + from + " to " + to);
> ------------------------------
> 
> New code ("buf" variable is declared at top of file as null):
> 
> ------------------------------
>     // Rename the old file to the new one. Unfortunately, the
> renameTo()
> method doesn't
>     // work reliably under some JVMs. Therefore, if the rename fails,
> we
> manually rename by
>     // copying the old file to the new one.
>     if (!old.renameTo(nu)) {
>       try {
>         java.io.InputStream in = new FileInputStream(old);
>         java.io.OutputStream out = new FileOutputStream(nu);
>         int len;
>         // See if the buf needs to be initialized. Initialization is
> only
> done on-demand
>         // since many VM's will never run into the renameTo bug and
> hence
> shouldn't waste
>         // 1K of mem for no reason.
>         if (buf == null) {
>           buf = new byte[1024];
>         }
>         while ((len = in.read(buf)) >= 0) {
>            out.write(buf, 0, len);
>         }
>         in.close();
>         out.close();
> 
>         // Delete the old file.
>         old.delete();
>       }
>       catch (IOException ioe) {
>         throw new IOException("couldn't rename " + from + " to " +
> to);
>       }
>     }
> ------------------------------
> 
> Advantages to this fix:
> 
>  * Makes indexing more reliable, especially under VM's where the
> renameTo
> method doesn't always work.
>  * Uses no additional memory or resources when File.renameTo works
> normally.
> 
> Disadvantages:
> 
>  * It always sucks to add workarounds to bugs in other libraries, but
> what
> is one to do?
>  * We've run into File.renameTo issues on multiple machines and VM
> versions,
> but it would be good if others could confirm they've had problems as
> well. 
> 
> I'd suggest this fix as a good candidate for Lucene 1.3 unless others
> see
> problems with it.
> 
> Regards,
> Matt
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 


__________________________________________________
Do you Yahoo!?
Yahoo! Shopping - Send Flowers for Valentine's Day
http://shopping.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message