lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Ganyo <scott.ga...@etapestry.com>
Subject Re: FSDirectory patch for file renaming
Date Mon, 24 Feb 2003 14:55:53 GMT
I'm ok with this, given the lack of a better option.  Please ensure that 
the input and output streams are closed in a finally block, though.

Scott

Otis Gospodnetic wrote:

>I hate having to make the suggested change for the reasons Matt
>mentioned as disadvantages below....but what is one to do?
>I'll change the code shortly.
>
>If any other developers are against this change, speak up.
>
>Otis
>
>
>--- Matt Tucker <matt@jivesoftware.com> wrote:
>  
>
>>Hello all,
>>
>>For quite some time, we've been experiencing intermittent indexing
>>failures
>>that look similar to the following:
>>
>>java.io.IOException: couldn't rename segments.new to segments 
>>     at
>>
>>    
>>
>org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
>  
>
>>String)V(Unknown Source) 
>>     at
>>
>>    
>>
>org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
>  
>
>>y)V(Unknown Source)
>>
>>This is caused by File.renameTo failing. From previous run-ins with
>>that
>>method, it's actually fairly flaky, especially under various Windows
>>JVMs.
>>The fix is pretty easy -- if File.renameTo fails, manually copy the
>>file
>>from the old to new destination.
>>
>>Old code (FSDirectory.renameFile): 
>>
>>------------------------------
>>    if (!old.renameTo(nu))
>>      throw new IOException("couldn't rename " + from + " to " + to);
>>------------------------------
>>
>>New code ("buf" variable is declared at top of file as null):
>>
>>------------------------------
>>    // Rename the old file to the new one. Unfortunately, the
>>renameTo()
>>method doesn't
>>    // work reliably under some JVMs. Therefore, if the rename fails,
>>we
>>manually rename by
>>    // copying the old file to the new one.
>>    if (!old.renameTo(nu)) {
>>      try {
>>        java.io.InputStream in = new FileInputStream(old);
>>        java.io.OutputStream out = new FileOutputStream(nu);
>>        int len;
>>        // See if the buf needs to be initialized. Initialization is
>>only
>>done on-demand
>>        // since many VM's will never run into the renameTo bug and
>>hence
>>shouldn't waste
>>        // 1K of mem for no reason.
>>        if (buf == null) {
>>          buf = new byte[1024];
>>        }
>>        while ((len = in.read(buf)) >= 0) {
>>           out.write(buf, 0, len);
>>        }
>>        in.close();
>>        out.close();
>>
>>        // Delete the old file.
>>        old.delete();
>>      }
>>      catch (IOException ioe) {
>>        throw new IOException("couldn't rename " + from + " to " +
>>to);
>>      }
>>    }
>>------------------------------
>>
>>Advantages to this fix:
>>
>> * Makes indexing more reliable, especially under VM's where the
>>renameTo
>>method doesn't always work.
>> * Uses no additional memory or resources when File.renameTo works
>>normally.
>>
>>Disadvantages:
>>
>> * It always sucks to add workarounds to bugs in other libraries, but
>>what
>>is one to do?
>> * We've run into File.renameTo issues on multiple machines and VM
>>versions,
>>but it would be good if others could confirm they've had problems as
>>well. 
>>
>>I'd suggest this fix as a good candidate for Lucene 1.3 unless others
>>see
>>problems with it.
>>
>>Regards,
>>Matt
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>>    
>>
>
>
>__________________________________________________
>Do you Yahoo!?
>Yahoo! Tax Center - forms, calculators, tips, more
>http://taxes.yahoo.com/
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>  
>

-- 
"Perfection in design is not achieved when there is nothing more to add, but when there is
nothing more to remove." - Antoine de Saint-Exupéry



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message