lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Serebrennikov <dmit...@earthlink.net>
Subject Re: FSDirectory patch for file renaming
Date Tue, 18 Feb 2003 00:11:33 GMT
Matt Tucker wrote:

>Otis,
>
>Yesterday, we had a data set on a particular Windows box that would
>consistently bomb out with the rename error. That's what finally motivated
>me to make the patch. After applying the patch the rebuild worked perfectly.
>I also removed the renameTo method call and tested with my manual copy code
>being used all the time (that also worked perfectly).
>
>The first time we ran into File.renameTo issues was with the config files in
>our product, Jive Forums. The algorithm was:
>
> * Write out new config file as jive_config.tmp
> * Delete old jive_config.xml
> * Rename jive_config.tmp to jive_config.xml using File.renameTo
>
>The rename would bomb out about once very two months for some customers, and
>much more frequently for others. At that point, we wrote a small testing app
>that performed renames again and again, and we were easily able to see the
>error consistently. Since we added the manual rename logic to our own code,
>we haven't seen a problem.
>
>The fact that another user had the error and bothered to email the list
>indicates to me that at least a few people are seeing the issue. Therefore,
>applying the patch seems like a very good idea, especially since it won't
>affect users that don't run into File.renameTo issues at all.
>
We've also seen errors related to the rename failures, especially when 
the Windows machine was heavily loaded (a lot of virtual memory use, 
swapping, etc.). Java bug database has numerous reports about this issue 
as well, and the general resolution (last time I looked at these bugs) 
was "too bad". So, I would agree that the patch makes sense, as long as 
it does not result in strange behavior when the error condition is 
legitimate.

>
>Regards,
>Matt
>
>  
>
>>-----Original Message-----
>>From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
>>Sent: Monday, February 17, 2003 3:19 PM
>>To: Lucene Developers List
>>Subject: Re: FSDirectory patch for file renaming
>>
>>
>>Hello Matt,
>>
>>Is there any way you can make this reproducable consistenly? 
>>You said renamaTo is 'fairly flaky' - can you make something 
>>that will always fail? I have never seen that fail, but I 
>>rarely use (Lucene on) Winblows.
>>
>>Adding your patch would not be a big problem, but I'd be much 
>>happier if we could find out what's causing this to fail. 
>>Does this ever fail in a single-threaded application? (yes, 
>>Lucene's locks and the fact renameFile method in FSDirectory 
>>is synchronized should prevent multiple threads from stepping 
>>at each other toes,
>>but...)
>>
>>Do other developers have suggestions and/or opionions regarding this?
>>
>>Thanks,
>>Otis
>>
>>
>>--- Matt Tucker <matt@jivesoftware.com> wrote:
>>    
>>
>>>Hello all,
>>>
>>>For quite some time, we've been experiencing intermittent indexing 
>>>failures that look similar to the following:
>>>
>>>java.io.IOException: couldn't rename segments.new to segments 
>>>     at
>>>
>>>      
>>>
>>org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.Stri
>>ng;Ljava.lang.
>>    
>>
>>>String)V(Unknown Source) 
>>>     at
>>>
>>>      
>>>
>>org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.
>>store.Director
>>    
>>
>>>y)V(Unknown Source)
>>>
>>>This is caused by File.renameTo failing. From previous run-ins with 
>>>that method, it's actually fairly flaky, especially under various 
>>>Windows JVMs.
>>>The fix is pretty easy -- if File.renameTo fails, manually copy the
>>>file
>>>from the old to new destination.
>>>
>>>Old code (FSDirectory.renameFile):
>>>
>>>------------------------------
>>>    if (!old.renameTo(nu))
>>>      throw new IOException("couldn't rename " + from + " 
>>>      
>>>
>>to " + to);
>>    
>>
>>>------------------------------
>>>
>>>New code ("buf" variable is declared at top of file as null):
>>>
>>>------------------------------
>>>    // Rename the old file to the new one. Unfortunately, the
>>>renameTo()
>>>method doesn't
>>>    // work reliably under some JVMs. Therefore, if the 
>>>      
>>>
>>rename fails, 
>>    
>>
>>>we manually rename by
>>>    // copying the old file to the new one.
>>>    if (!old.renameTo(nu)) {
>>>      try {
>>>        java.io.InputStream in = new FileInputStream(old);
>>>        java.io.OutputStream out = new FileOutputStream(nu);
>>>        int len;
>>>        // See if the buf needs to be initialized. Initialization is
>>>only
>>>done on-demand
>>>        // since many VM's will never run into the renameTo bug and
>>>hence
>>>shouldn't waste
>>>        // 1K of mem for no reason.
>>>        if (buf == null) {
>>>          buf = new byte[1024];
>>>        }
>>>        while ((len = in.read(buf)) >= 0) {
>>>           out.write(buf, 0, len);
>>>        }
>>>        in.close();
>>>        out.close();
>>>
>>>        // Delete the old file.
>>>        old.delete();
>>>      }
>>>      catch (IOException ioe) {
>>>        throw new IOException("couldn't rename " + from + " to " + 
>>>to);
>>>      }
>>>    }
>>>------------------------------
>>>
>>>Advantages to this fix:
>>>
>>> * Makes indexing more reliable, especially under VM's where the 
>>>renameTo method doesn't always work.
>>> * Uses no additional memory or resources when File.renameTo works
>>>normally.
>>>
>>>Disadvantages:
>>>
>>> * It always sucks to add workarounds to bugs in other 
>>>      
>>>
>>libraries, but 
>>    
>>
>>>what is one to do?
>>> * We've run into File.renameTo issues on multiple machines and VM
>>>versions,
>>>but it would be good if others could confirm they've had problems as
>>>well. 
>>>
>>>I'd suggest this fix as a good candidate for Lucene 1.3 
>>>      
>>>
>>unless others 
>>    
>>
>>>see problems with it.
>>>
>>>Regards,
>>>Matt
>>>
>>>
>>>
>>>      
>>>
>>---------------------------------------------------------------------
>>    
>>
>>>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>      
>>>
>>__________________________________________________
>>Do you Yahoo!?
>>Yahoo! Shopping - Send Flowers for Valentine's Day 
>>    
>>
>http://shopping.yahoo.com
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
>  
>



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message