lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: FSDirectory patch for file renaming
Date Mon, 24 Feb 2003 15:41:24 GMT
Yeah, I already changed that.  Thanks for checking!

Otis

--- Scott Ganyo <scott.ganyo@etapestry.com> wrote:
> I'm ok with this, given the lack of a better option.  Please ensure
> that 
> the input and output streams are closed in a finally block, though.
> 
> Scott
> 
> Otis Gospodnetic wrote:
> 
> >I hate having to make the suggested change for the reasons Matt
> >mentioned as disadvantages below....but what is one to do?
> >I'll change the code shortly.
> >
> >If any other developers are against this change, speak up.
> >
> >Otis
> >
> >
> >--- Matt Tucker <matt@jivesoftware.com> wrote:
> >  
> >
> >>Hello all,
> >>
> >>For quite some time, we've been experiencing intermittent indexing
> >>failures
> >>that look similar to the following:
> >>
> >>java.io.IOException: couldn't rename segments.new to segments 
> >>     at
> >>
> >>    
> >>
>
>org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
> >  
> >
> >>String)V(Unknown Source) 
> >>     at
> >>
> >>    
> >>
>
>org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
> >  
> >
> >>y)V(Unknown Source)
> >>
> >>This is caused by File.renameTo failing. From previous run-ins with
> >>that
> >>method, it's actually fairly flaky, especially under various
> Windows
> >>JVMs.
> >>The fix is pretty easy -- if File.renameTo fails, manually copy the
> >>file
> >>from the old to new destination.
> >>
> >>Old code (FSDirectory.renameFile): 
> >>
> >>------------------------------
> >>    if (!old.renameTo(nu))
> >>      throw new IOException("couldn't rename " + from + " to " +
> to);
> >>------------------------------
> >>
> >>New code ("buf" variable is declared at top of file as null):
> >>
> >>------------------------------
> >>    // Rename the old file to the new one. Unfortunately, the
> >>renameTo()
> >>method doesn't
> >>    // work reliably under some JVMs. Therefore, if the rename
> fails,
> >>we
> >>manually rename by
> >>    // copying the old file to the new one.
> >>    if (!old.renameTo(nu)) {
> >>      try {
> >>        java.io.InputStream in = new FileInputStream(old);
> >>        java.io.OutputStream out = new FileOutputStream(nu);
> >>        int len;
> >>        // See if the buf needs to be initialized. Initialization
> is
> >>only
> >>done on-demand
> >>        // since many VM's will never run into the renameTo bug and
> >>hence
> >>shouldn't waste
> >>        // 1K of mem for no reason.
> >>        if (buf == null) {
> >>          buf = new byte[1024];
> >>        }
> >>        while ((len = in.read(buf)) >= 0) {
> >>           out.write(buf, 0, len);
> >>        }
> >>        in.close();
> >>        out.close();
> >>
> >>        // Delete the old file.
> >>        old.delete();
> >>      }
> >>      catch (IOException ioe) {
> >>        throw new IOException("couldn't rename " + from + " to " +
> >>to);
> >>      }
> >>    }
> >>------------------------------
> >>
> >>Advantages to this fix:
> >>
> >> * Makes indexing more reliable, especially under VM's where the
> >>renameTo
> >>method doesn't always work.
> >> * Uses no additional memory or resources when File.renameTo works
> >>normally.
> >>
> >>Disadvantages:
> >>
> >> * It always sucks to add workarounds to bugs in other libraries,
> but
> >>what
> >>is one to do?
> >> * We've run into File.renameTo issues on multiple machines and VM
> >>versions,
> >>but it would be good if others could confirm they've had problems
> as
> >>well. 
> >>
> >>I'd suggest this fix as a good candidate for Lucene 1.3 unless
> others
> >>see
> >>problems with it.
> >>
> >>Regards,
> >>Matt
> >>
> >>
>
>>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> >>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >>
> >>    
> >>
> >
> >
> >__________________________________________________
> >Do you Yahoo!?
> >Yahoo! Tax Center - forms, calculators, tips, more
> >http://taxes.yahoo.com/
> >
>
>---------------------------------------------------------------------
> >To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >  
> >
> 
> -- 
> "Perfection in design is not achieved when there is nothing more to
> add, but when there is nothing more to remove." - Antoine de
> Saint-Exupéry
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 


__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message