lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <>
Subject Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown
Date Sun, 04 Nov 2007 10:21:34 GMT

I'll pull together a patch to allow FSDirectory/FSIndexOutput to
optionally do flush() & sync() before every close().

Lucene is already "atomic" (the "A" in ACID): it's very careful to do
the IO operations in order such that the index is never in an
inconsistent state ASSUMING the IO system completes or
fails-to-complete the operations *in order*.

The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
apparently future operations may have completed while some past
operations have not.  For example, the new segments_N file was
successfully written while say the _X.fdx file of the just-flushed
segment was not successfully written, even though Lucene had written &
closed _X.fdx before segments_N.

It's this out-of-order completion of the IO operations on a hard
shutdown that leads to an inconsistent index.

Using autoCommit=false just means your commits are less frequent, and,
the "atomic" transaction is all changes done during the lifetime of
that one writer, versus all changes done since the last flush() when
autoCommit=true.  However, autoCommit=false cannot fully eliminate the
chance of corruption due to out-of-order completion of IO operations.

It sounds like inserting a flush() then sync() call before every
close() *might* in fact force the IO system to attain in-order
completion of the operations Lucene sends it, at the cost of some
performance loss.  I say *might* because there are so many layers to
an IO system that it's not clear that the fsync() that the JVM is in
fact calling (and relying on) will really always do the right thing.

The performance hit could in practice be low, especially if you are
using a large RAM buffer in your writer.


"Mark Miller" <> wrote:
> I think it would be great if it where an option. Perhaps even a sandbox 
> implementation that could wrap or replace a few classes. Maybe that 
> complicates things too much, but that way you could just not use the 
> transaction system if you where on NFS (if that ends up being a problem) 
> or you didn't want to pay a performance cost.
> Maybe a Derby guy would pitch in.
> Chris Hostetter wrote:
> > : This is simple not true. See FileDescriptor.sync().
> > : 
> > : There are several options, but normally it is used so that when close
> > : completes, all data must be on disk. This is a much slower way to write data.
> > : It is very common in database systems when committing the log file.
> >
> > Ok.  I'll certainly take your word for it ... i've been trusting the docs 
> > for [File]OutputStream.flush()...
> >
> >   
> >>> If the intended destination of this stream is an abstraction provided 
> >>> by the underlying operating system, for example a file, then flushing 
> >>> the stream guarantees only that bytes previously written to the stream 
> >>> are passed to the operating system for writing; it does not guarantee 
> >>> that they are actually written to a physical device such as a disk drive.
> >>>       
> >
> > I haven't looked at the internals of FileOutputStream or FileDescriptor on 
> > any particular platforms to see how exactly they work, but if dealing with 
> > the FD directly and using FD.sync() the magic bullet then I'd love to see 
> > a patch that uses it in FSDirectory.
> >
> > I assume the SyncFailedException it throws is rare?  If it is always 
> > thrown when using things like NFS that may be a show stopper for using 
> > sync() in Lucene ... many people have jumped through a lot of hoops this 
> > past year to get Lucene working on NFS; I'd hate to see all that work go 
> > out the window in an effort to make Lucene ACID.  (I suspect there are 
> > more users interested in using Lucene on NFS then on using it as a 
> > transactional data store)
> >
> >   
> >>> Throws:
> >>>    SyncFailedException - Thrown when the buffers cannot be flushed, or 
> >>> because the system cannot guarantee that all the buffers have been 
> >>> synchronized with physical media.
> >>>       
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > For additional commands, e-mail:
> >
> >
> >   
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message