lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown
Date Sun, 04 Nov 2007 15:49:48 GMT

"Yonik Seeley" <yonik@apache.org> wrote:

> On 11/4/07, Michael McCandless <lucene@mikemccandless.com> wrote:
> > The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
> > apparently future operations may have completed while some past
> > operations have not.  For example, the new segments_N file was
> > successfully written while say the _X.fdx file of the just-flushed
> > segment was not successfully written, even though Lucene had written &
> > closed _X.fdx before segments_N.
> 
> That should be impossible except for a machine crash.  Kill -9 or a
> JVM crash should have no effect on data already written.

OK, right.  JVM crashing or getting killed should preserve
order-of-completion on the IO operations: those IO operations that
were handed off to the OS will eventually complete successfully.

But OS crashing, machine crashing or power-cord gets pulled can result
in out-of-order completion of IO operations, which is what can corrupt
the index.

> But a sync option would be both simple and useful for people trying to
> take live snapshots of an index, or to protect against machine
> crashes.  This isn't an absolute 100% guarantee either (so don't test
> for it) - the drives often lie to the OS about data being flushed.
> It's the best we can do at our level though.
> http://www.google.com/search?q=fsync+drive+lies

Right, the best the OS can do is get all writes out to the drives, but
if the drives then cache the writes (in non-stable storage) then we
are still at risk.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message