lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown
Date Sun, 04 Nov 2007 16:23:28 GMT

Well, by calling sync() on every file before closing it (the patch in
LUCENE-1044), we should achieve this, albeit with a possibly sizable
loss of indexing performance (I'm testing that now...).

Though, I still can't figure out how to sync a directory from Java.

In the meantime ... one simple way to be robust to machine/OS crashes
is to keep more than just the last commit point alive in the index.
You just have to create a deletion policy that keeps all commit points
younger than X amount of time (ther is an example of this in Lucene's
TestDeletionPolicy unit test).

This way if the machine crashes and segments_N is not usable you'd
still have segments_N-1 (and maybe segments_N-2, ..., if they are new
enough) to fall back to.

However, I'm not sure how large X would need to be, in practice, for
all write caches to be properly flushed.  And, this will necessarily
use more disk space in your index.

Mike


"Mark Miller" <markrmiller@gmail.com> wrote:
> Even if we cannot guarantee durability, it would be nice if we could 
> guarantee a consistent index. It sounds like the only problem in a 
> machine with a lying drive is that you could lose a number of committed 
> transactions. I would much prefer that to a corrupted index. I can 
> always re-add what was lost much quicker than rebuilding a 5 million doc 
> archive. In either case, I have my choice between the two as long as the 
> index is guaranteed to be corruption free.
> 
> robert engels wrote:
> > Usually you can configure the drives so that sync() ALWAYS syncs - 
> > drive jumpers, driver setup, or other methods. Some drives that are 
> > battery backed and such do not need it.
> >
> > Without sync() truly being a sync you could never write a database 
> > that was resilient.
> >
> > It will exact a heavier toll on performance that you might think. In 
> > order to do it properly, all filesystem metadata must be sync;d as 
> > well.  The biggest difference is that you lose the degree of 
> > multi-processing that is inherent when sync'ing is disabled - as the 
> > drive (or OS) does the physical write asynchronously while the system 
> > does other work - with sync() this is lost.
> >
> > This is why in a db system, the only file that is sync'd is the log 
> > file - all other files can be made "in sync" from the log file - and 
> > this file is normally striped for optimum write performance. Some 
> > systems have special "log file drives" (some even solid state, or 
> > battery backed ram) to aid the performance.
> >
> >
> > On Nov 4, 2007, at 8:30 AM, Yonik Seeley wrote:
> >
> >> On 11/4/07, Michael McCandless <lucene@mikemccandless.com> wrote:
> >>> The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
> >>> apparently future operations may have completed while some past
> >>> operations have not.  For example, the new segments_N file was
> >>> successfully written while say the _X.fdx file of the just-flushed
> >>> segment was not successfully written, even though Lucene had written &
> >>> closed _X.fdx before segments_N.
> >>
> >> That should be impossible except for a machine crash.  Kill -9 or a
> >> JVM crash should have no effect on data already written.
> >>
> >> But a sync option would be both simple and useful for people trying to
> >> take live snapshots of an index, or to protect against machine
> >> crashes.  This isn't an absolute 100% guarantee either (so don't test
> >> for it) - the drives often lie to the OS about data being flushed.
> >> It's the best we can do at our level though.
> >> http://www.google.com/search?q=fsync+drive+lies
> >>
> >> -Yonik
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message