lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown
Date Sun, 04 Nov 2007 15:37:59 GMT
Usually you can configure the drives so that sync() ALWAYS syncs -  
drive jumpers, driver setup, or other methods. Some drives that are  
battery backed and such do not need it.

Without sync() truly being a sync you could never write a database  
that was resilient.

It will exact a heavier toll on performance that you might think. In  
order to do it properly, all filesystem metadata must be sync;d as  
well.  The biggest difference is that you lose the degree of multi- 
processing that is inherent when sync'ing is disabled - as the drive  
(or OS) does the physical write asynchronously while the system does  
other work - with sync() this is lost.

This is why in a db system, the only file that is sync'd is the log  
file - all other files can be made "in sync" from the log file - and  
this file is normally striped for optimum write performance. Some  
systems have special "log file drives" (some even solid state, or  
battery backed ram) to aid the performance.


On Nov 4, 2007, at 8:30 AM, Yonik Seeley wrote:

> On 11/4/07, Michael McCandless <lucene@mikemccandless.com> wrote:
>> The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
>> apparently future operations may have completed while some past
>> operations have not.  For example, the new segments_N file was
>> successfully written while say the _X.fdx file of the just-flushed
>> segment was not successfully written, even though Lucene had  
>> written &
>> closed _X.fdx before segments_N.
>
> That should be impossible except for a machine crash.  Kill -9 or a
> JVM crash should have no effect on data already written.
>
> But a sync option would be both simple and useful for people trying to
> take live snapshots of an index, or to protect against machine
> crashes.  This isn't an absolute 100% guarantee either (so don't test
> for it) - the drives often lie to the OS about data being flushed.
> It's the best we can do at our level though.
> http://www.google.com/search?q=fsync+drive+lies
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message