lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown
Date Sun, 04 Nov 2007 15:51:28 GMT
Even if we cannot guarantee durability, it would be nice if we could 
guarantee a consistent index. It sounds like the only problem in a 
machine with a lying drive is that you could lose a number of committed 
transactions. I would much prefer that to a corrupted index. I can 
always re-add what was lost much quicker than rebuilding a 5 million doc 
archive. In either case, I have my choice between the two as long as the 
index is guaranteed to be corruption free.

robert engels wrote:
> Usually you can configure the drives so that sync() ALWAYS syncs - 
> drive jumpers, driver setup, or other methods. Some drives that are 
> battery backed and such do not need it.
>
> Without sync() truly being a sync you could never write a database 
> that was resilient.
>
> It will exact a heavier toll on performance that you might think. In 
> order to do it properly, all filesystem metadata must be sync;d as 
> well.  The biggest difference is that you lose the degree of 
> multi-processing that is inherent when sync'ing is disabled - as the 
> drive (or OS) does the physical write asynchronously while the system 
> does other work - with sync() this is lost.
>
> This is why in a db system, the only file that is sync'd is the log 
> file - all other files can be made "in sync" from the log file - and 
> this file is normally striped for optimum write performance. Some 
> systems have special "log file drives" (some even solid state, or 
> battery backed ram) to aid the performance.
>
>
> On Nov 4, 2007, at 8:30 AM, Yonik Seeley wrote:
>
>> On 11/4/07, Michael McCandless <lucene@mikemccandless.com> wrote:
>>> The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
>>> apparently future operations may have completed while some past
>>> operations have not.  For example, the new segments_N file was
>>> successfully written while say the _X.fdx file of the just-flushed
>>> segment was not successfully written, even though Lucene had written &
>>> closed _X.fdx before segments_N.
>>
>> That should be impossible except for a machine crash.  Kill -9 or a
>> JVM crash should have no effect on data already written.
>>
>> But a sync option would be both simple and useful for people trying to
>> take live snapshots of an index, or to protect against machine
>> crashes.  This isn't an absolute 100% guarantee either (so don't test
>> for it) - the drives often lie to the OS about data being flushed.
>> It's the best we can do at our level though.
>> http://www.google.com/search?q=fsync+drive+lies
>>
>> -Yonik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message