lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Getting fsync out of the loop
Date Wed, 07 Apr 2010 09:49:05 GMT
On Tue, Apr 6, 2010 at 7:26 PM, Earwin Burrfoot <> wrote:
>> Running out of disk space with fsync disabled won't lead to corruption.
>> Even kill -9 the JRE process with fsync disabled won't corrupt.
>> In these cases index just falls back to last successful commit.
>> It's "only" power loss / OS / machine crash where you need fsync to
>> avoid possible corruption (corruption may not even occur w/o fsync if
>> you "get lucky").
> Sorry to disappoint you, but running out of disk space is worse than kill -9.
> You can write down the file (to cache in fact), close it, all without
> getting any
> exceptions. And then it won't get flushed to disk because the disk is full.
> This can happen to segments file (and old one is deleted with default deletion
> policy). This can happen to fat freq/prox files mentioned in segments file
> (and yeah, the old segments file is deleted, so no falling back).

No, this doesn't make sense.  The OS detects a disk full on accepting
the write into the write cache, not [later] on flushing the write
cache to disk.  If the OS accepts the write, then disk is not full (ie
flushing the cache will succeed, unless some other not-disk-full
problem happens).

Hmmm, at least, normally.  What OS/IO system were you on when you saw
corruption due to disk full when fsync is disabled?

>> What if your background thread simply committed every couple of minutes?
>> What's the difference between taking the snapshot (which means you had
>> to call commit previously) and commit it, to call iw.commit by a backgroud merge?
> --
>> But: why do you need to commit so often?
> To see stuff on reopen? Yes, I know about NRT.
>> You've reinvented autocommit=true!
> ?? I'm doing regular commits, syncing down every Nth of it.
>> Doesn't this just BG the syncing?  Ie you could make a dedicated
>> thread to do this.
> Yes, exactly, this BGs the syncing to a dedicated thread. Threads
> doing indexation/merging can continue unhampered.

OK.  Or you can index with N+1 threads, and each indexer thread does
the commit if it's time...

>> One possible win with this aproach is.... the cost of fsync should go
>> way down the longer you wait after writing bytes to the file and
>> before calling fsync.  This is because typically OS write caches
>> expire by time (eg 30 seconds) so if you want long enough the bytes
>> will already at least be delivered to the IO system (but the IO system
>> can do further caching which could still take time).  On windows at
>> least I definitely noticed this effect -- wait some before fync'ing
>> and it's net/net much less costly.
> Yup. In fact you can just hold on to the latest commit for N seconds,
> than switch to the new latest commit.
> OS will fsync everything for you.

You're mixing up terminology a bit here -- you can't "hold on to the
latest commit then switch to it".  A commit (as sent to the deletion
policy) means a *real* commit (ie, IW.commit or IW.close was called).
So I think your BG thread would simply be calling IW.commit every N

> I'm just playing around with stupid idea. I'd like to have NRT
> look-alike without binding readers and writers. :)

I see... well binding durability & visibility will always be costly.
This is why Lucene decouples them (by making NRT readers available).

> Right now it's probably best for me to save my time and cut over to current NRT.
> But. An important lesson was learnt - no fsyncing blows up your index
> on out-of-disk-space.

I'm still skeptical that disk full even with fsync disabled can lead
to corruption.... I'd like to see some concrete proof :)

BTW, if you know your OS/IO system always persists cached writes w/in
N seconds, a safe way to avoid fsync is to use a by-time expiring
deletion policy.  Ie, a commit stays alive as long as its age is less
than X... DP's unit test has such a policy.  But you better really
know for sure that the OS/IO system guarantee that :)


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message