lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Getting fsync out of the loop
Date Tue, 06 Apr 2010 18:58:36 GMT
On Tue, Apr 6, 2010 at 10:11 AM, Earwin Burrfoot <> wrote:

> So, I want to pump my IndexWriter hard and fast with documents.


> Removing fsync from FSDirectory helps. But for that I pay with possibility of
> index corruption, not only if my node suddenly loses
> power/kernelpanics, but also if it
> runs out of disk space (which happens more frequently).

Running out of disk space with fsync disabled won't lead to corruption.

Even kill -9 the JRE process with fsync disabled won't corrupt.

In these cases index just falls back to last successful commit.

It's "only" power loss / OS / machine crash where you need fsync to
avoid possible corruption (corruption may not even occur w/o fsync if
you "get lucky").

But: why do you need to commit so often?

> I invented the following solution:
> We write a special deletion policy that resembles SnapshotDeletionPolicy.
> At all times it takes hold of "current synced commit" and preserves
> it. Once every N minutes
> a special thread takes latest commit, syncs it and nominates as
> "current synced commit". The
> previous one gets deleted.
> Now we are disastery-proof, and do fsync asynchronously from indexing
> threads. We pay for this with
> somewhat bigger transient disc usage, and probably losing a few
> minutes worth of updates in
> case of a crash, but that's acceptable.
> How does this sound?

You've reinvented autocommit=true!

Doesn't this just BG the syncing?  Ie you could make a dedicated
thread to do this.

One possible win with this aproach is.... the cost of fsync should go
way down the longer you wait after writing bytes to the file and
before calling fsync.  This is because typically OS write caches
expire by time (eg 30 seconds) so if you want long enough the bytes
will already at least be delivered to the IO system (but the IO system
can do further caching which could still take time).  On windows at
least I definitely noticed this effect -- wait some before fync'ing
and it's net/net much less costly.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message