lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Getting fsync out of the loop
Date Tue, 06 Apr 2010 14:11:53 GMT
So, I want to pump my IndexWriter hard and fast with documents.

Removing fsync from FSDirectory helps. But for that I pay with possibility of
index corruption, not only if my node suddenly loses
power/kernelpanics, but also if it
runs out of disk space (which happens more frequently).

I invented the following solution:
We write a special deletion policy that resembles SnapshotDeletionPolicy.
At all times it takes hold of "current synced commit" and preserves
it. Once every N minutes
a special thread takes latest commit, syncs it and nominates as
"current synced commit". The
previous one gets deleted.

Now we are disastery-proof, and do fsync asynchronously from indexing
threads. We pay for this with
somewhat bigger transient disc usage, and probably losing a few
minutes worth of updates in
case of a crash, but that's acceptable.

How does this sound?

-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message