lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: Getting fsync out of the loop
Date Tue, 06 Apr 2010 15:06:45 GMT
How often is fsync called? If it's just during calls to commit, then is that
that expensive? I mean, how often do you call commit?

If that's that expensive (do you have some numbers to share) then I think
that's be a neat idea. Though "losing a few minutes worth of updates" may
sometimes be unrecoverable, depending on the scenario, bur I guess for those
cases the 'standard way' should be used.

What if your background thread simply committed every couple of minutes?
What's the difference between taking the snapshot (which means you had to
call commit previously) and commit it, to call iw.commit by a backgroud
merge?

Shai

On Tue, Apr 6, 2010 at 5:11 PM, Earwin Burrfoot <earwin@gmail.com> wrote:

> So, I want to pump my IndexWriter hard and fast with documents.
>
> Removing fsync from FSDirectory helps. But for that I pay with possibility
> of
> index corruption, not only if my node suddenly loses
> power/kernelpanics, but also if it
> runs out of disk space (which happens more frequently).
>
> I invented the following solution:
> We write a special deletion policy that resembles SnapshotDeletionPolicy.
> At all times it takes hold of "current synced commit" and preserves
> it. Once every N minutes
> a special thread takes latest commit, syncs it and nominates as
> "current synced commit". The
> previous one gets deleted.
>
> Now we are disastery-proof, and do fsync asynchronously from indexing
> threads. We pay for this with
> somewhat bigger transient disc usage, and probably losing a few
> minutes worth of updates in
> case of a crash, but that's acceptable.
>
> How does this sound?
>
> --
> Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
> Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
> ICQ: 104465785
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message