lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <>
Subject Re: Getting fsync out of the loop
Date Wed, 07 Apr 2010 19:27:27 GMT
> No, this doesn't make sense.  The OS detects a disk full on accepting
> the write into the write cache, not [later] on flushing the write
> cache to disk.  If the OS accepts the write, then disk is not full (ie
> flushing the cache will succeed, unless some other not-disk-full
> problem happens).
> Hmmm, at least, normally.  What OS/IO system were you on when you saw
> corruption due to disk full when fsync is disabled?
> I'm still skeptical that disk full even with fsync disabled can lead
> to corruption.... I'd like to see some concrete proof :)

Linux 2.6.30-1-amd64, ext3, simple scsi drive
I checked with our resident DB brainiac, he says such things are possible.

Okay, I'm not 100% sure this is the cause of my corruptions. It just happened
that when the index got corrupted, disk space was also used up - several times.
I had that silent-fail-to-write theory and checked it up with some knowledgeable
people. Even if they are right, I can be mistaken and the root cause
is different.

> You're mixing up terminology a bit here -- you can't "hold on to the
> latest commit then switch to it".  A commit (as sent to the deletion
> policy) means a *real* commit (ie, IW.commit or IW.close was called).
> So I think your BG thread would simply be calling IW.commit every N
> seconds?
Under "hold on to" I meant - keep from being deleted, like SnapshotDP does.

>> I'm just playing around with stupid idea. I'd like to have NRT
>> look-alike without binding readers and writers. :)
> I see... well binding durability & visibility will always be costly.
> This is why Lucene decouples them (by making NRT readers available).
My experiments do the same, essentially.

But after I understood that to perform deletions IW has to load term indexes
anyway, I'm almost ready to give up and go for intertwined IW/IR mess :)

> BTW, if you know your OS/IO system always persists cached writes w/in
> N seconds, a safe way to avoid fsync is to use a by-time expiring
> deletion policy.  Ie, a commit stays alive as long as its age is less
> than X... DP's unit test has such a policy.  But you better really
> know for sure that the OS/IO system guarantee that :)
Yeah. I thought of it, but it is even more shady :)

Kirill Zakharenko/Кирилл Захаренко (
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message