lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lock-less commits
Date Sun, 20 Aug 2006 15:24:49 GMT
Yonik Seeley wrote:
> On 8/18/06, Michael McCandless <lucene@mikemccandless.com> wrote:
>> > One would also have to worry about partially deleted segments on
>> > Windows... while removing a segment, some of the files might fail to
>> > delete (due to still being open) and some might succeed.
>>
>> Yes, I think this case is handled correctly.  Once all searchers using
>> those old segments are closed, then the next commit that runs will
>> remove those files (just like it does today).
> 
> Unix systems don't have to worry about this.
> Windows systems use "deletable" to track what they should try to delete 
> later.
> How are you handling it?  Get a full directory listing and try to
> remove any older segment files?
> 
> I agree with the benefits of not requiring locking to open an
> IndexReader, but I wonder at the performance cost of the directory
> listing needed to "garbage collect" old segment files and to find the
> newest "segments.xxx"

Agreed, it looks like the big tradeoff here is possible performance
loss (due to directory listings & IndexReader having to retry) vs.
better robustness with a lock-less design.

I'm planning on doing benchmarks to measure net system throughput
(#docs indexed, #queries run) across different OS's, remote/local
filesystems, and across "high frequency" to "low frequency" of
re-opening the IndexSearchers.

The good news (in my testing so far) is that the lock-less design is
functionally correct.

On deletable: yes, I'm currently GC'ing unused segments by doing a
full directory listing.  But, this is actually a separable change:
reading/writing the deletable file only requires the write lock, so I
could keep the current approach.  I was wanting to avoid the
File.renameTo deletable.new -> deletable which has hit "Access Denied"
on Windows in the past.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message