lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Getting fsync out of the loop
Date Fri, 09 Apr 2010 00:05:42 GMT
On Thu, Apr 8, 2010 at 6:21 PM, Earwin Burrfoot <earwin@gmail.com> wrote:
>> But, IW doesn't let you "hold on to" checkpoints... only to commits.
>>
>> Ie SnapshotDP will only "see" actual commit/close calls, not
>> intermediate checkpoints like a random segment merge completing, a
>> flush happening, etc.
>>
>> Or... maybe you would in fact call commit frequently from the main
>> threads (but with fsync disabled), and then your DP holds onto these
>> "fake commits", periodically picking one of them to do the "real"
>> fsync ing?
> Yeah, that's exactly what I tried to describe in my initial post :)

Ahh ok then it makes more sense.  But still you shouldn't commit that
often (even with fake fsync) since it must flush the segment.

>>>>> I'm just playing around with stupid idea. I'd like to have NRT
>>>>> look-alike without binding readers and writers. :)
>>>> I see... well binding durability & visibility will always be costly.
>>>> This is why Lucene decouples them (by making NRT readers available).
>>> My experiments do the same, essentially.
>>> But after I understood that to perform deletions IW has to load term indexes
>>> anyway, I'm almost ready to give up and go for intertwined IW/IR mess :)
>> Hey if you really think it's a mess, post a patch that cleans it up :)
> Uh oh. Let me finish current one, first.

Heh, yes :)

> Second - I don't know yet how
> this should look like.
> Something along the lines of deletions/norms writers being extracted
> from segment reader
> and reader pool being made external to IW??

Yeah, reader pool should be pulled out of IW, and I think IW should be
split into "that which manages the segment infos", "that which
adds/deletes docs", and "the rest" (merging, addIndexes*)?  (There's
an issue open for this refactoring...).

I'm not sure about deletions/norms writers being extracted from SR....
I think delete ops would still go through IW?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message