lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <>
Subject Re: Per-Thread DW and IW
Date Wed, 21 Apr 2010 06:27:17 GMT
I'm not sure that a Parallel DW would work for PI because DW is too internal
to IW. Currently, the approach I've been thinking about for PI is to tackle
it from a high level, e.g. allow the application to pass a Directory, or
even an IW instance, and PI will play the coordinator role, ensuring that
merge of segments happens across all the slices in accordance, implementing
two-phase operations etc. A Parallel DW then does not fit nicely w/ that
approach (unless we want to refactor how IW works completely) because DW is
not aware of the Directory, and if PI indeed works over IW instances, then
each will have its own DW.

So there are two basic approaches we can take for PI (following current
architecture) - either let PI manage IW, or have PI a sort of IW itself,
which handles events at a much lower level. While the latter is more robust
(and based on current limitations I'm running into, might be even easier to
do), it lacks the flexibility of allowing the app to plug any IW it wants.
That requirement is also important, if the application wants to use PI in
scenarios where it keeps some slices in RAM and some on disk, or it wants to
control more closely which fields go to which slice, so that it can at some
point in time "rebuild" a certain slice outside PI and replace the existing
slice in PI w/ the new one ...

We should probably continue the discussion on PI, so I suggest we either
move it to another thread or on the issue directly.

Mike - I agree w/ you that we should keep the life of the application
developers easy and that having IW itself support concurrency is beneficial.
Like I said ... it was just a thought which was aimed at keeping our life
(Lucene developers) easier, but that probably comes second compared to
app-devs life :). I'm not at all sure also that that would have make our
life easier ...

So I'm good if you want to drop the discussion.


On Tue, Apr 20, 2010 at 8:16 PM, Michael Busch <> wrote:

> On 4/19/10 10:25 PM, Shai Erera wrote:
>> It will definitely simplify multi-threaded handling for IW extensions
>> like Parallel Index …
> I'm keeping Parallel indexing in mind.  After we have separate DWPT I'd
> like to introduce parallel DWPTs, that write different slices.
>  Synchronization should not be a big worry then, because writing is
> single-threaded.
> We could introduce a new abstract class SegmentWriter, which DWPT would
> implement.  An extension would be ParallelSegmentWriter, which would manage
> multiple SegmentWriters.   Or maybe SegmentSliceWriter would be a better
> name.
>  Michael
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message