lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Per-Thread DW and IW
Date Tue, 20 Apr 2010 11:27:57 GMT
I don't think an app should be required to manage multiple IWs, when
using multiple threads?

Ie, I think we should still offer the concurrency model we have today
-- you share a single IW across threads, you tell it how much RAM it's
allowed to use, and it has high internal concurrency.

There are elements of IW that still must be "centralized" -- managing
the merge policy/schedulers, deletion policy, writing/committing the
segments files, managing ongoing addIndexes, tracking pending
deletions, the reader pool, etc.

Maybe we can expose invidual control of each indexer (DWPT)... on an
advanced basis?  Ie, for convenience we have a single IW instance, single
RAM buffer, that manages the separate indexers, but advanced apps can
skip this convenience layer and directly control individual indexers?

If we make an "Indexer" interface (marked experimental!), that each
DWPT implements, but also the convenience layer implements, that could
be a clean way to achieve this?  So if an advance apps disagrees with
how the convenience layer manages concurrency (it's thread affinity &
flushing policy based on aggregate RAM used), it could go straight to
individual indexers.

Alternatively, we could make the thread affinity / flush selection
explicitly controllable with a policy?

Also thinking outloud :),


On Tue, Apr 20, 2010 at 1:25 AM, Shai Erera <> wrote:
> Hi
> I've been following the PTDW issue and so far the approach that's been
> proposed and agreed on sounds great! - each thread will have it's own
> DW, DWs will flush themselves independently of each other etc.
> I'm now wondering if we cannot simplify the approach even further by
> eliminating DW at all and let several IW instances open and index over
> the same Directory inside the same JVM (basically allowing multiple
> threads to open their own IW)?
> Would it simplify matters for the currently ongoing issue? Would it
> complicate matters for the app (init'ing IW per thread, controlling
> RAM settings etc.)?
> It will definitely simplify multi-threaded handling for IW extensions
> like Parallel Index …
> One thing we'll need to expose is a shared deleted doc IDs object
> which all IW will update. Anything else?
> I'm just thinking outloud here, hence why I didn't post it on the
> issue itself. What I'm thinking is taking all that per-thread thing
> outside IW and let the app more control. But having IW multi-threaded
> is also a great advantage and more convenient to the app.
> Shai
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message