lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Commented: (LUCENE-1879) Parallel incremental indexing
Date Thu, 04 Mar 2010 06:18:28 GMT


Shai Erera commented on LUCENE-1879:

(Warning, this post is long, and is easier to read in JIRA)

I've investigated the attached code a lot and I'd like to propose a different design and approach
to this whole Parallel Index solution. I'll start by describing the limitations of the current
design (whether its the approach or the code is debatable):
* Lucene is not built/designed properly to a Master/Slave architecture, where different indexes
share important files with others (such as segments_N, segments.gen and .del).
** I've realized this when I found that if tests (in this patch) are run with "-ea", there
are many assert exceptions that are printed from IndexWriter.startCommit. The reason is the
Master just updated one of the segments .del generation (and deleted the previous one), but
the Slave is not aware of that yet and looks for the wrong .del file. While this does not
run on production (e.g. "-ea" is usually not activated), it does affect the tests because
the assertion stops operations abruptly.
** Though someone can claim we can fix that, I think it points at a problem in the design,
and makes the whole solution fragile.
* I think it'd be really neat to introduce a ParallelWriter, equivalent to ParallelReader.
The latter does not have a Master/Slave notion and so I don't think PW should have.
* When I inspected the code carefully, I realized there are lots of hoola hoops done in order
to make the Master and Slave in sync. Such hoola hoops may be resolved if Lucene's IW API
would be more extensible, but still:
** The MergePolicy used is one that for the Slaves never checks the segments for which merges
should actually be done. Rather, it relies on the Master policy to set the proper merges.
Which is a must in this design because only the master needs to decide when to merge.
** However, and here I think it's because of lack of API on IW, the way the merge is done
is that the master first calls mergeInit(merge), then on all slaves .maybeMerge() and then
it merges that merge. maybeMerge() on the slaves consume all the merges that were decided
to be run by the master, while when that finished, the master didn't finish even one merge
** That works though because the MergeScheduler used is a Serial one (not SMS but still Serial)
and blocking. However that leads to inconsistencies - slaves' segments view is different at
one point in time from the master's.
* The current approach does not support multi-threaded indexing, but I think that's a limitation
that could be solved by exposing some API on IW or DW.
* Only SMS is supported on the slaves.
* Optimize, expungeDeletes are unsupported. Though the could and perhaps just not implemented.
* The current approach prevents having an architecture on which some of the parallels reside
on different machines, because they share the .del and segments file with the master. It's
not the worse limitation in the world, but still a limitation (of having any chance to do
it efficiently) I'd like to avoid.
* And I'm sure there are more disadvantages that I don't remember now.

I'd like to point out that even if the above limitations can be worked around, I still think
the Master and Slave notion is not the best approach. At least, I'd like to propose a different
* Introduce a ParallelWriter which serves as a manager/orchestrator on top of other IWs. It
is not a pure decorator because it drives everything that happens on the IWs, but it does
not contain any actual indexing logic (e.g. add/delete/update documents).
** The IWs PW will manage will be named hereinafter Slices.
* IW will expose enough API to perform two-phase operations, like the two-phase commit one
can achieve today. Example operations (and I don't cover all for the interest of space):
** addDocument - first obtain a doc ID, then proceed w/ addDocument on all Slices
** optimize - already exists
** merge - do the merge on all Slices and stamp it after all finished.
** deleteDocuments - here we would need to expose some API on IW for DW to get an IndexReader
so that IW can still return its readerPool.getReader but PW will return a ParallelSegmentReader
or something, to perform the deletes across all Slices.
** The idea is that we should do enough on the Slices so that if one fails we can still rollback,
and the final 'stamp' process will be very fast and less likely to fail.
* For correctness and protectiveness, PW will only accept a Directory and not IW. Reason is:
** We want all sorts of settings like MP, MS, RAM buffer usage to be controlled by PW and
not on the Slices. If we allow to pass an IW instance, one could override whatever we set,
which is wrong.
** Even though one could claim that someone 'can shoot himself in the leg freely', I think
that we should be resilient enough to protect stupid users from themselves.
** We will need to allow to pass in an IW Configuration object, so that we can still account
for settings such as Analyzer, MaxFieldLength etc., but discard other settings which PW will
control directly
*** Such Configuration was proposed in the past already and will eliminate lots of methods
on IW and ctors.
** On a side note, ParallelReader accepts IR today, which can lead the problems such as one
passes two IRs, one read-only and one not, and then deletes documents by the writable IR,
with PR not knowing about it. But it's a different issue, and I'll open a separate one for
* A special MergeScheduler and MergePolicy will be introduced to allow PW to drive merges
across the Slices. The idea is to support whatever MS/MP the application wants (SMS, CMS,
custom), and ensuring that when MP decides a merge should be performed, that merge is executed
by MS across all Slices. Few things:
** I think that a special MP is not needed, only MS. But need to validate that. If that's
true, then apps could use their own custom MPs freely.
** I think custom MS may be supported ... all that's required is for the MS to run on PW and
whenever it calls its merge(), let PW run the merges across all Slices? But I still need to
validate that code.
** CMS can introduce two-level concurrency. One like today which executes different merges
decided by MP concurrently. The other would control the concurrency level those merges are
executed on the Slices.
*** Hmm ... even SMS can benefit from that ...

I realize that accepting only Directory on PW might limit applications who want to pass in
their own IW extension, for whatever reason. But other than saying "if you pass in IW and
configure it afterwards, it's on your head", I don't think there is any other option ... Well
maybe except if we expose a package-private API for PW to turn off configuration on an IW
after it set it, so successive calls to the underlying IW's setters will throw an exception
... hmm might be doable. I'll look into that. If that will work, we might want to do the same
for the ParallelReader as well.

Michael mentioned a scenario above where one would want to rebuild an index Slice. That's
still achievable by this design - one should build the IW on the outside and then replace
the Directory instance on PW. We'll need to expose such API as well.

BTW, some of the things I've mentioned can be taken care of in different issues, as follow
on improvements, such as two-level concurrency, supporting custom MS etc. I've detailed them
here just so we all see the bigger picture that's going on in my head.

I think I wrote all (or most) of the high-level details. I'd like to start implementing this
soon. In my head it's all chewed and digested, so I feel I can start implementing today. If
possible, I'd like to get this out in 3.1. I'll try to break this issue down to as many issues
as I can, to make the contributions containable. We should just keep in mind for each such
issue the larger picture it solves.

I'd appreciate your comments.

> Parallel incremental indexing
> -----------------------------
>                 Key: LUCENE-1879
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>             Fix For: 3.1
>         Attachments: parallel_incremental_indexing.tar
> A new feature that allows building parallel indexes and keeping them in sync on a docID
level, independent of the choice of the MergePolicy/MergeScheduler.
> Find details on the wiki page for this feature:
> Discussion on java-dev:

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message