lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: addIndexes* blocks addDocuments calls
Date Tue, 21 Jul 2009 23:26:53 GMT
> EG you could imagine an addIndexes* call getting started,
> completing a few merges. Then, concurrently, CMS picks & chooses
> some of those added external segments to merge with some of the
> original segments. Then addIndexes hits an exception. What do we
> do?

An exception due to an IO error from the external dir? We can
abandon the merge and remove the external segments that failed
from segmentInfos?

> If IndexWriter maintained "branches" of the segmentInfos we
> could actually rollback all changes, ie, remove what was done by
> addIndexes but retroactively preserve any segments created by
> other methods (flushing, other addIndexes calls, etc.).

Branches stored in a TreeMap? With the keys being the
method+randomId that initiated them? (i.e. addIndexes12)

On Tue, Jul 14, 2009 at 12:56 PM, Michael
McCandless<lucene@mikemccandless.com> wrote:
> I agree, there's no real reason why addIndexes can run concurrently
> with other things.  It's just software ;)
>
> One challenge is the transactional guarantee that addIndexes provide,
> ie, it's all or none.  If there's an exception while adding, then
> nothing was added.
>
> But, that was added before autoCommit=false.  So perhaps we could
> relax that and expect the app to instead rely on the "global"
> transactional semantics provided by autoCommit=false.
>
> EG you could imagine an addIndexes* call getting started, completing a
> few merges.  Then, concurrently, CMS picks & chooses some of those
> added external segments to merge with some of the original segments.
> Then addIndexes hits an exception.  What do we do?
>
> If IndexWriter maintained "branches" of the segmentInfos we could
> actually rollback all changes, ie, remove what was done by addIndexes
> but retroactively preserve any segments created by other methods
> (flushing, other addIndexes calls, etc.).
>
> Mike
>
> On Tue, Jul 14, 2009 at 3:08 PM, Jason
> Rutherglen<jason.rutherglen@gmail.com> wrote:
>> For replicating and general system performance, it would be good to offer a
>> way to addIndexes* without blocking the addition of more docs. This seems
>> doable somehow?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message