lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: addIndexes* blocks addDocuments calls
Date Tue, 21 Jul 2009 23:56:57 GMT
On Tue, Jul 21, 2009 at 7:26 PM, Jason
Rutherglen<jason.rutherglen@gmail.com> wrote:
>> EG you could imagine an addIndexes* call getting started,
>> completing a few merges. Then, concurrently, CMS picks & chooses
>> some of those added external segments to merge with some of the
>> original segments. Then addIndexes hits an exception. What do we
>> do?
>
> An exception due to an IO error from the external dir?

Or OOME or a Lucene bug or something.

> We can
> abandon the merge and remove the external segments that failed
> from segmentInfos?

But that won't work in this case because some of the external segments
have been merged with "real" segments in the index.

>> If IndexWriter maintained "branches" of the segmentInfos we
>> could actually rollback all changes, ie, remove what was done by
>> addIndexes but retroactively preserve any segments created by
>> other methods (flushing, other addIndexes calls, etc.).
>
> Branches stored in a TreeMap? With the keys being the
> method+randomId that initiated them? (i.e. addIndexes12)

Actually I meant keeping track of the "genealogy" of how the merges
were done, plus holding an extra refCount against "real" segments that
had been merged with external-but-not-yet-committed segments.

So that if the addIndexes fails, yet we had merged [say] "real"
segments 1, 2, 3 with "external" segments 4 and 5, we could on failure
of addIndexes go "undo" that merge and put back segments 1, 2, 3.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message