lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: addIndexes* blocks addDocuments calls
Date Tue, 21 Jul 2009 23:56:57 GMT
On Tue, Jul 21, 2009 at 7:26 PM, Jason
Rutherglen<> wrote:
>> EG you could imagine an addIndexes* call getting started,
>> completing a few merges. Then, concurrently, CMS picks & chooses
>> some of those added external segments to merge with some of the
>> original segments. Then addIndexes hits an exception. What do we
>> do?
> An exception due to an IO error from the external dir?

Or OOME or a Lucene bug or something.

> We can
> abandon the merge and remove the external segments that failed
> from segmentInfos?

But that won't work in this case because some of the external segments
have been merged with "real" segments in the index.

>> If IndexWriter maintained "branches" of the segmentInfos we
>> could actually rollback all changes, ie, remove what was done by
>> addIndexes but retroactively preserve any segments created by
>> other methods (flushing, other addIndexes calls, etc.).
> Branches stored in a TreeMap? With the keys being the
> method+randomId that initiated them? (i.e. addIndexes12)

Actually I meant keeping track of the "genealogy" of how the merges
were done, plus holding an extra refCount against "real" segments that
had been merged with external-but-not-yet-committed segments.

So that if the addIndexes fails, yet we had merged [say] "real"
segments 1, 2, 3 with "external" segments 4 and 5, we could on failure
of addIndexes go "undo" that merge and put back segments 1, 2, 3.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message