lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: addIndexes* blocks addDocuments calls
Date Tue, 14 Jul 2009 21:25:24 GMT
> EG you could imagine an addIndexes* call getting started,
completing a few merges. Then, concurrently, CMS picks & chooses
some of those added external segments to merge with some of the
original segments. Then addIndexes hits an exception. What do we
do?

Right because we're rolling back all the segmentinfos at once
when using transactions.

> If IndexWriter maintained "branches" of the segmentInfos we
could actually rollback all changes, ie, remove what was done by
addIndexes but retroactively preserve any segments created by
other methods (flushing, other addIndexes calls, etc.).

I think on commit we resolve all external and (with LUCENE-1313)
ramdir segments in the foreground. Otherwise they are being
merged in the background as usual?

So we'd add a new method addIndexesNoCommit?


On Tue, Jul 14, 2009 at 12:56 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
>
> I agree, there's no real reason why addIndexes can run concurrently
> with other things.  It's just software ;)
>
> One challenge is the transactional guarantee that addIndexes provide,
> ie, it's all or none.  If there's an exception while adding, then
> nothing was added.
>
> But, that was added before autoCommit=false.  So perhaps we could
> relax that and expect the app to instead rely on the "global"
> transactional semantics provided by autoCommit=false.
>
> EG you could imagine an addIndexes* call getting started, completing a
> few merges.  Then, concurrently, CMS picks & chooses some of those
> added external segments to merge with some of the original segments.
> Then addIndexes hits an exception.  What do we do?
>
> If IndexWriter maintained "branches" of the segmentInfos we could
> actually rollback all changes, ie, remove what was done by addIndexes
> but retroactively preserve any segments created by other methods
> (flushing, other addIndexes calls, etc.).
>
> Mike
>
> On Tue, Jul 14, 2009 at 3:08 PM, Jason
> Rutherglen<jason.rutherglen@gmail.com> wrote:
> > For replicating and general system performance, it would be good to offer a
> > way to addIndexes* without blocking the addition of more docs. This seems
> > doable somehow?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message