Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.rutherglen@gmail.com
 designates 74.125.92.26 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=lsqbaejxi94EbCdnZ6oWr0f+4Xmc2DZ9+3pjd9y/YcwybOkBGJ4z75SlJhw25qjWoZ
         4UVEUQZAFr5aBQz+JvAjycrqwoADCwLWaRo8brBBofpHNfVTvAXUk8ti0FtsQQFtCG3j
         pHDsRPIM0DZeZpyyo7OgHk3Ja6Vhb73bSMukA=
MIME-Version: 1.0
In-Reply-To: <9ac0c6aa0907141256u2e247f55h9637b92fdc114f6e@mail.gmail.com>
References: <85d3c3b60907141208p4584b72bp8a3098623a1c117a@mail.gmail.com>
	 <9ac0c6aa0907141256u2e247f55h9637b92fdc114f6e@mail.gmail.com>
Date: Tue, 21 Jul 2009 16:26:53 -0700
Message-ID: <85d3c3b60907211626t717bdebbq5a00a7835030221e@mail.gmail.com>
Subject: Re: addIndexes* blocks addDocuments calls
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

> EG you could imagine an addIndexes* call getting started,
> completing a few merges. Then, concurrently, CMS picks & chooses
> some of those added external segments to merge with some of the
> original segments. Then addIndexes hits an exception. What do we
> do?

An exception due to an IO error from the external dir? We can
abandon the merge and remove the external segments that failed
from segmentInfos?

> If IndexWriter maintained "branches" of the segmentInfos we
> could actually rollback all changes, ie, remove what was done by
> addIndexes but retroactively preserve any segments created by
> other methods (flushing, other addIndexes calls, etc.).

Branches stored in a TreeMap? With the keys being the
method+randomId that initiated them? (i.e. addIndexes12)

On Tue, Jul 14, 2009 at 12:56 PM, Michael
McCandless<lucene@mikemccandless.com> wrote:
> I agree, there's no real reason why addIndexes can run concurrently
> with other things. =A0It's just software ;)
>
> One challenge is the transactional guarantee that addIndexes provide,
> ie, it's all or none. =A0If there's an exception while adding, then
> nothing was added.
>
> But, that was added before autoCommit=3Dfalse. =A0So perhaps we could
> relax that and expect the app to instead rely on the "global"
> transactional semantics provided by autoCommit=3Dfalse.
>
> EG you could imagine an addIndexes* call getting started, completing a
> few merges. =A0Then, concurrently, CMS picks & chooses some of those
> added external segments to merge with some of the original segments.
> Then addIndexes hits an exception. =A0What do we do?
>
> If IndexWriter maintained "branches" of the segmentInfos we could
> actually rollback all changes, ie, remove what was done by addIndexes
> but retroactively preserve any segments created by other methods
> (flushing, other addIndexes calls, etc.).
>
> Mike
>
> On Tue, Jul 14, 2009 at 3:08 PM, Jason
> Rutherglen<jason.rutherglen@gmail.com> wrote:
>> For replicating and general system performance, it would be good to offe=
r a
>> way to addIndexes* without blocking the addition of more docs. This seem=
s
>> doable somehow?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org