Mailing-List: contact lucy-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Wed, 18 Mar 2009 17:52:48 -0700
To: lucy-dev@lucene.apache.org
Subject: Re: real time updates
Message-ID: <20090319005248.GA7203@rectangular.com>
References: <20090315230128.GB23339@rectangular.com>
 <4B9436C9-4E76-4E11-9C55-0A15627B85FD@mikemccandless.com>
 <20090316004416.GC23339@rectangular.com>
 <9ac0c6aa0903160251j8a1451am64f51b704cf90b40@mail.gmail.com>
 <20090316172948.GA28202@rectangular.com>
 <9ac0c6aa0903161602icd8d9d8jcbf34b682481297@mail.gmail.com>
 <20090317003310.GA28694@rectangular.com>
 <9ac0c6aa0903170250t31e418f9pe6df3d072d9e4738@mail.gmail.com>
 <20090318195954.GA6047@rectangular.com>
 <9ac0c6aa0903181541i566c7e04ne86f3905b584aade@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <9ac0c6aa0903181541i566c7e04ne86f3905b584aade@mail.gmail.com>
User-Agent: Mutt/1.5.13 (2006-08-11)
From: Marvin Humphrey <marvin@rectangular.com>

On Wed, Mar 18, 2009 at 06:41:25PM -0400, Michael McCandless wrote:

> But at the end of the session you still insist on merging down to one
> segment before closing right?  So if my added docs all fit in RAM,
> closing is fast, but if I go a bit over 1 RAM buffer's worth, then
> closing is suddenly slower since it must merge those two runs before
> closing?

There's definitely a threshold effect.  But the time is still small relative
to the total run time of the indexer app.  10 or 15%, I think?

> I think having a close that's unexpectedly slow is irksome.  

It's not unexpectedly slow.  It's consistently slow.  :D  

Since KS *always* performs merging of runs at the end, the cost doesn't come
as a surprise.

Moving all the hard work into a Prepare_Commit() method would solve this and
would be a piece of cake to implement.  Then we could have a Commit() which
was always fast. 

Marvin Humphrey