Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (herse.apache.org: local policy)
Message-Id: <1178135948.7073.1187773683@webmail.messagingengine.com>
From: "Michael McCandless" <mail@mikemccandless.com>
To: java-dev@lucene.apache.org
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
MIME-Version: 1.0
References: <24175587.1175343325320.JavaMail.jira@brutus>
   <c7b302c50705020656s6ff7f8fta57bd0d326d32feb@mail.gmail.com>
Subject: Re: [jira] Created: (LUCENE-854) Create merge policy that doesn't
   periodically inadvertently optimize
In-Reply-To: <c7b302c50705020656s6ff7f8fta57bd0d326d32feb@mail.gmail.com>
Date: Wed, 02 May 2007 15:59:08 -0400


"Ning Li" <ning.li.li@gmail.com> wrote:
> On 3/31/07, Michael McCandless (JIRA) <jira@apache.org> wrote:
> > Create merge policy that doesn't periodically inadvertently optimize
> > --------------------------------------------------------------------
> > So we could make a small change to the policy by only merging the
> > first mergeFactor segments once we hit 2X the merge factor.  With
> > mergeFactor=10, when we have created the 20th level 0 (just flushed)
> > segment, we merge the first 10 into a level 1 segment.  Then on
> > creating another 10 level 0 segments, we merge the second set of 10
> > level 0 segments into a level 1 segment, etc.
> 
> Hi Mike,
> 
> When a 20th level 0 segment triggers a 20th level 1 segment which
> triggers a 20th level 2 segment... we are still optimizing, aren't we?
> Am I missing something here?

The merge would "cascade" in this case, but, would not optimize (you
will have > 1 segments in the end).

Each time you cascade you only merge the first 10 at each level, so
after cascading you would have 1 level 3 segment, 10 level 2 segments,
10 level 1 segments and 10 level 0 segments.

I'm actually using this merge policy in my patch for LUCENE-843 when
merging the flushed "partial" segments.  This is only used when
IndexWriter is opened with autoCommit=false, and, you add lots
and lots of documents (so RAM flushes many times).

But, I like the proposed merge policy at the end of LUCENE-845 even
better for Lucene's normal merges.

It would merge based on size (not # docs), would be free to merge
adjacent segments (not just rightmost segments), and would merge N
(configurable) at a time.  The part that's still unclear is how it
chooses when to "trigger" a merge and how specifically it picks which
N segments to merge (maybe: the series of N adjacent segments that are
"most similar" in size, but favoring smaller segments over larger
ones).

Mike
-- 
  Michael McCandless
  mail@mikemccandless.com


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org