Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of earwin@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=czE49kojjFyFq6q9rnLsH2gsaOA5XvfCbms0ACEvnO/7G1/CSRKNJl04WyAU0MkjvQ
         JtegGJAjQ2ZhyTQvf/65K/DqADqS2rB3A0ehkU+W1WmxhTP+MWErvCxOU+NHHlYaxODF
         OR2+bO/vpJQHd25g771cpmzwRJlkq3Ul6Ms1E=
MIME-Version: 1.0
In-Reply-To: <BANLkTinmE7FVksJAAMdeukRt9VAXQTVA9g@mail.gmail.com>
References: <BANLkTinmE7FVksJAAMdeukRt9VAXQTVA9g@mail.gmail.com>
Date: Mon, 2 May 2011 17:42:25 +0400
Message-ID: <BANLkTi=-YgvUjDN+n-vm3ps=+Xpcu+5M4g@mail.gmail.com>
Subject: Re: MergePolicy Thresholds
From: Earwin Burrfoot <earwin@gmail.com>
To: dev@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Have you checked BalancedSegmentMergePolicy? It has some more knobs :)

On Mon, May 2, 2011 at 17:03, Shai Erera <serera@gmail.com> wrote:
> Hi
>
> Today, LogMP allows you to set different thresholds for segments sizes,
> thereby allowing you to control the largest segment that will be
> considered for merge + the largest segment your index will hold (=3D~
> threshold * mergeFactor).
>
> So, if you want to end up w/ say 20GB segments, you can set
> maxMergeMB(ForOptimize) to 2GB and mergeFactor=3D10.
>
> However, this often does not achieve your desired goal -- if the index
> contains 5 and 7 GB segments, they will never be merged b/c they are
> bigger than the threshold. I am willing to spend the CPU and IO resources
> to end up w/ 20 GB segments, whether I'm merging 10 segments together or
> only 2. After I reach a 20GB segment, it can rest peacefully, at least
> until I increase the threshold.
>
> So I wonder, first, if this threshold (i.e., largest segment size you
> would like to end up with) is more natural to set than thee current
> thresholds,
> from the application level? I.e., wouldn't it be a simpler threshold to s=
et
> instead of doing weird calculus that depend on maxMergeMB(ForOptimize)
> and mergeFactor?
>
> Second, should this be an addition to LogMP, or a different
> type of MP. One that adheres to only those two factors (perhaps the
> segSize threshold should be allowed to set differently for optimize and
> regular merges). It can pick segments for merge such that it maximizes
> the result segment size (i.e., don't necessarily merge in sequential
> order), but not more than mergeFactor.
>
> I guess, if we think that maxResultSegmentSizeMB is more intuitive than
> the current thresholds, application-wise, then this change should go
> into LogMP. Otherwise, it feels like a different MP is needed, because
> LogMP is already complicated and another threshold would confuse things.
>
> What do you think of this? Am I trying to optimize too much? :)
>
> Shai
>
>


--=20
Kirill Zakharenko/=D0=9A=D0=B8=D1=80=D0=B8=D0=BB=D0=BB =D0=97=D0=B0=D1=85=
=D0=B0=D1=80=D0=B5=D0=BD=D0=BA=D0=BE
E-Mail/Jabber: earwin@gmail.com
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org