Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of serera@gmail.com designates
 209.85.160.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Gas0K1Vk70iGKOcZ9VaKTJJ6jS8H6AYunchxc1cUBnjqlVUQYbmhZGdaxEB2ebBMGE
         DS4aAdZgBIHpDwARSvhk6mGczvz4TGdohePSvQdw4C2lXPxjCcV35fD9nWlbPRhOQqD4
         eRRv31rv6y3hzipk/K0Ov8Aer5hKt5JiTX6nI=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=C-owwa92ykyOgfEhYiJZWZpdkEKgwN390ycUS@mail.gmail.com>
References: <AANLkTimysUzxuPv9++Y12kaDmq3-iOOnj+v2RzxCjPuP@mail.gmail.com>
	<AANLkTikuwq-U0_bLM4JEaiqxUDPUbSd=kj+Mj6THn_by@mail.gmail.com>
	<AANLkTimc5r_Xps6GcwqTW6kap0SF150ChJth+KDebw5-@mail.gmail.com>
	<AANLkTi=C-owwa92ykyOgfEhYiJZWZpdkEKgwN390ycUS@mail.gmail.com>
Date: Thu, 2 Dec 2010 13:19:05 +0200
Message-ID: <AANLkTi=yMRWoRJQpxwcEAPnXdsZqR6BCEFavCzbiM5LN@mail.gmail.com>
Subject: Re: Consolidate MP and LMP
From: Shai Erera <serera@gmail.com>
To: dev@lucene.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1b6eaf623f704966b96b7

--000e0cd1b6eaf623f704966b96b7
Content-Type: text/plain; charset=ISO-8859-1

>
> You can't remove it on 3x, it's used by a host of deprecated methods
> that access LMP's settings through IW.
>

Remove means deprecate in 3x and remove in trunk. Should have been more
clear about that.

For LMP is
> just returns the value of getUseCompoundFile (that is, until Mike's
> patch that switches off compounding for large segments).
>

As far as I can tell, getUseCompoundFile returns the same in trunk too. The
noCFS setting is not applied there.

Shai

On Thu, Dec 2, 2010 at 1:14 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Thu, Dec 2, 2010 at 4:43 AM, Simon Willnauer
> <simon.willnauer@googlemail.com> wrote:
>
> > During the work on Column Stride Fields I was actually thinking that
> > Compound vs. Non-Compound should not be a global decision since we now
> > have codecs and each codec should use its own way of writing files.
> > Maybe it would make things way easier if we expose CFS to codecs and
> > let them decide what to do. I can imagine that I want to use CFS for
> > some of the codecs like Column Stride or fields that are not  used for
> > searches but keep individual files per codec. Just an idea....
>
> +1!
>
> This would be a nice simplification.
>
> EG, it's bizarre today that on flushing a new segment, which has
> nothing to do with merging, we consult the MP to decide if we need CFS
> or not.
>
> Also, it's awkward we have getCF and also getCFDocStore.  In the
> future (docvalues) we may also want to separately build CFS for those
> files, or not.
>
> Making all these decisions private to the codec makes great sense.
> It's then free to CFS however it wants to.  But, the codec would need
> wider context, I think the full SegmentInfos, to base its decision on.
>  EG, LMP now conditionally builds CFS only if the segment is
> "smallish" relative to total index size.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

--000e0cd1b6eaf623f704966b96b7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><blockquote style=3D"margin: 0pt 0pt 0pt 0.8ex; border-lef=
t: 1px solid rgb(204, 204, 204); padding-left: 1ex;" class=3D"gmail_quote">=
You can&#39;t remove it on 3x, it&#39;s used by a host of deprecated method=
s<br>


that access LMP&#39;s settings through IW.<br>
</blockquote>
<br>
Remove means deprecate in 3x and remove in trunk. Should have been more cle=
ar about that.<br><br><blockquote style=3D"margin: 0pt 0pt 0pt 0.8ex; borde=
r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;" class=3D"gmail_qu=
ote">
 For LMP is<br>
just returns the value of getUseCompoundFile (that is, until Mike&#39;s<br>
patch that switches off compounding for large segments).<br></blockquote><b=
r>As far as I can tell, getUseCompoundFile returns the same in trunk too. T=
he noCFS setting is not applied there.<br><br>Shai<br><br><div class=3D"gma=
il_quote">
On Thu, Dec 2, 2010 at 1:14 PM, Michael McCandless <span dir=3D"ltr">&lt;<a=
 href=3D"mailto:lucene@mikemccandless.com">lucene@mikemccandless.com</a>&gt=
;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0=
pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;=
">
On Thu, Dec 2, 2010 at 4:43 AM, Simon Willnauer<br>
&lt;<a href=3D"mailto:simon.willnauer@googlemail.com">simon.willnauer@googl=
email.com</a>&gt; wrote:<br>
<br>
&gt; During the work on Column Stride Fields I was actually thinking that<b=
r>
&gt; Compound vs. Non-Compound should not be a global decision since we now=
<br>
&gt; have codecs and each codec should use its own way of writing files.<br=
>
&gt; Maybe it would make things way easier if we expose CFS to codecs and<b=
r>
&gt; let them decide what to do. I can imagine that I want to use CFS for<b=
r>
&gt; some of the codecs like Column Stride or fields that are not =A0used f=
or<br>
&gt; searches but keep individual files per codec. Just an idea....<br>
<br>
+1!<br>
<br>
This would be a nice simplification.<br>
<br>
EG, it&#39;s bizarre today that on flushing a new segment, which has<br>
nothing to do with merging, we consult the MP to decide if we need CFS<br>
or not.<br>
<br>
Also, it&#39;s awkward we have getCF and also getCFDocStore. =A0In the<br>
future (docvalues) we may also want to separately build CFS for those<br>
files, or not.<br>
<br>
Making all these decisions private to the codec makes great sense.<br>
It&#39;s then free to CFS however it wants to. =A0But, the codec would need=
<br>
wider context, I think the full SegmentInfos, to base its decision on.<br>
=A0EG, LMP now conditionally builds CFS only if the segment is<br>
&quot;smallish&quot; relative to total index size.<br>
<br>
Mike<br>
<div><div></div><div class=3D"h5"><br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:dev-unsubscribe@lucene.apache.org=
">dev-unsubscribe@lucene.apache.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:dev-help@lucene.apache.o=
rg">dev-help@lucene.apache.org</a><br>
<br>
</div></div></blockquote></div><br></div>

--000e0cd1b6eaf623f704966b96b7--