Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <47316FE3F6BA0D4DADF99663512552680B10035BA5@ITCS-ECLS-1-VS3.adsroot.itcs.umich.edu>
References: <BANLkTinmE7FVksJAAMdeukRt9VAXQTVA9g@mail.gmail.com>
	<BANLkTin3E4OzqyenCtdmBfdTWUViSsV-0A@mail.gmail.com>
	<BANLkTimQgewUPg49jbTaKSdUBC+G7utX3g@mail.gmail.com>
	<BANLkTik45Hjc--Uuj+Or00MgA5qhLrR0vA@mail.gmail.com>
	<47316FE3F6BA0D4DADF99663512552680B0F373C74@ITCS-ECLS-1-VS3.adsroot.itcs.umich.edu>
	<BANLkTikpkHR2f-w-2zM0h+tY4+QU4JgAnA@mail.gmail.com>
	<BANLkTimLuBc3AcOF1P9QTA814UmLv1rG0Q@mail.gmail.com>
	<BANLkTikhTtXzQ4vQipwi1=mvVGn7r3v6iQ@mail.gmail.com>
	<BANLkTikwOiNR_mGD3bnePxSVKEd4FF4gAQ@mail.gmail.com>
	<BANLkTimATC68Tw3qFXZTObh2cC9Zt-ATjA@mail.gmail.com>
	<BANLkTikjz3W5Sta2ioeMJ69L4QeiFVS7Cg@mail.gmail.com>
	<47316FE3F6BA0D4DADF99663512552680B0F373E27@ITCS-ECLS-1-VS3.adsroot.itcs.umich.edu>
	<BANLkTin2raZfCXgGLe++BjbcVyERZkzxKQ@mail.gmail.com>
	<47316FE3F6BA0D4DADF99663512552680B0F3743BC@ITCS-ECLS-1-VS3.adsroot.itcs.umich.edu>
	<47316FE3F6BA0D4DADF99663512552680B10035BA5@ITCS-ECLS-1-VS3.adsroot.itcs.umich.edu>
Date: Sat, 21 May 2011 06:46:38 -0400
Message-ID: <BANLkTik+3MwcWKZc6CgkzsAO0WgkKFhfdQ@mail.gmail.com>
Subject: Re: MergePolicy Thresholds
From: Michael McCandless <lucene@mikemccandless.com>
To: "Burton-West, Tom" <tburtonw@umich.edu>
Cc: "dev@lucene.apache.org" <dev@lucene.apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Thanks Tom!

Sounds like great fun working with such massive data sets :)

Mike

http://blog.mikemccandless.com

On Fri, May 20, 2011 at 7:03 PM, Burton-West, Tom <tburtonw@umich.edu> wrot=
e:
> Hi Mike and Shai,
>
>
>
> I was able to index=A0 a few documents with the tieredMergePolicy but I w=
as
> hoping to build a large test index of about 700,000 documents to compare =
the
> performance against our previous runs.=A0 I was hoping I would be able to
> report on my results in time for the Lucene Revolution conference.
> Unfortunately there was a power outage at our data center last week which
> resulted in a node failure in one of our storage nodes and node rebalanci=
ng
> for a cluster of 500 terabytes takes quite a while and totally messes up
> performance measurements.=A0 (Our 6-8 terabytes of large scale search ind=
exes
> shares storage with the repository that holds the 480+ terabytes of page
> images and metadata for the 8 million+ books).=A0 =A0Hopefully I will be =
able to
> run the tests when I get back.
>
>
>
> Tom
>
>
>
> From: Burton-West, Tom [mailto:tburtonw@umich.edu]
> Sent: Monday, May 09, 2011 4:10 PM
>
> To: dev@lucene.apache.org
> Subject: RE: MergePolicy Thresholds
>
>
>
> Thanks again Shai and Mike.
>
>
>
> Am in the process of downloading and building =A0=A0r1099998.=A0 Should b=
e able to
> build a test index sometime this week.=A0 I=92ll make some guesses on wha=
t
> parameters to use based on our previous tests.
>
>
>
> Tom
>
> From: Shai Erera [mailto:serera@gmail.com]
> Sent: Saturday, May 07, 2011 11:33 PM
> To: dev@lucene.apache.org
> Subject: Re: MergePolicy Thresholds
>
>
>
> Hey Tom,
>
> Mike back-ported the changes to 3x, so you can try it out.
>
> FYI,
> Shai
>
> On Tue, May 3, 2011 at 9:33 PM, Burton-West, Tom <tburtonw@umich.edu> wro=
te:
>
> Thanks Shai and Mike!
>
> I'll keep an eye on LUCENE-1076.
>
> Tom
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>
> Sent: Tuesday, May 03, 2011 11:15 AM
> To: dev@lucene.apache.org
> Subject: Re: MergePolicy Thresholds
>
> Thanks Shai!
>
> I'm way behind on my 3.x backports -- I'll try to do this soon.
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Tue, May 3, 2011 at 8:10 AM, Shai Erera <serera@gmail.com> wrote:
>> I uploaded a patch to LUCENE-1076.
>>
>> Tom, apparently the patch I've attached before cannot be used, because
>> there
>> are dependencies (in earlier commits on LUCENE-1076) that need to be
>> back-ported as well. So stay tuned on LUCENE-1076 for when it is safe to
>> use
>> this new MP.
>>
>> Shai
>>
>> On Tue, May 3, 2011 at 1:00 PM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>>
>>> That'd be great, thanks :)
>>>
>>> Yes, let's iterate on the issue! =A0But: it should still be open, I hop=
e
>>> (I didn't mean to close it yet, since it's not back ported)...
>>>
>>> Mike
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Tue, May 3, 2011 at 5:51 AM, Shai Erera <serera@gmail.com> wrote:
>>> > Mike, if you want, I can back-port it, as I've already started this
>>> > when
>>> > preparing the patch.
>>> >
>>> > I noticed that you added a "throws IOE" to IW.setInfoStream -- is it =
ok
>>> > on
>>> > 3x too? It'll be a backwards change.
>>> >
>>> > Maybe we should iterate on the issue? I can reopen.
>>> >
>>> > Shai
>>> >
>>> > On Tue, May 3, 2011 at 12:36 PM, Michael McCandless
>>> > <lucene@mikemccandless.com> wrote:
>>> >>
>>> >> Looks good Shai!
>>> >>
>>> >> Comments below too:
>>> >>
>>> >> On Tue, May 3, 2011 at 5:29 AM, Shai Erera <serera@gmail.com> wrote:
>>> >> > Hi
>>> >> >
>>> >> > I looked into porting it to 3x, and prepared the attached patch. I=
t
>>> >> > only
>>> >> > contains the new TieredMP and Test, as well as the necessary chang=
es
>>> >> > to
>>> >> > LuceneTestCase and IndexWriter. I guess you can start with it (eve=
n
>>> >> > just
>>> >> > the
>>> >> > MP and IW changes) to test it on your indexes.
>>> >> >
>>> >> > Mike, I saw that there were many more changes, as part of
>>> >> > LUCENE-1076,
>>> >> > done
>>> >> > to the code. In particular, this MP is now the default (on trunk),
>>> >> > so
>>> >> > I
>>> >> > guess many changes (to tests) were needed because of that. Do you
>>> >> > remember,
>>> >> > if apart from the changes I've included in the patch, other
>>> >> > important
>>> >> > changes w.r.t. this code?
>>> >>
>>> >> The only other changes I can think of were some verbosity improvemen=
ts
>>> >> to IndexWriter, to support the python script that can make a merge
>>> >> movie from an infoStream output; but that can wait for when I
>>> >> back-port to 3.x...
>>> >>
>>> >> > As we won't change the default MP on 3x, I'm guessing I don't need
>>> >> > to
>>> >> > port
>>> >> > all the changes to 3x.
>>> >>
>>> >> Right, I think.
>>> >>
>>> >> Mike
>>> >>
>>> >> --------------------------------------------------------------------=
-
>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org