lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: MergePolicy Thresholds
Date Sat, 21 May 2011 10:46:38 GMT
Thanks Tom!

Sounds like great fun working with such massive data sets :)

Mike

http://blog.mikemccandless.com

On Fri, May 20, 2011 at 7:03 PM, Burton-West, Tom <tburtonw@umich.edu> wrote:
> Hi Mike and Shai,
>
>
>
> I was able to index  a few documents with the tieredMergePolicy but I was
> hoping to build a large test index of about 700,000 documents to compare the
> performance against our previous runs.  I was hoping I would be able to
> report on my results in time for the Lucene Revolution conference.
> Unfortunately there was a power outage at our data center last week which
> resulted in a node failure in one of our storage nodes and node rebalancing
> for a cluster of 500 terabytes takes quite a while and totally messes up
> performance measurements.  (Our 6-8 terabytes of large scale search indexes
> shares storage with the repository that holds the 480+ terabytes of page
> images and metadata for the 8 million+ books).   Hopefully I will be able to
> run the tests when I get back.
>
>
>
> Tom
>
>
>
> From: Burton-West, Tom [mailto:tburtonw@umich.edu]
> Sent: Monday, May 09, 2011 4:10 PM
>
> To: dev@lucene.apache.org
> Subject: RE: MergePolicy Thresholds
>
>
>
> Thanks again Shai and Mike.
>
>
>
> Am in the process of downloading and building   r1099998.  Should be able to
> build a test index sometime this week.  I’ll make some guesses on what
> parameters to use based on our previous tests.
>
>
>
> Tom
>
> From: Shai Erera [mailto:serera@gmail.com]
> Sent: Saturday, May 07, 2011 11:33 PM
> To: dev@lucene.apache.org
> Subject: Re: MergePolicy Thresholds
>
>
>
> Hey Tom,
>
> Mike back-ported the changes to 3x, so you can try it out.
>
> FYI,
> Shai
>
> On Tue, May 3, 2011 at 9:33 PM, Burton-West, Tom <tburtonw@umich.edu> wrote:
>
> Thanks Shai and Mike!
>
> I'll keep an eye on LUCENE-1076.
>
> Tom
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>
> Sent: Tuesday, May 03, 2011 11:15 AM
> To: dev@lucene.apache.org
> Subject: Re: MergePolicy Thresholds
>
> Thanks Shai!
>
> I'm way behind on my 3.x backports -- I'll try to do this soon.
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Tue, May 3, 2011 at 8:10 AM, Shai Erera <serera@gmail.com> wrote:
>> I uploaded a patch to LUCENE-1076.
>>
>> Tom, apparently the patch I've attached before cannot be used, because
>> there
>> are dependencies (in earlier commits on LUCENE-1076) that need to be
>> back-ported as well. So stay tuned on LUCENE-1076 for when it is safe to
>> use
>> this new MP.
>>
>> Shai
>>
>> On Tue, May 3, 2011 at 1:00 PM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>>
>>> That'd be great, thanks :)
>>>
>>> Yes, let's iterate on the issue!  But: it should still be open, I hope
>>> (I didn't mean to close it yet, since it's not back ported)...
>>>
>>> Mike
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Tue, May 3, 2011 at 5:51 AM, Shai Erera <serera@gmail.com> wrote:
>>> > Mike, if you want, I can back-port it, as I've already started this
>>> > when
>>> > preparing the patch.
>>> >
>>> > I noticed that you added a "throws IOE" to IW.setInfoStream -- is it ok
>>> > on
>>> > 3x too? It'll be a backwards change.
>>> >
>>> > Maybe we should iterate on the issue? I can reopen.
>>> >
>>> > Shai
>>> >
>>> > On Tue, May 3, 2011 at 12:36 PM, Michael McCandless
>>> > <lucene@mikemccandless.com> wrote:
>>> >>
>>> >> Looks good Shai!
>>> >>
>>> >> Comments below too:
>>> >>
>>> >> On Tue, May 3, 2011 at 5:29 AM, Shai Erera <serera@gmail.com>
wrote:
>>> >> > Hi
>>> >> >
>>> >> > I looked into porting it to 3x, and prepared the attached patch.
It
>>> >> > only
>>> >> > contains the new TieredMP and Test, as well as the necessary changes
>>> >> > to
>>> >> > LuceneTestCase and IndexWriter. I guess you can start with it (even
>>> >> > just
>>> >> > the
>>> >> > MP and IW changes) to test it on your indexes.
>>> >> >
>>> >> > Mike, I saw that there were many more changes, as part of
>>> >> > LUCENE-1076,
>>> >> > done
>>> >> > to the code. In particular, this MP is now the default (on trunk),
>>> >> > so
>>> >> > I
>>> >> > guess many changes (to tests) were needed because of that. Do you
>>> >> > remember,
>>> >> > if apart from the changes I've included in the patch, other
>>> >> > important
>>> >> > changes w.r.t. this code?
>>> >>
>>> >> The only other changes I can think of were some verbosity improvements
>>> >> to IndexWriter, to support the python script that can make a merge
>>> >> movie from an infoStream output; but that can wait for when I
>>> >> back-port to 3.x...
>>> >>
>>> >> > As we won't change the default MP on 3x, I'm guessing I don't need
>>> >> > to
>>> >> > port
>>> >> > all the changes to 3x.
>>> >>
>>> >> Right, I think.
>>> >>
>>> >> Mike
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message