lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch in Solr ?
Date Mon, 21 Sep 2009 18:34:54 GMT
John,

It would be great if Lucene's benchmark were used so everyone
could execute the test in their own environment and verify. It's
not clear the settings or code used to generate the results so
it's difficult to draw any reliable conclusions.

The steep spike shows greater evidence for the IO cache being
cleared during large merges resulting in search performance
degradation. See:
http://www.lucidimagination.com/search/?q=madvise

Merging is IO intensive, less CPU intensive, if the
ConcurrentMergeScheduler is used, which defaults to 3 threads,
then the CPU could be maxed out. Using a single thread on
synchronous spinning magnetic media seems more logical. Queries
are usually the inverse, CPU intensive, not IO intensive when
the index is in the IO cache. After merging a large segment (or
during), queries would start hitting disk, and the results
clearly show that. The queries are suddenly more time consuming
as they seek on disk at a time when IO activity is at it's peak
from merging large segments. Using madvise would prevent usable
indexes from being swapped to disk during a merge, query
performance would continue unabated.

As we move to a sharded model of indexes, large merges will
naturally not occur. Shards will reach a specified size and new
documents will be sent to new shards.

-J

On Sun, Sep 20, 2009 at 11:12 PM, John Wang <john.wang@gmail.com> wrote:
> The current default Lucene MergePolicy does not handle frequent updates
> well.
>
> We have done some performance analysis with that and a custom merge policy:
>
> http://code.google.com/p/zoie/wiki/ZoieMergePolicy
>
> -John
>
> On Mon, Sep 21, 2009 at 1:08 PM, Jason Rutherglen <
> jason.rutherglen@gmail.com> wrote:
>
>> I opened SOLR-1447 for this
>>
>> 2009/9/18 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@corp.aol.com>:
>> > We can use a simple reflection based implementation to simplify
>> > reading too many parameters.
>> >
>> > What I wish to emphasize is that Solr should be agnostic of xml
>> > altogether. It should only be aware of specific Objects and
>> > interfaces. If users wish to plugin something else in some other way ,
>> > it should be fine
>> >
>> >
>> >  There is a huge learning involved in learning the current
>> > solrconfig.xml . Let us not make people throw away that .
>> >
>> > On Sat, Sep 19, 2009 at 1:59 AM, Jason Rutherglen
>> > <jason.rutherglen@gmail.com> wrote:
>> >> Over the weekend I may write a patch to allow simple reflection based
>> >> injection from within solrconfig.
>> >>
>> >> On Fri, Sep 18, 2009 at 8:10 AM, Yonik Seeley
>> >> <yonik@lucidimagination.com> wrote:
>> >>> On Thu, Sep 17, 2009 at 4:30 PM, Shalin Shekhar Mangar
>> >>> <shalinmangar@gmail.com> wrote:
>> >>>>> I was wondering if there is a way I can modify calibrateSizeByDeletes
>> just
>> >>>>> by configuration ?
>> >>>>>
>> >>>>
>> >>>> Alas, no. The only option that I see for you is to sub-class
>> >>>> LogByteSizeMergePolicy and set calibrateSizeByDeletes to true in
the
>> >>>> constructor. However, please open a Jira issue and so we don't forget
>> about
>> >>>> it.
>> >>>
>> >>> It's the continuing stuff like this that makes me feel like we should
>> >>> be Spring (or equivalent) based someday... I'm just not sure how we're
>> >>> going to get there.
>> >>>
>> >>> -Yonik
>> >>> http://www.lucidimagination.com
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > -----------------------------------------------------
>> > Noble Paul | Principal Engineer| AOL | http://aol.com
>> >
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message