Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 209.85.216.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <0E71CDF761B18841BF0AC7E45FAF96DF082C54A769@LONMXSMB04.win.dowjones.net>
References: 
 <0E71CDF761B18841BF0AC7E45FAF96DF082C54A207@LONMXSMB04.win.dowjones.net>
	<1335881528.58864.YahooMailNeo@web130101.mail.mud.yahoo.com>
	<0E71CDF761B18841BF0AC7E45FAF96DF082C54A6F0@LONMXSMB04.win.dowjones.net>
	<CAN4YXvcyvAHLZ8OiQmakbOP-D3A9CYjcNukTR9S+b9ZPNccZPg@mail.gmail.com>
	<0E71CDF761B18841BF0AC7E45FAF96DF082C54A769@LONMXSMB04.win.dowjones.net>
Date: Wed, 2 May 2012 08:15:13 -0400
Message-ID: 
 <CAN4YXvdFsnOgpPCoGZNF65Cm32g4k3HvD=9MrgUS50qYXn1DOQ@mail.gmail.com>
Subject: Re: Solr Merge during off peak times
From: Erick Erickson <erickerickson@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

But again, with a master/slave setup merging should
be relatively benign. And at 200M docs, having a M/S
setup is probably indicated.

Here's a good writeup of mergepolicy
http://juanggrande.wordpress.com/2011/02/07/merge-policy-internals/

If you're indexing and searching on a single machine, merging
is much less important than how often you commit. If a M/S
situation, then you're polling interval on the slave is important.

I'd look at commit frequency long before I worried about merging,
that's usually where people shoot themselves in the foot - by
committing too often.

Overall, your mergeFactor is probably less important than other
parts of how you perform indexing/searching, but it does have
some effect for sure...

Best
Erick

On Wed, May 2, 2012 at 7:54 AM, Prakashganesh, Prabhu
<Prabhu.Prakashganesh@dowjones.com> wrote:
> We have a fairly large scale system - about 200 million docs and fairly h=
igh indexing activity - about 300k docs per day with peak ingestion rates o=
f about 20 docs per sec. I want to work out what a good mergeFactor setting=
 would be by testing with different mergeFactor settings. I think the defau=
lt of 10 might be high, I want to try with 5 and compare. Unless I know whe=
n a merge starts and finishes, it would be quite difficult to work out the =
impact of changing mergeFactor. I want to be able to measure how long merge=
s take, run queries during the merge activity and see what the response tim=
es are etc..
>
> Thanks
> Prabhu
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: 02 May 2012 12:40
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Merge during off peak times
>
> Why do you care? Merging is generally a background process, or are
> you doing heavy indexing? In a master/slave setup,
> it's usually not really relevant except that (with 3.x), massive merges
> may temporarily stop indexing. Is that the problem?
>
> Look at the merge policys, there are configurations that make
> this less painful.
>
> In trunk, DocumentWriterPerThread makes merges happen in the
> background, which helps the long-pause-while-indexing problem.
>
> Best
> Erick
>
> On Wed, May 2, 2012 at 7:22 AM, Prakashganesh, Prabhu
> <Prabhu.Prakashganesh@dowjones.com> wrote:
>> Ok, thanks Otis
>> Another question on merging
>> What is the best way to monitor merging?
>> Is there something in the log file that I can look for?
>> It seems like I have to monitor the system resources - read/write IOPS e=
tc.. and work out when a merge happened
>> It would be great if I can do it by looking at log files or in the admin=
 UI. Do you know if this can be done or if there is some tool for this?
>>
>> Thanks
>> Prabhu
>>
>> -----Original Message-----
>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>> Sent: 01 May 2012 15:12
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Merge during off peak times
>>
>> Hi Prabhu,
>>
>> I don't think such a merge policy exists, but it would be nice to have t=
his option and I imagine it wouldn't be hard to write if you really just ba=
se the merge or no merge decision on the time of day (and maybe day of the =
week).
>>
>> Note that this should go into Lucene, not Solr, so if you decide to cont=
ribute your work, please see=A0http://wiki.apache.org/lucene-java/HowToCont=
ribute
>>
>> Otis
>> ----
>> Performance Monitoring for Solr - http://sematext.com/spm
>>
>>
>>
>>
>>>________________________________
>>> From: "Prakashganesh, Prabhu" <Prabhu.Prakashganesh@dowjones.com>
>>>To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>>Sent: Tuesday, May 1, 2012 8:45 AM
>>>Subject: Solr Merge during off peak times
>>>
>>>Hi,
>>>=A0 I would like to know if there is a way to configure index merge poli=
cy in solr so that the merging happens during off peak hours. Can you pleas=
e let me know if such a merge policy configuration exists?
>>>
>>>Thanks
>>>Prabhu
>>>
>>>
>>>