lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pouliot, Scott" <Scott.Poul...@peoplefluent.com>
Subject RE: SOLR 4.8.0 Master/Slave Replication Issues
Date Fri, 10 Mar 2017 16:09:27 GMT
Valid point on the soft commits.  I'll try disabling them to see if there if any change in
performance as well.  Worth a shot  ;-)

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Friday, March 10, 2017 11:01 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues

Glad to hear that. Do note that soft commits are irrelevant to replication, all that matters
are hard commits.

I'd go farther and disable soft commits, they don't serve much point in master/slave, at least
usually. The supposition in M/S setups is that the master indexes but does not serve queries
and the slaves serve queries but do not index documents. Since soft commits are about being
able to search documents before a hard commit
(openSearcher=true) happens and assuming you're not searching on the master, they do no good.
They won't cause updates to the slaves to happen for instance.

Or, put another way, soft commits are there to support NRT searches.
Master/slave setups are, by definition, not NRT so there's no reason to do soft commits.

Soft commits are cheaper than hard commits, but they're not free. So disabling them will decrease
the work the master has to do. Whether you'll even notice or not is an open question, I suspect
you'd have to be running your master at breakneck speed to be able to measure indexing throughput
differences with soft commits enabled/disabled, especially with a 60 second interval, more
a style thing I suppose.

Best,
Erick

On Fri, Mar 10, 2017 at 7:23 AM, Pouliot, Scott <Scott.Pouliot@peoplefluent.com> wrote:
> My apologies.  We're running 2 different sets of SOLR instances, the version with the
replication problems is 6.2.0 and NOT 4.8.0.
>
> After reading your questions, I double checked the config and we're set to autoCommit
after a LOOOOONG time.
> <autoCommit>
>        <maxTime>36000000</maxTime>
>        <openSearcher>true</openSearcher>
>      </autoCommit>
>
> Also using softCommit as well:
>
> <autoSoftCommit>
>        <maxTime>60000</maxTime>
>      </autoSoftCommit>
>
> After looking at our other servers, I changed the autoCommit maxTime down to 180000 to
match.  Hoping this might be our issue  ;-)  I'll report back in a bit though!
>
> Thanks for the reminder about commits.  We may have forgotten to change these values
when these new servers got setup.
>
> Scott
>
> -----Original Message-----
> From: Andrea Gazzarini [mailto:gxsubs@gmail.com]
> Sent: Friday, March 10, 2017 3:12 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
>
> Hi Scott,
> that could depend on a lot of things. Some questions:
>
>   * What is your commit policy? Explicit / auto / soft / hard ...
>   * "Other days things are off by a small amount between master and
>     slave"...what do you mean exactly? What is the behaviour you see in
>     terms of index versions between nodes?
>   * Any error / message in the log file?
>
> Honestly I never used Solr 4.8.0 (this specific version) but I have a lot of customers
with other 4.x (mostly 4.10.4) master / slave configuration and they don't have such problems.
>
> Best,
> Andrea
>
> On 09/03/17 22:26, Pouliot, Scott wrote:
>> So we've been having oddball replication issues between master and slave (Not SOLRCloud)
for a few weeks now.  Some days...everything lines up just fine.  Other days things are off
by a small amount between master and slave.
>>
>> Here is the config on the MASTER server replication config:
>>
>> <!-- ReplicationHandler -->
>>    <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>      <lst name="master">
>>        <!--Replicate on 'startup' and 'commit'. 'optimize' is also a valid value
for replicateAfter. -->
>>        <str name="replicateAfter">startup</str>
>>        <str name="replicateAfter">commit</str>
>>
>>        <!--The default value of reservation is 10 secs.See the documentation below
. Normally , you should not need to specify this -->
>>        <str name="commitReserveDuration">00:00:10</str>
>>      </lst>
>>      <!-- keep only 1 backup.  Using this parameter precludes using the "numberToKeep"
request parameter. (Solr3.6 / Solr4.0)-->
>>      <!-- (For this to work in conjunction with "backupAfter" with Solr 3.6.0,
see bug fix https://issues.apache.org/jira/browse/SOLR-3361 )-->
>>      <str name="maxNumberOfBackups">1</str>
>>    </requestHandler>
>>
>> Here is the SLAVE server replication config:
>>
>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>      <lst name="slave">
>>
>>        <!--fully qualified url to the master core. It is possible to pass on this
as a request param for the fetchindex command-->
>>        <str
>> name="masterUrl">http://server01:8080/solr/${solr.core.name}</str>
>>
>>        <!--Interval in which the slave should poll master .Format is HH:mm:ss
. If this is absent slave does not poll automatically.
>>           But a fetchindex can be triggered from the admin or the http API -->
>>        <str name="pollInterval">00:01:00</str>
>>      </lst>
>>    </requestHandler>
>>
>> It seems very sporadic at best.  Sometimes I can manually click "Replicate Now" from
the replication page in the SOLR UI and it will work, sometimes it does nothing.  Sometimes
I can go to the master server, and if the "Optimize" button is available and I click it....it
will replicate as soon as the Optimization is done.
>>
>> Is this a bug in SOLR?  Could we possibly have some settings out of whack here? 
I've been digging around online for a bit and not finding much info here.
>>
>> This is on an older version of SOLR though, so wondering if maybe it used to be a
bug?
>>
>> Thanks!
>>
>> Scott
>
Mime
View raw message