lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Solr crawls during replication
Date Fri, 03 Sep 2010 01:11:42 GMT
Yes, the rsync scripts are still there. And they still work fine. It
definitely helps to be a Unix shell wiz.

You would add an option to the rsync call in the scripts that does
rsync throttling.

Rsync is just a standard copying tool in the SSH toolsuite. It's 12
years old and works quite well.

Lance

On Thu, Sep 2, 2010 at 8:31 AM, Mark <static.void.dev@gmail.com> wrote:
>  On 8/6/10 5:03 PM, Chris Hostetter wrote:
>>
>> : We have an index around 25-30G w/ 1 master and 5 slaves. We perform
>> : replication every 30 mins. During replication the disk I/O obviously
>> shoots up
>> : on the slaves to the point where all requests routed to that slave take
>> a
>> : really long time... sometimes to the point of timing out.
>> :
>> : Is there any logical or physical changes we could make to our
>> architecture to
>> : overcome this problem?
>>
>> If the problem really is disk I/O then perhaps you don't have enough RAM
>> set asside for the filesystem cache to keep the "current" index in memory?
>>
>> I've seen people have this type of problem before, but usually it's
>> network I/O that is the bottleneck, in which case using multiple NICs on
>> your slaves (one for client requests, one for replication)
>>
>> I think at one point there was also talk about leveraging an rsync option
>> to force snappuller to throttle itself an only use a max amount of
>> bandwidth -- but then we moved away from script based replication to java
>> based replication and i don't think the Java Network/IO system suports
>> that type of throttling.  However: you might be able to configure it in
>> your switches/routers (ie: only let the slaves use X% of their total
>> badwidth to talk to the master)
>>
>>
>> -Hoss
>>
> Thanks for the suggestions. Our slaves have 12G with 10G dedicated to the
> JVM.. too much?
>
> Are the rysnc snappuller featurs still available in 1.4.1? I may try that to
> see if helps. Configuration of the switches may also be possible.
>
> Also, would you mind explaining your second point... using dual NIC cards.
> How can this be accomplished/configured. Thanks for you help
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message