lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Performance vs. maxBufferedAddsPerServer=10
Date Tue, 30 Jul 2013 03:04:38 GMT
Yes, the internal document forwarding path is different and does not use the CloudSolrServer.
It currently works with a buffer of 10.

- Mark

On Jul 29, 2013, at 3:10 PM, Erick Erickson <erickerickson@gmail.com> wrote:

> Why wouldn't it? Or are you saying that the routing to replicas
> from the leader also 10/packet? Hmmm, hadn't thought of that...
> 
> On Mon, Jul 29, 2013 at 7:58 AM, Mark Miller <markrmiller@gmail.com> wrote:
>> SOLR-4816 won't address this - it will just speed up *different* parts. There are
other things that will need to be done to speed up that part.
>> 
>> - Mark
>> 
>> On Jul 26, 2013, at 3:53 PM, Erick Erickson <erickerickson@gmail.com> wrote:
>> 
>>> This is current a hard-coded limit from what I've understood. From what
>>> I remember, Mark said Yonik said that there are reasons to make the
>>> packets that size. But whether this is empirically a Good Thing I don't know.
>>> 
>>> SOLR-4816 will address this a different way by making SolrJ batch up
>>> the docs and send them to the right leader, which should pretty much remove
>>> any performance consideration here.
>>> 
>>> There's some anecdotal evidence that changing that in the code might
>>> improve throughput, but I don't remember the details.
>>> 
>>> FWIW
>>> Erick
>>> 
>>> On Thu, Jul 25, 2013 at 7:09 AM, Otis Gospodnetic
>>> <otis.gospodnetic@gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> Context:
>>>> * https://issues.apache.org/jira/browse/SOLR-4956
>>>> * http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
>>>> 
>>>> As you can see, maxBufferedAddsPerServer = 10.
>>>> 
>>>> We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
>>>> We batch 20K docs for performance reasons. But then the receiving node
>>>> ends up sending VERY small batches of just 10 docs around for indexing
>>>> and we lose the benefit of batching those 20K docs in the first place.
>>>> 
>>>> Our app is "add only".
>>>> 
>>>> Is there anything one can do to avoid performance loss associated with
>>>> maxBufferedAddsPerServer=10?
>>>> 
>>>> Thanks,
>>>> Otis
>>>> --
>>>> Solr & ElasticSearch Support -- http://sematext.com/
>>>> Performance Monitoring -- http://sematext.com/spm
>> 


Mime
View raw message