lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prasenjit mukherjee <prasen....@gmail.com>
Subject Re: Solr on netty
Date Wed, 22 Feb 2012 16:19:17 GMT
Thanks for the response.

Yes we have 16 shards/partitions each on 16 different nodes and a
separate master Solr receiving continuous parallel requests from 10
client threads running on a single separate machine.
Our observation was that the perf degraded non linearly as the load (
no of concurrent clients ) increased.

Have some followup questions :

1.  What is the default maxnumber of threads configured when a Solr
instance make calls to other 16 partitions ?

2. How do I increase the max no of connections for solr<-->solr
interactions as u mentioned in your mail ?



On 2/22/12, Yonik Seeley <yonik@lucidimagination.com> wrote:
> On Wed, Feb 22, 2012 at 9:27 AM, prasenjit mukherjee
> <prasen.bea@gmail.com> wrote:
>> Is anybody aware of any effort regarding porting solr to a netty ( or
>> any other async-io based framework ) based framework.
>>
>> Even on medium load ( 10 parallel clients )  with 16 shards
>> performance seems to deteriorate quite sharply compared another
>> alternative ( async-io based ) solution as load increases.
>
> By "16 shards" do you mean you have 16 nodes and each single client
> request causes a distributed search across all of them them?  How many
> concurrent requests are your 10 clients making to each node?
>
> NIO works well when there are many clients, but when servicing those
> client requests only needs intermittent CPU.  That's not the pattern
> we see for search.
> You *can* easily configure Solr's Jetty to use NIO when accepting
> client connections, but it won't do you any good, just as switching to
> Netty wouldn't do anything here.
>
> Where NIO could help a little is with the requests that Solr makes to
> other Solr instances.  Solr is already architected for async
> request-response to other nodes, but the current underlying
> implementation uses HttpClient 3 (which doesn't have NIO).
>
> Anyway, it's unlikely that NIO vs BIO will make much of a difference
> with the numbers you're talking about (16 shards).
>
> Someone else reported that we have the number of connections per host
> set too low, and they saw big gains by increasing this.  There's an
> issue open to make this configurable in 3x:
> https://issues.apache.org/jira/browse/SOLR-3079
> We should probably up the max connections per host by default.
>
> -Yonik
> lucidimagination.com
>

-- 
Sent from my mobile device

Mime
View raw message