lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: Slow qTime for distributed search
Date Tue, 09 Apr 2013 20:38:36 GMT
On 4/9/2013 2:10 PM, Manuel Le Normand wrote:
> Thanks for replying.
> My config:
>
>     - 40 dedicated servers, dual-core each
>     - Running Tomcat servlet on Linux
>     - 12 Gb RAM per server, splitted half between OS and Solr
>     - Complex queries (up to 30 conditions on different fields), 1 qps rate
>
> Sharding my index was done for two reasons, based on 2 servers (4shards)
> tests:
>
>     1. As index grew above few million of docs qTime raised greatly, while
>     sharding the index to smaller pieces (about 0.5M docs) gave way better
>     results, so I bound every shard to have 0.5M docs.
>     2. Tests showed i was cpu-bounded during queries. As i have low qps rate
>     (emphasize: lower than expected qTime) and as a query runs single-threaded
>     on each shard, it made sense to accord a cpu to each shard.
>
> For the same amount of docs per shards I do expect a raise in total qTime
> for the reasons:
>
>     1. The response should wait for the slowest shard
>     2. Merging the responses from 40 different shards takes time
>
> What i understand from your explanation is that it's the merging that takes
> time and as qTime ends only after the second retrieval phase, the qTime on
> each shard will take longer. Meaning during a significant proportion of the
> first query phase (right after the [id,score] are retieved), all cpu's are
> idle except the response-merger thread running on a single cpu. I thought
> of the merge as a simple sorting of [id,score], way more simple than
> additional 300 ms cpu time.
>
> Why would a RAM increase improve my performances, as it's a
> "response-merge" (CPU resource) bottleneck?

If you have not tweaked the Tomcat configuration, that can lead to 
problems, but if your total query volume is really only one query per 
second, this is probably not a worry for you.  A tomcat connector can be 
configured with a maxThreads parameter.  The recommended value there is 
10000, but Tomcat defaults to 200.

You didn't include the index sizes.  There's half a million docs per 
shard, but I don't know what that translates to in terms of MB or GB of 
disk space.

On another email thread you mention that your documents are about 50KB 
each.  That would translate to an index that's at least 25GB, possibly 
more.  That email thread also says that optimization for you takes an 
hour, further indications that you've got some really big indexes.

You're saying that you have given 6GB out of the 12GB to Solr, leaving 
only 6GB for the OS and caching.  Ideally you want to have enough RAM to 
cache the entire index, but in reality you can usually get away with 
caching between half and two thirds of the index.  Exactly what ratio 
works best is highly dependent on your schema.

If my numbers are even close to right, then you've got a lot more index 
on each server than available RAM.  Based on what I can deduce, you 
would want 24 to 48GB of RAM per server.  If my numbers are wrong, then 
this estimate is wrong.

I would be interested in seeing your queries.  If the complexity can be 
expressed as filter queries that get re-used a lot, the filter cache can 
be a major boost to performance.  Solr's caches in general can make a 
big difference.  There is no guarantee that caches will help, of course.

Thanks,
Shawn


Mime
View raw message