lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: strange performance issue with many shards on one server
Date Wed, 28 Sep 2011 14:40:25 GMT
On Wed, 2011-09-28 at 12:58 +0200, Frederik Kraus wrote:
> - 10 shards per server (needed for response times) running in a single tomcat instance

Have you tested that sharding actually decreases response times in your
case? I see the idea in decreasing response times with sharding at the
cost of decreasing throughput, but the added overhead of merging is
non-trivial.

> - each query queries all 20 shards (distributed search)
> 
> - each shard holds about 1.5 mio documents (small shards are needed due to rather complex
queries)
> - all caches are warmed / high cache hit rates (99%) etc.

> Now for some reason we cannot seem to fully utilize all CPU power (no disk IO), ie. increasing
concurrent users doesn't increase CPU-Load at a point, decreases throughput and increases
the response times of the individual queries.

It sounds as if there's a hard limit on the number of concurrent users
somewhere. I am no expert in httpclient, but the blocked threads in your
thread dump seems to indicate that they wait for connections to be
established rather than for results to be produced.

I seem to remember that tomcat has a default limit on 200 concurrent
connections and with 10 shards/search, that is just 200 / (10
shard_connections + 1 incoming_connection) = 18 concurrent searches.

> Also 1-2% of the queries take significantly longer: avg somewhere at 100ms while 1-2%
take 1.5s or longer. 

Could be garbage collection, especially since it shows under high load
which might result in more old objects and thereby trigger full gc.


Mime
View raw message