lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <>
Subject Re: SolrCloud scaling/optimization for high request rate
Date Tue, 30 Oct 2018 20:04:58 GMT
On 10/29/2018 7:24 AM, Sofiya Strochyk wrote:
> Actually the smallest server doesn't look bad in terms of performance, 
> it has been consistently better that the other ones (without 
> replication) which seems a bit strange (it should be about the same or 
> slightly worse, right?). I guess the memory being smaller than index 
> doesn't cause problems due to the fact that we use SSDs.

SSD, while fast, is nowhere near as fast as main memory. As I said, the 
memory numbers might cause performance problems, or they might not.  
Glad you're in the latter category.

> What if we are sending requests to machine which is part of the 
> cluster but doesn't host any shards? Does it handle the initial 
> request and merging of the results, or this has to be handled by one 
> of the shards anyway?
> Also i was thinking "more shards -> each shard searches smaller set of 
> documents -> search is faster". Or is the overhead for merging results 
> bigger than overhead from searching larger set of documents?

If every shard is on its own machine, many shards might not be a 
performance bottleneck with a high query rate.  The more shards you 
have, the more the machine doing the aggregation must do to produce results.

SolrCloud complicates the situation further.  It normally does load 
balancing of all requests that come in across the cloud.  So the machine 
handling the request might not be the machine where you SENT the request.

>> Very likely the one with a higher load is the one that is aggregating 
>> shard requests for a full result.
> Is there a way to confirm this? Maybe the aggregating shard is going 
> to have additional requests in its solr.log?

The logfiles on your servers should be verbose enough to indicate what 
machines are handling which parts of the request.

>> Most Solr performance issues are memory related.  With an extreme 
>> query rate, CPU can also be a bottleneck, but memory will almost 
>> always be the bottleneck you run into first.
> This is the advice i've seen often, but how exactly can we run out of 
> memory if total RAM is 128, heap is 8 and index size is 80. Especially 
> since node with 64G runs just as fine if not better.

Even when memory is insufficient, "running out" of memory generally 
doesn't happen unless the heap is too small.Java will work within the 
limits imposed by the system if it can. For OS disk cache, the OS tries 
to be as smart as it can about which data stays in the cache and which 
data is discarded.

>> A lot of useful information can be obtained from the GC logs that 
>> Solr's built-in scripting creates.  Can you share these logs?
>> The screenshots described here can also be very useful for 
>> troubleshooting:
> I have attached some GC logs and screenshots, hope these are helpful 
> (can only attach small files)

Only one attachment made it to the list.  I'm surprised that ANY of them 
made it -- usually they don't.  Generally you need to use a file sharing 
website and provide links.  Dropbox is one site that works well.  Gist 
might also work.

The GC log that made it through (solr_gc.log.7.1) is only two minutes 
long.  Nothing useful can be learned from a log that short.  It is also 
missing the information at the top about the JVM that created it, so I'm 
wondering if you edited the file so it was shorter before including it.


View raw message