lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sofiya Strochyk>
Subject Re: SolrCloud scaling/optimization for high request rate
Date Fri, 26 Oct 2018 17:21:52 GMT
Thanks Erick,

1. We already use Solr 7.5, upgraded some of our nodes only recently to 
see if this eliminates the difference in performance (it doesn't, but 
I'll test and see if the situation with replicas syncing/recovery has 
improved since then)

2. Yes, we only open searcher once every 30 minutes so it is not an NRT 
case. But it is only recommended 

to use NRT/TLOG/TLOG+PULL replica types together (currently we have all 
NRT replicas), would you suggest we change leaders to TLOG and slaves to 
PULL? And this would also eliminate the redundancy provided by 
replication because PULL replicas can't become leaders, right?

3. Yes but then it would be reflected in iowait metric, which is almost 
always near zero on our servers. Is there anything else Solr could be 
waiting for, and is there a way to check it? If we are going to need 
even more servers for faster response and faceting then there must be a 
way to know which resource we should get more of.

5. Yes, docValues are enabled for the fields we sort on (except score 
which is an internal field); _version_ is left at default i think 
(type="long" indexed="false" stored="false", and it's also marked as 
having DocValues in the admin UI)

6. QPS and response time seem to be about the same with and without 
indexing; server load also looks about the same so i assume indexing 
doesn't take up a lot of resources (a little strange, but possible if it 
is limited by network or some other things from point 3).

7. Will try using G1 if nothing else helps... Haven't tested it yet 
because it is considered unsafe and i'd like to have all other options 
exhausted first. (And even then it is probably going to be a minor 
improvement? How much more efficient could it possibly be?)

On 26.10.18 19:18, Erick Erickson wrote:
> Some ideas:
> 1> What version of Solr? Solr 7.3 completely re-wrote Leader Initiated
> Recovery and 7.5 has other improvements for recovery, we're hoping
> that the recovery situation is much improved.
> 2> In the 7x code line, there are TLOG and PULL replicas. As of 7.5,
> you can set up so the queries are served by replica type, see:
> This might help you
> out. This moves all the indexing to the leader and reserves the rest
> of the nodes for queries only, using old-style replication. I'm
> assuming from your commit rate that latency between when updates
> happen and the updates are searchable isn't a big concern.
> 3> Just because the CPU isn't 100% doesn't mean Solr is running flat
> out. There's I/O waits while sub-requests are serviced and the like.
> 4> As for how to add faceting without slowing down querying, there's
> no way. Extra work is extra work. Depending on _what_ you're faceting
> on, you may be able to do some tricks, but without details it's hard
> to say. You need to get the query rate target first though ;)
> 5> OOMs Hmm, you say you're doing complex sorts, are all fields
> involved in sorts docValues=true? They have to be to be used in
> function queries of course, but what about any fields that aren't?
> What about your _version_ field?
> 6>  bq. "...indexed 2 times/day, as fast as the SOLR allows..." One
> experiment I'd run is to test your QPS rate when there was _no_
> indexing going on. That would give you a hint as to whether the
> TLOG/PULL configuration would be helpful. There's been talk of
> separate thread pools for indexing and querying to give queries a
> better shot at the CPU, but that's not in place yet.
> 7> G1GC may also help rather than CMS, but as you're well aware GC
> tuning "is more art than science" ;).
> Good luck!
> Erick
> On Fri, Oct 26, 2018 at 8:55 AM Sofiya Strochyk <> wrote:
>> Hi everyone,
>> We have a SolrCloud setup with the following configuration:
>> 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon E5-1650v2, 12 cores,
with SSDs)
>> One collection, 4 shards, each has only a single replica (so 4 replicas in total),
using compositeId router
>> Total index size is about 150M documents/320GB, so about 40M/80GB per node
>> Zookeeper is on a separate server
>> Documents consist of about 20 fields (most of them are both stored and indexed),
average document size is about 2kB
>> Queries are mostly 2-3 words in the q field, with 2 fq parameters, with complex sort
expression (containing IF functions)
>> We don't use faceting due to performance reasons but need to add it in the future
>> Majority of the documents are reindexed 2 times/day, as fast as the SOLR allows,
in batches of 1000-10000 docs. Some of the documents are also deleted (by id, not by query)
>> autoCommit is set to maxTime of 1 minute with openSearcher=false and autoSoftCommit
maxTime is 30 minutes with openSearcher=true. Commits from clients are ignored.
>> Heap size is set to 8GB.
>> Target query rate is up to 500 qps, maybe 300, and we need to keep response time
at <200ms. But at the moment we only see very good search performance with up to 100 requests
per second. Whenever it grows to about 200, average response time abruptly increases to 0.5-1
second. (Also it seems that request rate reported by SOLR in admin metrics is 2x higher than
the real one, because for every query, every shard receives 2 requests: one to obtain IDs
and second one to get data by IDs; so target rate for SOLR metrics would be 1000 qps).
>> During high request load, CPU usage increases dramatically on the SOLR nodes. It
doesn't reach 100% but averages at 50-70% on 3 servers and about 93% on 1 server (random server
each time, not the smallest one).
>> The documentation mentions replication to spread the load between the servers. We
tested replicating to smaller servers (32GB RAM, Intel Core i7-4770). However, when we tested
it, the replicas were going out of sync all the time (possibly during commits) and reported
errors like "PeerSync Recovery was not successful - trying replication." Then they proceed
with replication which takes hours and the leader handles all requests singlehandedly during
that time. Also both leaders and replicas started encountering OOM errors (heap space) for
unknown reason. Heap dump analysis shows that most of the memory is consumed by [J (array
of long) type, my best guess would be that it is "_version_" field, but it's still unclear
why it happens. Also, even though with replication request rate and CPU usage drop 2 times,
it doesn't seem to affect mean_ms, stddev_ms or p95_ms numbers (p75_ms is much smaller on
nodes with replication, but still not as low as under load of <100 requests/s).
>> Garbage collection is much more active during high load as well. Full GC happens
almost exclusively during those times. We have tried tuning GC options like suggested here
and it didn't change things though.
>> My questions are
>> How do we increase throughput? Is replication the only solution?
>> if yes - then why doesn't it affect response times, considering that CPU is not 100%
used and index fits into memory?
>> How to deal with OOM and replicas going into recovery?
>> Is memory or CPU the main problem? (When searching on the internet, i never see CPU
as main bottleneck for SOLR, but our case might be different)
>> Or do we need smaller shards? Could segments merging be a problem?
>> How to add faceting without search queries slowing down too much?
>> How to diagnose these problems and narrow down to the real reason in hardware or
>> Any help would be much appreciated.
>> Thanks!
>> --
>> Sofiia Strochyk

Email Signature
*Sofiia Strochyk
* <>
	InterLogic <>

Facebook icon <> LinkedIn 
icon <>

View raw message