lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Stoney <Karl.Sto...@autotrader.co.uk.INVALID>
Subject Re: Solr Searcher 100% Latency Spike
Date Wed, 29 Jan 2020 21:11:22 GMT
Hey Shawn,
Thanks for the reply - funnily enough that is exactly what i'm trialing now.  I've significantly
lowered the autoWarm (as well as the size) and still have a 0.95+ cache hit rate through searcher
loads.

I'm going to continue to tweak these values down so long as i keep the hit rate above 90,
which should reduce some memory pressure at least.

Thanks
Karl
________________________________
From: Shawn Heisey <apache@elyograg.org>
Sent: 29 January 2020 21:01
To: solr-user@lucene.apache.org <solr-user@lucene.apache.org>
Subject: Re: Solr Searcher 100% Latency Spike

On 1/29/2020 12:44 PM, Karl Stoney wrote:
> Looking for a bit of support here.  When we soft commit (every 10
> minutes), we get a latency spike that means response times for solr are
> loosely double, as you can see in this screenshot:

Attachments almost never make it to the list.  We cannot see any of your
screenshots.

> They do correlate to filterCache warmup, which seem to take between 10s
> and 30s:
>
> We don't have any other caches enabled, due to the high level of
> cardinality of the queries.
>
> The spikes are specifically on /select
>
>
> We have the following autowarm configuration for the filterCache:
>
>          <filterCache class="solr.FastLRUCache"
>                       size="8192"
>                       initialSize="8192"
>                       cleanupThread="true"
>                       autowarmCount="900"/>

Autowarm, especially on filterCache, can be an extremely lengthy
process.  What Solr must do in order to warm the cache here is execute
up to 900 queries, sequentially, on the new index.  That can take a lot
of time and use a lot of resources like CPU and I/O.

In order to reduce the impact of cache warming, I had to reduce my own
autowarmCount on the filterCache to 4.

Thanks,
Shawn

This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place,
Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files
transmitted with it are confidential and may be legally privileged, and intended solely for
the use of the individual or entity to whom they are addressed. If you have received this
email in error please notify the sender. This email message has been swept for the presence
of computer viruses.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message