lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wun...@wunderwood.org>
Subject Re: firstSearcher cache warming with own QuerySenderListener
Date Fri, 25 Sep 2015 16:43:13 GMT
Right.

I chose the twenty most frequent terms from our documents and use those for cache warming.
The list of most frequent terms is pretty stable in most collections.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Sep 25, 2015, at 8:38 AM, Erick Erickson <erickerickson@gmail.com> wrote:
> 
> That's what the firstSearcher event in solrconfig.xml is for, exactly the
> case of autowarming Solr when it's just been started. The queries you put
> in that event are fired only when the server starts.
> 
> So I'd just put my queries there. And you do not have to put a zillion
> queries here. Start with one that mentions all the facets you intend to
> use, sorts by all the various sort fields you use, perhaps (if you have any
> _very_ common filter queries) put those in too.
> 
> Then analyze the queries that are still slow when issued the first time
> after startup and add what you suspect are the relevant bits to the
> firstSearcher query (or queries).
> 
> I suggest that this is a much easier thing to do, and focus efforts on why
> you are shutting down your Solr servers often enough that anyone notices..
> 
> Best,
> Erick
> 
> 
> 
> On Fri, Sep 25, 2015 at 8:31 AM, Christian Reuschling <
> christian.reuschling@gmail.com> wrote:
> 
>> Hey all,
>> 
>> we want to avoid cold start performance issues when the caches are cleared
>> after a server restart.
>> 
>> For this, we have written a SearchComponent that saves least recently used
>> queries. These are
>> written to a file inside a closeHook of a SolrCoreAware at server shutdown.
>> 
>> The plan is to perform these queries at server startup to warm up the
>> caches. For this, we have
>> written a derivative of the QuerySenderListener and configured it as
>> firstSearcher listener in
>> solrconfig.xml. The only difference to the origin QuerySenderListener is
>> that it gets it's queries
>> from the formerly dumped lru queries rather than getting them from the
>> config file.
>> 
>> It seems that everything is called correctly, and we have the impression
>> that the query response
>> times for the dumped queries are sometimes slightly better than without
>> this warming.
>> 
>> Nevertheless, there is still a huge difference against the times when we
>> manually perform the same
>> queries once, e.g. from a browser. If we do this, the second time we
>> perform these queries they
>> respond much faster (up to 10 times) than the response times after the
>> implemented warming.
>> 
>> It seems that not all caches are warmed up during our warming. And because
>> of these huge
>> differences, I doubt we missed something.
>> 
>> The index has about 25M documents, and is splitted into two shards in a
>> cloud configuration, both
>> shards are on the same server instance for now, for testing purposes.
>> 
>> Does anybody have an idea? I tried to disable lazy field loading as a
>> potential issue, but with no
>> success.
>> 
>> 
>> Cheers,
>> 
>> Christian
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message