lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <erik.hatc...@gmail.com>
Subject Re: are facets or MatchAllDocsQuery not cached?
Date Thu, 06 Aug 2015 13:29:40 GMT
> Do you think an all_parameters -> complete_response cache is possible?
> It could be initialized right before or during warmup and would not take to much memory.

This is along the lines of Solr’s 304 capabilities.  See <https://cwiki.apache.org/confluence/display/solr/RequestDispatcher+in+SolrConfig#RequestDispatcherinSolrConfig-httpCachingElement
<https://cwiki.apache.org/confluence/display/solr/RequestDispatcher+in+SolrConfig#RequestDispatcherinSolrConfig-httpCachingElement>>


Hooking some kind of caching layer in front of Solr that can leverage the 304 responses is
something I’ve heard of folks doing.

—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com <http://www.lucidworks.com/>




> On Aug 6, 2015, at 8:32 AM, Bernd Fehling <bernd.fehling@uni-bielefeld.de> wrote:
> 
> Thanks a lot, your statement makes me feel better :-)
> 
> It "feels" like this behavior showed up after changing to docValues for sorting,
> because before the 99 percentile for qtime was at 550ms average and 1.4 seconds at max.
> 
> So my assumption is that the inverted index on the sort fields (when _not_ using
> docValues) helps here with the disadvantage of using a lot of heap :-(
> 
> Do you think an all_parameters -> complete_response cache is possible?
> It could be initialized right before or during warmup and would not take to much memory.
> 
> Regards
> Bernd
> 
> 
> 
> Am 06.08.2015 um 13:38 schrieb Toke Eskildsen:
>> On Thu, 2015-08-06 at 13:00 +0200, Bernd Fehling wrote:
>>> Single Index Solr 4.10.4, optimized Index, 76M docs, 235GB index size.
>>> 
>>> I was analysing my solr logs and it turned out that I have some queries
>>> which are above 30 seconds qtime while normally the qtime is below 1 second.
>>> Looking closer about the queries it turned out that this is for MatchAllDocsQuery(*:*).
>>> Next was to turn debugQuery on and see where the bottleneck is.
>>> The result was that the facetting is consuming most of the qtime.
>>> 
>>> So the question is, are facets or is facetting not cached?
>> 
>> As far as I know it is not. 35 seconds for a match-all faceting sounds
>> fairly on par with what we are seeing (250M docs, 900GB shard).
>> 
>> Of course response time is very depending on the field itself. If you
>> have very few unique values in your facet field(s), you might try
>> facet.method=enum. If that is not the case, your best bet would probably
>> be to cache the match-all outside of Solr.
>> 
>>> My assumption is that the queryResultCache is catching such a MatchAllDocsQuery(*:*).
>> 
>> It only stores the docIDs.
>> 
>> I don't know why there is is no all_parameters -> complete_response
>> cache in Solr.
>> 
>> - Toke Eskildsen, State and University Library, Denmark
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message