lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Upayavira ...@odoko.co.uk>
Subject Re: are facets or MatchAllDocsQuery not cached?
Date Thu, 06 Aug 2015 17:01:23 GMT
Also, attempting to facet across a large number of docs is going to take
some time. Perhaps you might gain some performance benefit by sharding
your index?

Upayavira

On Thu, Aug 6, 2015, at 04:48 PM, Mikhail Khludnev wrote:
> On Thu, Aug 6, 2015 at 3:56 PM, Bernd Fehling <
> bernd.fehling@uni-bielefeld.de> wrote:
> 
> >
> >
> > Am 06.08.2015 um 14:33 schrieb Upayavira:
> > > Typically such performance issues with faceting are to do with the time
> > > spend uninverting the index before calculating the facet counts.
> > >
> > > If you indexed the fields with docValues enabled, perhaps you could then
> > > use them for faceting, which might improve performance.
> >
> > Well, this is against my observations. When I used uninverted fields
> > without
> > docValues I had a much better 99 percentile qtime but a very high heap
> > consumption.
> > Now with docValues the heap usage went down, but the 99 percentile
> > qtime for MatchAllDocsQuery(*:*) went up to above 33 seconds.
> >
> 
>  Note about performance optimization for DocValues faceting in
>  forthcoming
> 5.3
> 
> 
> >
> > >
> > > If you are using a non-docValues field, and the second query is faster,
> > > then you could add the query to your static warming, look for
> > > newSearcher in your solrconfig.xml. That will execute your query,
> > > warming the caches used by faceting, before a new searcher is made
> > > available for searches.
> >
> > The q=*.* with sorting and facetting is always the first query I'm doing
> > at static warming and it helped until switching to docValues :-(
> >
> > Bernd
> >
> > >
> > > Upayavira
> > >
> > > On Thu, Aug 6, 2015, at 12:38 PM, Toke Eskildsen wrote:
> > >> On Thu, 2015-08-06 at 13:00 +0200, Bernd Fehling wrote:
> > >>> Single Index Solr 4.10.4, optimized Index, 76M docs, 235GB index size.
> > >>>
> > >>> I was analysing my solr logs and it turned out that I have some queries
> > >>> which are above 30 seconds qtime while normally the qtime is below
1
> > second.
> > >>> Looking closer about the queries it turned out that this is for
> > MatchAllDocsQuery(*:*).
> > >>> Next was to turn debugQuery on and see where the bottleneck is.
> > >>> The result was that the facetting is consuming most of the qtime.
> > >>>
> > >>> So the question is, are facets or is facetting not cached?
> > >>
> > >> As far as I know it is not. 35 seconds for a match-all faceting sounds
> > >> fairly on par with what we are seeing (250M docs, 900GB shard).
> > >>
> > >> Of course response time is very depending on the field itself. If you
> > >> have very few unique values in your facet field(s), you might try
> > >> facet.method=enum. If that is not the case, your best bet would probably
> > >> be to cache the match-all outside of Solr.
> > >>
> > >>> My assumption is that the queryResultCache is catching such a
> > MatchAllDocsQuery(*:*).
> > >>
> > >> It only stores the docIDs.
> > >>
> > >> I don't know why there is is no all_parameters -> complete_response
> > >> cache in Solr.
> > >>
> > >> - Toke Eskildsen, State and University Library, Denmark
> > >>
> > >>
> >
> 
> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
> 
> <http://www.griddynamics.com>
> <mkhludnev@griddynamics.com>

Mime
View raw message