lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rogalon <nico.beche...@me.com>
Subject Re: Function Query performance in combination with filters
Date Tue, 16 Apr 2013 13:25:44 GMT


Am 16. April 2013 um 14:46 schrieb "Yonik Seeley-4 [via Lucene]" <ml-node+s472066n4056299h21@n3.nabble.com>:

> On Tue, Apr 16, 2013 at 7:51 AM, Rogalon <[hidden email]> wrote:
>
> > Hi,
> > I am using pretty complex function queries to completely customize (not only
> > boost) the score of my result documents that are retrieved from an index of
> > approx 10e7 documents. To get to an acceptable level of performance I
> > combine my query with filters in the following way (very short example):
> >
> > q=_val_:"sum(termfreq(fieldname,`word`),termfreq(fieldname2,`word2`))"&fq=fieldname:`word`&fq=fieldname2:`word2`
> >
> > Although I always have (because of the filter) approx 50.000 docs in the
> > result set, the search times vary (depending on the actual query) between
> > 100ms and 6000ms.
> >
> > My understanding was that the scoring function is only applied to the result
> > set from the filters.
>
> That should be the case.
>
> > But based on what I am seeing it seems that a lot more
> > documents are actually put through the _val_ function.
>
> How did you verify this? 
Thanks for taking a look at my problem.

For now - I verified just by taking a look at the query times and doing some simple experiments.

If I am not using the function query at all (q=*:*&fq=...), the approx. 50.000 results
from the filters are always returned within 200-300ms. This is pretty stable. If I have a
(test) index of 50.000 documents (instead of the the 10e7 index) only and I pass every document
through the _val_ query (without any filters), this takes about 150ms which in my case would
be ok.

Applying no filters to the function query on the 10e7 index leads to search times at about
6000ms which is too much.

But if I use the filters as stated above I get returned 50.000 documents but the query times
suddenly start to vary between 100ms and 6000ms. Some of my filters might actually be on stop
words which appear in every other document in the index but that seems to really hurt performance
only if the function query is used.

     Greetings, Nico     
>
>
> -Yonik
> http://lucidworks.com
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://lucene.472066.n3.nabble.com/Function-Query-performance-in-combination-with-filters-tp4056283p4056299.html
>  
> To unsubscribe from Function Query performance in combination with filters, click here.
> NAML
>  



--
View this message in context: http://lucene.472066.n3.nabble.com/Function-Query-performance-in-combination-with-filters-tp4056283p4056312.html
Sent from the Solr - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message