lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wettin <karl.wet...@gmail.com>
Subject Re: Slow queries with lots of hits
Date Fri, 05 Dec 2008 06:25:45 GMT
Hi Tim,

is it possible that the slow queries contains terms that are very  
common in your index? If so you could replace those clauses with a  
filter. This would impact the score as filters does nothing with that,  
but if your query contains enough other clauses that should not be a  
problem.

That is how I've in several applications managed to solve the problem  
you describe.


      karl

4 dec 2008 kl. 21.27 skrev Tim Sturge:

> Hi all,
>
> I have an interesting problem with my query traffic. Most of the  
> queries run
> in a fairly short amount of time (< 100ms) but a few take over  
> 1000ms. These
> queries are predominantly those with a huge number of hits (>1  
> million hits
> in a >100 million document index). The time taken (as far as I can  
> tell) is
> for lucene to sit there while it scores and sorts all these results.
>
> However it turns out these queries really don’t have top results.  
> That is,
> of the million documents, there are easily 10000 which are decent  
> results
> (basically those above some threshold score). Frankly, just  
> returning some
> consistent (so paging and reload work) but
> otherwise arbitrary ranking of these 10000 results would be more  
> than good
> enough.
>
> It seems to me that a solution would be to impose some sort of  
> pseudo-random
> filter (e.g. consider only every n-th document assuming they are  
> uniformly
> distributed). I’m wondering if anyone else has experience with this  
> sort of
> issue and what solutions they have found to work well in practice.
>
> Thanks,
>
> Tim


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message