lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Created: (LUCENE-1195) Performance improvement for TermInfosReader
Date Tue, 26 Feb 2008 23:54:08 GMT
This can only possibly work if all 50,000 terms are held in memory,  
otherwise the cache management overhead is going to matter.

What is the number of terms in the database? What is the distribution  
of the terms used in the test case (differ for each query)?

Performance tests like this without the detailed data is not very  
accurate (or useful).

On Feb 26, 2008, at 5:00 PM, Michael Busch (JIRA) wrote:

> Performance improvement for TermInfosReader
> -------------------------------------------
>
>                  Key: LUCENE-1195
>                  URL: https://issues.apache.org/jira/browse/ 
> LUCENE-1195
>              Project: Lucene - Java
>           Issue Type: Improvement
>           Components: Index
>             Reporter: Michael Busch
>             Assignee: Michael Busch
>             Priority: Minor
>              Fix For: 2.4
>
>
> Currently we have a bottleneck for multi-term queries: the  
> dictionary lookup is being done
> twice for each term. The first time in Similarity.idf(), where  
> searcher.docFreq() is called.
> The second time when the posting list is opened (TermDocs or  
> TermPositions).
>
> The dictionary lookup is not cheap, that's why a significant  
> performance improvement is
> possible here if we avoid the second lookup. An easy way to do this  
> is to add a small LRU
> cache to TermInfosReader.
>
> I ran some performance experiments with an LRU cache size of 20,  
> and an mid-size index of
> 500,000 documents from wikipedia. Here are some test results:
>
> 50,000 AND queries with 3 terms each:
> old:                  152 secs
> new (with LRU cache): 112 secs (26% faster)
>
> 50,000 OR queries with 3 terms each:
> old:                  175 secs
> new (with LRU cache): 133 secs (24% faster)
>
> For bigger indexes this patch will probably have less impact, for  
> smaller once more.
>
> I will attach a patch soon.
>
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message