lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2829) improve termquery "pk lookup" performance
Date Tue, 21 Dec 2010 21:14:01 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973889#action_12973889
] 

Robert Muir commented on LUCENE-2829:
-------------------------------------

Hoss Man, well I think if you surely know its a PK field you can definitely do something better,
starting with a custom collector that does something like what you mentioned, with no PQ at
all etc.

But in this case, though i categorized it as PK, the general problem is this:
* in lots of cases we do redundant seeks, like to get the docFreq, then to get the DocsEnum
* in most cases the term dictionary cache helps here because the 2nd time (e.g. getting DocsEnum)
is cached.

Here's the problem with "PK" or "PK-ish" (low freq terms like what wildcards/fuzzies/range
queries hit too):
* our cache doesn't cache "negative" hits, the fact that a term *doesnt* exist in some segment.
* For example in the PK case, if there are 15 segments we always get at most 1 cache hit and

at least 14 misses when getting the DocsEnum, so we do at least 14 wasted seeks always.
* For other low frequency terms that don't exist in all segments (very precise dates or what
have you) 
the same idea applies, just to a lesser extent: the PK is the worst.


> improve termquery "pk lookup" performance
> -----------------------------------------
>
>                 Key: LUCENE-2829
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2829
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Robert Muir
>         Attachments: LUCENE-2829.patch
>
>
> For things that are like primary keys and don't exist in some segments (worst case is
primary/unique key that only exists in 1)
> we do wasted seeks.
> While LUCENE-2694 tries to solve some of this issue with TermState, I'm concerned we
could every backport that to 3.1 for example.
> This is a simpler solution here just to solve this one problem in termquery... we could
just revert it in trunk when we resolve LUCENE-2694,
> but I don't think we should leave things as they are in 3.x

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message