lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Commented: (LUCENE-2829) improve termquery "pk lookup" performance
Date Tue, 21 Dec 2010 21:14:01 GMT


Robert Muir commented on LUCENE-2829:

Hoss Man, well I think if you surely know its a PK field you can definitely do something better,
starting with a custom collector that does something like what you mentioned, with no PQ at
all etc.

But in this case, though i categorized it as PK, the general problem is this:
* in lots of cases we do redundant seeks, like to get the docFreq, then to get the DocsEnum
* in most cases the term dictionary cache helps here because the 2nd time (e.g. getting DocsEnum)
is cached.

Here's the problem with "PK" or "PK-ish" (low freq terms like what wildcards/fuzzies/range
queries hit too):
* our cache doesn't cache "negative" hits, the fact that a term *doesnt* exist in some segment.
* For example in the PK case, if there are 15 segments we always get at most 1 cache hit and

at least 14 misses when getting the DocsEnum, so we do at least 14 wasted seeks always.
* For other low frequency terms that don't exist in all segments (very precise dates or what
have you) 
the same idea applies, just to a lesser extent: the PK is the worst.

> improve termquery "pk lookup" performance
> -----------------------------------------
>                 Key: LUCENE-2829
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Robert Muir
>         Attachments: LUCENE-2829.patch
> For things that are like primary keys and don't exist in some segments (worst case is
primary/unique key that only exists in 1)
> we do wasted seeks.
> While LUCENE-2694 tries to solve some of this issue with TermState, I'm concerned we
could every backport that to 3.1 for example.
> This is a simpler solution here just to solve this one problem in termquery... we could
just revert it in trunk when we resolve LUCENE-2694,
> but I don't think we should leave things as they are in 3.x

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message