lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LUCENE-5879) Add auto-prefix terms to block tree terms dict
Date Fri, 08 Aug 2014 09:31:12 GMT
Michael McCandless created LUCENE-5879:
------------------------------------------

             Summary: Add auto-prefix terms to block tree terms dict
                 Key: LUCENE-5879
                 URL: https://issues.apache.org/jira/browse/LUCENE-5879
             Project: Lucene - Core
          Issue Type: New Feature
          Components: core/codecs
            Reporter: Michael McCandless
            Assignee: Michael McCandless
             Fix For: 5.0, 4.10


This cool idea to generalize numeric/trie fields came from Adrien:

Today, when we index a numeric field (LongField, etc.) we pre-compute
(via NumericTokenStream) outside of indexer/codec which prefix terms
should be indexed.

But this can be inefficient: you set a static precisionStep, and
always add those prefix terms regardless of how the terms in the field
are actually distributed.  Yet typically in real world applications
the terms have a non-random distribution.

So, it should be better if instead the terms dict decides where it
makes sense to insert prefix terms, based on how dense the terms are
in each region of term space.

This way we can speed up query time for both term (e.g. infix
suggester) and numeric ranges, and it should let us use less index
space and get faster range queries.
 
This would also mean that min/maxTerm for a numeric field would now be
correct, vs today where the externally computed prefix terms are
placed after the full precision terms, causing hairy code like
NumericUtils.getMaxInt/Long.  So optos like LUCENE-5860 become
feasible.

The terms dict can also do tricks not possible if you must live on top
of its APIs, e.g. to handle the adversary/over-constrained case when a
given prefix has too many terms following it but finer prefixes
have too few (what block tree calls "floor term blocks").




--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message