lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1420) Similarity.lengthNorm and positionIncrement=0
Date Thu, 16 Oct 2008 09:30:46 GMT


Michael McCandless commented on LUCENE-1420:

Re 1 & 2: I agree.  Even though the back compat is "looser" for contribs, we still should
not break it unless we have to.  In this case we definitely don't have to.

Re 3 & 4: I like this in principle, but I'm nervous about making FieldInvertState public
since it's such a new API.  I guess if we put the "this API is new & subject to change"
caveat on there, to reserve the right to change it, that's OK.  I also like having Similiarity.computeNorm
factor in the boost, and extending FieldInverState to track the number of field occurrences
that had been processed.

Andrzej, can you update the patch to address these issues?

> Similarity.lengthNorm and positionIncrement=0
> ---------------------------------------------
>                 Key: LUCENE-1420
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.3, 2.9
>            Reporter: Andrzej Bialecki 
>            Assignee: Michael McCandless
>             Fix For: 2.3.3, 2.9
>         Attachments: similarity.patch
> Calculation of lengthNorm factor should in some cases take into account the number of
tokens with positionIncrement=0. This should be made optional, to support two different scenarios:
> * when analyzers insert artificially constructed tokens into TokenStream (e.g. ASCII-fied
versions of accented terms, stemmed terms), and it's unlikely that users submit queries containing
both versions of tokens: in this case lengthNorm calculation should ignore the tokens with
> * when analyzers insert synonyms, and it's likely that users may submit queries that
contain multiple synonymous terms: in this case the lengthNorm should be calculated as it
is now, i.e. it should take into account all terms no matter what is their positionIncrement.
> The default should be backward-compatible, i.e. it should count all tokens.
> (See also the discussion here: )

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message