lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-1908) Similarity javadocs for scoring function to relate more tightly to scoring models in effect
Date Sat, 12 Sep 2009 05:38:57 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754470#action_12754470
] 

Mark Miller edited comment on LUCENE-1908 at 9/11/09 10:37 PM:
---------------------------------------------------------------

Looks great!

bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing performance considerstions
(?).

Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)| normalization
factor with our document length factor?

That's how it appears to me anyway -

for |V(d)| you have many options right?

the cosine normalization - your standard euclidean length - |V(d)| 
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene uses by default
or Okapi's formula, 
or ...

So we are replacing (which everyone generally does) not dropping right?

And I don't think we are replacing for performance reasons (though it is complicated to calculate)
- we are replacing because its not a great normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer documents will
still have higher tf's and more distinct terms - so it unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics). So its not
generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and short preferred
too much)

Again, I'm not an IR guy, but thats what my modest take is.

* edit: I suppose you could argue that you could do cosine normalization and then a further
normalization approach on that, and in that sense we are dropping the cosine normalization
because its too expensive. But from what I can see, it appears more the case that you try
and use a normalization factor that can just replace cosine normalization - like the pivoted
normalization which rotates the cosine normalization curve. I think pivoted is something like
1/root(stuff, ie unique terms), so our norm of 1/root(L) is of a similar, simpler, vein. So
thats why I think we are not really dropping it - we are choosing one of the variety of replacements.

      was (Author: markrmiller@gmail.com):
    Looks great!

bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing performance considerstions
(?).

Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)| normalization
factor with our document length factor?

That's how it appears to me anyway -

for |V(d)| you have many options right?

the cosine normalization - your standard euclidean length - |V(d)| 
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene uses by default
or Okapi's formula, 
or ...

So we are replacing (which everyone generally does) not dropping right?

And I don't think we are replacing for performance reasons (though it is complicated to calculate)
- we are replacing because its not a great normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer documents will
still have higher tf's and more distinct terms - so it unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics). So its not
generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and short preferred
too much)

Again, I'm not an IR guy, but thats what my modest take is.
  
> Similarity javadocs for scoring function to relate more tightly to scoring models in
effect
> -------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1908
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1908
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Doron Cohen
>            Assignee: Doron Cohen
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1908.patch
>
>
> See discussion in the related issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message