Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <1753716727.1252733937685.JavaMail.jira@brutus>
Date: Fri, 11 Sep 2009 22:38:57 -0700 (PDT)
From: "Mark Miller (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Issue Comment Edited: (LUCENE-1908) Similarity javadocs for
 scoring function to relate more tightly to scoring models in effect
In-Reply-To: <1529898827.1252698057805.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754470#action_12754470 ] 

Mark Miller edited comment on LUCENE-1908 at 9/11/09 10:37 PM:
---------------------------------------------------------------

Looks great!

bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing performance considerstions (?).

Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)| normalization factor with our document length factor?

That's how it appears to me anyway -

for |V(d)| you have many options right?

the cosine normalization - your standard euclidean length - |V(d)| 
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene uses by default
or Okapi's formula, 
or ...

So we are replacing (which everyone generally does) not dropping right?

And I don't think we are replacing for performance reasons (though it is complicated to calculate) - we are replacing because its not a great normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer documents will still have higher tf's and more distinct terms - so it unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics). So its not generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and short preferred too much)

Again, I'm not an IR guy, but thats what my modest take is.

* edit: I suppose you could argue that you could do cosine normalization and then a further normalization approach on that, and in that sense we are dropping the cosine normalization because its too expensive. But from what I can see, it appears more the case that you try and use a normalization factor that can just replace cosine normalization - like the pivoted normalization which rotates the cosine normalization curve. I think pivoted is something like 1/root(stuff, ie unique terms), so our norm of 1/root(L) is of a similar, simpler, vein. So thats why I think we are not really dropping it - we are choosing one of the variety of replacements.

      was (Author: markrmiller@gmail.com):
    Looks great!

bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing performance considerstions (?).

Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)| normalization factor with our document length factor?

That's how it appears to me anyway -

for |V(d)| you have many options right?

the cosine normalization - your standard euclidean length - |V(d)| 
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene uses by default
or Okapi's formula, 
or ...

So we are replacing (which everyone generally does) not dropping right?

And I don't think we are replacing for performance reasons (though it is complicated to calculate) - we are replacing because its not a great normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer documents will still have higher tf's and more distinct terms - so it unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics). So its not generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and short preferred too much)

Again, I'm not an IR guy, but thats what my modest take is.
  
> Similarity javadocs for scoring function to relate more tightly to scoring models in effect
> -------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1908
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1908
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Doron Cohen
>            Assignee: Doron Cohen
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1908.patch
>
>
> See discussion in the related issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org