lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <>
Subject [jira] Commented: (LUCENE-1908) Similarity javadocs for scoring function to relate more tightly to scoring models in effect
Date Sat, 12 Sep 2009 18:09:57 GMT


Mark Miller commented on LUCENE-1908:

bq. Is it really better? It seems to "punish" the same for length due to distinct terms, and
to punish less for length due to duplicate terms. Is this really a desired behavior? My intuition
says no, but I am not sure.

Its only desired behavior if you have a corpus that favors it, but most do. I think you can
think of the |V(d)| as taking out information about the document length - you start with an
m space vector, with each term given a weight depending on how many times it occurs - in other
words, there is information about the length of that document there, and when you normalize
by |V(d)|, you will take out that information - but it will appear more similar the more unique
terms it started with and the higher the tf's. So that method favors long docs, witch will
naturally have more of both, though of course not always be more similar.

All of the normalizations I have seen appear in the vein of |V(d)| -eg 1/root(something).
All of the others also try and make up for this doc length problem - by messing with the curve
so that ultra long docs are not favored too highly. Our default method is a known not very
good one - buts its also very fast and efficient compared to the better ones. Sweetspot is
much better and I think still efficient - but you need to tune the right params I believe.

> Similarity javadocs for scoring function to relate more tightly to scoring models in
> -------------------------------------------------------------------------------------------
>                 Key: LUCENE-1908
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Doron Cohen
>            Assignee: Doron Cohen
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1908.patch
> See discussion in the related issue.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message