lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Organisciak <organisc...@gmail.com>
Subject Re: Scoring function in LMDirichletSimilarity Class
Date Thu, 04 Apr 2013 19:44:15 GMT
I think this is the problem that you're running into, though maybe a person
with more expertise can confirm...

ZP, If you look at section 5.1 of the Zhai Lafferty paper (
http://www.cs.cmu.edu/~lafferty/pub/smooth-tois.ps), they note that the
"term weight is log(1+(1-\lambda)p_ml(q_i|d) / \lamdba p(q_i|C)". P_ml is
freq/docLen, so it looks right, no?

The formula that you are looking at smooths on p(q_i|d), but if you look at
equation 6 (disregarding the constant at the end, that we don't need) and
read the paragraph below it, you can see that a term weight in the full log
p(q|d) calculation is more that just p(q_i|d).

The same goes for Dong's question on Dirichlet smoothing, which also uses a
non-constant \alpha_d, making the math a bit trickier.

Peter


On Tue, Apr 2, 2013 at 12:46 PM, Zeynep P. <zpvie@yahoo.com> wrote:

> Hi,
>
> I have the same question related to LMJelinekMercerSimiliarity class.
>
>   protected float score(BasicStats stats, float freq, float docLen) {
>     return stats.getTotalBoost() *
>         (float)Math.log(1 +  ((1 - lambda) * freq / docLen) / (lambda *
> ((LMStats)stats).getCollectionProbability()));
>   }
>
>  score = Math.log( (1 - lambda) *  freq / docLen * + *lambda *
> ((LMStats)stats).getCollectionProbability()) )
>
> I am also getting much worse results by updating the code like above.
>
> Why is it calculated this way?
>
> Thanks in advance,
>
> Best regards,
> ZP
>
> P.S: Instead of creating a new question, I used your question because I
> believe that the reason should be the same.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Scoring-function-in-LMDirichletSimilarity-Class-tp4052488p4053267.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message