Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of DORONC@il.ibm.com designates
 195.212.29.153 as permitted sender)
In-Reply-To: <Pine.LNX.4.58.0707181623480.2109@hal.rescomp.berkeley.edu>
Subject: Re: search quality - assessment & improvements
To: java-dev@lucene.apache.org
Message-ID: 
 <OFC7EE6FF2.AC3A4E6F-ON8825731D.00693A5B-8825731D.006AFB64@il.ibm.com>
From: Doron Cohen <DORONC@il.ibm.com>
Date: Thu, 19 Jul 2007 12:28:31 -0700
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII

> However ... i still think that if you realy want
> a length norm that takes into account the average
> length of the docs, you want one that rewards docs
> for being near the average ...

... like SweetSpotSimilarity (SSS)

> it doesn't seem to make a lot of sense to me to say
> that a doc whose length is N% longer longer then the
> average length is significantly worse the docs whose
> length is N% shorter then the average length.

I don't understand why a doc should be punished for
just having length different from the average length
(i.e. no matter longer or shorter).

The (evolving) way I understand it:
(a) Very long docs are likely to contain everything,
    let's punish them to relax this;
(b) This is what the original doc-length-norm
    actually does;
(c) But then very short docs might be
    rewarded too much;
(d) Now we might get stupid (or erroneous)
    few words docs as top results;
(e) To solve this, pivoted doc-length-norm punishes too
    long docs (longer than the average) but only slightly
    rewards docs that are shorter than the average.

It makes sense to me (IR'ishly if I may say so).
The SSS way does not make sense to me that way.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org