Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: error (idunn.apache.osuosl.org: domain yahoo.co.uk from
 217.12.11.96 cause and error)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.uk;
  h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding;
  b=H/uonYAeYu8moPxTANz0KUw8Vp+Mlb0m3mjEZtJrkcYGFTehUo4YvLvmPsFQLdgqTHI910aFmJK/glFWPkOfLBAl02ncYY34VLCukBRdQ+NcsBTXbVb/Ooi748NOg3XARnto4bGxQDVFxqt0LIqVIASgnr1hIFOV612xRzH5u0s=
  ;
Message-ID: <4518D290.9030205@yahoo.co.uk>
Date: Tue, 26 Sep 2006 08:11:12 +0100
From: markharw00d <markharw00d@yahoo.co.uk>
User-Agent: Thunderbird 1.5.0.7 (Windows/20060909)
MIME-Version: 1.0
To: java-dev@lucene.apache.org
Subject: Re: highlight - scoring fragments with more of the same token
References: 
 <OF22294083.97AE0B97-ON882571F5.00087C01-882571F5.000A41FE@il.ibm.com>
In-Reply-To: 
 <OF22294083.97AE0B97-ON882571F5.00087C01-882571F5.000A41FE@il.ibm.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

If you were to score repeated terms then I suspect it would have to be 
done so that the repetitions didn't score as highly as the first 
occurrence - otherwise f2 could be selected as a better fragment than f3 
for the query q1 in your example.
Repetitions of a term in a fragment could be scored as a very small 
fraction of the score given to the first occurrence. This would at least 
rank  f2 higher than f1 for query q2.
Another potentially useful ranking factor may be to boost fragments 
found at the beginning of a document - that's where people tend to write 
summaries or introductions.


Doron Cohen wrote:
> This question was raised in the user's list -
> http://www.nabble.com/highlighting-tf2322109.html
>
> Assume three fragments and two queries:
>   f1 = aa  11  bb  33  cc
>   f2 = aa  11  bb  11  cc
>   f3 = aa  11  bb  22  cc
>   q1 = 11 22
>   q2 = 11
> Now we call highlighter.getBestFragment(q);
> For q1, f3 is returned, as expected.
> For q2, f1 is returned, although "11" appears twice in f2 but only once in
> f1.
>
> This is because QueryScorer.getTokenScore(Token) counts only unique
> fragment tokens.
>
> Would it make sense to make this behavior controllable?
> (It is easily done but I am not sure about the consequences.)
>
> Or perhaps there is a way to achieve this behavior (preferring f2 on f1 for
> q2 above) that I missed?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>   


___________________________________________________________ 
Copy addresses and emails from any email account to Yahoo! Mail - quick, easy and free. http://uk.docs.yahoo.com/trueswitch2.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org