lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen
Date Wed, 06 Feb 2008 12:15:57 GMT
Hey Doron,

I see you recommend that we think about making SweetSpot the default  
similarity.  Do you have numbers showing for running that alone?  Or  
for that matter, any of the other combinations of #3 individually?


On Jan 31, 2008, at 4:09 AM, Doron Cohen wrote:

> Hi Otis,
> On Thu, Jan 31, 2008 at 7:21 AM, Otis Gospodnetic <
>> wrote:
>> Doron - this looks super useful!
>> Can you give an example for the lexical affinities you mention here?
>> ("Juru creates posting lists for lexical affinities")
> Sure, - simply put, denote {X} as the posting list of term X, then  
> for a
> query - A B C D - in addition to the four posting lists {A}, {B},  
> {C}, {D}
> which are processed ignoring position info (i.e. Lucene's  
> termDocs()) Juru
> also computes combined posting lists {A,B}, {A,C}, {A,D}, {B,C},  
> {B,D} and
> {C,D} in which a (virtual) term {X,Y} is said to exist in a document  
> D if
> the two words X and Y are found in that document within a sliding  
> window of
> size L (say 5).
> (You can also require LA's in order which is useful in some  
> scenarios.)
> Juru's tokenization detects sentences and so the two words must be  
> in the
> same sentence. The term-freq of that LA-term in the doc is as usual  
> the
> number of matches in that doc satisfying this sliding window rule.
> The IDF of this term is not known in advance, and so it is first  
> estimated
> based on the DF of X and Y, and this estimate is later tuned as more
> documents are processed and more statistics are available.
> You can see the resemblance to SpanNear queries. Note that the IDF  
> of this
> virtual term is going to be high and as such it is "focusing" the  
> query
> search on the more relevant documents.
> In my Lucene implementation for this I used a window size of 7, and  
> note
> that (1) there was no sentence boundaries knowledge in my Lucene
> implementation and (2) the IDF was fixed all along, estimated by the
> involved terms IDF, as computed once in SpanNear query. The default
> computation is their sum. This is in most cases too low an IDF, I  
> think.
> Phrase query btw behaves the same.
> So in both cases (Phrase, Span) I think it would be interesting to
> experiment with adaptive IDF computation that updates the IDF as more
> documents are processed. When the query is made of only a single  
> span or
> only a single phrase element this is a waste of time. But when the  
> query is
> more complex (as the query we built) and you have in the query both
> multi-term parts and single-term parts, or several multi-term parts,  
> then a
> more accurate IDF can improve the quality I would think.  
> Implementation wise
> the "Weight.value" would need to be updated and might raise  
> questions about
> the normalizing of other query parts, but I am not sure about this  
> now.
> Well I hope this makes sense - I will update the Wiki page with  
> similar
> info...
> Also:
>> "Normalized term-frequency, as in Juru.
>> Here, tf(freq) is normalized by the average term frequency of the
>> document."
>> I've never seen this mentioned anywhere except here and once here  
>> on the
>> ML (was it you who mentioned this?), but this sounds intuitive.
> Yes I think I mentioned this - I think it is not our idea - Juru  
> uses it but
> it was used before in the SMART system - see "Length Normalization in
> Degraded Text Collections (1995)" - 
> ,
> and "New Retrieval Approaches Using SMART : TREC 4" -
>> What do others think?
>> Otis

Grant Ingersoll

Lucene Helpful Hints:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message