lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chong, Herb" <HCho...@bloomberg.com>
Subject RE: Probabilistic Model in Lucene - possible?
Date Thu, 04 Dec 2003 13:57:16 GMT
oh, i forgot to mention, the interterm correlation model has been tested on TREC and deliver
up to 0.10 improved precision at 0 recall. however, i used a single pass search engine without
automatic relevance feedback and no linguistic analysis of the documents or queries, so i
suffered in overall performance. the submission i made did perform the best of the single
pass methods though, and middle of the pack in the multipass ones.

Herb...

-----Original Message-----
From: Chong, Herb [mailto:HCHONG3@bloomberg.net]
Sent: Thursday, December 04, 2003 8:53 AM
To: Lucene Users List
Subject: RE: Probabilistic Model in Lucene - possible?

the implementations that i have done are with a former company and that means secret and protected
by various intellectual property rights. however, i can sketch here the general approach one
has to take and an outline of the derivation that unifies probabilistic models with vector
space models and at the same time incorporate pairwise interterm correlation. in fact, the
pairwise interterm correlations are a fundamental assumption. once you do all this, you can
show that the traditional vector space model is a special case of a pairwise interterm correlation
model. for those that are interested in advanced matrix algebra and some basic statistics,
it should be very interesting. if only i had a published paper, i would post it. unfortunately,
what i have is very obtuse because it's protected. the only paper that started out was submitted
to SIGIR but rejected by all but one referee. that one thought this was a tremendous unification
of the two methods, but academic journals being what they are, when 4 out of 5 referees can't
understand the paper, it doesn't get published. i may brush it off and enlarge into a much
longer paper for the Journal of IR, but once again, unless you are comfortable with probability
theory and matrix theory, you are not going to follow it.

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message