lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From markharw00d <markharw...@yahoo.co.uk>
Subject Re: Find related question
Date Sat, 10 Mar 2007 09:51:41 GMT
 >>most of the body text is the same, but I want to group them all under 
one result.

I created this analyzer class to identify content that was "mostly 
similar" but not necessarily identical.
http://issues.apache.org/jira/browse/LUCENE-725

If you feed a small set of documents through it (say your search 
results) it will only emit tokens for sequences of words that it hasn't 
seen before.  Documents that have significantly more tokens going into 
NovelAnalyzer than coming out of NovelAnalyzer are near-duplicates

Cheers
Mark


sdeck wrote:
> Hello,
> I run Nutch and get a whole slew of articles and when I display search
> results, there may be 5-6 articles that have different titles, and most of
> the body text is the same, but I want to group them all under one result. 
> These are usually AP articles that all newspapers repurpose.
>
> When using the MoreLikeThis functionality, the articles that are returned
> may or may not be similar. When I run the query, the scores returned can
> range from .1 to .4 for the first 2 hits and it usually will return around
> 50 results, with the last score coming in fairly close to 0. Usually, the
> first hit is the exact same article as what I am trying to determine related
> articles for.  I know that the score value has no real meaning though,
> because it is done based upon the query, and other factors and then
> normalized.
>
> So, should I be taking (hit score/1) to use as a percentage value to see
> what other articles might be similar after that first hit? Try and normalize
> the similarity basically? Am I off my rocker?
>
> Or, is there possibly a way to use Carrot2 to find related articles for a
> given document?
>
> Thanks,
> Scott
>
>   



	
	
		
___________________________________________________________ 
All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC
Magazine 
http://uk.docs.yahoo.com/nowyoucan.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message