Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.uk;
  h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
  b=bpMqPWd0Eh8BoGaZwAsWYxO+dOHK4GfX5sgGLr4csc9Vs26d6bQDSNCoTy+Ay6SGo7aer1A16qlv6GBkZ/zZxZINUE8KonOYScoPV5Ymld8XSuHYFDZtGfotzrsvGr4ffXMKZtsO923QgMWkidDyk2i0DO7K7W3S3nEaa/MXa0g=;
Date: Fri, 9 Feb 2007 11:52:23 +0000 (GMT)
From: mark harwood <markharw00d@yahoo.co.uk>
Subject: Re: Reduction based "more like this"?
To: java-user@lucene.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Message-ID: <181449.5358.qm@web26011.mail.ukl.yahoo.com>

The distinguishing characteristics you mark out and put in a field may not =
be so distinguishing as more content is added to an index (e.g. use of new =
terminology like "podcast" becomes more prevalent). Maintaining/regeneratin=
g this field in anything other than a static index then starts to look like=
 a non-trivial overhead.=0A=0AWhile we are musing on this, I'm not sure tha=
t with things like MoreLikeThis (or the BooleanQuery scoring?) we have cons=
idered the true value of *coincidences* of terms rather than independently =
summing their individual IDFs. For example, given terms "female", "John" an=
d "London" - all 3 may have equal IDF but should a document representing a =
female in London be given equal weighting to a document representing  the r=
arer example of a female who happens to be called "John"? Considering these=
 pairings adds extra complexity/cost but might be an interesting avenue to =
explore for some apps when selecting distinguishing characteristics or weig=
hting query results.=0A=0ACheers=0AMark=0A=0A=0A=0A=0A----- Original Messag=
e ----=0AFrom: karl wettin <karl.wettin@gmail.com>=0ATo: java-user@lucene.a=
pache.org=0ASent: Friday, 9 February, 2007 8:31:05 AM=0ASubject: Reduction =
based "more like this"?=0A=0AI just woke up thinking it would be cool to at=
tempt reducing the data  =0Aof all documents using PCA (or so) and store th=
e output in a new  =0Afield per dimention introduced in order to find simil=
air documents by  =0Aplacing a simple proximity query. Did anyone attempt s=
omething like  =0Athis?=0A=0AI did not think this through that much. Nor do=
 I need this feature.  =0AJust think it would be a cool experiment.=0A=0A--=
 =0Akarl=0A=0A-------------------------------------------------------------=
--------=0ATo unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org=
=0AFor additional commands, e-mail: java-user-help@lucene.apache.org=0A=0A=
=0A=0A=0A=0A=0A=09=09=0A___________________________________________________=
________ =0AInbox full of unwanted email? Get leading protection and 1GB st=
orage with All New Yahoo! Mail. http://uk.docs.yahoo.com/nowyoucan.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org