Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.uk;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Qkdjm/0S43gRGd3swZ3+QXEYAPN31IJ+kuZfZXYFvtE1zTJpyj1tp1qjLMGBvarhK3BgM4soMFuOPTxpKbzrSkJt+Fx4E/3/atuhL5xE8Tz53k7J7sIbA7uwprtjzr/xPgWIxrvMeGKKRVdhI3Q8bCLAKPm4hKtDhLCRfy2WPv0=;
Message-ID: <671780.48925.qm@web24603.mail.ird.yahoo.com>
References: <24175732.post@talk.nabble.com>
 <e2e029610906231540j7659cb27xc372cabe41750add@mail.gmail.com>
  <24175800.post@talk.nabble.com>
 <e2e029610906231548g13c7d6a4ia506faa5b22f13a3@mail.gmail.com>
  <c7d45fc70906231617k394194a6we07b873995445e29@mail.gmail.com>
  <e2e029610906231620n51e84ab6l493e5e048321b193@mail.gmail.com>
  <347836.95459.qm@web24605.mail.ird.yahoo.com>
 <c7d45fc70906231752u56ae92ach937366ee0a13a92b@mail.gmail.com>
Date: Tue, 23 Jun 2009 20:05:25 -0700 (PDT)
From: Paul Jones <paul_jonez99@yahoo.co.uk>
Subject: Re: LSI, cosine and others which use vectors
To: mahout-user@lucene.apache.org
In-Reply-To: <c7d45fc70906231752u56ae92ach937366ee0a13a92b@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1206415185-1245812725=:48925"

--0-1206415185-1245812725=:48925
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

tks Ted, but if its a live system, and you have 10 million documents, then =
isn't the computation on the fly going to be a pain, if you add say 1000 do=
cs per hour or whatever, which is why I was assuming that its a batch proce=
ss.=0A=0AAlso I think I have worked out what I meant about the relationship=
s between the words themselves, I think I was looking to build a term-term =
matrix instead of a term-doc, whereby I have the freq of occurence of each =
word alongside each other word in a doc.(I guess easy way to start is that =
the two words can co-occur anywhere in the doc). If done, hopefully the 'di=
stance' between the two vectors should give me a relative relationship. I r=
ealise lots of problems with this approach. i.e how don't know how the word=
s are related...I just know that they are.=0A=0APaul=0A=0A=0A=0A=0A________=
________________________=0AFrom: Ted Dunning <ted.dunning@gmail.com>=0ATo: =
mahout-user@lucene.apache.org=0ASent: Wednesday, 24 June, 2009 1:52:41=0ASu=
bject: Re: LSI, cosine and others which use vectors=0A=0AThere are two kind=
s of changes here.=0A=0AThe first kind is when a single document changes.  =
That will change the=0Adistances between that document and others, but it w=
on't change the=0Adistances between two other documents.  Most importantly,=
 it won't change=0Athe distance between queries and other documents.=0A=0AT=
he second kind of change is due to the first and is relatively=0Aunavoidabl=
e.  When a document changes, almost inevitably the corpus word=0Afrequencie=
s will change as a result.  This changes the weightings applied to=0Apartic=
ular terms in documents.  When you have many documents of which few=0Achang=
e these changes will be small enough to ignore.=0A=0AIn practice, you don't=
 much care about what has changed because a live=0Asystem computes all simi=
larities or distances on the fly based on the=0Acurrent state.   If the sim=
ilarities that you have not yet computed change,=0Ayou don't care.=0A=0AOn =
Tue, Jun 23, 2009 at 5:01 PM, Paul Jones <paul_jonez99@yahoo.co.uk>wrote:=
=0A=0A> Yes another question, am going through a rapid learning curve...=0A=
>=0A> All these vector based systems, which require you to build a term-doc=
 etc,=0A> are they of any use in a system where the data is changing, i.e l=
ets assume=0A> the docs are webpages, which are being crawled, and hence up=
dated. Surely if=0A> there is a vector diagram being formed, then the posit=
ion of these vectors=0A> changes based on the changes (size, content) of th=
e entire matrix, or am I=0A> missing something here.=0A>=0A> If the above i=
s correct, then is a actual live project how is this done,=0A> are distance=
s worked out on a per-day type of basis, and the indexes then=0A> updated ?=
=0A>=0A> Paul=0A>=0A>=0A>=0A>=0A=0A=0A=0A=0A-- =0ATed Dunning, CTO=0ADeepDy=
ve=0A=0A111 West Evelyn Ave. Ste. 202=0ASunnyvale, CA 94086=0Ahttp://www.de=
epdyve.com=0A858-414-0013 (m)=0A408-773-0220 (fax)=0A=0A=0A=0A      
--0-1206415185-1245812725=:48925--