Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: neutral (nike.apache.org: 98.138.90.46 is neither permitted nor
 denied by domain of danielhelm@verizon.net)
References: <loom.20120729T022528-982@post.gmane.org>
 <1343526059.35337.YahooMailNeo@web84508.mail.ne1.yahoo.com>
 <CACYXym-QV-tnemu_OJb7fPSuOqX1U4T+3v4-a4-peZNotSEhdg@mail.gmail.com>
 <CAHD0Z=Nozb-jgwS4vcqgx4Z1Ld1a7NyCKUsUvNaYwi0OpKj1jg@mail.gmail.com>
 <1343584189.6944.YahooMailNeo@web84501.mail.ne1.yahoo.com>
 <CAHD0Z=Ng-AmJ59Y0x20=mGKF6bctXKAf7HNLq9VozRuUbuj6=g@mail.gmail.com>
 <1343593782.17723.YahooMailNeo@web84516.mail.ne1.yahoo.com>
 <CAHD0Z=MWpnVU=X0busP5hcFWzBUir1WKJNb0wsLmLNrSUswiAA@mail.gmail.com>
 <CAHD0Z=NrAWe9O71Mguwx=iFG7wNJ4uh9Py+PWnLoaSLTqdTatg@mail.gmail.com>
 <CAHD0Z=MDVooRE3SLEOt90CmMm0ugvHkQ=+sKhcD=qeHwTNPs7A@mail.gmail.com>
Message-ID: <1343775588.54902.YahooMailNeo@web84512.mail.ne1.yahoo.com>
Date: Tue, 31 Jul 2012 15:59:48 -0700 (PDT)
From: DAN HELM <danielhelm@verizon.net>
Reply-To: DAN HELM <danielhelm@verizon.net>
Subject: Re: Using Mahout to train an CVB and retrieve it's topics
To: "user@mahout.apache.org" <user@mahout.apache.org>,
  Folcon Red <folcon@gmail.com>
In-Reply-To: 
 <CAHD0Z=MDVooRE3SLEOt90CmMm0ugvHkQ=+sKhcD=qeHwTNPs7A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="-821251416-1099181302-1343775588=:54902"

---821251416-1099181302-1343775588=:54902
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Folcon,=0A=C2=A0=0Aseqdirectory should also read files in subfolders.=0A=C2=
=A0=0ADid you verify that recent seqdirectory command=C2=A0did in fact gene=
rate non-empty sequence files?=C2=A0 I believe seqdirectory=C2=A0command ju=
st assumes=C2=A0 each file contains a single document=C2=A0(no concatenated=
 documents per file), and=C2=A0that each file contains basic=C2=A0text.=0A=
=C2=A0=0AIf it did generate sequence files this time, I am assume your fold=
er "/user/sgeadmin/text_seq" was copied to hdfs (if not already there) befo=
re you ran seq2sparse on it?=0A=C2=A0=0ADan=0A =0A=0A______________________=
__________=0A From: Folcon Red <folcon@gmail.com>=0ATo: DAN HELM <danielhel=
m@verizon.net> =0ACc: Jake Mannix <jake.mannix@gmail.com>; "user@mahout.apa=
che.org" <user@mahout.apache.org> =0ASent: Tuesday, July 31, 2012 1:34 PM=
=0ASubject: Re: Using Mahout to train an CVB and retrieve it's topics=0A  =
=0ASo part-r-00000 inside text_vec is=0Astill SEQorg.apache.hadoop.io.Text%=
org.apache.mahout.math.VectorWritable=0Aeven after moving all the training =
files into a single folder.=0A=0ARegards,=0AFolcon=0A=0AOn 31 July 2012 18:=
18, Folcon Red <folcon@gmail.com> wrote:=0A=0A> Hey Everyone,=0A>=0A> Ok no=
t certain why=C2=A0  $MAHOUT_HOME/bin/mahout seqdirectory --input /user/=0A=
> sgeadmin/text_train --output /user/sgeadmin/text_seq -c UTF-8 -ow=C2=A0 d=
idn't=0A> produce sequence files, just looking inside text_seq only gives m=
e:=0A>=0A> SEQ org.apache.hadoop.io.Text org.apache.hadoop.io.Text=0A>=0A> =
and that's it. Any ideas what I've been doing wrong? Maybe it's because I=
=0A> have the files nested in the folder by class, for example a tree view =
of=0A> the directory would look like.=0A>=0A> text_train -+=0A>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0  | A -+=0A>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 100=0A=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 | 101=0A>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 103=0A>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0  | B -+=0A>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 102=0A>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 105=0A>=C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 | 106=0A>=0A> So it's not picking them up? Or perhaps something else? I=
'm going to try=0A> some variations to see what happens.=0A>=0A> Thanks for=
 the help so far!=0A>=0A> Regards,=0A> Folcon=0A>=0A>=0A> On 29 July 2012 2=
2:10, Folcon Red <folcon@gmail.com> wrote:=0A>=0A>> Right, well here's some=
thing promising, running $MAHOUT_HOME/bin/mahout=0A>> seqdumper -i /user/sg=
eadmin/text_lda/part-m-00000 produced:=0A>>=0A>>=0A>> 7:NaN,29478:NaN,29479=
:NaN,29480:NaN,29481:NaN,29482:NaN,29483:NaN,29484:NaN,29485:NaN,29486:NaN,=
29487:NaN,29488:NaN,29489:NaN,29490:NaN,29491:NaN,29492:NaN,29493:NaN,29494=
:NaN,29495:NaN,29496:NaN,29497:NaN,29498:NaN,29499:NaN,29500:NaN,29501:NaN,=
29502:NaN,29503:NaN,29504:NaN,29505:NaN,29506:NaN,29507:NaN,29508:NaN,29509=
:NaN,29510:NaN,29511:NaN,29512:NaN,29513:NaN,29514:NaN,29515:NaN,29516:NaN,=
29517:NaN,29518:NaN,29519:NaN,29520:NaN,29521:NaN,29522:NaN,29523:NaN,29524=
:NaN,29525:NaN,29526:NaN,29527:NaN,29528:NaN,29529:NaN,29530:NaN,29531:NaN,=
29532:NaN,29533:NaN,29534:NaN,29535:NaN}=0A>>=0A>> And $MAHOUT_HOME/bin/mah=
out seqdumper -i=0A>> /user/sgeadmin/text_vec/tf-vectors/part-r-00000 produ=
ced:=0A>>=0A>> MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath=
.=0A>> Running on hadoop, using /usr/lib/hadoop/bin/hadoop and=0A>> HADOOP_=
CONF_DIR=3D/usr/lib/hadoop/conf=0A>> MAHOUT-JOB: /home/sgeadmin/mahout/maho=
ut-examples-0.7-job.jar=0A>> 12/07/29 21:09:17 INFO common.AbstractJob: Com=
mand line arguments:=0A>> {--endPhase=3D[2147483647],=0A>> --input=3D[/user=
/sgeadmin/text_vec/tf-vectors/part-r-00000],=0A>> --startPhase=3D[0], --tem=
pDir=3D[temp]}=0A>> Input Path: /user/sgeadmin/text_vec/tf-vectors/part-r-0=
0000=0A>> Key class: class org.apache.hadoop.io.Text Value Class: class=0A>=
> org.apache.mahout.math.VectorWritable=0A>> Count: 0=0A>>=0A>> Kind Regard=
s,=0A>> Folcon=0A>>=0A>> On 29 July 2012 21:29, DAN HELM <danielhelm@verizo=
n.net> wrote:=0A>>=0A>>> Yep something went wrong, most likely with the clu=
stering.=C2=A0 part file is=0A>>> empty.=C2=A0 Should look something like t=
his:=0A>>>=0A>>> Key class: class org.apache.hadoop.io.IntWritable Value Cl=
ass: class=0A>>> org.apache.mahout.math.VectorWritable=0A>>> Key: 0: Value:=
=0A>>> {0:0.06475650422868284,1:0.010728747158503565,2:0.005463535698651016=
,3:0.023451709705466457}=0A>>> Key: 1: Value:=0A>>> {0:0.01838885430227092,=
1:0.05068404879399544,2:0.002110418548647133,3:0.005566514441743756}=0A>>> =
Key: 2: Value:=0A>>> {0:0.018575587065216153,1:1.236602313900785E-5,2:8.654=
629660837919E-6,3:5.820637306957196E-6}=0A>>> ...=0A>>> ...=0A>>>=0A>>> Key=
 refers to a document id and the Value are topic ids:weights assigned=0A>>>=
 to document id.=0A>>>=0A>>> So you need to figure out where things went wr=
ong.=C2=A0 I'm assume folder=0A>>> /user/sgeadmin/text_lda also has empty p=
art files?=C2=A0 Assuming parts=0A>>> files are there run seqdumper on one.=
=C2=A0 Should have data like the above=0A>>> except in this case the key wi=
ll be a topic id and the vector will be term=0A>>> ids:weights.=0A>>>=0A>>>=
 You can also check folder /user/sgeadmin/text_vec/tf-vectors to make=0A>>>=
 sure sparse vectors were generated for your input to cvb.=0A>>>=0A>>> Dan=
=0A>>>=0A>>>=C2=A0 =C2=A0 *From:* Folcon Red <folcon@gmail.com>=0A>>> *To:*=
 DAN HELM <danielhelm@verizon.net>=0A>>> *Cc:* Jake Mannix <jake.mannix@gma=
il.com>; "user@mahout.apache.org" <=0A>>> user@mahout.apache.org>=0A>>> *Se=
nt:* Sunday, July 29, 2012 3:35 PM=0A>>>=0A>>> *Subject:* Re: Using Mahout =
to train an CVB and retrieve it's topics=0A>>>=0A>>> Thanks Dan and Jake,=
=0A>>>=0A>>> The output I got from $MAHOUT_HOME/bin/mahout seqdumper -i /us=
er/=0A>>> sgeadmin/text_cvb_document/part-m-00000 is:=0A>>>=0A>>> Input Pat=
h: /user/sgeadmin/text_cvb_document/part-m-00000=0A>>> Key class: class org=
.apache.hadoop.io <http://org.apache.hadoop.io.int/>=0A>>> .IntWritable Val=
ue Class: class org.apache.mahout.math.VectorWritable=0A>>> Count: 0=0A>>>=
=0A>>> I'm not certain what went wrong.=0A>>>=0A>>> Kind Regards,=0A>>> Fol=
con=0A>>>=0A>>> On 29 July 2012 18:49, DAN HELM <danielhelm@verizon.net> wr=
ote:=0A>>>=0A>>> Folcon,=0A>>>=0A>>> I'm still using Mahout 0.6 so don't kn=
ow much about changes in 0.7.=0A>>>=0A>>> Your output folder for "dt" looks=
 correct.=C2=A0 The relevant data would be=0A>>> in=C2=A0 /user/sgeadmin/te=
xt_cvb_document/part-m-00000 which is what I would=0A>>> be passing to a "-=
s" option.=C2=A0 But I see it says size is only 97 so that=0A>>> looks susp=
icious.=C2=A0 So you can just view file (for starters) as: mahout=0A>>> seq=
dumper -s /user/sgeadmin/text_cvb_document/part-m-00000.=C2=A0 And the=0A>>=
> vector dumper command (as Jake pointed out) has a lot more options to=0A>=
>> post-process the data but you may want to first just see what is in=0A>>=
> that file.=0A>>>=0A>>> Dan=0A>>>=0A>>>=C2=A0 =C2=A0 *From:* Folcon Red <f=
olcon@gmail.com>=0A>>> *To:* Jake Mannix <jake.mannix@gmail.com>=0A>>> *Cc:=
* user@mahout.apache.org; DAN HELM <danielhelm@verizon.net>=0A>>> *Sent:* S=
unday, July 29, 2012 1:08 PM=0A>>> *Subject:* Re: Using Mahout to train an =
CVB and retrieve it's topics=0A>>>=0A>>> Hi Guys,=0A>>>=0A>>> Thanks for re=
plying, the problem is whenever I use any -s flag I get the=0A>>> error "Un=
expected -s while processing Job-Specific Options:"=0A>>>=0A>>> Also I'm no=
t sure if this is supposed to be the output of -dt=0A>>>=0A>>> sgeadmin@mas=
ter:~$ dumbo ls /user/sgeadmin/text_cvb_document -hadoop=0A>>> starcluster=
=0A>>> Found 3 items=0A>>> -rw-r--r--=C2=A0  3 sgeadmin supergroup=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2012-07-29 16:51 /user/=0A>>> sgeadmin/text_c=
vb_document/_SUCCESS=0A>>> drwxr-xr-x=C2=A0  - sgeadmin supergroup=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2012-07-29 16:50 /user/=0A>>> sgeadmin/text_c=
vb_document/_logs=0A>>> -rw-r--r--=C2=A0  3 sgeadmin supergroup=C2=A0 =C2=
=A0 =C2=A0 =C2=A0  97 2012-07-29 16:51 /user/=0A>>> sgeadmin/text_cvb_docum=
ent/part-m-00000=0A>>>=0A>>> Should I be using a newer version of mahout? I=
've just been using the=0A>>> 0.7 distribution so far as apparently the com=
piled versions are missing=0A>>> parts that the distributed ones have.=0A>>=
>=0A>>> Kind Regards,=0A>>> Folcon=0A>>>=0A>>> PS: Thanks for the help so f=
ar!=0A>>>=0A>>> On 29 July 2012 04:52, Jake Mannix <jake.mannix@gmail.com> =
wrote:=0A>>>=0A>>>=0A>>>=0A>>> On Sat, Jul 28, 2012 at 6:40 PM, DAN HELM <d=
anielhelm@verizon.net>wrote:=0A>>>=0A>>> Hi Folcon,=0A>>>=0A>>> In the fold=
er you specified for the =E2=80=93dt option for cvb command=0A>>> there sho=
uld be sequence files with the document to topic associations=0A>>> (Key:=
=0A>>> IntWritable, Value: VectorWritable).=0A>>>=0A>>>=0A>>> Yeah, this is=
 correct, although this:=0A>>>=0A>>>=0A>>> You can dump in text format as: =
mahout seqdumper =E2=80=93s <sequence file>=0A>>>=0A>>>=0A>>> is not as goo=
d as using vectordumper:=0A>>>=0A>>>=C2=A0 =C2=A0 mahout vectordump -s <seq=
uence file> --dictionary <path to dictionary.file-0>=0A>>> \=0A>>>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 --dictionaryType seqfile --vectorSize <num entries per=
 topic you=0A>>> want to see> -sort=0A>>>=0A>>> This joins your topic vecto=
rs with the dictionary, then picks out the=0A>>> top k terms (with their=0A=
>>> probabilities) for each topic and prints them to the console (or to the=
=0A>>> file you specify with=0A>>> an --output option).=0A>>>=0A>>> *althou=
gh* I notice now that in trunk when I just checked, VectorDumper.java=0A>>>=
 had a bug=0A>>> in it for "vectorSize" - line 175 asks for cmdline option =
"=0A>>> numIndexesPerVector" not=0A>>> vectorSize, ack!=C2=A0 So I took the=
 liberty of fixing that, but you'll need=0A>>> to "svn up" and rebuild=0A>>=
> your jar before using vectordump like this.=0A>>>=0A>>>=0A>>>=C2=A0 So in=
 text output from seqdumper, the key is a document id and the=0A>>> vector =
contains=0A>>> the topics and associated scores associated with the documen=
t.=C2=A0 I think=0A>>> all topics are listed for each=0A>>> document but ma=
ny with near zero score.=0A>>> In my case I used rowid to convert keys of o=
riginal sparse=0A>>> document vectors from Text to Integer before running c=
vb and this=0A>>> generates a mapping file so I know the textual=0A>>> keys=
 that correspond to the numeric document ids (since my original=0A>>> docum=
ent ids were file names and I created named vectors).=0A>>> Hope this helps=
.=0A>>> Dan=0A>>>=0A>>> ________________________________=0A>>>=0A>>>=C2=A0 =
From: Folcon <folcon@gmail.com>=0A>>> To: user@mahout.apache.org=0A>>> Sent=
: Saturday, July 28, 2012 8:28 PM=0A>>> Subject: Using Mahout to train an C=
VB and retrieve it's topics=0A>>>=0A>>> Hi Everyone,=0A>>>=0A>>> I'm postin=
g this as my original message did not seem to appear on the=0A>>> mailing=
=0A>>> list, I'm very sorry if I have done this in error.=0A>>>=0A>>> I'm d=
oing this to then use the topics to train a maxent algorithm to=0A>>> predi=
ct the=0A>>> classes of documents given their topic mixtures. Any further a=
id in this=0A>>> direction would be appreciated!=0A>>>=0A>>> I've been tryi=
ng to extract the topics out of my run of cvb. Here's=0A>>> what I did=0A>>=
> so far.=0A>>>=0A>>> Ok, so I still don't know how to output the topics, b=
ut I have worked=0A>>> out how to=0A>>> get the cvb and what I think are th=
e document vectors, however I'm not=0A>>> having=0A>>> any luck dumping the=
m, so help here would still be appreciated!=0A>>>=0A>>> I set the values of=
:=0A>>>=C2=A0 =C2=A0  export MAHOUT_HOME=3D/home/sgeadmin/mahout=0A>>>=C2=
=A0 =C2=A0  export HADOOP_HOME=3D/usr/lib/hadoop=0A>>>=C2=A0 =C2=A0  export=
 JAVA_HOME=3D/usr/lib/jvm/java-6-openjdk=0A>>>=C2=A0 =C2=A0  export HADOOP_=
CONF_DIR=3D$HADOOP_HOME/conf=0A>>> on the master otherwise none of this wor=
ks.=0A>>>=0A>>> So first I uploaded the documents using starclusters put:=
=0A>>>=C2=A0 =C2=A0  starcluster put mycluster text_train /home/sgeadmin/=
=0A>>>=C2=A0 =C2=A0  starcluster put mycluster text_test /home/sgeadmin/=0A=
>>>=0A>>> Then I added them to hadoop's hbase filesystem:=0A>>>=C2=A0 =C2=
=A0  dumbo put /home/sgeadmin/text_train /user/sgeadmin/ -hadoop=0A>>> star=
cluster=0A>>>=0A>>> Then I called Mahout's seqdirectory to turn the text in=
to sequence files=0A>>>=C2=A0 =C2=A0  $MAHOUT_HOME/bin/mahout seqdirectory =
--input /user/sgeadmin/text_train=0A>>> --=0A>>> output /user/sgeadmin/text=
_seq -c UTF-8 -ow=0A>>>=0A>>> Then I called Mahout's seq2parse to turn them=
 into vectors=0A>>>=C2=A0 =C2=A0  $MAHOUT_HOME/bin/mahout seq2sparse -i tex=
t_seq -o /user/sgeadmin=0A>>> /text_vec -=0A>>> wt tf -a org.apache.lucene.=
analysis.WhitespaceAnalyzer -ow=0A>>>=0A>>> Finally I called cvb, I believe=
 that the -dt flag states where the=0A>>> inferred=0A>>> topics should go, =
but because I haven't yet been able to dump them I=0A>>> can't=0A>>> confir=
m this.=0A>>>=C2=A0 =C2=A0  $MAHOUT_HOME/bin/mahout cvb -i /user/sgeadmin/t=
ext_vec/tf-vectors -o=0A>>> /user/sgeadmin/text_lda -k 100 -nt 29536 -x 20 =
-dict=0A>>> /user/sgeadmin/text_vec/dictionary.file-0 -dt /user/sgeadmin/te=
xt_cvb_document=0A>>> -=0A>>> mt /user/sgeadmin/text_states=0A>>>=0A>>> The=
 -k flag is the number of topics, the -nt flag is the size of the=0A>>> dic=
tionary,=0A>>> I computed this by counting the number of entries of the=0A>=
>> dictionary.file-0=0A>>> inside the vectors(in this case under /user/sgea=
dmin/text_vec) and -x=0A>>> is the=0A>>> number of iterations.=0A>>>=0A>>> =
If you know how to get what the document topic probabilities are from=0A>>>=
 here, help=0A>>> would be most appreciated!=0A>>>=0A>>> Kind Regards,=0A>>=
> Folcon=0A>>>=0A>>>=0A>>>=0A>>>=0A>>> --=0A>>>=0A>>>=C2=A0  -jake=0A>>>=0A=
>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>>=0A>>=0A>>=0A>
---821251416-1099181302-1343775588=:54902--