mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Folcon Red <fol...@gmail.com>
Subject Re: Using Mahout to train an CVB and retrieve it's topics
Date Sun, 29 Jul 2012 17:08:02 GMT
Hi Guys,

Thanks for replying, the problem is whenever I use any -s flag I get the
error "Unexpected -s while processing Job-Specific Options:"

Also I'm not sure if this is supposed to be the output of -dt

sgeadmin@master:~$ dumbo ls /user/sgeadmin/text_cvb_document -hadoop
starcluster
Found 3 items
-rw-r--r--   3 sgeadmin supergroup          0 2012-07-29 16:51
/user/sgeadmin/text_cvb_document/_SUCCESS
drwxr-xr-x   - sgeadmin supergroup          0 2012-07-29 16:50
/user/sgeadmin/text_cvb_document/_logs
-rw-r--r--   3 sgeadmin supergroup         97 2012-07-29 16:51
/user/sgeadmin/text_cvb_document/part-m-00000

Should I be using a newer version of mahout? I've just been using the 0.7
distribution so far as apparently the compiled versions are missing parts
that the distributed ones have.

Kind Regards,
Folcon

PS: Thanks for the help so far!

On 29 July 2012 04:52, Jake Mannix <jake.mannix@gmail.com> wrote:

>
>
> On Sat, Jul 28, 2012 at 6:40 PM, DAN HELM <danielhelm@verizon.net> wrote:
>
>> Hi Folcon,
>>
>> In the folder you specified for the –dt option for cvb command
>> there should be sequence files with the document to topic associations
>> (Key:
>> IntWritable, Value: VectorWritable).
>
>
> Yeah, this is correct, although this:
>
>
>> You can dump in text format as: mahout seqdumper –s <sequence file>
>>
>
> is not as good as using vectordumper:
>
>    mahout vectordump -s <sequence file> --dictionary <path to dictionary.file-0>
> \
>        --dictionaryType seqfile --vectorSize <num entries per topic you
> want to see> -sort
>
> This joins your topic vectors with the dictionary, then picks out the top
> k terms (with their
> probabilities) for each topic and prints them to the console (or to the
> file you specify with
> an --output option).
>
> *although* I notice now that in trunk when I just checked,
> VectorDumper.java had a bug
> in it for "vectorSize" - line 175 asks for cmdline option "numIndexesPerVector"
> not
> vectorSize, ack!  So I took the liberty of fixing that, but you'll need to
> "svn up" and rebuild
> your jar before using vectordump like this.
>
>
>> So in text output from seqdumper, the key is a document id and the vector
>> contains
>> the topics and associated scores associated with the document.  I think
>> all topics are listed for each
>> document but many with near zero score.
>> In my case I used rowid to convert keys of original sparse
>> document vectors from Text to Integer before running cvb and this
>> generates a mapping file so I know the textual
>> keys that correspond to the numeric document ids (since my original
>> document ids were file names and I created named vectors).
>> Hope this helps.
>> Dan
>>
>> ________________________________
>>
>>  From: Folcon <folcon@gmail.com>
>> To: user@mahout.apache.org
>> Sent: Saturday, July 28, 2012 8:28 PM
>> Subject: Using Mahout to train an CVB and retrieve it's topics
>>
>> Hi Everyone,
>>
>> I'm posting this as my original message did not seem to appear on the
>> mailing
>> list, I'm very sorry if I have done this in error.
>>
>> I'm doing this to then use the topics to train a maxent algorithm to
>> predict the
>> classes of documents given their topic mixtures. Any further aid in this
>> direction would be appreciated!
>>
>> I've been trying to extract the topics out of my run of cvb. Here's what
>> I did
>> so far.
>>
>> Ok, so I still don't know how to output the topics, but I have worked out
>> how to
>> get the cvb and what I think are the document vectors, however I'm not
>> having
>> any luck dumping them, so help here would still be appreciated!
>>
>> I set the values of:
>>     export MAHOUT_HOME=/home/sgeadmin/mahout
>>     export HADOOP_HOME=/usr/lib/hadoop
>>     export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
>>     export HADOOP_CONF_DIR=$HADOOP_HOME/conf
>> on the master otherwise none of this works.
>>
>> So first I uploaded the documents using starclusters put:
>>     starcluster put mycluster text_train /home/sgeadmin/
>>     starcluster put mycluster text_test /home/sgeadmin/
>>
>> Then I added them to hadoop's hbase filesystem:
>>     dumbo put /home/sgeadmin/text_train /user/sgeadmin/ -hadoop
>> starcluster
>>
>> Then I called Mahout's seqdirectory to turn the text into sequence files
>>     $MAHOUT_HOME/bin/mahout seqdirectory --input
>> /user/sgeadmin/text_train --
>> output /user/sgeadmin/text_seq -c UTF-8 -ow
>>
>> Then I called Mahout's seq2parse to turn them into vectors
>>     $MAHOUT_HOME/bin/mahout seq2sparse -i text_seq -o
>> /user/sgeadmin/text_vec -
>> wt tf -a org.apache.lucene.analysis.WhitespaceAnalyzer -ow
>>
>> Finally I called cvb, I believe that the -dt flag states where the
>> inferred
>> topics should go, but because I haven't yet been able to dump them I can't
>> confirm this.
>>     $MAHOUT_HOME/bin/mahout cvb -i /user/sgeadmin/text_vec/tf-vectors -o
>> /user/sgeadmin/text_lda -k 100 -nt 29536 -x 20 -dict
>> /user/sgeadmin/text_vec/dictionary.file-0 -dt
>> /user/sgeadmin/text_cvb_document -
>> mt /user/sgeadmin/text_states
>>
>> The -k flag is the number of topics, the -nt flag is the size of the
>> dictionary,
>> I computed this by counting the number of entries of the dictionary.file-0
>> inside the vectors(in this case under /user/sgeadmin/text_vec) and -x is
>> the
>> number of iterations.
>>
>> If you know how to get what the document topic probabilities are from
>> here, help
>> would be most appreciated!
>>
>> Kind Regards,
>> Folcon
>>
>
>
>
> --
>
>   -jake
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message