mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szymon Chojnacki (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAHOUT-588) Benchmark Mahout's clustering performance on EC2 and publish the results
Date Tue, 01 Feb 2011 14:47:29 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989223#comment-12989223
] 

Szymon Chojnacki commented on MAHOUT-588:
-----------------------------------------

Collocations bottle-neck.

As I see, so far both of us did vectorization with 1-grams. I must admit that I am affraid
to calculate 2-grams or 3-grams. My understanding of:
DictionaryVectorizer
CollocDriver
CollocMapper
CollocReducer

is that, firstly all collocations are built and counted and only those with freq>minCount
are restored. If it is so, I think that one more preprocessing step would make it faster in
our setting (huge amount of typos and a long,long-tail) - i.e. building collocations only
for tokens (1-grams) with freq>minCount. I'll experiment with this issue few hours today.


> Benchmark Mahout's clustering performance on EC2 and publish the results
> ------------------------------------------------------------------------
>
>                 Key: MAHOUT-588
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-588
>             Project: Mahout
>          Issue Type: Task
>            Reporter: Grant Ingersoll
>         Attachments: SequenceFilesFromMailArchives.java, SequenceFilesFromMailArchives2.java,
Top1000Tokens_maybe_stopWords, Uncompress.java, clusters_kMeans.txt, distcp_large_to_s3_failed.log,
seq2sparse_small_failed.log, seq2sparse_xlarge_ok.log
>
>
> For Taming Text, I've commissioned some benchmarking work on Mahout's clustering algorithms.
 I've asked the two doing the project to do all the work in the open here.  The goal is to
use a publicly reusable dataset (for now, the ASF mail archives, assuming it is big enough)
and run on EC2 and make all resources available so others can reproduce/improve.
> I'd like to add the setup code to utils (although it could possibly be done as a Vectorizer)
and the publication of the results will be put up on the Wiki as well as in the book.  This
issue is to track the patches, etc.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message