mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Maturity level annotations
Date Wed, 28 Dec 2011 20:42:53 GMT
Or you can take a small set of good data and generate variations to
get a big set with the same disribution curves.

On Wed, Dec 28, 2011 at 10:47 AM, Ted Dunning <ted.dunning@gmail.com> wrote:
> I have nearly given up on getting publicly available large data sets and
> have started to specify synthetic datasets for development projects.   The
> key is to build reasonably realistic generation algorithms and for that
> there are always some serious difficulties.
>
> For simple scaling tests, however, synthetic data is often just the ticket.
>  You still need some sophistication about the data, but it doesn't take
> much.  For k-means clustering of text documents, for instance, you can use
> re-sample from real text to generate new text with desired properties or
> you can define an LDA-like generator to generate data with known clustering
> properties.  Similarly, to test scaling of classification algorithms, it is
> easy to generate text-like data with known properties.
>
> The primary virtues of synthetic data are that a synthetic data set is easy
> to carry around and it can be any size at all.
>
> As an example of a potential pitfall, I wrote tests for the sequential
> version of the SSVD codes by building low rank matrices and testing the
> reconstruction error.  This is a fine test for correctness and some scaling
> attributes, but it ignores the truncation error that Radim was fulminating
> about recently.  It would be good to additionally explore large matrices
> that are more realistic because they are generated as count data from a
> model that has a realistic spectrum.
>
> On Wed, Dec 28, 2011 at 10:35 AM, Grant Ingersoll <gsingers@apache.org>wrote:
>
>> To me, the big thing we continue to be missing is the ability for those of
>> us working on the project to reliably test the algorithms at scale.  For
>> instance, I've seen hints of several places where our clustering algorithms
>> don't appear to scale very well (which are all M/R -- K-Means does scale)
>> and it isn't clear to me whether it is our implementation, Hadoop, or
>> simply that the data set isn't big enough or the combination of all three.
>>  To see this in action, try out the ASF email archive up on Amazon with 10,
>> 15 or 30 EC2 double x-large nodes and try out fuzzy k-means, dirichlet,
>> etc.  Now, I realize EC2 isn't ideal for this kind of testing, but it all
>> many of us have access to.  Perhaps it's also b/c 7M+ emails isn't big
>> enough (~100GB), but in some regards that's silly since the whole point is
>> supposed to be it scales.  Or perhaps my tests were flawed.  Either way, it
>> seems like it is an area we need to focus on more.
>>
>> Of course, the hard part with all of this is debugging where the
>> bottlenecks are.  In the end, we need to figure out how to reliably get
>> compute time available for testing along with a real data sets that we can
>> use to validate scalability.
>>
>>
>> On Dec 27, 2011, at 10:22 PM, Ted Dunning wrote:
>>
>> > On Tue, Dec 27, 2011 at 3:24 PM, Tom Pierce <tcp@cloudera.com> wrote:
>> >
>> >> ...
>> >>
>> >> They discover Mahout, which does specifically bill itself as scalable
>> >> (from http://mahout.apache.org, in some of the largest letters: "What
>> >> is Apache Mahout?  The Apache Mahout™ machine learning library's goal
>> >> is to build scalable machine learning libraries.").  They sniff check
>> >> it by massaging some moderately-sized data set into the same format as
>> >> an example from the wiki and they fail to get a result - often because
>> >> their problem has some very different properties (more classes, much
>> >> larger feature space, etc.) and the implementation has some limitation
>> >> that they trip over.
>> >>
>> >
>> > I have worked with users of Mahout who had 10^9 possible features and
>> > others who are classifying
>> > into 60,000 categories.
>> >
>> > Neither of these implementations uses Naive Bayes.  Both work very well.
>> >
>> > They will usually try one of the simplest methods available under the
>> >> assumption "well, if this doesn't scale well, the more complex methods
>> >> are surely no better".
>> >
>> >
>> > Silly assumption.
>> >
>> >
>> >> This may not be entirely fair, but since the
>> >> docs they're encountering on the main website and wiki don't warn them
>> >> that certain implementations don't necessarily scale in different
>> >> ways, it's certainly not unreasonable.
>> >
>> >
>> > Well, it is actually silly.
>> >
>> > Clearly the docs can be better.  Clearly the code quality can be better
>> > especially in terms of nuking capabilities that have not found an
>> audience.
>> > But clearly also just trying one technique without asking anybody what
>> the
>> > limitations are isn't going to work as an evaluation technique.  This is
>> > exactly analogous to somebody finding that a matrix in R doesn't do what
>> a
>> > data frame is supposed to do.  It doesn't and you aren't going to find
>> out
>> > why or how from the documentation very quickly.
>> >
>> > In both cases of investigating Mahout or investigating R you will find
>> out
>> > plenty if you ask somebody who knows what they are talking about.
>> >
>> > They're at best going to
>> >> conclude the scalability will be hit-and-miss when a simple method
>> >> doesn't work.  Perhaps they'll check in again in 6-12 months.
>> >>
>> >
>> > Maybe so.  Maybe not.  I have little sympathy with people who make
>> > scatter-shot decisions like this.
>> >
>> >
>> >> ...
>> >> I see your analogy to R or sciPy - and I don't disagree.  But those
>> >> projects do not put scaling front and center; if Mahout is going to
>> >> keep scalability as a "headline feature" (which I would like to see!),
>> >> I think prominently acknowledging how different methods fail to scale
>> >> would really help its credibility.  For what it's worth, of the people
>> >> I know who've tried Mahout 100% of them were using R and/or sciPy
>> >> already, but were curious about Mahout specifically for better
>> >> scalability.
>> >>
>> >
>> > Did they ask on the mailing list?
>> >
>> >
>> >> I'm not sure where this information is best placed - it would be great
>> >> to see it on the Wiki along with the examples, at least.
>> >
>> >
>> > Sounds OK.  Maybe we should put it in the book.
>> >
>> > (oh... wait, we already did that)
>> >
>> >
>> >> It would be
>> >> awesome to see warnings at runtime ("Warning: You just trained a model
>> >> that you cannot load without at least 20GB of RAM"), but I'm not sure
>> >> how realistic that is.
>> >
>> >
>> > I think it is fine that loading the model fails with a fine error message
>> > but putting yellow warning tape all over the user's keyboard isn't going
>> to
>> > help anything.
>> >
>> >
>> >> I would like it to be easier to determine, at some very high level, why
>> >> something didn't work when an experiment fails.  Ideally, without
>> having to
>> >> dive into the code at all.
>> >>
>> >
>> > How about you ask an expert?
>> >
>> > That really is easier.  It helps the community to hear about what other
>> > people need and it helps the new user to hear what other people have
>> done.
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message