Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.222.194 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=himzAe7DJ4mVEcEeYveCx3hzMDOIu4NljcxIttvVuMvradBXrGr71vYZn4KlAgve+D
         bQB2JvWG/ccLg1g7VmW3rzN6YyBGPkc9x+tYE/66UqnqE99VMKtU59ISC1JB9g2qtAEy
         txyVzG9JacPmA9yxFOz/6TJX3qLCQIuRH03XQ=
MIME-Version: 1.0
In-Reply-To: <201001142209.31527.janert@ieee.org>
References: <201001141957.38062.janert@ieee.org>
 <c7d45fc71001142204q6b2b6c35ua728b5500ba142f8@mail.gmail.com>
	<201001142209.31527.janert@ieee.org>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Thu, 14 Jan 2010 22:26:17 -0800
Message-ID: <c7d45fc71001142226y8f2d7efkeef5c08aef8dbc91@mail.gmail.com>
Subject: Re: [Slightly OT] SVD etc on Map/Reduce?
To: janert@ieee.org
Cc: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00504502d609f55ee9047d2e15aa

--00504502d609f55ee9047d2e15aa
Content-Type: text/plain; charset=UTF-8

On Thu, Jan 14, 2010 at 10:09 PM, Philipp K. Janert <janert@ieee.org> wrote:

> > If you mean matrix factorization, take a look at this:
> > http://arxiv.org/abs/0909.4061v1
>
> That seems to support my earlier hunch that
> efficient implementations of such factorizations
> on M/R would likely be approximate only or
> partial (ie yielding the largest of the eigenvalues,
> not necessarily the entire spectrum).
>

For very large sparse problems, approximate decompositions are generally
preferred.  Due to limited accuracy in the input, only the first several
eigenvectors can be extracted at all.  Moreover, many important problems
have very large apparent dimensionality, but limited actual rank.   Neither
of these characteristics is a characteristic of map-reduce in the slightest.


 > It is the common sense of those on this mailing list that these kinds of
> > algorithms could be done using map-reduce.
>
> I am not sure what you are trying to tell me here.


I am trying to say that we don't yet have working implementations.  There
should be a k-means implementation that use these techniques before long.
You would be very welcome to try your hand at some other of the algorithms
and I am sure that you would have quite a lot of support from the mailing
list.

You comments puzzle me, though.  Do you have an application in mind?  Was
there something you were particularly looking for?

-- 
Ted Dunning, CTO
DeepDyve

--00504502d609f55ee9047d2e15aa--