mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei Li <wei.le...@gmail.com>
Subject Re: is it possible to compute the SVD for a large scale matrix
Date Fri, 25 Mar 2011 10:25:18 GMT
Actually, I would like to perform the spectral clustering on a large scale
sparse matrix, but it failed due to the OutOfMemory error when creating the
DenseMatrix for SVD decomposition.

Best
Wei

On Fri, Mar 25, 2011 at 4:05 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:

> SSVD != Lanczos. if you do PCA or LSI it is perhaps what you need. it
> can take on these things. Well at least some of my branches can, if
> not the official patch.
>
> -d
>
> On Thu, Mar 24, 2011 at 11:09 PM, Wei Li <wei.lee04@gmail.com> wrote:
> >
> > thanks for your reply
> >
> > my matrix is not very dense, a sparse matrix.
> >
> > I have tried the svd of Mahout, but failed due to the OutOfMemory error.
> >
> > Best
> > Wei
> >
> >
> >
> > On Fri, Mar 25, 2011 at 2:03 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
> >>
> >> you can certainly try to write it out into a DRM (distributed row
> >> matrix) and run stochastic SVD on  hadoop (off the trunk now). see
> >> MAHOUT-593. This is suitable if you have a good decay of singular
> >> values (but if you don't it probably just means you have so much noise
> >> that it masks the problem you are trying to solve in your data).
> >>
> >> Current committed solution is not most efficient yet, but it should be
> >> quite capable.
> >>
> >> If you do, let me know how it went.
> >>
> >> thanks.
> >> -d
> >>
> >> On Thu, Mar 24, 2011 at 10:59 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> >> wrote:
> >> > Are you sure your matrix is dense?
> >> >
> >> > On Thu, Mar 24, 2011 at 9:59 PM, Wei Li <wei.lee04@gmail.com> wrote:
> >> >> Hi All:
> >> >>
> >> >>    is it possible to compute the SVD factorization for a 600,000 *
> >> >> 600,000
> >> >> matrix using Mahout?
> >> >>
> >> >>    I have got the OutOfMemory error when creating the DenseMatrix.
> >> >>
> >> >> Best
> >> >> Wei
> >> >>
> >> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message