hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Matrix multiplication in Hadoop
Date Fri, 18 Nov 2011 19:46:47 GMT
A problem with matrix multiplication in hadoop is that hadoop is row
oriented for the most part. I have thought about this use case however and
you can theoretically turn a 2D matrix into a 1D matrix and then that fits
into the row oriented nature of hadoop. Also being that the typical mapper
can have fairly large chunks of memory like 1024MB I have done work like
this before were I loaded such datasets into memory to process them. That
usage does not really fit the map reduce model.

I have been wanting to look at:

On Fri, Nov 18, 2011 at 1:48 PM, Ayon Sinha <ayonsinha@yahoo.com> wrote:

> I'd really be interested in a comparison of Numpy/Octave/Matlab kind of
> tools with a Hadoop (lets say 4-10 large cloud servers) implementation with
> growing size of the matrix. I want to know the scale at which Hadoop really
> starts to pull away.
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
> ________________________________
> From: Michel Segel <michael_segel@hotmail.com>
> To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
> Sent: Friday, November 18, 2011 9:33 AM
> Subject: Re: Matrix multiplication in Hadoop
> Is Hadoop the best tool for doing large matrix math.
> Sure you can do it, but, aren't there better tools for these types of
> problems?
> Sent from a remote device. Please excuse any typos...
> Mike Segel
> On Nov 18, 2011, at 10:59 AM, Mike Spreitzer <mspreitz@us.ibm.com> wrote:
> > Who is doing multiplication of large dense matrices using Hadoop?  What
> is
> > a good way to do that computation using Hadoop?
> >
> > Thanks,
> > Mike

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message