Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.210.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <FCA91A92EE52B041906A0358FC28FCC38F36E1389E@FRE1EXCH02.hq.exar.com>
References: 
 <OF784DFC6D.2508B92D-ON8525794C.005D39E8-8525794C.005D51B6@us.ibm.com>
	<BLU0-SMTP113BD5C754CC05F10518B188FC40@phx.gbl>
	<OF09D318B2.8C69FC21-ON8525794C.0060DE66-8525794C.0060F484@us.ibm.com>
	<BAY170-W139EED6AEEBC90770152E578FC40@phx.gbl>
	<OF942A7F3B.EDD95BA8-ON8525794C.006CB759-8525794C.006D23C5@us.ibm.com>
	<CAK-3We_c=7pnaB6P1fNeKrzErCvx7yNRWP73=mXqX-DW_6OSDA@mail.gmail.com>
	<FCA91A92EE52B041906A0358FC28FCC38F36E1389E@FRE1EXCH02.hq.exar.com>
Date: Sat, 19 Nov 2011 11:53:30 -0500
Message-ID: 
 <CAENxBwybuKrF+DL_+Y9OTS4p4wC8VV55vqQpuwxamOXkR95kBA@mail.gmail.com>
Subject: Re: Matrix multiplication in Hadoop
From: Edward Capriolo <edlinuxguru@gmail.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=14dae93411f916cc9d04b2194bb6

--14dae93411f916cc9d04b2194bb6
Content-Type: text/plain; charset=ISO-8859-1

Sounds like a job for next gen map reduce native libraries and gpu's. A
modern day Dr frankenstein for sure.

On Saturday, November 19, 2011, Tim Broberg <Tim.Broberg@exar.com> wrote:
> Perhaps this is a good candidate for a native library, then?
>
> ________________________________________
> From: Mike Davis [xmikedavis@gmail.com]
> Sent: Friday, November 18, 2011 7:39 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Matrix multiplication in Hadoop
>
> On Friday, November 18, 2011, Mike Spreitzer <mspreitz@us.ibm.com> wrote:
>>  Why is matrix multiplication ill-suited for Hadoop?
>
> IMHO, a huge issue here is the JVM's inability to fully support cpu vendor
> specific SIMD instructions and, by extension, optimized BLAS routines.
> Running a large MM task using intel's MKL rather than relying on generic
> compiler optimization is orders of magnitude faster on a single multicore
> processor. I see almost no way that Hadoop could win such a CPU intensive
> task against an mpi cluster with even a tenth of the nodes running with a
> decently tuned BLAS library. Racing even against a single CPU might be
> difficult, given the i/o overhead.
>
> Still, it's a reasonably common problem and we shouldn't murder the good
in
> favor of the best. I'm certain a MM/LinAlg Hadoop library with even
> mediocre performance, wrt C, would get used.
>
> --
> Mike Davis
>
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged.  The message is
> intended solely for the addressee(s).  If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful.  If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>

--14dae93411f916cc9d04b2194bb6--