spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Freeman <freeman.jer...@gmail.com>
Subject Re: [mllib] Add multiplying large scale matrices
Date Fri, 05 Sep 2014 17:29:18 GMT
Hey all, 

Definitely agreed this would be nice! In our own work we've done element-wise addition, subtraction,
and scalar multiplication of similarly partitioned matrices very efficiently with zipping.
We've also done matrix-matrix multiplication with zipping, but that only works in certain
circumstances, and it's otherwise very communication intensive (as Shivaram says). Another
tricky thing with addition / subtraction is how to handle sparse vs. dense arrays.

Would be happy to contribute anything we did, but definitely first worth knowing what progress
has been made from the AMPLab.

-- Jeremy

---------------------
jeremy freeman, phd
neuroscientist
@thefreemanlab

On Sep 5, 2014, at 12:23 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> Hey There,
> 
> I believe this is on the roadmap for the 1.2 next release. But
> Xiangrui can comment on this.
> 
> - Patrick
> 
> On Fri, Sep 5, 2014 at 9:18 AM, Yu Ishikawa
> <yuu.ishikawa+spark@gmail.com> wrote:
>> Hi Evan,
>> 
>> That's sounds interesting.
>> 
>> Here is the ticket which I created.
>> https://issues.apache.org/jira/browse/SPARK-3416
>> 
>> thanks,
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Add-multiplying-large-scale-matrices-tp8291p8296.html
>> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message