spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanbo Liang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21591) Implement treeAggregate on Dataset API
Date Tue, 01 Aug 2017 07:01:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yanbo Liang updated SPARK-21591:
--------------------------------
    Description: 
The Tungsten execution engine substantially improved the efficiency of memory and CPU for
Spark application. However, in MLlib we still not migrate the internal computing workload
from {{RDD}} to {{DataFrame}}.
The main block issue is there is no {{treeAggregate}} on {{DataFrame}}. It's very important
for MLlib algorithms, since they do aggregate on {{Vector}} which may has millions of elements.
As we all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an order of
magnitude for  lots of MLlib algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API and do the
performance benchmark related issues. And I think other scenarios except MLlib will also benefit
from this improvement if we get it done.

  was:
The Tungsten execution engine substantially improved the efficiency of memory and CPU for
Spark application. However, in MLlib we still not migrate the internal computing workload
from {{RDD}} to {{DataFrame}}.
The main block issue is there is no {{treeAggregate}} on {{DataFrame}}. It's very important
for MLlib algorithms, since they do aggregate on vector who may has millions of elements.
As we all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an order of
magnitude for  lots of MLlib algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API and do the
performance benchmark related issues. And I think other scenarios except MLlib will also benefit
from this improvement if we get it done.


> Implement treeAggregate on Dataset API
> --------------------------------------
>
>                 Key: SPARK-21591
>                 URL: https://issues.apache.org/jira/browse/SPARK-21591
>             Project: Spark
>          Issue Type: Brainstorming
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory and CPU
for Spark application. However, in MLlib we still not migrate the internal computing workload
from {{RDD}} to {{DataFrame}}.
> The main block issue is there is no {{treeAggregate}} on {{DataFrame}}. It's very important
for MLlib algorithms, since they do aggregate on {{Vector}} which may has millions of elements.
As we all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an order of
magnitude for  lots of MLlib algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API and do
the performance benchmark related issues. And I think other scenarios except MLlib will also
benefit from this improvement if we get it done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message