spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [17/50] git commit: Merge pull request #220 from rxin/zippart
Date Thu, 12 Dec 2013 07:11:48 GMT
Merge pull request #220 from rxin/zippart

Memoize preferred locations in ZippedPartitionsBaseRDD

so preferred location computation doesn't lead to exponential explosion.

This was a problem in GraphX where we have a whole chain of RDDs that are ZippedPartitionsRDD's,
and the preferred locations were taking eternity to compute.

(cherry picked from commit e36fe55a031d2c01c9d7c5d85965951c681a0c74)
Signed-off-by: Reynold Xin <rxin@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/87676a6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/87676a6a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/87676a6a

Branch: refs/heads/scala-2.10
Commit: 87676a6af2c8fc33c5b5d4e7eb45e3e8558f3c33
Parents: 0780498 9cf7f31
Author: Matei Zaharia <matei@eecs.berkeley.edu>
Authored: Fri Dec 6 11:01:42 2013 -0800
Committer: Matei Zaharia <matei@eecs.berkeley.edu>
Committed: Fri Dec 6 11:01:42 2013 -0800

----------------------------------------------------------------------
 .../apache/spark/rdd/ZippedPartitionsRDD.scala  | 27 ++++++++------------
 1 file changed, 11 insertions(+), 16 deletions(-)
----------------------------------------------------------------------



Mime
View raw message