spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-4467] Partial fix for fetch failure in sort-based shuffle (1.1)
Date Tue, 18 Nov 2014 02:11:12 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.1 e4f5695ec -> aa9ebdaa2


[SPARK-4467] Partial fix for fetch failure in sort-based shuffle (1.1)

This is the 1.1 version of #3302. There has been some refactoring in master so we can't cherry-pick
that PR.

Author: Andrew Or <andrew@databricks.com>

Closes #3330 from andrewor14/sort-fetch-fail and squashes the following commits:

486fc49 [Andrew Or] Reset `elementsRead`


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/aa9ebdaa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/aa9ebdaa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/aa9ebdaa

Branch: refs/heads/branch-1.1
Commit: aa9ebdaa28bebccc8f65a323d7c6fc34cf68ef73
Parents: e4f5695
Author: Andrew Or <andrew@databricks.com>
Authored: Mon Nov 17 18:10:49 2014 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Mon Nov 17 18:10:49 2014 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/util/collection/ExternalSorter.scala     | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/aa9ebdaa/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
index 3136306..97ddd96 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
@@ -288,6 +288,7 @@ private[spark] class ExternalSorter[K, V, C](
     myMemoryThreshold = 0
 
     _memoryBytesSpilled += memorySize
+    elementsRead = 0
   }
 
   /**


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message