spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ma...@apache.org
Subject git commit: SPARK-1775: Unneeded lock in ShuffleMapTask.deserializeInfo
Date Sat, 10 May 2014 05:58:42 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-0.9 9e2c59efe -> bea2be308


SPARK-1775: Unneeded lock in ShuffleMapTask.deserializeInfo

This was used in the past to have a cache of deserialized ShuffleMapTasks, but that's been
removed, so there's no need for a lock. It slows down Spark when task descriptions are large,
e.g. due to large lineage graphs or local variables.

Author: Sandeep <sandeep@techaddict.me>

Closes #707 from techaddict/SPARK-1775 and squashes the following commits:

18d8ebf [Sandeep] SPARK-1775: Unneeded lock in ShuffleMapTask.deserializeInfo This was used
in the past to have a cache of deserialized ShuffleMapTasks, but that's been removed, so there's
no need for a lock. It slows down Spark when task descriptions are large, e.g. due to large
lineage graphs or local variables.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bea2be30
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bea2be30
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bea2be30

Branch: refs/heads/branch-0.9
Commit: bea2be3082b3fb43c1eb1e84d1a1fa7a2dab6593
Parents: 9e2c59e
Author: Sandeep <sandeep@techaddict.me>
Authored: Thu May 8 22:30:17 2014 -0700
Committer: Matei Zaharia <matei@databricks.com>
Committed: Fri May 9 22:58:32 2014 -0700

----------------------------------------------------------------------
 .../org/apache/spark/scheduler/ShuffleMapTask.scala | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/bea2be30/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
index a37ead5..0ed28b9 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
@@ -61,15 +61,13 @@ private[spark] object ShuffleMapTask {
   }
 
   def deserializeInfo(stageId: Int, bytes: Array[Byte]): (RDD[_], ShuffleDependency[_,_])
= {
-    synchronized {
-      val loader = Thread.currentThread.getContextClassLoader
-      val in = new GZIPInputStream(new ByteArrayInputStream(bytes))
-      val ser = SparkEnv.get.closureSerializer.newInstance()
-      val objIn = ser.deserializeStream(in)
-      val rdd = objIn.readObject().asInstanceOf[RDD[_]]
-      val dep = objIn.readObject().asInstanceOf[ShuffleDependency[_,_]]
-      (rdd, dep)
-    }
+    val loader = Thread.currentThread.getContextClassLoader
+    val in = new GZIPInputStream(new ByteArrayInputStream(bytes))
+    val ser = SparkEnv.get.closureSerializer.newInstance()
+    val objIn = ser.deserializeStream(in)
+    val rdd = objIn.readObject().asInstanceOf[RDD[_]]
+    val dep = objIn.readObject().asInstanceOf[ShuffleDependency[_,_]]
+    (rdd, dep)
   }
 
   // Since both the JarSet and FileSet have the same format this is used for both.


Mime
View raw message