spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject git commit: [SPARK-2350] Don't NPE while launching drivers
Date Fri, 04 Jul 2014 06:00:46 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-0.9 0d3d5ce2b -> c37e9ed83


[SPARK-2350] Don't NPE while launching drivers

Prior to this change, we could throw a NPE if we launch a driver while another one is waiting,
because removing from an iterator while iterating over it is not safe.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1289 from aarondav/master-fail and squashes the following commits:

1cf1cf4 [Aaron Davidson] SPARK-2350: Don't NPE while launching drivers
(cherry picked from commit 586feb5c9528042420f678f78bacb6c254a5eaf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c37e9ed8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c37e9ed8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c37e9ed8

Branch: refs/heads/branch-0.9
Commit: c37e9ed833c07fef8bd132da7014aab08434fd43
Parents: 0d3d5ce
Author: Aaron Davidson <aaron@databricks.com>
Authored: Thu Jul 3 22:31:41 2014 -0700
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Thu Jul 3 23:00:30 2014 -0700

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/deploy/master/Master.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/c37e9ed8/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index d72bb7a..ef2ee81 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -450,7 +450,7 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int) extends
Act
     // First schedule drivers, they take strict precedence over applications
     val shuffledWorkers = Random.shuffle(workers) // Randomization helps balance drivers
     for (worker <- shuffledWorkers if worker.state == WorkerState.ALIVE) {
-      for (driver <- waitingDrivers) {
+      for (driver <- List(waitingDrivers: _*)) { // iterate over a copy of waitingDrivers
         if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores)
{
           launchDriver(worker, driver)
           waitingDrivers -= driver


Mime
View raw message