spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [20/20] git commit: Merge pull request #127 from kayousterhout/consolidate_schedulers
Date Wed, 25 Dec 2013 00:35:51 GMT
Merge pull request #127 from kayousterhout/consolidate_schedulers

Deduplicate Local and Cluster schedulers.

The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.

Unfortunately the diff makes this change look much more invasive than it is -- TaskScheduler.scala
is only superficially changed (names updated, overrides removed) from the old ClusterScheduler.scala,
and the same with
TaskSetManager.scala.

Thanks @rxin for suggesting this change!


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/85a344b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/85a344b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/85a344b4

Branch: refs/heads/master
Commit: 85a344b4f0cd149c6e6f06f8b942c34146b302be
Parents: c2dd6bc 1efe3ad
Author: Patrick Wendell <pwendell@gmail.com>
Authored: Tue Dec 24 16:35:06 2013 -0800
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Tue Dec 24 16:35:06 2013 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/SparkContext.scala   |  41 +-
 .../spark/scheduler/ExecutorLossReason.scala    |  38 +
 .../spark/scheduler/SchedulerBackend.scala      |  37 +
 .../spark/scheduler/TaskResultGetter.scala      | 107 +++
 .../apache/spark/scheduler/TaskScheduler.scala  |  11 +-
 .../spark/scheduler/TaskSchedulerImpl.scala     | 474 ++++++++++++
 .../apache/spark/scheduler/TaskSetManager.scala | 704 +++++++++++++++++-
 .../apache/spark/scheduler/WorkerOffer.scala    |  24 +
 .../scheduler/cluster/ClusterScheduler.scala    | 473 ------------
 .../cluster/ClusterTaskSetManager.scala         | 713 -------------------
 .../cluster/CoarseGrainedSchedulerBackend.scala |   6 +-
 .../scheduler/cluster/ExecutorLossReason.scala  |  38 -
 .../scheduler/cluster/SchedulerBackend.scala    |  37 -
 .../cluster/SimrSchedulerBackend.scala          |   4 +-
 .../cluster/SparkDeploySchedulerBackend.scala   |   6 +-
 .../scheduler/cluster/TaskResultGetter.scala    | 108 ---
 .../spark/scheduler/cluster/WorkerOffer.scala   |  24 -
 .../mesos/CoarseMesosSchedulerBackend.scala     |   5 +-
 .../cluster/mesos/MesosSchedulerBackend.scala   |   7 +-
 .../spark/scheduler/local/LocalBackend.scala    | 108 +++
 .../spark/scheduler/local/LocalScheduler.scala  | 222 ------
 .../scheduler/local/LocalTaskSetManager.scala   | 191 -----
 .../scala/org/apache/spark/FailureSuite.scala   |   4 +-
 .../SparkContextSchedulerCreationSuite.scala    |  58 +-
 .../spark/scheduler/ClusterSchedulerSuite.scala | 265 +++++++
 .../org/apache/spark/scheduler/FakeTask.scala   |  26 +
 .../spark/scheduler/SparkListenerSuite.scala    |  19 +-
 .../spark/scheduler/TaskResultGetterSuite.scala | 112 +++
 .../spark/scheduler/TaskSetManagerSuite.scala   | 319 +++++++++
 .../cluster/ClusterSchedulerSuite.scala         | 267 -------
 .../cluster/ClusterTaskSetManagerSuite.scala    | 319 ---------
 .../spark/scheduler/cluster/FakeTask.scala      |  27 -
 .../cluster/TaskResultGetterSuite.scala         | 114 ---
 .../scheduler/local/LocalSchedulerSuite.scala   | 227 ------
 .../cluster/YarnClusterScheduler.scala          |  10 +-
 35 files changed, 2287 insertions(+), 2858 deletions(-)
----------------------------------------------------------------------



Mime
View raw message