Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6832910E10 for ; Wed, 25 Dec 2013 00:35:59 +0000 (UTC) Received: (qmail 89696 invoked by uid 500); 25 Dec 2013 00:35:59 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 89658 invoked by uid 500); 25 Dec 2013 00:35:59 -0000 Mailing-List: contact commits-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list commits@spark.incubator.apache.org Received: (qmail 89651 invoked by uid 99); 25 Dec 2013 00:35:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Dec 2013 00:35:59 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 25 Dec 2013 00:35:54 +0000 Received: (qmail 89376 invoked by uid 99); 25 Dec 2013 00:35:33 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Dec 2013 00:35:33 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id EB6BD89434F; Wed, 25 Dec 2013 00:35:32 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: pwendell@apache.org To: commits@spark.incubator.apache.org Date: Wed, 25 Dec 2013 00:35:51 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [20/20] git commit: Merge pull request #127 from kayousterhout/consolidate_schedulers X-Virus-Checked: Checked by ClamAV on apache.org Merge pull request #127 from kayousterhout/consolidate_schedulers Deduplicate Local and Cluster schedulers. The code in LocalScheduler/LocalTaskSetManager was nearly identical to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy made making updating the schedulers unnecessarily painful and error- prone. This commit combines the two into a single TaskScheduler/ TaskSetManager. Unfortunately the diff makes this change look much more invasive than it is -- TaskScheduler.scala is only superficially changed (names updated, overrides removed) from the old ClusterScheduler.scala, and the same with TaskSetManager.scala. Thanks @rxin for suggesting this change! Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/85a344b4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/85a344b4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/85a344b4 Branch: refs/heads/master Commit: 85a344b4f0cd149c6e6f06f8b942c34146b302be Parents: c2dd6bc 1efe3ad Author: Patrick Wendell Authored: Tue Dec 24 16:35:06 2013 -0800 Committer: Patrick Wendell Committed: Tue Dec 24 16:35:06 2013 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/SparkContext.scala | 41 +- .../spark/scheduler/ExecutorLossReason.scala | 38 + .../spark/scheduler/SchedulerBackend.scala | 37 + .../spark/scheduler/TaskResultGetter.scala | 107 +++ .../apache/spark/scheduler/TaskScheduler.scala | 11 +- .../spark/scheduler/TaskSchedulerImpl.scala | 474 ++++++++++++ .../apache/spark/scheduler/TaskSetManager.scala | 704 +++++++++++++++++- .../apache/spark/scheduler/WorkerOffer.scala | 24 + .../scheduler/cluster/ClusterScheduler.scala | 473 ------------ .../cluster/ClusterTaskSetManager.scala | 713 ------------------- .../cluster/CoarseGrainedSchedulerBackend.scala | 6 +- .../scheduler/cluster/ExecutorLossReason.scala | 38 - .../scheduler/cluster/SchedulerBackend.scala | 37 - .../cluster/SimrSchedulerBackend.scala | 4 +- .../cluster/SparkDeploySchedulerBackend.scala | 6 +- .../scheduler/cluster/TaskResultGetter.scala | 108 --- .../spark/scheduler/cluster/WorkerOffer.scala | 24 - .../mesos/CoarseMesosSchedulerBackend.scala | 5 +- .../cluster/mesos/MesosSchedulerBackend.scala | 7 +- .../spark/scheduler/local/LocalBackend.scala | 108 +++ .../spark/scheduler/local/LocalScheduler.scala | 222 ------ .../scheduler/local/LocalTaskSetManager.scala | 191 ----- .../scala/org/apache/spark/FailureSuite.scala | 4 +- .../SparkContextSchedulerCreationSuite.scala | 58 +- .../spark/scheduler/ClusterSchedulerSuite.scala | 265 +++++++ .../org/apache/spark/scheduler/FakeTask.scala | 26 + .../spark/scheduler/SparkListenerSuite.scala | 19 +- .../spark/scheduler/TaskResultGetterSuite.scala | 112 +++ .../spark/scheduler/TaskSetManagerSuite.scala | 319 +++++++++ .../cluster/ClusterSchedulerSuite.scala | 267 ------- .../cluster/ClusterTaskSetManagerSuite.scala | 319 --------- .../spark/scheduler/cluster/FakeTask.scala | 27 - .../cluster/TaskResultGetterSuite.scala | 114 --- .../scheduler/local/LocalSchedulerSuite.scala | 227 ------ .../cluster/YarnClusterScheduler.scala | 10 +- 35 files changed, 2287 insertions(+), 2858 deletions(-) ----------------------------------------------------------------------