spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-5816] Add huge compatibility warning in DriverWrapper
Date Thu, 19 Feb 2015 17:56:39 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 fbcb949c5 -> f93d4d992


[SPARK-5816] Add huge compatibility warning in DriverWrapper

The stability of the new submission gateway assumes that the arguments in `DriverWrapper`
are consistent across multiple Spark versions. However, this is not at all clear from the
code itself. In fact, this was broken in 20a6013106b56a1a1cc3e8cda092330ffbe77cc3, which is
fortunately OK because both that commit and the original commit that added this gateway are
part of the same release.

To prevent this from happening again we should at the very least add a huge warning where
appropriate.

Author: Andrew Or <andrew@databricks.com>

Closes #4687 from andrewor14/driver-wrapper-warning and squashes the following commits:

7989b56 [Andrew Or] Add huge compatibility warning

(cherry picked from commit 38e624a732b18e01ad2e7a499ce0bb0d7acdcdf6)
Signed-off-by: Andrew Or <andrew@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f93d4d99
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f93d4d99
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f93d4d99

Branch: refs/heads/branch-1.3
Commit: f93d4d99204e3d83eaaaa802e326c55416e1b7e2
Parents: fbcb949
Author: Andrew Or <andrew@databricks.com>
Authored: Thu Feb 19 09:56:25 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Thu Feb 19 09:56:37 2015 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/deploy/worker/DriverWrapper.scala  | 7 +++++++
 1 file changed, 7 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/f93d4d99/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala b/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala
index ab467a5..deef6ef 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala
@@ -26,10 +26,17 @@ import org.apache.spark.util.{AkkaUtils, ChildFirstURLClassLoader, MutableURLCla
 
 /**
  * Utility object for launching driver programs such that they share fate with the Worker
process.
+ * This is used in standalone cluster mode only.
  */
 object DriverWrapper {
   def main(args: Array[String]) {
     args.toList match {
+      /*
+       * IMPORTANT: Spark 1.3 provides a stable application submission gateway that is both
+       * backward and forward compatible across future Spark versions. Because this gateway
+       * uses this class to launch the driver, the ordering and semantics of the arguments
+       * here must also remain consistent across versions.
+       */
       case workerUrl :: userJar :: mainClass :: extraArgs =>
         val conf = new SparkConf()
         val (actorSystem, _) = AkkaUtils.createActorSystem("Driver",


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message