spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [spark] yifeih commented on a change in pull request #25823: [SPARK-28211][Core][Shuffle] Propose Shuffle Driver Components API
Date Mon, 30 Sep 2019 23:59:09 GMT
yifeih commented on a change in pull request #25823: [SPARK-28211][Core][Shuffle] Propose Shuffle
Driver Components API

 File path: core/src/main/scala/org/apache/spark/SparkContext.scala
 @@ -524,6 +528,19 @@ class SparkContext(config: SparkConf) extends Logging {
     executorEnvs ++= _conf.getExecutorEnv
     executorEnvs("SPARK_USER") = sparkUser
+    val configuredPluginClasses = conf.get(SHUFFLE_IO_PLUGIN_CLASS)
+    val maybeIO = Utils.loadExtensions(
+      classOf[ShuffleDataIO], Seq(configuredPluginClasses), conf)
+    require(maybeIO.nonEmpty, s"At least one valid shuffle plugin must be specified by config
" +
+      s"${SHUFFLE_IO_PLUGIN_CLASS.key}, but $configuredPluginClasses resulted in zero valid
" +
+      s"plugins.")
+    require(maybeIO.size == 1,
+        s"Specified shuffle plugin(s) $configuredPluginClasses resulted in more than one
valid " +
+        s"plugin, but only one valid plugin should be specified")
+    _shuffleDriverComponents = maybeIO.head.driver()
+    _shuffleDriverComponents.initializeApplication().asScala.foreach {
+      case (k, v) => _conf.set(ShuffleDataIO.SHUFFLE_SPARK_CONF_PREFIX + k, v) }
 Review comment:
   That's one use case I didn't think about. Intuitively, I guess it would make sense for
the user to be able to override the configs set by the plugin, since the alternative might
be confusing to a user. In that case, I think it would make sense to log the configs, and
also log when the value returned by the driver won't take effect due to user overrides for
that particular application. 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message