spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JoshRosen <>
Subject [GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Date Fri, 07 Nov 2014 00:28:52 GMT
Github user JoshRosen commented on a diff in the pull request:
    --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
    @@ -179,6 +182,30 @@ class SparkContext(config: SparkConf) extends SparkStatusAPI with
Logging {
       conf.setIfMissing("", Utils.localHostName())
       conf.setIfMissing("spark.driver.port", "0")
    +  // This is placed after the configuration validation so that common configuration errors,
    +  // forgetting to pass a master url or app name, don't prevent subsequent SparkContexts
from being
    +  // constructed.
    +  SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
    +    SparkContext.activeSparkContextCreationSite.foreach { creationSite =>
    +      val errMsg = "Only one SparkContext may be active in this JVM (see SPARK-2243)."
    +      val errDetails = if (SparkContext.activeSparkContextIsFullyConstructed) {
    +        s"The currently active SparkContext was created at:\n${creationSite.longForm}"
    +      } else {
    +        s"Another SparkContext is either being constructed or threw an exception from
its" +
    +        " constructor; please restart your JVM in order to create a new SparkContext."
    +        s"The current SparkContext was created at:\n${creationSite.longForm}"
    +      }
    +      val exception = new SparkException(s"$errMsg $errDetails")
    +      if (conf.getBoolean("spark.driver.disableMultipleSparkContextsErrorChecking", false))
    --- End diff --
    Yeah, I picked a kinda verbose name because I wanted to reserve the better names in case
we added a public configuration that does something similar.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message