spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shiva...@apache.org
Subject spark git commit: [SPARK-22315][SPARKR] Warn if SparkR package version doesn't match SparkContext
Date Mon, 06 Nov 2017 16:58:57 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 e35c53a97 -> 2695b9213


[SPARK-22315][SPARKR] Warn if SparkR package version doesn't match SparkContext

## What changes were proposed in this pull request?

This PR adds a check between the R package version used and the version reported by SparkContext
running in the JVM. The goal here is to warn users when they have a R package downloaded from
CRAN and are using that to connect to an existing Spark cluster.

This is raised as a warning rather than an error as users might want to use patch versions
interchangeably (e.g., 2.1.3 with 2.1.2 etc.)

## How was this patch tested?

Manually by changing the `DESCRIPTION` file

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes #19624 from shivaram/sparkr-version-check.

(cherry picked from commit 65a8bf6036fe41a53b4b1e4298fa35d7fa4e9970)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2695b921
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2695b921
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2695b921

Branch: refs/heads/branch-2.2
Commit: 2695b9213d590201cb9937736134e94a11f48ba2
Parents: e35c53a
Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Authored: Mon Nov 6 08:58:42 2017 -0800
Committer: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Committed: Mon Nov 6 08:58:55 2017 -0800

----------------------------------------------------------------------
 R/pkg/R/sparkR.R | 12 ++++++++++++
 1 file changed, 12 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/2695b921/R/pkg/R/sparkR.R
----------------------------------------------------------------------
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index d0a12b7..9ebd344 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -420,6 +420,18 @@ sparkR.session <- function(
                                 enableHiveSupport)
     assign(".sparkRsession", sparkSession, envir = .sparkREnv)
   }
+
+  # Check if version number of SparkSession matches version number of SparkR package
+  jvmVersion <- callJMethod(sparkSession, "version")
+  # Remove -SNAPSHOT from jvm versions
+  jvmVersionStrip <- gsub("-SNAPSHOT", "", jvmVersion)
+  rPackageVersion <- paste0(packageVersion("SparkR"))
+
+  if (jvmVersionStrip != rPackageVersion) {
+    warning(paste("Version mismatch between Spark JVM and SparkR package. JVM version was",
+                  jvmVersion, ", while R package version was", rPackageVersion))
+  }
+
   sparkSession
 }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message