spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From felixcheung <>
Subject [GitHub] spark pull request #14179: [SPARK-16055][SPARKR] warning added while using s...
Date Fri, 15 Jul 2016 00:00:16 GMT
Github user felixcheung commented on a diff in the pull request:
    --- Diff: R/pkg/R/sparkR.R ---
    @@ -155,6 +155,9 @@ sparkR.sparkContext <- function(
       existingPort <- Sys.getenv("EXISTING_SPARKR_BACKEND_PORT", "")
       if (existingPort != "") {
    +    if(sparkPackages != ""){
    +        warning("--packages flag should be used with with spark-submit")
    --- End diff --
    @shivaram maybe it should but sparkR.session() is already called in sparkR shell, and
calling SparkSession again with the sparkPackages does nothing:
    > sparkR.session(sparkPackages = "com.databricks:spark-avro_2.10:2.0.1")
    Java ref type org.apache.spark.sql.SparkSession id 1
    > read.df("", source = "avro")
    16/07/14 23:55:43 ERROR RBackendHandler: loadDF on org.apache.spark.sql.api.r.SQLUtils
    Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
      org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Please use
Spark package;
    @krishnakalyan3 something like "sparkPackages has no effect when using spark-submit or
sparkR shell, please use the --packages commandline instead"

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message