spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewor14 <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-5654] Integrate SparkR
Date Sat, 04 Apr 2015 01:30:09 GMT
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5096#discussion_r27765296
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
---
    @@ -243,6 +258,36 @@
         return pyargs;
       }
     
    +  private List<String> buildSparkRCommand(Map<String, String> env) throws
IOException {
    +    if (!appArgs.isEmpty() && appArgs.get(0).endsWith(".R")) {
    +      appResource = appArgs.get(0);
    +      appArgs.remove(0);
    +      return buildCommand(env);
    +    }
    +
    +    Properties props = loadPropertiesFile();
    +    mergeEnvPathList(env, getLibPathEnvName(),
    +            firstNonEmptyValue(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH, conf, props));
    +
    +    // Store spark-submit arguments in an environment variable, since there's no way
to pass
    +    // them to sparkR on the command line.
    +    StringBuilder submitArgs = new StringBuilder();
    +    for (String arg : buildSparkSubmitArgs()) {
    +      if (submitArgs.length() > 0) {
    +        submitArgs.append(" ");
    +      }
    +      submitArgs.append(quoteForPython(arg));
    --- End diff --
    
    not a big deal, but these few lines are duplicated with python. Could we have a common
method that puts Spark submit args, escapes them properly, and puts them in an environment
variable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message