spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <...@git.apache.org>
Subject [GitHub] spark pull request #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files...
Date Fri, 25 May 2018 07:14:27 GMT
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21426#discussion_r190808099
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala ---
    @@ -153,4 +154,25 @@ object PythonRunner {
           .map { p => formatPath(p, testWindows) }
       }
     
    +  /**
    +   * Resolves the ".py" files. ".py" file should not be added as is because PYTHONPATH
does
    +   * not expect a file. This method creates a temporary directory and puts the ".py"
files
    +   * if exist in the given paths.
    +   */
    +  private def resolvePyFiles(pyFiles: Array[String]): Array[String] = {
    +    val dest = Utils.createTempDir(namePrefix = "localPyFiles")
    +    pyFiles.map { pyFile =>
    +      // In case of client with submit, the python paths should be set before context
    +      // initialization because the context initialization can be done later.
    +      // We will copy the local ".py" files because ".py" file shouldn't be added
    +      // alone but its parent directory in PYTHONPATH. See SPARK-24384.
    +      if (pyFile.endsWith(".py")) {
    +        val source = new File(pyFile)
    --- End diff --
    
    Yeap


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message