spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tgravescs <...@git.apache.org>
Subject [GitHub] spark pull request #21468: [SPARK-22151] : PYTHONPATH not picked up from the...
Date Thu, 07 Jun 2018 18:16:06 GMT
Github user tgravescs commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21468#discussion_r193842887
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
    @@ -813,8 +813,14 @@ private[spark] class Client(
         if (pythonPath.nonEmpty) {
           val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath)
             .mkString(ApplicationConstants.CLASS_PATH_SEPARATOR)
    -      env("PYTHONPATH") = pythonPathStr
    -      sparkConf.setExecutorEnv("PYTHONPATH", pythonPathStr)
    +      val newValue =
    --- End diff --
    
    good questions
    
    - precedence: So right now you can work around this issue by exporting PYTHONPATH before
you launch spark-submit, I think this is something that could just be in someone's env on
the launcher box and might not be what you want in a yarn container.  I would think that specifying
explicit pythonpath via spark.yarn.appMasterEnv would take precedence over that since you
explicitly configured. Now the second question is where that fails with the py-files and that
one isn't as clear to me since like you said its explicitly specified.    Maybe we do py-files
then spark.yarn.appMasterEnv.PYTHONPATH and then last env PYTHONPATH.  that is different from
the way it is now though. thoughts?
    
    - agree this should not be reflected in the executors so if it is we shouldn't do that.
We should make sure the spark. executorEnv.PYTHONPATH works


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message