spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tgravescs <>
Subject [GitHub] spark pull request #21468: [SPARK-22151] : PYTHONPATH not picked up from the...
Date Thu, 07 Jun 2018 18:16:06 GMT
Github user tgravescs commented on a diff in the pull request:
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
    @@ -813,8 +813,14 @@ private[spark] class Client(
         if (pythonPath.nonEmpty) {
           val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath)
    -      env("PYTHONPATH") = pythonPathStr
    -      sparkConf.setExecutorEnv("PYTHONPATH", pythonPathStr)
    +      val newValue =
    --- End diff --
    good questions
    - precedence: So right now you can work around this issue by exporting PYTHONPATH before
you launch spark-submit, I think this is something that could just be in someone's env on
the launcher box and might not be what you want in a yarn container.  I would think that specifying
explicit pythonpath via spark.yarn.appMasterEnv would take precedence over that since you
explicitly configured. Now the second question is where that fails with the py-files and that
one isn't as clear to me since like you said its explicitly specified.    Maybe we do py-files
then spark.yarn.appMasterEnv.PYTHONPATH and then last env PYTHONPATH.  that is different from
the way it is now though. thoughts?
    - agree this should not be reflected in the executors so if it is we shouldn't do that.
We should make sure the spark. executorEnv.PYTHONPATH works


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message