spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh
Date Sun, 11 Sep 2016 09:20:02 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 bde545268 -> d293062a4


[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh

## What changes were proposed in this pull request?
During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH
and can be sourced many times, causing duplicates in the path.  This change adds a env flag
that is set when the PYTHONPATH is appended so it will happen only one time.

## How was this patch tested?
Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries.

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #15028 from BryanCutler/fix-duplicate-pythonpath-SPARK-17336.

(cherry picked from commit c76baff0cc4775c2191d075cc9a8176e4915fec8)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d293062a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d293062a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d293062a

Branch: refs/heads/branch-2.0
Commit: d293062a4cd04acf48697b1fd6a70fef97b338da
Parents: bde5452
Author: Bryan Cutler <cutlerb@gmail.com>
Authored: Sun Sep 11 10:19:39 2016 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Sun Sep 11 10:19:49 2016 +0100

----------------------------------------------------------------------
 sbin/spark-config.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d293062a/sbin/spark-config.sh
----------------------------------------------------------------------
diff --git a/sbin/spark-config.sh b/sbin/spark-config.sh
index a7a44cd..b728448 100755
--- a/sbin/spark-config.sh
+++ b/sbin/spark-config.sh
@@ -26,5 +26,8 @@ fi
 
 export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}/conf"}"
 # Add the PySpark classes to the PYTHONPATH:
-export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
-export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
+if [ -z "${PYSPARK_PYTHONPATH_SET}" ]; then
+  export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
+  export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
+  export PYSPARK_PYTHONPATH_SET=1
+fi


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message