spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From van...@apache.org
Subject spark git commit: [SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkContext.
Date Wed, 25 Jan 2017 20:08:15 GMT
Repository: spark
Updated Branches:
  refs/heads/master f6480b146 -> 92afaa93a


[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkContext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #16682 from vanzin/SPARK-19307.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/92afaa93
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/92afaa93
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/92afaa93

Branch: refs/heads/master
Commit: 92afaa93a0b67f561a790822ccdd2b814455edcc
Parents: f6480b1
Author: Marcelo Vanzin <vanzin@cloudera.com>
Authored: Wed Jan 25 12:08:08 2017 -0800
Committer: Marcelo Vanzin <vanzin@cloudera.com>
Committed: Wed Jan 25 12:08:08 2017 -0800

----------------------------------------------------------------------
 python/pyspark/context.py |  3 +++
 python/pyspark/tests.py   | 20 ++++++++++++++++++++
 2 files changed, 23 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/92afaa93/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 5c4e79c..ac4b2b0 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -132,6 +132,9 @@ class SparkContext(object):
             self._conf = conf
         else:
             self._conf = SparkConf(_jvm=SparkContext._jvm)
+            if conf is not None:
+                for k, v in conf.getAll():
+                    self._conf.set(k, v)
 
         self._batchSize = batchSize  # -1 represents an unlimited batch size
         self._unbatched_serializer = serializer

http://git-wip-us.apache.org/repos/asf/spark/blob/92afaa93/python/pyspark/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py
index c383d9a..e908b1e 100644
--- a/python/pyspark/tests.py
+++ b/python/pyspark/tests.py
@@ -2035,6 +2035,26 @@ class SparkSubmitTests(unittest.TestCase):
         self.assertEqual(0, proc.returncode)
         self.assertIn("[2, 4, 6]", out.decode('utf-8'))
 
+    def test_user_configuration(self):
+        """Make sure user configuration is respected (SPARK-19307)"""
+        script = self.createTempFile("test.py", """
+            |from pyspark import SparkConf, SparkContext
+            |
+            |conf = SparkConf().set("spark.test_config", "1")
+            |sc = SparkContext(conf = conf)
+            |try:
+            |    if sc._conf.get("spark.test_config") != "1":
+            |        raise Exception("Cannot find spark.test_config in SparkContext's conf.")
+            |finally:
+            |    sc.stop()
+            """)
+        proc = subprocess.Popen(
+            [self.sparkSubmit, "--master", "local", script],
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT)
+        out, err = proc.communicate()
+        self.assertEqual(0, proc.returncode, msg="Process failed with error:\n {0}".format(out))
+
 
 class ContextTests(unittest.TestCase):
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message