spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Farrellee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2003) SparkContext(SparkConf) doesn't work in pyspark
Date Fri, 27 Jun 2014 17:06:28 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046152#comment-14046152
] 

Matthew Farrellee commented on SPARK-2003:
------------------------------------------

first up - reproducer should be closer to:

from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName("blah").setMaster('local')
sc = SparkContext(conf)

documentation says -

conf = SparkConf().setAppName(appName).setMaster(master)
sc = SparkContext(conf)


however, there's a confounding issue:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/context.py", line 94,
in __init__
    SparkContext._ensure_initialized(self, gateway=gateway)
  File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/context.py", line 197,
in _ensure_initialized
    currentMaster = SparkContext._active_spark_context.master
AttributeError: 'SparkContext' object has no attribute 'master'

this additional issue appears to be the result of adding an _ensure_initialized call before
self.master is set

> SparkContext(SparkConf) doesn't work in pyspark
> -----------------------------------------------
>
>                 Key: SPARK-2003
>                 URL: https://issues.apache.org/jira/browse/SPARK-2003
>             Project: Spark
>          Issue Type: Bug
>          Components: Documentation, PySpark
>    Affects Versions: 1.0.0
>            Reporter: Diana Carroll
>
> Using SparkConf with SparkContext as described in the Programming Guide does NOT work
in Python:
> conf = SparkConf.setAppName("blah")
> sc = SparkContext(conf)
> When I tried I got 
> AttributeError: 'SparkConf' object has no attribute '_get_object_id'
> [This equivalent code in Scala works fine:
> val conf = new SparkConf().setAppName("blah")
> val sc = new SparkContext(conf)]
> I think this is because there's no equivalent for the Scala constructor SparkContext(SparkConf).
 
> Workaround:
> If I explicitly set the conf parameter in the python call, it does work:
> sconf = SparkConf.setAppName("blah")
> sc = SparkContext(conf=sconf)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message