spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17261) Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"
Date Mon, 29 Aug 2016 06:25:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15444961#comment-15444961
] 

Rahul Jain commented on SPARK-17261:
------------------------------------

i looked into 'zjffdu' pull request and the changed code. From my understanding, this code
change particularly addresses to SparkSession only and doesn't provide backward compatibility
for previous spark version codes(Spark Context). I ran the below code in Spark 2.0 and it
failed. 

"""
>>> sc.stop()
>>> sc._instantiatedContext = None
>>> from  pyspark import SparkContext
>>> from pyspark import HiveContext
>>> sc = SparkContext()
16/08/29 06:20:18 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port
4041.
16/08/29 06:20:18 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling
back to uploading libraries under SPARK_HOME.
>>> sqlContext = HiveContext(sc)
>>> sqlContext.sql("show databases").collect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/spark/python/pyspark/sql/context.py", line 350, in sql
    return self.sparkSession.sql(sqlQuery)
  File "/usr/lib/spark/python/pyspark/sql/session.py", line 541, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/lib/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", line 933, in
__call__
  File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/usr/lib/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py", line 312, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o44.sql.
: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:
"""

> Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException:
Cannot call methods on a stopped sparkContext"
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-17261
>                 URL: https://issues.apache.org/jira/browse/SPARK-17261
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.0.0
>         Environment: Amazon AWS EMR 5.0
>            Reporter: Rahul Jain
>             Fix For: 2.0.0
>
>
> After stopping SparkSession if we recreate it and use HiveContext in it. it will throw
error.
> Steps to reproduce:
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> spark.stop()
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"
> Above error occurs only in case of Pyspark not in SparkShell



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message