spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark
Date Sun, 24 Jan 2016 20:32:39 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114523#comment-15114523
] 

Josh Rosen commented on SPARK-12120:
------------------------------------

Fixed in 1.6.1 and 2.0.0.

> Improve exception message when failing to initialize HiveContext in PySpark
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-12120
>                 URL: https://issues.apache.org/jira/browse/SPARK-12120
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>            Priority: Minor
>             Fix For: 1.6.1, 2.0.0
>
>
> I get the following exception message when failing to initialize HiveContext. This is
hard to figure out why HiveContext failed to initialize. Actually I build spark with hive
profile enabled. The reason the HiveContext failed is due to I didn't start hdfs service.
And actually I can see the full stacktrace in spark-shell. And I also can see the full stack
trace in python2. The issue only exists in python2.x
> {code}
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line 430, in createDataFrame
>     jdf = self._ssql_ctx.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
>   File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line 691, in _ssql_ctx
>     "build/sbt assembly", e)
> Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt
assembly", Py4JJavaError(u'An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.\n',
JavaObject id=o34))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message