spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17538) sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0
Date Sun, 08 Oct 2017 17:15:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196236#comment-16196236
] 

Hyukjin Kwon commented on SPARK-17538:
--------------------------------------

I can't reproduce this. Would you be able to provide more steps to reproduce this? For example,
how the initial dataframe was created?

> sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-17538
>                 URL: https://issues.apache.org/jira/browse/SPARK-17538
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>         Environment: os - linux
> cluster -> yarn and local
>            Reporter: Srinivas Rishindra Pothireddi
>
> I have a production job in spark 1.6.2 that registers several dataframes as tables. 
> After testing the job in spark 2.0.0, I found that one of the dataframes is not getting
registered as a table.
> Line 353 of my code --> self.sqlContext.registerDataFrameAsTable(anonymousDF, "anonymousTable")
> line 354 of my code --> df = self.sqlContext.sql("select AnonymousFiled1, AnonymousUDF(
AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable")
> my stacktrace
>  File "anonymousFile.py", line 354, in anonymousMethod
>     df = self.sqlContext.sql("select AnonymousFiled1, AnonymousUDF( AnonymousFiled1 )
as AnonymousFiled3 from anonymousTable")
>   File "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/context.py",
line 350, in sql
>     return self.sparkSession.sql(sqlQuery)
>   File "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/session.py",
line 541, in sql
>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>   File "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
line 933, in __call__
>     answer, self.gateway_client, self.target_id, self.name)
>   File "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/utils.py",
line 69, in deco
>     raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> AnalysisException: u'Table or view not found: anonymousTable; line 1 pos 61'
> The same code is working perfectly fine in spark-1.6.2 
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message