spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Asif Hussain Shahid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22192) An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API
Date Wed, 04 Oct 2017 05:49:02 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190852#comment-16190852
] 

Asif Hussain Shahid commented on SPARK-22192:
---------------------------------------------

Not at all. I have added a bug test in org.apache.spark.sql.SQLContextSuite  in the branch
SPARK-22192 

> An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame
API
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22192
>                 URL: https://issues.apache.org/jira/browse/SPARK-22192
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>         Environment: Independent of OS / platform
>            Reporter: Asif Hussain Shahid
>            Priority: Minor
>
> If an RDD contains nested POJO objects, then SQLContext.createDataFrame(RDD, Class) api
only handles the top level POJO object. It throws ScalaMatchError exception when handling
the nested POJO object as the code does not recursively handle the nested POJOs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message