spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-12439) Fix toCatalystArray and MapObjects
Date Sat, 19 Dec 2015 13:33:46 GMT


Apache Spark reassigned SPARK-12439:

    Assignee: Apache Spark

> Fix toCatalystArray and MapObjects
> ----------------------------------
>                 Key: SPARK-12439
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>            Assignee: Apache Spark
> In toCatalystArray, we should look at the data type returned by dataTypeFor instead of
silentSchemaFor, to determine if the element is native type. An obvious problem is when the
element is Option[Int] class, catalsilentSchemaFor will return Int, then we will wrongly recognize
the element is native type.
> There is another problem when using Option as array element. When we encode data like
Seq(Some(1), Some(2), None) with encoder, we will use MapObjects to construct an array for
it later. But in MapObjects, we don't check if the return value of lambdaFunction is null
or not. That causes a bug that the decoded data for Seq(Some(1), Some(2), None) would be Seq(1,
2, -1), instead of Seq(1, 2, null).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message