spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Umesh Chaudhary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20299) NullPointerException when null and string are in a tuple while encoding Dataset
Date Mon, 17 Apr 2017 08:46:41 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970857#comment-15970857
] 

Umesh Chaudhary commented on SPARK-20299:
-----------------------------------------

[~lwlin], I want to work on this but waiting on inputs from [~marmbrus]. If you want to continue,
just to ahead. 

> NullPointerException when null and string are in a tuple while encoding Dataset
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-20299
>                 URL: https://issues.apache.org/jira/browse/SPARK-20299
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Jacek Laskowski
>            Priority: Minor
>
> When creating a Dataset from a tuple with {{null}} and a string, NPE is reported. When
either is removed, it works fine.
> {code}
> scala> Seq((1, null.asInstanceOf[Int]), (2, 1)).toDS
> res43: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, _2: int]
> scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS
> java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException
> staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString,
assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product input object),
- root class: "scala.Tuple2")._1, true) AS _1#474
> assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product input object),
- root class: "scala.Tuple2")._2 AS _2#475
>   at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
>   at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
>   at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
>   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.immutable.List.map(List.scala:285)
>   at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454)
>   at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:377)
>   at org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:246)
>   ... 48 elided
> Caused by: java.lang.NullPointerException
>   at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_1$(Unknown
Source)
>   at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source)
>   at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
>   ... 58 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message