Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 4 Apr 2017 09:15:41 +0000 (UTC)
From: "Adrian Ionescu (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13060965.1491131136000.199324.1491297341898@Atlassian.JIRA>
In-Reply-To: <JIRA.13060965.1491131136000@Atlassian.JIRA>
References: <JIRA.13060965.1491131136000@Atlassian.JIRA> <JIRA.13060965.1491131136974@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-20193) Selecting empty struct causes
 ExpressionEncoder error.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 04 Apr 2017 09:15:46 -0000


    [ https://issues.apache.org/jira/browse/SPARK-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954878#comment-15954878 ] 

Adrian Ionescu commented on SPARK-20193:
----------------------------------------

Thanks for the workaround, but, sorry, this is not good enough. I agree that an empty struct is not very useful, but if it's not supported then the docs should say so and the error message should be clear.

In my case, I'm building this struct dynamically, based on user input, so it may or may not be empty. Right now I have to special case it, but that introduces unnecessary complexity and makes the code less readable.


> Selecting empty struct causes ExpressionEncoder error.
> ------------------------------------------------------
>
>                 Key: SPARK-20193
>                 URL: https://issues.apache.org/jira/browse/SPARK-20193
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.0
>            Reporter: Adrian Ionescu
>              Labels: struct
>
> {{def struct(cols: Column*): Column}}
> Given the above signature and the lack of any note in the docs saying that a struct with no columns is not supported, I would expect the following to work:
> {{spark.range(3).select(col("id"), struct().as("empty_struct")).collect}}
> However, this results in:
> {quote}
> java.lang.AssertionError: assertion failed: each serializer expression should contains at least one `BoundReference`
>   at scala.Predef$.assert(Predef.scala:170)
>   at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$11.apply(ExpressionEncoder.scala:240)
>   at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$11.apply(ExpressionEncoder.scala:238)
>   at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>   at scala.collection.immutable.List.flatMap(List.scala:344)
>   at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.<init>(ExpressionEncoder.scala:238)
>   at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:63)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
>   at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2837)
>   at org.apache.spark.sql.Dataset.select(Dataset.scala:1131)
>   ... 39 elided
> {quote}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org