spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.
Date Wed, 11 Jul 2018 06:37:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539630#comment-16539630
] 

Saisai Shao commented on SPARK-24781:
-------------------------------------

Thanks Felix. Does this have to be in 2.3.2? [~ueshin]

> Using a reference from Dataset in Filter/Sort might not work.
> -------------------------------------------------------------
>
>                 Key: SPARK-24781
>                 URL: https://issues.apache.org/jira/browse/SPARK-24781
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.1
>            Reporter: Takuya Ueshin
>            Priority: Blocker
>
> When we use a reference fromĀ {{Dataset}} in {{filter}} or {{sort}}, which was not used
in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing from name#5
in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>    +- AnalysisBarrier
>       +- Project [name#5]
>          +- Project [_1#2 AS name#5, _2#3 AS id#6]
>             +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message