spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery
Date Thu, 24 Aug 2017 13:48:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan reassigned SPARK-21759:
-----------------------------------

    Assignee: Liang-Chi Hsieh

> In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21759
>                 URL: https://issues.apache.org/jira/browse/SPARK-21759
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Liang-Chi Hsieh
>            Assignee: Liang-Chi Hsieh
>             Fix For: 2.3.0
>
>
> With the check for structural integrity proposed in SPARK-21726, I found that an optimization
rule {{PullupCorrelatedPredicates}} can produce unresolved plans.
> For a correlated IN query like:
> {code}
> Project [a#0]
> +- Filter a#0 IN (list#4 [b#1])
>    :  +- Project [c#2]
>    :     +- Filter (outer(b#1) < d#3)
>    :        +- LocalRelation <empty>, [c#2, d#3]
>    +- LocalRelation <empty>, [a#0, b#1]
> {code}
> After {{PullupCorrelatedPredicates}}, it produces query plan like:
> {code}
> 'Project [a#0]
> +- 'Filter a#0 IN (list#4 [(b#1 < d#3)])
>    :  +- Project [c#2, d#3]
>    :     +- LocalRelation <empty>, [c#2, d#3]
>    +- LocalRelation <empty>, [a#0, b#1]
> {code}
> Because the correlated predicate involves another attribute {{d#3}} in subquery, it has
been pulled out and added into the {{Project}} on the top of the subquery.
> When {{list}} in {{In}} contains just one {{ListQuery}}, {{In.checkInputDataTypes}} checks
if the size of {{value}} expressions matches the output size of subquery. In the above example,
there is only {{value}} expression and the subquery output has two attributes {{c#2, d#3}},
so it fails the check and {{In.resolved}} returns {{false}}.
> We should not let {{In.checkInputDataTypes}} wrongly report unresolved plans to fail
the structural integrity check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message