spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join
Date Tue, 29 May 2018 12:41:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-24385:
------------------------------------

    Assignee: Apache Spark

> Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-24385
>                 URL: https://issues.apache.org/jira/browse/SPARK-24385
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.1, 2.3.0
>            Reporter: Daniel Shields
>            Assignee: Apache Spark
>            Priority: Major
>
> Dataset.join(right: Dataset[_], joinExprs: Column, joinType: String) has special logic
for resolving trivially-true predicates to both sides. It currently handles regular equals
but not null-safe equals; the code should be updated to also handle null-safe equals.
> Pyspark example:
> {code:java}
> df = spark.range(10)
> df.join(df, 'id').collect() # This works.
> df.join(df, df['id'] == df['id']).collect() # This works.
> df.join(df, df['id'].eqNullSafe(df['id'])).collect() # This fails!!!
> # This is a workaround that works.
> df2 = df.withColumn('id', F.col('id'))
> df.join(df2, df['id'].eqNullSafe(df2['id'])).collect(){code}
> The relevant code in Dataset.join should look like this:
> {code:java}
> // Otherwise, find the trivially true predicates and automatically resolves them to both
sides.
> // By the time we get here, since we have already run analysis, all attributes should've
been
> // resolved and become AttributeReference.
> val cond = plan.condition.map { _.transform {
>   case catalyst.expressions.EqualTo(a: AttributeReference, b: AttributeReference) if
a.sameRef(b) =>
>     catalyst.expressions.EqualTo(
>       withPlan(plan.left).resolve(a.name),
>       withPlan(plan.right).resolve(b.name))
>   // This case is new!!!
>   case catalyst.expressions.EqualNullSafe(a: AttributeReference, b: AttributeReference)
if a.sameRef(b) =>
>     catalyst.expressions.EqualNullSafe(
>       withPlan(plan.left).resolve(a.name),
>       withPlan(plan.right).resolve(b.name))
> }}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message