spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yh...@apache.org
Subject spark git commit: [SPARK-18070][SQL] binary operator should not consider nullability when comparing input types
Date Tue, 25 Oct 2016 19:08:32 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 1c1e847bc -> 7c8d9a557


[SPARK-18070][SQL] binary operator should not consider nullability when comparing input types

## What changes were proposed in this pull request?

Binary operator requires its inputs to be of same type, but it should not consider nullability,
e.g. `EqualTo` should be able to compare an element-nullable array and an element-non-nullable
array.

## How was this patch tested?

a regression test in `DataFrameSuite`

Author: Wenchen Fan <wenchen@databricks.com>

Closes #15606 from cloud-fan/type-bug.

(cherry picked from commit a21791e3164f4e6546fbe0a90017a4394a05deb1)
Signed-off-by: Yin Huai <yhuai@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c8d9a55
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c8d9a55
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7c8d9a55

Branch: refs/heads/branch-2.0
Commit: 7c8d9a55707ceb1b93ce1f91bcedb10aea8d5c3c
Parents: 1c1e847
Author: Wenchen Fan <wenchen@databricks.com>
Authored: Tue Oct 25 12:08:17 2016 -0700
Committer: Yin Huai <yhuai@databricks.com>
Committed: Tue Oct 25 12:08:28 2016 -0700

----------------------------------------------------------------------
 .../apache/spark/sql/catalyst/expressions/Expression.scala  | 2 +-
 .../test/scala/org/apache/spark/sql/DataFrameSuite.scala    | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/7c8d9a55/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
index 7abbbe2..0f6a896 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
@@ -511,7 +511,7 @@ abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
{
 
   override def checkInputDataTypes(): TypeCheckResult = {
     // First check whether left and right have the same type, then check if the type is acceptable.
-    if (left.dataType != right.dataType) {
+    if (!left.dataType.sameType(right.dataType)) {
       TypeCheckResult.TypeCheckFailure(s"differing types in '$sql' " +
         s"(${left.dataType.simpleString} and ${right.dataType.simpleString}).")
     } else if (!inputType.acceptsType(left.dataType)) {

http://git-wip-us.apache.org/repos/asf/spark/blob/7c8d9a55/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index f8d7ddd..4478a9a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -1601,4 +1601,13 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
     dates.except(widenTypedRows).collect()
     dates.intersect(widenTypedRows).collect()
   }
+
+  test("SPARK-18070 binary operator should not consider nullability when comparing input
types") {
+    val rows = Seq(Row(Seq(1), Seq(1)))
+    val schema = new StructType()
+      .add("array1", ArrayType(IntegerType))
+      .add("array2", ArrayType(IntegerType, containsNull = false))
+    val df = spark.createDataFrame(spark.sparkContext.makeRDD(rows), schema)
+    assert(df.filter($"array1" === $"array2").count() == 1)
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message