spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dhruve ashar <dhruveas...@gmail.com>
Subject Re: Question about equality of o.a.s.sql.Row
Date Mon, 20 Jun 2016 20:19:22 GMT
In scala, "==" and "!=" are not operators but methods which are defined here
<http://www.scala-lang.org/api/current/#scala.Any> as :

The expression x == that is equivalent to if (x eq null) that eq null else
x.*equals*(that).
The expression x != that is equivalent to true if !(this == that)

So its recommended that you override equals method but check for equality
using == and !=.




On Mon, Jun 20, 2016 at 2:03 PM, Michael Armbrust <michael@databricks.com>
wrote:

> This is because two objects are compared by "o1 != o2" instead of
>> "o1.equals(o2)" at
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala#L408
>
>
> Even equals(...) does not do what you want on the JVM:
>
> scala> Array(1,2).equals(Array(1,2))
> res1: Boolean = false
>
>
>> (2) If (1) is expected, where is this behavior is described or defined? I
>> cannot find the description in the API document.
>> https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/Row.html
>>
>> https://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/api/scala/index.html#org.apache.spark.sql.Row
>
>
> Pull requests for documentation welcome!
>
>
>> (3) If (3) is expected, is there any recommendation to write code of
>> equality of two Rows that have an Array or complex types (e.g. Map)?
>
>
> Internally for tests, we usually compare the string representation of the
> Row.
>



-- 
-Dhruve Ashar

Mime
View raw message