Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8BD71200B16 for ; Mon, 20 Jun 2016 22:19:35 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8A785160A55; Mon, 20 Jun 2016 20:19:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AC049160A26 for ; Mon, 20 Jun 2016 22:19:34 +0200 (CEST) Received: (qmail 6474 invoked by uid 500); 20 Jun 2016 20:19:33 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 6463 invoked by uid 99); 20 Jun 2016 20:19:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2016 20:19:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5C69318060F for ; Mon, 20 Jun 2016 20:19:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id zGDAy6od8zbu for ; Mon, 20 Jun 2016 20:19:30 +0000 (UTC) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id D9C1E5F260 for ; Mon, 20 Jun 2016 20:19:29 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id xp5so560015lbb.0 for ; Mon, 20 Jun 2016 13:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=CswQELAKbYsa4l+4cd0GkXjQyErTXIawwmqEPvo/+Ik=; b=VgkwsA5tWRNEMIUGXff7ZCdcJF9E/3ATFDSYWubUUxgNBW4FSNz67hCPRKHLzwV7qa 1lIjda8MCeHvx0/3LgFV37JFSDPQmiNVklZHvN/C/D7efrhQFwGCgiS7sq/JlQd4BeQW d9HSHPFDmxnEkQ0IGejoIfDRipcAyWyKhk9cGII/CAuxIkNO34gEN1NMerNYYXWn/K1e 1WMNluCIAaYZKV2dR0xGi4nQgNBUElA92wBCxIxHcrFJHyhJHjF4GqUnH8HWyGcggHlm ifcup7JglWHgRhA5rqEBjFa3iXTNfOo0APN7MY74NQbE54ZstfiBeU6vtrkD41Qkn7LQ jUaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=CswQELAKbYsa4l+4cd0GkXjQyErTXIawwmqEPvo/+Ik=; b=VOtqD25X5XaTHD4nqj+1u0JjfNZirnrImpnyxXk39ARbhaatGUdGi+mKKMtblQu8GL HmdIwWIdS5annU5UpYOVSBusF7nXwp9pQNCrYrWPM4mKRy5ONMMgZbb46k22Zl8pbRNj ghDmR6RckrbBowEfgSR3fr6PetLo2FzuLh23Dmp6tO0nxZ+At+TsUQwQsbv424idgeN2 7xJUC6h8nrI/vAQJGaAUoayav2Qq+3RLLM5mObntVPnHDZloGUIxy9G852ViQBUjZDK2 DoCic6PvePYhsI7Zpw6D148xB8VwZxqxAmPEj2/Nkv5ael1vX31CzjEQE/UX3K7XtcyY VzkQ== X-Gm-Message-State: ALyK8tLokYqikE3+t0FkmYD8Ec092HMP369OZd88SSn+eYkzYFKB1SyKkPzVXnmcslpMHqG7AqYRZVgdbkJtuQ== X-Received: by 10.194.105.41 with SMTP id gj9mr5493573wjb.22.1466453963395; Mon, 20 Jun 2016 13:19:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.195.1.137 with HTTP; Mon, 20 Jun 2016 13:19:22 -0700 (PDT) In-Reply-To: References: From: dhruve ashar Date: Mon, 20 Jun 2016 15:19:22 -0500 Message-ID: Subject: Re: Question about equality of o.a.s.sql.Row To: Michael Armbrust Cc: Kazuaki Ishizaki , "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=001a1130cd269157d30535bb6b88 archived-at: Mon, 20 Jun 2016 20:19:35 -0000 --001a1130cd269157d30535bb6b88 Content-Type: text/plain; charset=UTF-8 In scala, "==" and "!=" are not operators but methods which are defined here as : The expression x == that is equivalent to if (x eq null) that eq null else x.*equals*(that). The expression x != that is equivalent to true if !(this == that) So its recommended that you override equals method but check for equality using == and !=. On Mon, Jun 20, 2016 at 2:03 PM, Michael Armbrust wrote: > This is because two objects are compared by "o1 != o2" instead of >> "o1.equals(o2)" at >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala#L408 > > > Even equals(...) does not do what you want on the JVM: > > scala> Array(1,2).equals(Array(1,2)) > res1: Boolean = false > > >> (2) If (1) is expected, where is this behavior is described or defined? I >> cannot find the description in the API document. >> https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/Row.html >> >> https://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/api/scala/index.html#org.apache.spark.sql.Row > > > Pull requests for documentation welcome! > > >> (3) If (3) is expected, is there any recommendation to write code of >> equality of two Rows that have an Array or complex types (e.g. Map)? > > > Internally for tests, we usually compare the string representation of the > Row. > -- -Dhruve Ashar --001a1130cd269157d30535bb6b88 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
In scala, "=3D=3D" and "!=3D" are not = operators but methods which are defined here=C2=A0as :

The = expression=C2=A0x =3D=3D that=C2=A0is equivalent to=C2=A0if (x eq null) that eq null else x.equals(that)= .
The expression=C2=A0x !=3D that=C2=A0is equivalent to=C2=A0true=C2=A0if !(this = =3D=3D that)

So its recommended = that you override equals method but check for equality using =3D=3D and != =3D.=C2=A0


=


On Mon, Jun 20, 2016 at 2:03 PM, Michael Armbrust <= span dir=3D"ltr"><michael@databricks.com> wrote:
This is because two objects are compared by "o1 !=3D o2" instead of "o1.equals(o2)" at https://github.com/apache/spark/blob/= master/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala#L408

Even equals(...) does not do = what you want on the JVM:

scala> Array(1,2).equals(Array(1,2))
res1: Boolean =3D false
=C2=A0
(2) If (1) is expected, where is this behavior is described or defined? I cannot find the description in the API document.
https://spark.apache.org/docs/1.6.1/api/java= /org/apache/spark/sql/Row.html
https://home.apache.org/~pwendell/spark-releases/spar= k-2.0.0-preview-docs/api/scala/index.html#org.apache.spark.sql.Row

=C2=A0
(3) If (3) is expected, is there any recommendation to write code of equality of two Rows that have an Array or complex types (e.g. Map)?

= Internally for tests, we usually compare the string representation of the R= ow.=C2=A0



--
-D= hruve Ashar

--001a1130cd269157d30535bb6b88--